protobuf-net v2 on the iPhone

It might not look very impressive to you, but this is pure joy to me. It is the first evidence I’ve seen that the revised rewritten protobuf-net can work in MonoTouch for use on the iPhone – and presumably MonoDroid in the future.

A big thanks here to Christian Weyer and Andrew Rimmer who both offered some help here (since I don’t actually own a mac, an iPhone or MonoTouch!). The grab is from Andrew (he assures me it works fine on the actual device too).

This piece of work has been a slog, and I kept putting it off (because frankly, re-writing the entire stack is a huge job). The goals have not been trivial:

• Continue to support all the existing runtime-only usage…
• ..while also making it support fully static / AOT compatible
• Allow it to write serialization dlls in advance (like sgen)
• Fix the stability issues on Compact Framework (caused by overuse of generics)
• Remove some performance bottlenecks on Compact Framework
• Support POCO (or more accurately, work without requiring special [attributes]) so you can use it with types outside your influence, or for inheritance that isn’t known at compile-time
• Make it work on iPhone etc
• Make it even faster
• Make it more extensible (for example, what if I want to write a proxy that writes a DataTable to a protobuf stream?)
• Make it support mutable structs (yes I know that they are usually evil, but people simply keep asking for this, in particular for XNA)
• Remove a heap of code-debt caused by an evolving project that grew without a 100% clear direction
• Throw in some other feature-requests and bug-fixes for good measure
• And as a side-benefit, it even compiles and runs on .NET 1.1 or via mcs on Mono (with minimal #if usage)

I should stress that it is not yet complete – but the core of the engine is there and working, demonstrating most of the key points above (albeit in a very raw state). There’s still a way to go, but it is getting there.

Everything “is” an object?

This nugget of the .NET unified type system. And the cause of oh so much misunderstanding.

My way of looking at it is that it isn’t helpful to think of value-types (structs in C# parlance) “as objects”. You can treat them as though they are, but you need to fundamentally change them (box them) in order to do it. Which has lots of uses for sure; but just make sure you understand what you are doing and why. If a value-type truly was an object you wouldn’t have to do this.

It is a bit like saying all people are meat; technically true, but if you start manipulating me like you would beef, don’t expect me to be quite the same afterwards.

Instead, I try to emphasize that everything can be treated as an object.

Oh, and mutable structs are evil, but you already knew that.

New spike - looking hopeful

After a few false starts, I've finally found some time to spike the much-needed re-work for protobuf-net; this pretty much means "re-write it from scratch, learning from design mistakes and copying the occasional block of code".

Key changes / goals:

• Generics: for the most part, gone from the guts of the implementation; it now uses a non-generic "decorator" approach - you wouldn't believe how much pain they have caused me (here)
• This should hopefully fix compact-framework (here, here), and might (no promises at this stage) even allow it to work on .NET 1.1 and micro-framework
• Build around a runtime model rather than attribute reflection (but with attribute reflection to be added as a model provider)
• Much easier to unit-test individual decorators
• Double implementation; vanilla decorator for "light" frameworks, and compiled implementation for the full-fat versions
• I lack the hardware to test, but I'm hopeful that the vanilla decorators will work on iPhone - I don't suppose you have an iPhone and iMac to spare?
• Potential to emit the compiled version to disk (think: sgen)
• Added buffer-pooling (makes it possible to use a larger buffer with minimal allocation costs)
• Re-written the string handling to use pointer (unsafe) code where appropriate, and better incremental encoder handling

There is still masses to do - but I'm feeling pretty happy with the initial results... I've got the core engine in place, and enough decorators to write some basic messages. Timings below, with meanings:

• New (decorator) - vanilla decorator implementation - no optimisations (yet; although I'm sure we can improve this once I can tell what works/where)
• New (compiled) - compile/flatten the decorators into a single operation
• New (delegate) - same, but to a delegate rather than class (just for fun)
• Old: the existing protobuf-net "generic" Serializer

   New (decorator): 3399ms
  New (compiled): 788ms
  New (delegate): 619ms
  Old: 758ms
  New (decorator): 2569ms
  New (compiled): 642ms
  New (delegate): 604ms
  Old: 754ms
  New (decorator): 2549ms
  New (compiled): 631ms
  New (delegate): 623ms
  Old: 740ms
  New (decorator): 2589ms
  New (compiled): 640ms
  New (delegate): 608ms
  Old: 750ms
  New (decorator): 2587ms
  New (compiled): 638ms
  New (delegate): 614ms
  Old: 750ms
  New (decorator): 2433ms
  New (compiled): 519ms
  New (delegate): 504ms
  Old: 664ms
  New (decorator): 2318ms
  New (compiled): 512ms
  New (delegate): 506ms
  Old: 596ms

Oh, and I need to fix all the other things, too... sigh.