Tuesday, 2 February 2010

New spike - looking hopeful

After a few false starts, I've finally found some time to spike the much-needed re-work for protobuf-net; this pretty much means "re-write it from scratch, learning from design mistakes and copying the occasional block of code".

Key changes / goals:
  • Generics: for the most part, gone from the guts of the implementation; it now uses a non-generic "decorator" approach - you wouldn't believe how much pain they have caused me (here)
  • This should hopefully fix compact-framework (here, here), and might (no promises at this stage) even allow it to work on .NET 1.1 and micro-framework
  • Build around a runtime model rather than attribute reflection (but with attribute reflection to be added as a model provider)
  • Much easier to unit-test individual decorators
  • Double implementation; vanilla decorator for "light" frameworks, and compiled implementation for the full-fat versions
  • I lack the hardware to test, but I'm hopeful that the vanilla decorators will work on iPhone - I don't suppose you have an iPhone and iMac to spare?
  • Potential to emit the compiled version to disk (think: sgen)
  • Added buffer-pooling (makes it possible to use a larger buffer with minimal allocation costs)
  • Re-written the string handling to use pointer (unsafe) code where appropriate, and better incremental encoder handling
There is still masses to do - but I'm feeling pretty happy with the initial results... I've got the core engine in place, and enough decorators to write some basic messages. Timings below, with meanings:
  • New (decorator) - vanilla decorator implementation - no optimisations (yet; although I'm sure we can improve this once I can tell what works/where)
  • New (compiled) - compile/flatten the decorators into a single operation
  • New (delegate) - same, but to a delegate rather than class (just for fun)
  • Old: the existing protobuf-net "generic" Serializer

   New (decorator): 3399ms
New (compiled): 788ms
New (delegate): 619ms
Old: 758ms
New (decorator): 2569ms
New (compiled): 642ms
New (delegate): 604ms
Old: 754ms
New (decorator): 2549ms
New (compiled): 631ms
New (delegate): 623ms
Old: 740ms
New (decorator): 2589ms
New (compiled): 640ms
New (delegate): 608ms
Old: 750ms
New (decorator): 2587ms
New (compiled): 638ms
New (delegate): 614ms
Old: 750ms
New (decorator): 2433ms
New (compiled): 519ms
New (delegate): 504ms
Old: 664ms
New (decorator): 2318ms
New (compiled): 512ms
New (delegate): 506ms
Old: 596ms
Oh, and I need to fix all the other things, too... sigh.


Timothy Parez said...


There seem to be a number of
implementations for .NET available.

How safe is it to use your implementation? Would you suggest it for use in a real project?

I see you might support the .NET micro framework, that would be really cool :p

Marc Gravell said...

The v1 download is pretty solid; there are a few known edge cases, but it is being used by a good number of real projects.

The v2 stuff is barely "hello world" at the moment; don't touch that.

I can't *guarantee* MF, but we'll see.

Anonymous said...

Hi Marc,
I am in the early stages of looking into cross platform (mostly Java, C#, C++) serialization options and came across your c# implementation.
Do you have, or plan to support, different "formatters" - namely JSON and XML, on top of the binary format?

Thrift claims they have that, but I was put off by the lack of documentation and was thinking to give protobuf a shot

Marc Gravell said...

Well, most of the *point* seems to be the wire format. There are already perfectly good answers for json and xml...? In short - no, that isn't something I had on my "todo" list - but I'm open to reasons that it should be?

Anonymous said...

In Thrift there is a nice abstraction - Protocol, separating data structure, from transport (which allows for pluggable data encodings).

Here is doc which explains it nicely:

I think the Protocol generalization is very nice - can expose the same service in binary, json, xml, using the same technology. I can see it being useful at least for debugging.

I am not aware of a tool which supports JSON, XML, and binary out of the box supporting at least java and c#.

I have used XStream for xml and json, but this is only java.