Sunday 6 June 2010

protobuf-net; a status update

I keep promising “v2” is just around the corner, and (quite reasonably) I get a steady stream of “when” e-mails / tweets etc, so it is time for a status update…

…but first I must apologise for the delay; the short version is that in the last few weeks I’ve changed jobs, and it has taken a bit of time to get my daily pattern (work time, family time, geek time, etc) sorted. Since I no longer spend 4 hours a day commuting, in theory I have more time available. In reality I’m spending lots more time with my family, which I don’t regret. But I’m getting back into the “crazy geek” thing ;p

So where is protobuf-net?

The first thing to note that “v2” is not a minor change; it completely changes the crazy-complex generics pipeline (which killed CF, and was generally a bad design choice for lots of reasons), opting instead for a non-generic pipeline coupled with an IL emitter to get the maximum performance. It also introduces a whole new metadata abstraction layer, and static dll usage (so it works on iPhone (but don’t ask me how the SDK rules affect this; IANAL), Phone 7, and so that CF doesn’t have to use reflection).

The numbers

  • Tests for new new features (metadata abstraction, IL generation, etc): 100%
  • Compatibility tests:79% (228 / 287, plus some skipped that won’t be in the first alpha release)

I should note: my unit tests for this tend, right or wrong, to each cover a number of features; it is very hard to make the regression tests overly granular, but to be honest I think we can all agree that the main objective is not to have the world’s most beautiful tests, but rather: to check that the system works robustly.

The gaps

  • Mapped enums; it’ll handle enums, but it currently only treats them as pass-thrus to the underlying primitive value. It doesn’t currently process [ProtoEnum]. Not hard, just time.
  • Member inference (which was the best “v1” could do to allow serialization of types outside your control); this doesn’t involve IL, so not too tricky – and not used by the majority.
  • Roundtrip for unexpected fields (currently dropped; this also impacts the memcached transcoder)
  • Parse/ToString fallback for unknown types (another “v1” feature)
  • Packed arrays (an optional, alternative wire-format for lists/arrays; this won’t affect you if you haven’t heard of it, since it is opt-in)
  • *WithLengthPrefix – for reading/writing multiple objects to/from a single stream
  • GetProto (this is deferred from the alpha release, as it isn’t “core” functionality, and is significant work to do properly)

No show-stoppers; just work to do. Not long now. I’m reluctant to promise a “when” since I’ve delayed already…