Objects, Graphs and all that Jazz

protobuf-net v2 ticks along slowly; I’m embarrassed to say that due to a combination of factors progress has been slower than I would have liked – and for purely human reasons (availability etc).

But; I thought I’d better update with the tweaks I’m looking at currently; they have all been ticking along in the back of head for ages, but frankly people kept nagging me to provide them, so who am I to argue?

Caveat

This is all work-in-progress; don’t try to pull the trunk and shove it into production! Soon… I should also stress that all of the thoughts described below are outside the interoperable protobuf spec; it’ll still be a valid protobuf stream, but this only applies to protobuf-net talking to protobuf-net.

So; here’s some common questions I get…

Type meta

The serialization is fine, but I don’t know (and cannot know) all of my types up front. How can I do this?

Well, protobuf is a contract based format; if you don’t know the types, it will struggle – as will any contract based serializer…

Yes, I get that; now: how do I do it?

Now, I’ve held off putting any meta in the stream for various reasons:

• it steps far outside the core protobuf spec
• it flashes the warning signs of BinaryFormatter, my nemesis

But, so many people seem to want this that I think I have to buckle; but on my terms! So in v2, I’m adding the ability to indicate that (on a per-member basis) objects should resolve their type information from the stream. By default, by embedding the assembly-qualified-name, but providing an abstraction layer over that allowing you to provide your own string<===>Type map (and thus avoiding the knots in by stomach caused by too much type dependency).

Full graphs

The serialization is fine, but my data is a graph, not a tree. How can I do this?

Well, protobuf is a tree format; it doesn’t work like that…

Yes, I get that; now: how do I do it?

(by the way, are you spotting a pattern in the questions I get?)

Type meta is something I didn’t want to add, but graph support is something I have really wanted to sneak in; breaking all the rules with type meta seems a reasonable excuse. So in v2, I’m adding the ability (on a per-member basis) to use reference tracking to (de)serialize a complete object graph (except events; don’t get me started on that…). A minor complication here is that for technical reasons it is a nightmare to support this concurrently with inheritance (which, btw, the core protobuf doesn’t support – I feel quite the anarchist here…); but I can support it in conjunction with type meta; so you still get to keep inheritance, but implemented differently.

Repeated strings

It is fairly common to have repeated string data in a large graph. As an offshoot of full-graph support, we also get a mechanism to support string re-use for free; woot! So again, in v2 I’ll be enabling this on a per-member basis.

Footnote

All of this goes so far outside of the protobuf spec that I have little right to even call it protobuf-related any more; maybe I should force the caller to explicitly set an EnableImplementationSpecificOptions flag? So there is an portable core with some opt-in hacks for these random options.

And again; this whole area is under development; I’m working on it, honest!

That’s great! My full-graph of dynamic objects using inheritance and string re-use works fine! Now how do I load it into my c++/java/php/python/whatever client?

Don’t make me hurt you…

DataTable – life in the old beast?

Updated: Billy referred me to the binary remoting format for DataTable; stats for BinaryFormatter now include both xml and binary.

---

This week, someone asked the fastest way to send a lot of data, only currently available in a DataTable (as xml), over the wire.

My immediate thought was simply GZIP the xml and have done with it, but I was intrigued… can we do better? In particular (just for a complete surprise) I was thinking whether protobuf-net could be used to write them. Now don’t get me wrong; I’m not a big supporter of DataTable, but they are still alive in the wild. And protobuf-net is a general-purpose serializer, so… why not try to be helpful, eh?

v1 won’t help at all with this, since the object-model just isn’t primed for extension, but in v2 we have many options. I thought I’d look at some typical data (SalesOrderDetail from AdventureWorks2008R2; 121317 rows and 11 columns), comparing the inbuilt xml support, BinaryFormatter support, and protobuf-net. And each of those uncompressed, via gzip, and via deflate.

At this point my new friend Lucy (right) is pestering me for a walk, but… it worked; purely experimental, but committed (r356). Results shown below:

Table loaded    11 cols 121317 rows
DataTable (xml) (vanilla)               2269ms/6039ms
                                       64,150,771 bytes
DataTable (xml) (gzip)                  4881ms/6714ms
                                       7,136,821 bytes
DataTable (xml) (deflate)               4475ms/6351ms
                                       7,136,803 bytes
BinaryFormatter (rf:xml) (vanilla)      3710ms/6623ms
                                       74,969,470 bytes
BinaryFormatter (rf:xml) (gzip)         6879ms/8312ms
                                       12,495,791 bytes
BinaryFormatter (rf:xml) (deflate)      5979ms/7472ms
                                       12,495,773 bytes
BinaryFormatter (rf:binary) (vanilla)   2006ms/3366ms
                                       11,272,592 bytes
BinaryFormatter (rf:binary) (gzip)      3332ms/4267ms
                                       8,265,057 bytes
BinaryFormatter (rf:binary) (deflate)   3216ms/4130ms
                                       8,265,039 bytes
protobuf-net v2 (vanilla)               316ms/773ms
                                       8,708,846 bytes
protobuf-net v2 (gzip)                  932ms/1096ms
                                       4,797,856 bytes
protobuf-net v2 (deflate)               872ms/1050ms
                                       4,797,838 bytes

Since “walkies” beckons, I’ll be brief:

• the inbuilt xml and BinaryFormatter (as xml) are both very large natively and compress pretty well, but with noticeable additional CPU costs
• with BinaryFormatter using the binary encoding it does much less work, getting (before compression) into the same size-bracket as the xml encoding managed with compression
• protobuf-net is much faster in terms of CPU, but slightly larger than the gzip xml output
• unusually for protobuf, it compresses quite will – presumably there is enough text in there to make it worthwhile; this then takes roughly half the size and much less CPU (compared to gzip xml)

Overall the current experimental code is exceptionally rough and ready, mainly as an investigation. But… should I pursue this? do people still care about DataTable enough? Or just use gzip/xml in this scenario?

Other news…

Sorry if this is turning into a bit of a serialization fest; I do have other coding hobbies; simply this is the thing I’ve been spending my spare time on lately ;p

Silverlight WCF

It recently came to my attention that possibly Silverlight does now support the WCF extension points for swapping the serializer at runtime. Not via xml configuration, but via code configuration. Full details on Carlos’ Blog, but that is definitely an area I want to investigate – at least, if @abdullin will let me.

Interface serialization

Something a few people have asked for, and definitely not outside reach. In fact, Wallace Turner kindly donated a v1 patch to enable interface-based serialization. I’m keeping this “on ice” for now, simply to keep my main focus on finishing v2. But this exists and is certainly on the table for consideration.

Anyway, enough for now; the day job starts again in not-a-lot-of-hours…

protobuf-net on MonoDroid

I was very pleased to get an invite to the MonoDroid (Mono tools / environment for Android) beta earlier in the week, so in addition to a few v2 commits I’ve been playing with shiny new toys.

Wow; decent reflection!

Initially I made the mistake of thinking that MonoDroid would have the same reflection limitations as (say) MonoTouch. I even tied myself in knots trying to get around them! Then Jb Evian pointed out that actually MonoDroid can do full meta-programming at runtime! That is a huge plus for me… so; whack in a few conditional-compilation symbols and – wow! It works!

A taster

Don’t get too excited – this is only going to be brief…

Only intended to show something working, but here is v2 on MonoDroid, compiled at runtime, round-tripping some trivial data.

A small (but significant) step. Actually, the time taken to compile the model and run it the first time is still a bit slow for my liking – not sure if that is the emulator, the beta, or a general JIT issue – I’ll dig more and investigate rebasing further.

And of course as a fallback, I can still use runtime usage without “emit” – it is noticeably slower, but still works.

Again, without an actual device it is hard to gauge typical performance at this point, but I’m pretty hopeful.

So… if that works, what are you waiting for?

I still need to finish v2; some decent commits lately now that I’m all refreshed. The finishing line is within sight… oh, and MonoDroid needs to be released ;p

protobuf-net; ushort glitch before r274

Dammit, I hate it when a gremlin shows up. I hate it even more when they show up a year later…

It turns out that a bug “fix” to ushort (unsigned 16-bit integers) handling back in October 2009 introduced a slight… “nuance” when moving between versions < r274 and >= r274. Basically, due to a braindead bug on my part, it was treating ushort data as strings rather than handling appropriately. This was fixed a year ago, but if you are moving between versions is a PITA.

Sorry for any inconvenience folks, but… well, it happened. I can’t change that. Anyway, how to fix?! The good news is NO DATA IS LOST - it is just a bit trickier to access. You wouldn't have encountered the issue if working against rigid .proto / cross-platform definitions (since .proto doesn't have direct ushort support), so I’ll limit myself to the pure .NET scenario. In which case, IMO the easiest fix is to introduce a shim property, so:

[ProtoMember(1)]
public ushort Foo {get;set;}

might be updated to:

[ProtoMember(1)]
private string FooLegacy { // a pass-thru
    get {return Foo.ToString();}
    set {Foo = ushort.Parse(value);}
}
private bool FooLegacySpecified { // suppress serialization
    get {return false;}
    set {}
}
[ProtoMember(42)] // any unused field number
public ushort Foo {get;set;}

This allows old-style and the correct data to be used side-by-side (always writing to the new format). For arrays, see my longer answer here, which also switches to the more efficient "packed" encoding, introduced later to the protobuf spec.

Sorry for any inconvenience folks. Fortunately use of ushort is fairly uncommon in .NET, so my hope is that there aren't vast hordes of people impacted by this unfortunate foible.

As a side note, I should add: I considered trying to add automatic handling of old data, but it transpires that the borked data is so similar to "packed" encoding as to be ambiguous in some cases, so trying to guess was too risky. And it would compound the issue by layering hack upon bug.

Truer than true

(edit: apols for the formatting; google ate my images - have replaced with source, but raw)

I’ve been quiet for a little while, due to general fatigue, changing job etc – but after a holiday I’m back refreshed and renewed.

So let’s start with a fun edge condition, inspired by my colleague balpha, who asked me “does C#/.NET/LINQ guarantee false < true?”.

So, is it?

I’m leaving LINQ aside, as for anything other than LINQ-to-Objects, all bets are off; so let’s just limit ourselves to C#; in that case the answer is pretty simple, “yes”. System.Boolean implements IComparable and IComparable<bool> such that this will always work. But not all .NET is C# ;p So how can we be evil here?

What is a boolean, anyways?

First, we need to understand that (in common with many runtimes/languages) the CLI doesn’t really know much about booleans – under the hood they are treated pretty-much the same as integers, with special treatment for “zero” and “not zero”. That means that technically we aren’t restricted to the expected values – it is simply that C# handles all the logic to make sure you only get sane values. But we can write our own IL – here I’m using DynamicMethod to create a bool with underlying value 2:

// write an evil non-C# function that returns something 
// unexpected for a bool (but legal IL)
var dm = new DynamicMethod("mwahaha", typeof(bool), null);
var il = dm.GetILGenerator();
il.Emit(OpCodes.Ldc_I4_2);
il.Emit(OpCodes.Ret);
var func = (Func<bool>)dm.CreateDelegate(typeof(Func<bool>));
var superTrue = func();

The first thing to note is that you shouldn’t do this. So what does such a beast do? The first thing to note is that at first glance it appears to work as expected:

// is it true/false?
Console.WriteLine(superTrue); // prints: true
Console.WriteLine(!superTrue); // prints: false
// and does it *equal* true/false?
Console.WriteLine(superTrue == true); // prints: true
Console.WriteLine(superTrue == false); // prints: false

so the non-zero / zero handling is just like we expect? Well, no. The compiler isn’t helping us here, as it spots the constants. However, if we compare to variables, it uses the underlying value:

// really?
bool normalTrue = bool.Parse("true"), normalFalse = bool.Parse("false");
Console.WriteLine(superTrue == normalTrue); // prints: false
Console.WriteLine(superTrue == normalFalse); // prints: false

(I’m using bool.Parse here to avoid the compiler getting clever)

And ordering?

We could postulate that this “super-true” is more than “true”, or maybe we might suppose the implementation is going to just treat it as true. In fact, neither is correct – the implementation of CompareTo (testing in .NET 4) means that it simply breaks:

// how about sorting...
Console.WriteLine(superTrue.CompareTo(true)); // prints: 1
Console.WriteLine(true.CompareTo(superTrue)); // prints: 1 - oops!

That is going to make sorting very painful.

So what is your point?

Well, the first thing to note is that I haven’t disproven the original question; as far as I can tell, false is always less than true. However, you can get some very bizarrely behaved booleans. You would have to be crazy to deliberately introduce such shenanigans into your code, but it is something to perhaps at least know about as part of defensive programming / data sanitization – I’m fairly sure you could cause some nasty effects by passing 17 in as a bool to a few places…

Extending ASP.NET MVC with custom binders and results

Recently, I was putting together a brief sample for someone who wanted to use protobuf-net with ASP.NET as part of a basic HTTP transport; fairly simple stuff – just using protobuf as the binary body of the HTTP request/response.

It was a simple enough demo, but it got me wondering: what would be required to do this (more cleanly) in ASP.NET MVC? In many ways we just want something pretty similar to how many people currently use JSON: forget the formal API – just tell me the routes and let me know what data I need.

It turns out that this is astonishingly easy…

The controller

A good way of understanding the big picture is to look at the controller. So let’s take a peek:

One very basic controller, with one action. Things to note:

• We want to read the request parameter (“req”) from the request body
• To do that, we’re using a custom binder, via [ProtoPost] which we’ll see shortly
• We could have exposed different input arguments separately – it would have added complexity, and isn’t very compatible with the way that requests are written in the .proto language, so I’ve restricted myself to binding a single (composite) parameter
• We run some logic, same as normal
• We return an action-result that will write the response back as protobuf

The nice thing about this is that (no matter which option you use to control your routes) it is very simple to add actions (as different urls).

That might be all you need to know!

I’ll include more explanation for completeness, but if you want a simple way of throwing objects over http efficiently and in an interoperable way, that might be enough.

The binder

The binder’s job is it understand the incoming request, and map that into things the controller can understand, such as method variables. There are two parts to a binder: writing the code to handle the input, and telling the system to use it. Writing a binder, it turns out, can be pretty simple (well – if you already have the serialization engine available); here’s all there is:

This simply checks that the caller is doing a POST, and if they are it passes the input stream to protobuf-net, based on the parameter type (which is available from the binding-context).

The other part is to tell ASP.NET MVC to use it; in my case I’m using a new [ProtoPost] attribute which simply creates the binder:

The result

Just as a controller doesn’t need to know how to parse input, it also doesn’t know how to format output – that is the job of an action-result; we’ll write our own, that we can re-use to write objects to the response stream. Which is actually less work than writing the last sentence!

Again – some very basic code that simply takes an object and writes it to the output stream via protobuf-net.

Anything else?

OK, I’ve taken some shortcuts there. To do it properly you might want to check for an expected content-type, etc – and things like security is left open for your own implementation (just like it would be for any other similar object-passing API). I’m just showing the core object <===> stream code here. But it works, and is painless.

The example code (include the ASP.NET / IHttpHandler equivalent, and an example client) is all available in the protobuf-net trunk, here.