Fun with dynamic - how to build a dynamic object

The project: build a redis client

The dynamic feature in C# 4 is powerful when used appropriately. I thought it was about time I wrote a piece on how to do that. And … because I like redis so much I thought it would make a perfect example.

Now, the observant among you may be thinking “but Marc, you’ve already written a redis client” – in which case I agree (and incidentally I congratulate you on getting hyperlinks into speech – no mean feat); but that isn’t the point! This toy client isn’t meant to compete: BookSleeve is heavily optimised to allow really fast and efficient usage. This one is just for fun. If you like it and it works for you, great! All the code for this is available on google-code.

What we want

I want to be able to do things like:

// increment and fetch new
int hitCount = client.incr("hits");
// fetch next, if any (else null)
string nextWorkItem = client.lpop("pending");

but without the client knowing anything about redis except the binary wire protocol – those commands are entirely dynamic. Actually a nice advantage of this is that the client doesn’t need to be updated as new redis features are released… but I digress!

Getting started

The key in implementing a dynamic API is implementing IDynamicMetaObjectProvider – although frankly I don’t propose doing that; I’m just going to subclass DynamicObject which does a lot of the work for us. So here's our first step:

public sealed class RedisClient : DynamicObject, IDisposable {...}

This gives us the start of a client that will respond to dynamic; although it doesn't actually do anything yet - we have to tell our object to handle dynamic method calls, which we do by overriding TryInvokeMember. Again, keep in mind that this is only a toy, and we’ll do this by simply writing the command name and parameters (in redis format) down the wire, and reading one result in redis format (note that this means that we’ll pay the full latency price per operation, which isn’t ideal – and that we can’t act as a redis pub/sub subscriber – that would simply not work, since replies don’t match neatly to commands then):

public override bool TryInvokeMember(
    InvokeMemberBinder binder,
    object[] args, out object result)
{
    WriteCommand(binder.Name, args);
    result = ReadResult();
    var err = result as RedisExceptionResult;
    if (err != null) throw err.GetException();
    return true;
}

In particular, note that the name of the method requested is available as binder.Name, and the parameters are just an object[].

Writing the command

I won't dwell on the redis protocol details (feel free to read the specification), but basically we need to write the number of arguments for the command (where the command-name itself counts as an argument), followed by the command-name, followed by each of the parameters. To avoid packet-fragmentation, we’ll use some buffering into a BufferedStream which we hold in outStream, which in turn writes to a NetworkStream which we hold in netStream - and obviously after each command we need to flush those to ensure they get to the server, so we get:

private void WriteCommand(string name, object[] args)
{
    WriteRaw(outStream, '*');
    WriteRaw(outStream, 1 + args.Length);
    WriteEndLine();
    WriteArg(name);
    for (int i = 0; i < args.Length; i++)
    {
        WriteArg(args[i]);
    }
    // and make sure we aren't holding onto any data...
    outStream.Flush(); // flushes to netStream
    netStream.Flush(); // just to be sure! (although this is a no-op, IIRC)
}

And for each argument, we need to write the length of the data, followed by the data itself. I won't detail each the various formats for different types of data, but: to avoid having to test "what type of data is this?", we'll cheat by having a few overloads of a method we'll call WriteRaw, and use dynamic to get the runtime to pick between the overloads for us... sneaky:

private void WriteArg(object value)
{
    // need to know the length, so: write to our memory-stream
    // first
    buffer.SetLength(0);
    WriteRaw(buffer, (dynamic)value);
    // now write that to the (bufferred) output
    WriteRaw(outStream, '$');
    WriteRaw(outStream, (int)buffer.Length);
    WriteEndLine();
    WriteRaw(outStream, new ArraySegment(buffer.GetBuffer(), 0, (int)buffer.Length));
    WriteEndLine();
}

Did you spot the cheeky dynamic in there? Since we're already in dynamic-land, it is hard to say that this is going to have any negative impact... so; why not? It means that if I need to support a new data-type, I just add a new WriteRaw to match, and: job done - and frankly, that's about it for sending the data.

Reading the response

After that, we need to refer to the specification again to see what replies look like - it turns out that they're basically the same format as the outbound data. But we have some ambiguity - does the user want their data as a byte[]? or as a decoded string? or maybe they want int? The nice thing is: we can let them tell us, by providing a result that supports conversions via dynamic - so when they type:

byte[] blob = client.get("my_image");
string name = client.get("name");

they get the binary and text correctly. So we can subclass DynamicObject again for a class that holds the raw result, and override another method - TryConvert this time. We get passed in a different binder, this time with access to the requested type in binder.Type - which we can then use to unscramble the data accordingly. This implementation is a less interesting and more tedious (testing different matched types), so I’ll leave that out of the blog. The only thing left to do (as shown in the TryInvokeMember) is to check if the response is an error-message from the server, and turn that into a thrown Exception, so that it feels intuitively .NET. The reason I can’t do this directly when reading the reply comes down to some implementation details – some replies are themselves composed of multiple nested replies, and I want to re-use the code internally for reading those. We can’t do that if it throws when hitting an error a few levels down – the stream could be left in an incomplete state (i.e. we might not have finished reading the outer-most reply).

Summary

And: that's it! A redis client written from scratch in a little over an hour; but more importantly, a complete dynamic API illustration. Well, maybe not complete, but as you can imagine: the other available operations (properties, indexers, operators, etc) work very similarly. I hope it is illustrative. Again, all the code for this is available on google-code.