Monday, 4 July 2011

BookSleeve, transactions, hashes, and flukes

Sometimes, you just get lucky.

tl;dr; version : BookSleeve now has transactions and Hashes; new version on nuget

Transactions

For some time now I’ve been meaning to add multi/exec support to BookSleeve (which is how redis manages transactions). The main reason it wasn’t there already is simply: we don’t use them within StackExchange.

To recap, because BookSleeve is designed to be entirely asynchronous, the API almost-always returns some kind of Task or (more often) Task-of-T; from there you can hook a “continue-with” handler, you can elect to wait for a result (or error), or you can just drop it on the floor (fire-and-forget).

Over the weekend, I finally got around to looking at multi/exec. The interesting thing here is that once you’ve issued a MULTI, redis changes the output – return “QUEUED” for pretty much every command. If I was using a synchronous API, I’d be pretty much scuppered at this point; I’d need to duplicate the entire (quite large) API to provide a way of using it. I’d love to say it was by-design (I’d be lying), but the existing asynchronous API made this a breeze (relatively speaking) – all I had to do was wrap the messages in a decorator that expects to see “QUEUED”, and then (after the EXEC) run the already-existing logic against the wrapped message. Or more visually:

conn.Remove(db, "foo"); // just to reset
using(var tran = conn.CreateTransaction())
{ // deliberately ignoring INCRBY here
tran.Increment(db, "foo");
tran.Increment(db, "foo");
var val = tran.GetString(db, "foo");

tran.Execute(); // this *still* returns a Task

Assert.AreEqual("2", conn.Wait(val));
}

Here all the operations happen as an atomic (but pipelined) unit, allowing for more complex integrity conditions. Note that Execute() is still asynchronous (returning a Task), so you can still carry on doing useful work while the network and redis server does some thinking (although redis is shockingly quick).

So what is CreateTransaction?

Because a BookSleeve connection is a thread-safe multiplexer, it is not a good idea to start sending messages down the wire until we have everything we need. If we did that, we’d have to block all the other clients, which is not a good thing. Instead, CreateTransaction() creates a staging area to build commands (using exactly the same API) and capture future results. Then, when Execute() is called the buffered commands are assembled into a MULTI/EXEC unit and sent down in a contiguous block (the multiplexer will send all of these together, obviously).

As an added bonus, we can also use the Task cancellation metaphor to cleanly handle discarding the transaction without execution.

The only disadvantage of the multiplexer approach here is that it makes it pretty hard to use WATCH/UNWATCH, since we wouldn’t be sure what we are watching. Such is life; I’ll think some more on this.

Hashes

Another tool we don’t use much at StackExchange is hashes; however, for your convenience this is now fully implemented in BookSleeve. And since we now have transactions, we can solve the awkward varadic/non-varadic issue of HDEL between 2.2+ and previous versions; if BookSleeve detects a 2.2+ server it will automatically use the varadic version, otherwise it will automatically use a transaction (or enlist in the existing transaction). Sweet!

Go play

A new build is on nuget; I hope you find it useful.

Wednesday, 8 June 2011

Profiling with knobs on

A little while ago, I mentioned the core part of the profiling tool we use at stackexchange – our mini-profiler. Well, I’m delighted to say that with the input from Jarrod and Sam this tool has been evolving at an alarming rate, and has now gained:

  • full integration with our bespoke SQL profiler, which is now also included
    • basically, this is a custom ADO.NET DbConnection that you wrap around your existing connection, and it adds logging
  • an awesome UI (rather than html comments), including a much clearer timeline
  • full support for AJAX activity
  • some very basic highlighting of duplicated queries, etc
  • webform support
  • and a ton of other features and enhancements

To illustrate with Jarrod’s image from the project page:

miniprofiler

The custom connection has been exercised extensively with dapper-dot-net, LINQ-to-SQL and direct ADO.NET, but should also work with Entity Framework (via ObjectContextUtils, provided) and hopefully any other DB frameworks built on top of ADO.NET.

We use it constantly. It is perhaps our primary tool in both understanding problems and just keeping a constant eye on the system while we use it – since this is running (for our developers) constantly, 24×7.

All too often such fine-grain profiling is just “there’s a problem, ask a DBA to attach a profiler to the production server for 2 minutes while I validate something”.

Enjoy.

http://code.google.com/p/mvc-mini-profiler/

Wednesday, 25 May 2011

So soon, beta 2

I’m delighted by the level of response to my protobuf-net v2 beta; I have been kept really busy with a range of questions, feature requests and bug fixes (almost exclusively limited to the new features).

So, before I annoy people with too many “fixed in source” comments, I thought I’d better re-deploy. Beta 2 gives the same as beta 1, but with:

  • reference-tracked objects
    • root object now included (BREAKING CHANGE)
    • fixed false-positive recursion issue
  • full type metadata
    • support custom formatting/parsing
  • interface serialization
    • support serialization from interfaces
    • support known-implementations of interfaces
    • support default concrete type
  • WCF
    • expose WCF types on public API to allow custom models
  • misc
    • support shadow set-methods
    • fix IL glitch
    • fix cold-start thread-race condition
    • fix missing method-forwarding from legacy non-generic API

In addition to a very encouraging level of interest, I’m also pleased that things like the interface-based support mentioned above were pretty effortless to push through the v2 type-model. These are things that had been proposed previously, but there was just no way to push them into the codebase without hideous (and cumulative) hackery. With v2, it was pretty much a breeze.

But for v2 fun see the project download page

And please, keep pestering me ;p

Note that the BREAKING CHANGE marked above only applies to data serialized with the new full-graph AsReference option. No v1 data is impacted, but if you have stored data from earlier v2 alpha/beta code, please drop me a line and I can advise.

Thursday, 19 May 2011

protobuf-net v2, beta

It has been a long time in the coming, I know. I make no excuses, but various other things have meant that it didn’t get much input for a while…

But! I’m happy to say that I’ve just pushed a beta download up onto the project site

So… this v2… what is it?

Basically, it is still the core protobuf stream, but against all the safe advice of my employer I rewrote the entire core. For many reasons:

  • the original design grew organically, and ideas that seemed good turned to make for some problematic code later
  • the vast overuse of generics (and in particular, generics at runtime via reflection) was really hurting some platforms
  • there were many things the design could not readily support – structs, runtime models, pre-built serialization dlls, usage on mobile devices, etc
  • the code path at runtime was just not as minimal as I would like
  • and a myriad of other things

So… I rewrote it. Fortunately I had a barrage of integration tests, which grew yet further during this exercise. The way I figured, I had 2 choices here:

  • go with a richer code-generator to use as a build task (not runtime)
  • go crazy with meta-programming

After some thought, I opted for the latter. In particular, this would (in my reasoning) give me lots of flexibility for keeping things runtime based (where you are able to, at least), and would be a good opportunity to have some fun and learn IL emit to dangerous levels (which, incidentally, turns out to be very valuable – hence mine and Sam’s work on dapper-dot-net).

So what is new?

I don’t claim this list is exhaustive:

  • the ability to define models at runtime (also useful for working with types you don’t control) (example)
  • the ability to have separate models for the same types
  • pre-generation to dll
  • runtime performance improvements
  • support for structs
  • support for immutable objects via surrogates (example)
  • the ability to run on mobile platforms (unity, wp7 – and perhaps moot for a while: MonoDroid, MonoTouch – but presumably Xamarin)
  • the ability to serialize object graphs preserving references (example)
  • the ability to work with types not known in advance (same)
  • probably about 40 other things that slip my mind

I’ll try to go into each of these in detail when I can

And what isn’t there yet?

This is beta; at some point I had to make a milestone cut – but some things aren’t quite complete yet; these are not usually needed (hint: I’m happy to use protobuf-net v2 on my employer’s code-base, so I have confidence in it, and confidence that the core paths have been smoke-tested)

But things that definitely aren’t there in the beta

  • WCF hooks
  • VS tooling (it is just the core runtime dlls)
  • round-trip safe / “extension” fields
  • extraction of .proto (schema) files from a model
  • my build / deploy script needs an overhaul
  • tons of documentation / examples
  • tooling to make the “mobile” story easier (but; it should work – it is being used in a few, for example)

But; have at it

If you want to play with it, go ahead. I only ask that if something behaves unexpectedly, drop me a line before declaring loudly “it sux, dude!”. It might just need some guidance, or maybe even a code fix (I’m far from perfect).

Likewise, any feature suggestions, etc; let me know.

Wednesday, 27 April 2011

Completion tasks - the easy way

One of the good things about doing things in public – people point out when you’ve missed a trick.

Just the other day, I was moaning about how Task seemed too tightly coupled to schedulers, and wouldn’t it be great if you could have a Task that just allowed you to indicate success/failure – I even went so far as to write some experimental code to do just that.

So you can predict what comes next; it already existed – I just didn’t know about it. I give you TaskCompletionSource<T>, which has SetResult(…), SetException(…) and a Task property. Everything I need (except maybe support for result-free tasks, but I can work around that).

And wouldn’t you believe it, the TPL folks shine again, easily doubling my lousy performance, and trebling (or more) the performance compared to “faking it” via RunSynchronously:

Future (uncontested): 1976ms // my offering
Task (uncontested): 4149ms // TPL Task via RunSynchronously
Source (uncontested): 765ms // via TaskCompletionSource
Future (contested): 5608ms
Task (contested): 6982ms
Source (contested): 2389ms


I am hugely indebted to Sam Jack, who corrected me via a comment on the blog entry. Cheers!

Monday, 25 April 2011

Musings on async

OUT OF DATE: SEE UPDATE


For BookSleeve, I wanted an API that would work with current C#/.NET, but which would also mesh directly into C# 5 with the async/await pattern. The obvious option there was the Task API, which is familiar to anyone who has used the TPL in .NET 4.0, but which gains extension methods with the Async CTP to enable async/await.

This works, but in the process I found a few niggles that made me have a few doubts:

Completion without local scheduling

In my process, I never really want to schedule a task; the task is performed on a separate server, and what I really want to do is signal that something is now complete. There isn’t really an API for that on Task (since it is geared more for operations you are running locally, typically in parallel). The closest you can do is to ask the default scheduler to run your task immediately, but this feels a bit ungainly, not least because task-schedulers are not required to offer synchronous support.

In my use-case, I’m not even remotely interested in scheduling; personally I’d quite like it if Task supported this mode of use in isolation, perhaps via a protected method and a subclass of Task.

(yes, there is RunSynchronously, but that just uses the current scheduler, which in library code you can’t assume is capable of actually running synchronously).

Death by exception

The task API is also pretty fussy about errors – which isn’t unreasonable. If a task fails and you don’t explicitly observe the exception (by asking it for the result, etc), then it intentionally re-surfaces that exception in a finalizer. Having a finalizer is note-worthy in itself, and you get one last chance to convince the API that you’re sane – but if you forget to hook that exception it is a process-killer.

So what would it take to do it ourselves?

So: what is involved in writing our own sync+async friendly API? It turns out it isn’t that hard; in common with things like LINQ (and foreach if you really want), the async API is pattern-based rather than interface-based; this is convenient for retro-fitting the async tools onto existing APIs without changing existing interfaces.

What you need (in the Async CTP Refresh for VS2010 SP1) is:

  • Some GetAwaiter() method (possibly but not necessarily an extension method) that returns something with all of:
  • A boolean IsCompleted property (get)
  • A void OnCompleted(Action callback)
  • A GetResult() method which returns void, or the desired outcome of the awaited operation

So this isn’t a hugely challenging API to implement if you want to write a custom awaitable object. I have a working implementation that I put together with BookSleeve in mind. Highlights:

  • Acts as a set-once value with wait (sync) and continuation (async) support
  • Thread-safe
  • Isolated from the TPL, and scheduling in particular
  • Based on Monitor, but allowing efficient re-use of the object used as the sync-lock (my understanding is that once used/contested in a Monitor, the object instance obtains additional cost; may as well minimise that)
  • Supporting typed (Future<T>) or untyped (Future) usage – compares to Task<T> and Task respectively

My local tests aren’t exhaustive, but (over 500,000 batches of 10 operations each), I get:

Future (uncontested): 1993ms
Task (uncontested): 4126ms
Future (contested): 5487ms
Task (contested): 6787ms

So our custom awaitable object is faster… but I’m just not convinced that it is enough of an improvement to justify changing away from the Task API. This call density is somewhat artificial, and we’re talking less than a µs per-operation difference.

Conclusions

In some ways I’m pleasantly surprised with the results; if Task is keeping up (more or less), even outside of it’s primary design case, then I think we should forget about it; use Task, and move on to the actual meat of the problem we are trying to solve.

However, I’ll leave my experimental Future/Future<T> code as reference only off on the side of BookSleeve – in case anybody else feels the need for a non-TPL implementation. I’m not saying mine is ideal, but it works reasonably.

But: I will not be changing away from Task / Task<T> at this time. I’m passionate about performance, but I’m not (quite) crazy; I’ll take the more typical and more highly-tested Task API that has been put together by people who really, really understand threading optimisation, to quite ludicrous levels.

Tuesday, 19 April 2011

Practical Profiling

Profiling the hard wayIf you don’t know what is causing delays, you are doomed to performance woes. Performance is something I care about deeply, and there are no end of performance / profiling tools available for .NET. Some invasive and detailed, some at a higher level. And quite often, tools that you wouldn’t leave plugged into your production code 24/7.

Yet… I care about my production environment 24/7; and trying to reproduce a simulated load for the sites I work on can be  somewhat challenging. So how can we get realistic and detailed data without adversely impacting the system?

Keep It Simple, Stupid

A common strapline in agile programming states:

Do the simplest thing that could possibly work

Now, I don’t profess to be strictly “agile”, or indeed strictly anything (except maybe “pragmatic”) in my development process, but there is much wisdom in the above. And it makes perfect sense when wanting to add constant (live) profiling capabilities.

So what is the simplest thing that could possibly work with profiling? Automated instrumentation? Process polling on the debugging API? Way too invasive. IoC/DI chaining with profiling decorators? Overkill. AOP with something like PostSharp? Unnecessary complexity. How about we just tell the system openly what we are doing?

Heresy! That isn’t part of the system! It has no place in the codebase!

Well, firstly – remember I said I was “pragmatic”, and secondly (more importantly) performance is both a requirement and a feature, so I have no qualms whatsoever changing my code to improve our measurements.

So what are you talking about?

Frustrated by the inconvenience of many of the automated profiling tools, I cobbled together the simplest, hackiest, yet fully working mini-profiler – and I thought I’d share. What I want is as a developer, to be able to review the performance of pages I’m viewing in the production environment – sure, this doesn’t cover every scenario, but it certainly does the job on sites that are read-intensive. So say I browse to “http://mysite/grobbits/MK2-super-grobit” – I want immediate access to how that page was constructed (and where the pain was). And in particular, I want it live so I can hit “refresh” a few times and watch how it behaves as different caches expire. Nothing rocket-science, just a basic tool that will let me hone in on the unexpected performance bumps. Finally, it can’t impact performance the 99.99% of regular users who will never see that data.

I’m currently having great success using this mini tool; the concept is simple – you have a MiniProfiler object (which would be null for most users), and you just surround the interesting code:

using (profiler.Step("Set page title"))
{
ViewBag.Message = "Welcome to ASP.NET MVC!";
}

using (profiler.Step("Doing complex stuff"))
{
using (profiler.Step("Step A"))
{ // something more interesting here
Thread.Sleep(100);
}
using (profiler.Step("Step B"))
{ // and here
Thread.Sleep(250);
}
}

(the Step(…) method is implemented as an extension method, so it is perfectly happy operating on a null reference; which is a crude but simple way of short-circuiting the timings for regular users)

Obviously this isn’t very sophisticated, and it isn’t meant to be – but it is very fast, and easy to make as granular as you like as you focus in on some specific knot of code that is hurting. But for the simplicity, it is remarkably useful in finding trouble spots on your key pages, and reviewing ongoing performance.


So what do I see?

The output is very basic functional – you simply get a call tree of the code you’ve marked as interesting (above), to whatever granularity you marked it to; no more, no less. So with the sample project (see below), the home-page displays (in the html markup):


<!--
MYPC at 19/04/2011 11:28:12
Path: http://localhost:3324/
http://localhost:3324/ = 376.9ms
> Set page title = 0ms
> Doing complex stuff = 349.3ms
>> Step A = 99.3ms
>> Step B = 250ms
> OnResultExecuting = 27.5ms
>> Some complex thinking = 24.8ms
-->

(for your developers only; there is no change for users without the profiler enabled)

Stop waffling, Man!

Anyway, if you have similar (high-level, but live) profiling needs, I’ve thrown the mini-profiler onto google-code and NuGet, along with a tweaked version of the MVC3 (razor) sample project to show typical usage. Actually, you only need a single C# file (it really is basic code, honest).

  • If you find it useful, let me know!
  • If it sucks, let me know!
  • If I’ve gone mad, let me know!
  • I’ve you’ve gone mad, keep that to yourself.