State Assassins with experience of Reporting Services

No, this isn’t one of the more unlikely job listings on careers – instead, a bit of work from last week that I found pretty interesting.

I was essentially adding some consultancy input to an unfamiliar codebase; the problem was that the site would be running very smoothly, then page-times would go crazy for a while, and then suddenly go back to normal.

It turned out (after putting state into SQL-Server for analysis) that some of the state records were unexpectedly large. Damagingly huge. So what were they? After all, the app only used state to store one tiny innocent object…

One advantage of using a database for state is that you can get the data back out, and it turns out that it isn’t too hard to reverse a BLOB back into the state objects (it helped that stackoverflow had half the answer already).

The cuplrit? You might guess (from the post title): reporting services and the web-forms report viewer control. As soon as a user viewed a report, it was using state to cache the data – which might work great for small reports, but if the report is large it gets… fun.

Perhaps a design failing in ASP.NET state; once a large object is in state, it goes up and down the wire (between the web-server and the state storage) on every state-enabled request for that user, even when no longer using that data. So if they used the report 10 minutes ago, and are now just looking at nice slim tidy pages, they are still dragging a huge state object over the network all the time, potentially crippling the site for lots of other users.

If the page had been highly AJAX-enabled it could have been much worse. And the reason the problem ended so abruptly? My reasoning is that the user in question (who is harming the site through no fault of their own) got bored of sluggish responses and did something else for a while.

Learning points:

• Even if you aren’t using SQL Server for state normally, it makes analysis much easier as a temporary swap-out
• Watch out for web-controls (especially reports) that might (ab)use state
• You can decipher a regular state BLOB – it isn’t tricky (I’ll update that SO question with an example if anyone is interested)
• Consider partitioning your app, or disabling state in some scenarios
• We aware of how much data your reports are dragging around with them