Friday, 28 March 2014

Windows Redis-64

As you know, I’m a huge fan of Redis. This week, a team at the Microsoft Open Tech group released a new drop of redis-64 for windows (note: all links there are old – github is fresher), with binaries available on both NuGet and chocolatey. It is even marked by them as “production ready”. The current drop is 2.8.4 in Redis terms (the last public drop was 2.6.12).

So congratulations, thanks and kudos to the team involved: an important milestone.

In production, at Stack Exchange we use CentOS to host Redis, so I’m simply not in a position to comment on how well it works in production, but for windows developers using Redis it is really handy having convenient access to a Redis server without having to spin up a VM.

One really important thing to watch

From reading the "Redis Release Notes.docx" and "redis.windows.conf" file in the packages/Redis-64.2.8.4 folder, the appoach they used to solve the fork problem was to introduce a shared, memory-mapped file, and it defaults to the size of your physical memory (and that is before it has forked, when it can grow due to copy-on-write). So whether you are using it as a production server or a dev tool, you might want to pay particular attention the the “maxheap” setting in your .conf file. For my local dev work I usually spin up 5 servers, and I currently have 24GB physical memory in my machine – so by default, when running 5 completely empty databases, it instantly chewed up 120GB of hard disk space (you get it back when the server exits). If you don’t have enough disk space: the server won’t start. You'll probably see:

QForkMasterInit: system error caught. error code=0x000005af, message=VirtualAllocEx failed.

Fortunately it is simple to configure a memory bound:

maxheap 4gb # or whatever

Please note – this really isn’t a criticism; I understand the “why” – they need the shared memory-mapped file for the pseudo-fork, and they need to allocate the “maxheap” amount from the outset because IIRC you can’t grow such a map very flexibly. My intent is merely to flag it in flashing colours that if you’re playing with Redis-64, you want to think about your memory vs disk.

Now go, play, have some Redis fun.

20 comments:

PanKak said...

I did a stress test several times using 40 different threads each one requesting a different page from a pool of 10.000 real life pages. Every thread requests a different page every 5ms and each test lasted 15 minutes.
The pages do heavy redis work and they utilize Stackexchange.Redis on MsOpen tech redis x64.

I performed the test several times in a production web server (4 cpu's - 8gb memory - ssd's) and It worked like a charm!!! I got 300 pages/sec throughput.... I suppose that's a good score...

The funny think was that I did the tests before I read your post and I did noticed indeed the large file that you mentioned. By reading your post I understood why this is happening. Thanks!

Pankak said...

Maybe I could achieve even better throughput by using the async api's but our site currently has about 40 pages/sec request ratio so I suppose that it's meaningless to change al my repositories in order to use async.

Pankak said...

Maybe I could achieve even better throughput by using the async api's but our site currently has about 40 pages/sec request ratio so I suppose that it's meaningless to change al my repositories in order to use async.

Marc Gravell said...

@Pankak I did not suggest bending pages out of shape just to use async. We actually primarily use the sync API at stack exchange. The async API is awesome for pipelining and for true async/await code. But sync works great too - and due to multiplexing, you get concurrency for free with the convenience of procedural code.

Pankak said...

Yes you have done superb work with the muxer!!!
I'm very grateful for sharing the code.

Anonymous said...

Thanks Mark, you're a life-saver.

Having made a custom build of Redis (the latest bins are not compatible with Server 2008 R1), I was hitting the QFork error only on one box (which happens to have lots of RAM).

That left me scratching my head...until I found your post!

Anonymous said...

Where do I add the line
maxheap 4gb?

I have modified the redis.windows.conf file with the following: maxheap 104857600

But I'm still receiving the same error message: QForkMasterInit: system error caught. error code=0x000005af, message=VirtualAllocEx failed.

Anonymous said...

I've also tried other values such as 100mb but to no avail.

Anonymous said...

OK got it working. I just had to close all the programs running on my machine.

Marc Gravell said...

@Anon you don't edit that file. You usually create your own .conf file, and run: redis-server your.conf (with paths etc)

Baz said...

we have two servers of 220gb of ram with 6 redis instance each on windows redis msopentech 2.6.12 and its run like a charm.
2.8.4 is currently in one of this instance and memory optimization is great (less fragmentation)
However a bug exist with the bgsave with the 2.8.4 version that don't reset the counter of last save

Marc Gravell said...

@Baz please report that to the MS team (which is: not me). I've had success reporting issues via the contact link on the NuGet package.

Marc Gravell said...

please report that to the MS team (which is: not me). I've had success reporting issues via the contact link on the NuGet package

Pankak said...

@baz
That's good news because I didn't know anyone until now that is using in production the win version of redis and benchmarks are very difficult to setup in order to test realtime like performace.

Adam Kozmic said...

Awesome! However, I'm getting the same error message when connecting to localhost.

"It was not possible to connect to the redis server(s); to create a disconnected multiplexer, disable AbortOnConnectFail"

I passed a string writer in as the log and here's the output:

//CODE
this.Redis = ConnectionMultiplexer.Connect(this.ConnectionString, sw);

//OUTPUT
"{localhost:6379
1 unique nodes specified
Requesting tie-break from localhost:6379 > __Booksleeve_TieBreak...
Allowing endpoints 00:00:01 to respond...
localhost:6379 did not respond
localhost:6379 failed to nominate (WaitingForActivation)
No masters detected
localhost:6379: Standalone v2.0.0, master; keep-alive: 00:01:00; int: Connecting; sub: ConnectedEstablished, 1 active; not in use: DidNotRespond
localhost:6379: int ops=0, qu=4, qs=0, qc=0, wr=0, socks=1; sub ops=2, qu=0, qs=0, qc=0, wr=0, subs=1, sync=2, socks=1
Circular op-count snapshot; int: 0 (0.00 ops/s; spans 10s); sub: 0+2=2 (0.20 ops/s; spans 10s)
Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago
Starting heartbeat...
}"

This does not happen every time, but enough to be problematic.

Any thoughts?

Adam Kozmic said...

To follow-up here's what a success looks like:

"{localhost:6379

1 unique nodes specified
Requesting tie-break from localhost:6379 > __Booksleeve_TieBreak...
Allowing endpoints 00:00:01 to respond...
localhost:6379 returned with success
localhost:6379 had no tiebreaker set
Single master detected: localhost:6379
localhost:6379: Standalone v2.4.6, master; keep-alive: 00:01:00; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active
localhost:6379: int ops=10, qu=0, qs=1, qc=0, wr=1, sync=8, socks=1; sub ops=2, qu=0, qs=0, qc=0, wr=0, subs=1, sync=2, socks=1
Circular op-count snapshot; int: 0+10=10 (1.00 ops/s; spans 10s); sub: 0+2=2 (0.20 ops/s; spans 10s)
Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago
Starting heartbeat...
}"

Adam Kozmic said...

Sorry those last two posts were meant for a different thread... My apologies.

Anonymous said...

(What can I say , you're the man) ----Marc, I just downloaded the 2.8.9 ( latest from msOpenTech and I had this exact problem. changing maxHeap (4294967296) solved it. thanks you :-)))

(Royi-Namir)

Gerwin said...

Very helpful post. My windows 7 machine was failing to start due to this error but my windows 8 would go ahead and allocate 4gb automatically. Not sure why the difference but it's all working now. Cheers.

Johan Svedberg said...

Do you know if you can have Redis save the file in another directory? Can't find a config entry for this.