Matts Sanctuary

For the past few days now I've been hiding myself away from the world. As I said last post, I've chosen to work on a compressed cache for NetBSD's virtual memory system. Seeing as there's a linux implementation I decided to look through the code and see how it was interacting with the Linux memory manager (mm). So for most of this past week I've been sifting through the source code and the patches provided at http://linuxcompresed.sourceforge.net.

Let me break down the way that the cache is going to work (and currently does work in the linux patches):

Usually, when memory in a computer system becomes scarce, the mm starts swapping pages to disk. Now, over the past few years the difference in speed between the CPU accessing data from main memory and accessing data from the hard disk has widened. Thus, swapping pages to disk is a very 'slow' process. This is where the compressed cache comes in.

When the system starts up, a portion of memory is put aside for the compressed cache. Ideally, this cache has an initial size and should be able to grow or shrink depending on how much it is being used (adaptability policy), because naturally, if there are no compressed pages in the compressed cache it's a complete waste of RAM space and should be 'shrunk'.

Ok, so we've got some space put aside for our compresssed cache.. and the system is chugging along nicely using up more and more free pages but not really freeing any. This can be caused by having lots of large processes in memory at once time, each requesting more and more pages. Naturally, there are only enough a finite amout of pages free. Suddenly the pagedaemon (the process that handles paging in NetBSD) can't keep up with the demand any more, it can't free pages fast enough and other processes are demanding more and more free pages all the time. Something must be done...

So the pagedaemon comes in and starts swapping pages out to 'backing store'. Now, this is a very slow process so what happens is, instead of writing these pages straight to disk, they're compressed and placed in the compressed cache. The page that contains the data can now be freed and used to meet the free page demand.

When the compressed cache becomes full the compressed pages are decompressed and finally written to swap. This has the advantage that if the free page demand is satisfied, and a process pagefaults on a page that we have stored in the compressed cache, we've just saved ourself an expensive write to disk..

I _think_ that makes sense. Please feel free to leave any comments if you didn't understand or (and this is totally probable) that I got something wrong.

Until next time...