tag:blogger.com,1999:blog-238633042024-03-08T03:00:57.768+00:00Matts SanctuaryMatt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.comBlogger30125tag:blogger.com,1999:blog-23863304.post-1160761283561497782006-10-13T18:38:00.000+01:002006-10-13T18:41:23.576+01:00Journaling for FFSAfter seeing the failed Google Summer of Code project of integrating journaling support for BSD's FFS, I decided that I'd quite like to work on that project. So, time and university work allowing, I'm going to try and make a start on the code soon. Stay tuned for updates.<br /><br />p.s I also received my SoC t-shirt and certificate yesterday, they're both pretty cool.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com1tag:blogger.com,1999:blog-23863304.post-1157469260888177062006-09-05T15:24:00.000+01:002006-09-05T16:14:20.950+01:00kauth(9) regression test frameworkSeeing as this year's SoC is over I've been spending some time working on a project for Netbsd. As some of you may know, Elad Efrat has been working on a new authorization model for the NetBSD kernel, kauth(9). It's based on the kauth framework from MacOSX and it's my job to help Ober to produce a regression test framework to ensure that the transition from a traditional security model (refered to in literature as "bsd44") has no adverse side-effects (such as allowing a non-superuser to reboot the system) because the kauth framework hasn't implemented something properly. I won't discuss how the framework works here, refer to the kauth(9) man page for more details and the kernel authorization technical note from apple <a href="http://developer.apple.com/technotes/tn2005/tn2127.html">here</a>.<br /><br />The real purpose of this blog entry is to stress the need for regression tests in every project :P The regression framework and tests are written in Python as I found it the quickest language to write designs up in. I wrote the framework in two days and began implementing the tests and running them with a kernel that had Elad's patches compiled in. Within a few runs we'd found bugs in Elad's implementation and in fact, in the NetBSD date(1) command. This bug in the date(1) had been around for years and no one had ever reported it before. These tests have been invaluable and it still fails to amaze me how important good regression tests are to the success of a project, especially when you're implementing new policies or changing existing implementations of interfaces. The tests are also adaptable for different models.<br /><br />Everyone write unit tests ;-)Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1156280403307981132006-08-22T21:18:00.000+01:002006-08-22T22:00:03.346+01:00End of SoCWell that's it now. Summer of Code is over for another year. I've had a great time working on it and managed to get some nice stuff into pdb/pydb. Whilst I didn't complete all of the goals I'd set myself there's definately been some progress. One new thing that pydb has is gdb-like signal handling.<br /><br />* gdb-like signal handling<br /><br />Also yet to be merged but working is the ability to debug a process from another process and the ability to debug a process remotely. These will be neat features when they go in.<br /><br />The thing that I didn't complete though, that the ability to debug threaded applications. This proved to be much more involved than either I or my mentor had first thought. However, we're still working on this and hope to get it into pydb soon.<br /><br />If anyone's thinking about taking part next year, I whole-heartedly recommend it! Having a mentor to guide you is great, and you get to take advantage of that fountain of experience. A big thanks to my mentor, Rocky!<br /><br />MattMatt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1154393898997724352006-08-01T01:53:00.000+01:002006-08-01T01:58:19.010+01:00Deadlock in the Python LibI spent most of today debugging a deadlock case in the Python Standard Library. I'm not sure if anyone's ever tried, but if you write a tracing function and pass it to threading.settrace, and within this function call 'threading.currentThread()', when the trace functions traces the thread code through threading.Thread.__delete, if will try to call threading.currentThread() *AFTER* _active[_get_ident()] has been deleted (threading.currentThread() just returns _active[_get_ident()]). Now the Python developers thought of the possibilty of asking for the currentThread() when it had been removed from _active, so they decided to create a _DummyThread object. Only this object tries to acquire a lock held in the threading.Thread.__delete method. *BOOM* deadlock.<br /><br />Anywho, I'm tired, it's nearly 2am...Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1153868959345956692006-07-25T23:34:00.000+01:002006-07-26T00:09:19.363+01:00Week 9Been working on allowing debugging by a debugger running in a separate process for most of this week. I pushed this forward in the list of things to do (in front of thread debugging) as I'd spent quite a long time working on allowing thread debugging and was getting a little frustrated. Ok,<br />so, 'debugging an already running process?', I hear you say. Yes! I'll give a quick overview, if you're interested, check out the documentation in the sandbox/pdb/Doc directory.<br /><br />So, you've got your simple chat application, that allows clients to connect to a server and messages are broadcast to each of the clients. But you're a little unsure if the code you've written is totally stable, you'd better be able to debug it. But you realise you want to debug the program from a different terminal, but still on the same machine (think, logging in to a machine at work from home). Add this little bit of code to enable debugging from another process<br /><br /><pre><br />import mpdb ; mpdb.process_debugging()<br /></pre><br /><br />What this does is sets up a signal handler and when it catches a signal a pdbserver is started. You can specify the pdbserver parameters by passing them to process_debugging. The default<br />connection parameters are a FIFO connection with a filename composed of the temp directory on your system (as returned by tempfile.gettempdir()), the PID of the process and the word 'mpdb'. For, for example, running this on my NetBSD machine I get,<br /><br /><pre><br />>>> import mpdb ; mpdb.process_debugging()<br />'mconnection.MConnectionServerFIFO /tmp/25813mpdb'<br /></pre><br /><br />But say, you wanted to start a pdbserver using TCP on localhost:8000 (you can only use localhost as a hostname for a TCP connection with process_debugging),<br /><br /><pre><br />>>> mpdb.process_debugging(protocol='tcp', addr=':8000')<br />'tcp :8000'<br /></pre><br /><br />Now a debugger can start a debugging session by issuing the 'attach' command, which takes a PID as an argument and sends (by default) a SIGUSR1 signal to that PID. This gets handled by the handler in mpdb and starts a pdbserver. The debugger then connects. *BAM* have a debugging-good-time. By the way, use the 'detach' command to detach from the debugger, as it allows the program being debugged to continue execution. Typing 'quit' quits the program :PMatt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1152479703528548272006-07-09T22:07:00.000+01:002006-07-09T22:15:03.553+01:00Tim Peters' current-frames branchTim posted a message to python-dev today asking if he could get some changes into the threading code of Python, namely, exposing a mechanism for mapping thread names to thread frame objects in a dict. This is the original conversation back in March 2005, <a href="http://mail.python.org/pipermail/python-dev/2005-March/051856.html">http://mail.python.org/pipermail/python-dev/2005-March/051856.html</a>.<br /><br />Since the Python trunk is in "feature-freeze" and Tim's patches can't be incorporated, it looks as though he's made his own branch for these features, tim-current_frames. The actual code is going to be used to get the functionality of a module called 'threadframe' into the Python core. This could really be useful to my SoC project for using these features to help debug threaded applications :-) Who knows, if I can think of some useful things to add, maybe Tim will accept them?<br /><br />I'll keep y'all posted.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com1tag:blogger.com,1999:blog-23863304.post-1152019258219503502006-07-04T14:17:00.000+01:002006-07-04T14:20:58.230+01:00Non-Soc related post.Well yesterday one of my lecturers at <a href="http://www.comp.leeds.ac.uk">university</a> emailed me to tell me that he's changed my degree programme to computer science from computing, which was at my request. This is a good thing for me a) because computer science has much more interesting classes than computing b) I would have wasted option-points choosing comp sci classes and would still have had to do the boring computing classes! So, all in all, I'm a happy man..Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1151167434775385852006-06-24T17:42:00.000+01:002006-06-24T19:05:52.896+01:00Week 5Well this is week 5 and I'm currently working on thread debugging (I changed the order in my proposal but the times are still the same). Committed some code this morning to allow a user to call mpdb.py with a switch specifying either a target to connect to or a pdbserver instance to start. Also got the code working to set up thread debugging, i.e. setting up our tracer object for each thread that follows the threads execution. When I'm eventually done you'll be able to set break points and inspect frames, all the usual cool shizzle.<br /><br />Although I did make a horrible discovery. It seems that the unit tests I had written for the mpdb module were passing purely by 'coincidence' and shouldn't have been passing at all. It was a problem with the way I'd written the code to set up a pdbserver and connect a client debugger to it. It's actually still not fixed, as I don't have time tonight. But I'll trying to fix it tomorrow.<br /><br />Also checkout the README.txt sandbox/trunk/pdb for things left 'todo'.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com1tag:blogger.com,1999:blog-23863304.post-1150754454501107982006-06-19T22:08:00.000+01:002006-06-19T23:00:54.526+01:00TAO thread debuggingFor the past few days I've been sifting through Python's thread handling code trying to figure out how I'm going to facilitate the debugging of threads in mpdb. Now, the Python interpreter provides no way for Python code to switch between threads, nor does it provide a way to suspend, stop, resume, interrupt any thread. So how, you may ask, are we going to enable thread debugging in mpdb? The answer is simply, 'I haven't decided yet'. I'm still running through some possible solutions, which I'll discuss now.<br /><br />1] This is the least desirable solution. Code a Python module in C to look at the state of each of the threads and try to get some more control over it.<br /><br />2] Have a main MPdb object that creates, initially sets the settrace method for all threads that are created from the threading module to some helper function inside mpdb that creates a new tracer object and then, from within the thread reassigns the settrace method to that objects trace method. Sound confusing? It kinda is. The main is that, because these tracer objects are under our control (i.e. the debugger's) we can handle all the synchronisation of how the main MPdb object wants to receive data from the tracer object.<br /><br />3] Similar as above except using tracer threads instead of objects. This is how it's done in rpdb2 (Nir Aides debugger) with positive results.<br /><br />4] Make calls to mpdb.settrace() inside thread code which sets the tracer function to one defined in the mpdb module. Whilst this would work for both threads created with thread.start_new_thread() and the threading module, it's not all happy-days because the programmer has to modify their buggy code in order to monitor the threads.<br /><br />At the moment I'm leaning in the direction of #2, but we should be have confirmation by the end of the week as to exactly which design is going to appear in mpdb.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1150071323604313232006-06-12T00:54:00.000+01:002006-06-12T01:17:21.363+01:00Checking in..Well, I haven't posted here for quite a few days. But I've started making some progress with the<br />code in the tree. There's now a pdbserver command analogous to the gdbserver command in gdb, and a target command to connect to a pdbserver. Here's some docstrings,<br /><br />target,<br /><pre><br />""" Connect to a target machine or process.<br />The first argument is the type or protocol of the target machine<br />(which can be the name of a class that is avaible either in the current<br />working directory or in Python's PYTHONPATH environtment variable).<br />Remaining arguments are interpreted by the target protocol. For more<br />information on the arguments for a particular protocol, type<br />`help target ' followed by the protocol name.<br /><br />List of target subcommands:<br /><br />target serial -- Use a remote computer via a serial line<br />target tcp -- Use a remote computer via a socket connection<br />"""<br /></pre><br /><br />So, from a MPdb prompt you can do<br /><br />`target tcp localhost:8000'<br /><br />which connects to a pdbserver running on port 8000 on localhost.<br /><br />pdbserver,<br /><br /><pre><br />""" Allow a debugger to connect to this session.<br />The first argument is the type or protocol that is used for this connection<br />(which can be the name of a class that is avaible either in the current<br />working directory or in Python's PYTHONPATH environtment variable).<br />The next argument is protocol specific arguments (e.g. hostname and<br />port number for a TCP connection, or a serial device for a serial line<br />connection). The next argument is the filename of the script that is<br />being debugged. The rest of the arguments are passed as arguments to the<br />script file and are optional. For more information on the arguments for a<br />particular protocol, type `help pdbserver ' followed by the protocol name.<br />The syntax for this command is,<br /><br />`pdbserver ConnectionClass comm scriptfile [args ...]'<br /><br />"""<br /></pre><br /><br />And we can do<br /><br />`pdbserver tcp localhost:8000 broken_script.py'<br /><br /><br />One thing that you can do with this is instead of naming 'tcp' as the protocol, you can name a class as the protocol, for example, SocketServer.TCPServer (not that this would actually work, but you get the idea). More importantly, there's an abstract interface that connection classes must implement in order to be used by pdbserver. This class is called MServerConnectionInterface so somebody could derive from this class in a class named MyNewConnectionClass and (as long as this class was in the current directory of in PYTHONPATH environment variable) if this class was ina file named connclasses.py a user could do<br /><br />`pdbserver connclasses.MyNewConnectionClass localhost:8000 broken_script.py'<br /><br />I've found this pretty useful.<br /><br />Right, bedtime.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com4tag:blogger.com,1999:blog-23863304.post-1149635788027389182006-06-06T23:22:00.000+01:002006-06-07T00:16:28.040+01:00Pydb, unit tests and file descriptorsUnfortunately, the first topic of this post is an annoying one. I couldn't access www.blogger.com<br />_again_ the other day, and if this happens I'm going to move my blog to someone who can provide<br />a half-decent service.<br /><br />Right, anyway, there's been a few developments over the past few days. I've started to write up <br />some unit tests mainly to test the connection classes to make sure that they work by themselves. <br />This is a key idea that the connection classes that we've provided should be 'generic' to the <br />extent that they can be used for other projects.<br /><br />Also, I'm now using pydb on top of pdb. Pydb is the Extended Python Debugger and is written by<br />Rocky Bernstein. It provides some excellent enhancements to pdb's features and whilst my project<br />aims to do the same, it does so with different goals in mind.<br /><br />I've also come across another design decision. How do I get the debugger client to send its <br />commands to the server? I could override some of the pydb methods so that all the commands typed<br />on the stdin of the client are sent to the server, but what if some of the commands being typed<br />should be run on client-side? This is something I'm gonna have to think about tomorrow. <br /><br />Anyway, time for bed for me..Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1149293029872172812006-06-03T00:35:00.000+01:002006-06-03T01:04:59.316+01:00Day 8I just posted a link to the <a href="http://groups.google.com/group/pyxides/browse_thread/thread/677507a32c8cd50f">pyxides group</a> regarding the status of my project. In the post I asked two questions to the pyxides community..<br /><br />1. Whether they thought that subclassing two classes, one the specified the interface of a MPdb target and one that specified the protocol used for communication, was a good design. In this design the target abstract class, which we'll call MTargetInterface looks something like that<br /><pre><br />class MTargetInterface(object):<br /> def accept(self, client):<br /> """ Accept a connection from a debugging console. """<br /> raise NotImplementedError, "Override this method in a subclass"<br /><br /> def disconnect(self):<br /> """ Close a connection to a debugging console. """<br /> raise NotImplementedError<br /><br /> def listen(self):<br /> """ Listen for incoming connections from debugging consoles. """<br /> raise NotImplementedError<br /><br /></pre><br />And this is what I meant about the 'interface' for a target. This is just an abstract class that provides an interface that a subclass must implement. How these classes are implemented for different protocols is of no concern to the debugger core. So, an implementation of a target may be something along the lines of<br /><br /><pre><br />from SocketServer import TCPServer<br /><br />class MTCPTarget(MTargetInterface, TCPServer):<br /> """ Allow incoming connections from debugging consoles using the TCP protocol. """<br /> def __init__(self, addr):<br /> MTargetInterface.__init__(self)<br /> TCPServer.__init__(self, addr, TCPRequestHandler)<br /> <br /> def accept(self, client):<br /> """ Accept incoming connections from a debugging console. """<br /> # This is just a TCPServer method, although the method signature is wrong I think<br /> self.handle_request(client)<br /><br /> def listen(self):<br /> """ Listen for incoming connections """<br /> # This would probably be handled by TCPServer anyway<br /><br /> def disconnect(self):<br /> """ Close all connections. """<br /> # Here we would close all connections<br /><br /></pre><br />I've left out the real _meat_ of this class because the other stuff would just be getting TCPServer to properly.<br /><br /><br />2. What techniques are currently being used to keep the source file that is local to a GUI front-end for a debugger and the source code the debugger is working on. Front-ends usually keep a 'local' copy of the source code so that they can perform their own parsing and whatnot, and I was curious to find out from the community how exactly they do this.<br /><br />Well that's all for this time, be sure to check the pyxides group for more information and replies form people.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com1tag:blogger.com,1999:blog-23863304.post-1148919458430741042006-05-29T17:06:00.000+01:002006-05-29T17:17:38.446+01:00Day 4Well me and Rocky have been passing e-mails back and forth for a few days and a design is beginning to emerge from the discussions. Rocky has made the importance of a good design very clear to me, when my initial idea was to charge in and start coding. Needless to say, it didn't take long for me to come around to his idea. He is a really cool guy and my motivation and enthusiasm for this project is ever increasing!<br /><br />Whilst there's nothing of use yet in the Python trunk sandbox, after a few more e-mails between myself and Rocky, and when we're both fully agreed on the design, I'm sure something will appear.<br /><br />I've been purposefully vague in this post because I don't want to corner myself into a design or features that don't end up in the code.. more information will be posted here later.<br /><br />Thanks, and happy coding!Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1148599884079394702006-05-26T00:10:00.000+01:002006-05-26T00:31:24.090+01:00Day 1: Let it begin!Well this is the first day I've had to work on my SoC project. After speaking with Rocky I tried to get write access to the Python svn repository. A. M. Kuchling was very accomodating and gave me access, so my code is now in <a href="http://svn.python.org/projects/sandbox/trunk/pdb">http://svn.python.org/projects/sandbox/trunk/pdb</a> for people that want to see it. I've been looking through some of Nir Aires code from the <a href="http://www.digitalpeers.com/pythondebugger/">winpdb</a> project which is a Python debugger project. More source grepping tomorrow, perhaps even some initial code will find its what into the tree...<br /><br />Happy coding!Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1148495778817180582006-05-24T19:33:00.000+01:002006-05-24T19:36:18.826+01:00Summer of Code acceptenceWoot, I was accepted to this year's summer of code. The Python software Foundation accepted my proposal, so I'll be working on the python debugger. I've just spoken with my mentor and I'm making preparations for getting started on the code. After waiting for so long it's such a relief to know that I'm finally through. Phew...<br /><br />Although I have one exam tomorrow and a last exam in 7 days time, I can soon start concentrating solely on this project.<br /><br />Let the good times roll! (read good times as sleepless nights and never-ending coding sessions).Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1147801733150240012006-05-16T18:44:00.000+01:002006-05-16T18:48:53.160+01:00Interesting maths primerI just read <a href="http://triplebuffer.devmaster.net/file.php?id=5&page=0">this article</a> over on devmaster.com and thought it was really good. Definately worth the read. It goes through a basic maths principles like fields, vectors, matrices. It's mainly a guide to maths for game developers.<br /><br />First exam at university tomorrow, can't wait till they're over and done with ;)<br /><br />Also awaiting my copy of Kameo! Which play.com managed to fuck up and send me Archer Macalean's Mercury for the PSP... don't get me started. Oh, I also bought Splinter Cell for the original Xbox on Sunday and have been playing it non-stop on my Xbox 360 since. It's an excellent game and seeing as I never own an Xbox 1 I thought I'd give some of the 'classics' a try. It's an awesome game.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1147363196131046242006-05-11T16:54:00.000+01:002006-05-11T16:59:56.143+01:00I need to get away from this computerWell, I'm supposed to be revising this week but instead I've spent the entire day sat in front of my computer working on two games.<br /><br />The first is the remnants of a project for university, a Texas Hold 'Em game written with Python. I've decided to continue its development and even make it network-based. Of course, adding network functionality to an existing project, but thusfar it's been fine. I've looked into a few things and I think I'm gonna just Twisted. It may take a little work because I still need to basically change the entire architecture, passing all commands through a Game Connection. Oh well..<br /><br />And the second game I'm working on is a detective game. It's only a simple 2D game (yes, written in Python) where a character can move from room to room picking up clues and solving mysteries. I've got a room rendered on screen now and am just working something together to allow the player to move between rooms easily.<br /><br />If either of these projects become useful I may stick them on sourceforge.net.<br /><br />Now, I really have to do some studying, bye!Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1147093819331442782006-05-08T14:03:00.000+01:002006-05-08T14:10:19.350+01:00Summer Of CodeWell I tried posting here the other day but blogger.com started going all screwy and timing out. Not much fun I can tell you. Eventually I gave up, but I thought I'd might as well post today what I meant to post the other day.<br /><br />Google SUMMER OF CODE! I've written two proposals,<br /><br />1. A compressed cache for NetBSD's UVM (blogged about earlier)<br />2. Adding some features for Python's debugger (Pdb)<br /><br />I've had positive feedback on both of these and seeing as today is the last day for applications the coming events are placed in the hands of the powers-that-be. I actually had to change my 2nd proposal because the features I said I would write were already written in a package call pydb, the author of this package contacted me and told me so :P. Rocky was (the author) very helpful in fact and gave me ideas of other things that I could implement that were lacking in pdb, such as:<br /><br /> - Debugging threads<br /> - Debugging from another process<br /> - Remote debugging<br /><br />all of which sound super cool. So fingers crossed that I actually get to spend my summer doing something worthwhile!<br /><br />Also looked at the tcsh man page yesterday and found its got a load of features that I never knew about. Spent most of yesterday playing with them :D<br /><br />Well, more studying before the exams at university.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1146510648583731532006-05-01T20:08:00.000+01:002006-05-01T20:10:48.593+01:00Applications away!!Well, after waiting for so long to send in my applications, the day is finally and I've just pressed 'submit'.<br /><br />I've sent two applications,<br /><br />1. NetBSD - A compressed cache for NetBSD's virtual memory manager<br />2. Python Software Foundation - A rewrite and fixing of the Python debugger (Pdb)<br /><br />So fingers crossed that I actually get one of them. Both would be good to work on. I'll keep y'all posted.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1145808693508797962006-04-23T15:47:00.000+01:002006-04-23T17:11:33.550+01:00In the cave.For the past few days now I've been hiding myself away from the world. As I said last post, I've chosen to work on a compressed cache for NetBSD's virtual memory system. Seeing as there's a linux implementation I decided to look through the code and see how it was interacting with the Linux memory manager (mm). So for most of this past week I've been sifting through the source code and the patches provided at <a href="http://linuxcompresed.sourceforge.net">http://linuxcompresed.sourceforge.net</a>.<br /><br />Let me break down the way that the cache is going to work (and currently does work in the linux patches):<br /><br />Usually, when memory in a computer system becomes scarce, the mm starts swapping pages to disk. Now, over the past few years the difference in speed between the CPU accessing data from main memory and accessing data from the hard disk has widened. Thus, swapping pages to disk is a very 'slow' process. This is where the compressed cache comes in.<br /><br />When the system starts up, a portion of memory is put aside for the compressed cache. Ideally, this cache has an initial size and should be able to grow or shrink depending on how much it is being used (adaptability policy), because naturally, if there are no compressed pages in the compressed cache it's a complete waste of RAM space and should be 'shrunk'.<br /><br />Ok, so we've got some space put aside for our compresssed cache.. and the system is chugging along nicely using up more and more free pages but not really freeing any. This can be caused by having lots of large processes in memory at once time, each requesting more and more pages. Naturally, there are only enough a finite amout of pages free. Suddenly the pagedaemon (the process that handles paging in NetBSD) can't keep up with the demand any more, it can't free pages fast enough and other processes are demanding more and more free pages all the time. Something must be done...<br /><br />So the pagedaemon comes in and starts swapping pages out to 'backing store'. Now, this is a very slow process so what happens is, instead of writing these pages straight to disk, they're compressed and placed in the compressed cache. The page that contains the data can now be freed and used to meet the free page demand.<br /><br />When the compressed cache becomes full the compressed pages are decompressed and finally written to swap. This has the advantage that if the free page demand is satisfied, and a process pagefaults on a page that we have stored in the compressed cache, we've just saved ourself an expensive write to disk..<br /><br />I _think_ that makes sense. Please feel free to leave any comments if you didn't understand or (and this is totally probable) that I got something wrong.<br /><br />Until next time...Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1145200260820478172006-04-16T16:03:00.000+01:002006-04-16T16:11:00.833+01:00cats emulationWoo hoo! I got gxemul working with a NetBSD/cats image last night. Going to use this emulator to work on NetBSD's vm manager in an embedded architecture environment. It was actually pretty simple to set up as the documentation leading from the NetBSD <a href="http://www.netbsd.org/Ports/emulators.html">software emulators page</a> gave a great walk-through. Also, I've compiled the latest NetBSD -current source tree (after fixing a problem with libc) and now have a -current kernel running on an emulated cats CPU.<br /><br />And just in case anyone was wondering, no, I didn't build the source tree on the cats CPU, I built it on my AMD Athlon 64-bit 3000+ machine ;]<br /><br />Also, the reason I'm working on this embedded system is that for this year's Google Summer Of Code I'm wanting to work on a compressed cache subsystem for NetBSD's virtual memory layer. This idea is along the lines of <a href="http://linuxcompressed.sourceforge.net">http://linuxcompressed.sourceforge.net</a><br /><br />So fingers crossed that I get accepted! Even if I don't, I'm sure I'll have some information to post here soon.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com2tag:blogger.com,1999:blog-23863304.post-1144705437534584262006-04-10T22:36:00.000+01:002006-04-10T22:43:57.556+01:00Games, games, books?As the title no doubt suggests.. this post is a games related one. I've been playing Elder Scrolls IV - Oblivion practically non-stop since it's been released! It's an excellent game, there's just so much to do, so many little quests that seem to fill your time! Not only that, but the 'main quest' is enough for any person. Having lost one of the items that is a key part of the main quest, I'v recently restarted with a new character. <br /><br />Not usually being one to play games over and over again, the fact that I even thought of starting this game with a new character and redoing all those quests that I've spent around 40 hours so far completing, is obviously a sure sign of a classic.<br /><br />I also got my hands on a copy of Final Fantasy VII, (having never played any of the Final Fantasy series) I'm hoping how this one turns out. A friend of mine told me that this one in particular was the best one of the series, so we shall certainly see. <br /><br />I'm currently reading The Hitchiker's guide To The Galaxy, it's a funny book, although I'm slightly annoyed at how watching the movie _before_ reading the book has seemed to lessen the enjoyment that I might have actually gotten out of it had I read it first. Oh well..<br /><br />Until next time.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1144410234235067982006-04-07T12:39:00.000+01:002006-04-07T12:43:54.250+01:00I suffer from NADDNADD may sound like a severe disease, but its not all bad, it means I have embraced the multimedia age. After reading <a href="http://www.randsinrepose.com/archives/2003/07/10/nadd.html">this post</a> I've realised that I exhibit similar symptoms. Do you have more than 10 windows open on your desktop right now? Might want to check it out and see if you too, are affliced with NADD.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1143902055019913422006-04-01T15:30:00.000+01:002006-04-01T15:34:15.030+01:00Python Bug dayWell yesterday's Python bug day was fun.. apparently more people turned up then previous years which is cool! I had quite a good time, spent about 10 hours on it in total. Will definately go to the rest.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0tag:blogger.com,1999:blog-23863304.post-1143041599398113222006-03-22T13:18:00.000+00:002006-03-22T15:33:19.423+00:00DIE DIE DIE Insert-mode!I may finally be able to put my foot down and consider myself an emacs user. I'm sick and tired of being in insert-mode and not being able to navigate around the document I'm editing at the same time! This discussion was started by someone in a maths lecture today, and all it's done is put me off of vim.. maybe even for good. In emacs, there is *NO* insert-mode.. at least not specifically. I can insert text anywhere I want and still have the ability to move around freely and to be frank, easily. Emacs is the way forward!<br /><br />Also, found this article which made me laugh<br /><br /><a href="http://www.lysator.liu.se/c/ten-commandments.html">The Ten Commandments for C Programmers (Annotated Edition)</a>.<br /><br />Good for a giggle.Matt Fleminghttp://www.blogger.com/profile/14213643280794051707noreply@blogger.com0