I dream of being able to write I/O code in Python in the synchronous, blocking style, then switching the code to asynchronous and non-blocking with minimal changes. Synchronous I/O is easier to write, but asynchronous avoids the need for threads and is often faster. Can it be done? The challenge is this: when the software is about to do some blocking I/O operation, you have to convert the operation to a non-blocking operation (fairly easy), unwind the stack (probably not hard), transfer control to an event loop (easy), and restore the stack when the I/O finishes (hard).

Stackless Python provides a way to unwind and restore stacks. Stackless is the easiest way to find out whether the idea is a good one. However, Stackless is not part of the core Python and may never be, so even if the idea turns out successful, not many people will use it.

The new coroutines in Python 2.5 might help if you code everything that does any I/O as a generator and simulate a stack using a stack of generators. Unfortunately, I think I'd have to write a ton of "yield" statements in that case. I wouldn't be allowed to call any function that does any I/O; I'd have to ask the simulated stack to call the function.

The coroutine-based solution might work, though, if there's a decorator that can turn a function into an I/O aware coroutine. This decorator would turn calls to blocking functions into yield statements that transfer control back to the event loop.

I wonder if anyone is trying to do this already.

Bit Mountain is a digital archiving system I've been building. I've mentioned it before. I'm working to make it scale from from a few small files on a few hard drives to billions of files (consisting of multiple petabytes) on thousands of media. I believe I've come to understand the storage problems that occur at that scale and have coded many of the solutions in Bit Mountain.

I recently presented a paper about Bit Mountain. You can download the slides from the Family History Technology Workshop 2006 archive. I'll also post the paper I presented if anyone requests it.

I believe I just found the solution to a nagging problem in Bit Mountain. Bit Mountain currently creates 3-4 relational database rows for every file in the archive. This is no problem if there are millions of files, but it's an ugly problem if there are billions of files. Databases typically don't support billions of rows without big hardware and deep magic tuning.

The obvious solution is a distributed hash table, but that would require a complete rewrite of the software, and a DHT does not take advantage of natural file groupings. Many of the images represent pages of books, so it would make sense to keep book pages together in the archive. Keeping pages together means that when you flip pages in a digital book, loading a page is likely to transparently pre-fetch information about the other pages, and turning pages will be quick.

What I decided to try is uploading bundles of files rather than individual files. The database will record only 3-4 rows for each bundle. To read an individual file, clients will fetch it from the bundle that contains it. Bundles are encoded as multipart MIME messages and can span gigabytes. For each bundle, there is a metadata file that holds an index of the files it contains. If bundles contain an average of 1000 files, then the billion file problem suddenly becomes a mere million file problem and life is easy again. Yesterday and today, I worked out the code for uploading bundles. Now I need to work on the download portion. I'm optimistic that it will perform well.

Everyone I talk to at work is in favor of releasing Bit Mountain as open source software, but I can't do it without official approval. The idea of releasing it keeps stalling for some reason. I ache to release it. I want feedback, I want people to try it out, I want people to improve upon it. I want people to test my theory that Bit Mountain can provide higher reliability and better storage utilization than what is achievable through replication. I want as many people as possible to run it, harden it, and prove that it's reliable.

Gentoo's emerge command is powerful, but it stops if you attempt to run it in the background, and it often takes a long time. Particularly when compiling stuff on a server, it's useful to have emerge work in the background even after I log out.

Solution: the screen utility. Before starting a long compile, I type screen, which opens a shell session that I can detach and reattach later. I start emerge, wait a little to make sure I typed the command correctly, then push Ctrl-A followed by D. At that point, I can log out while the screen session continues in the background. A few hours later, I type screen -x to reattach the session, often using a different computer than the one I used to start the emerge.

In general, the screen utility makes it easier to perform long running automated tasks, like compiling software, downloading large files, or converting media files.

I bought a train model kit for Esther at Christmas. After we painted and assembled it, the motor ran fine, but the wheels stuck often. The mechanism relies on the left and right sides being a quarter turn out of phase, but the nylon connectors often slip on the metal axles, causing the wheels to lock up.

I found a nice solution to the problem. Anyone who buys the MWK-06 kit and has problems with slippage probably ought to do this. The strategy is to ensure the nylon "L" connectors never slip, and I did that by grinding the ends of the axles.

1. Use a small metal grinder (I used a rotary tool) to grind the ends of every axle into flattened surfaces, with the two ends of each axle oriented exactly out of phase with each other. In other words, grind one end in a horizontal orientation and the other end in a vertical orientation.
2. Squeeze the axle shafts on the nylon L connectors using pliers, flattening the round holes and turning them into ellipses. Flatten all of them the same way.
3. Press the flattened L connectors onto the axles. The connectors should fit easily, but snugly; if they are loose, squeeze the L connectors some more.
4. To allow free movement, tighten the screws connected to the pulleys and the second pair of wheels from the back as much as possible without causing friction. Loosen the other 4 screws as much as possible without letting the nut fall off.

I can supply pictures if these directions don't suffice. The train moves pretty fast once it's running properly.

I'm happy to announce PGStorage, a new ZODB backend that stores pickles in a PostgreSQL database. It supports undo and packing.

The intent of this storage is to be a good ZODB pickle store, and PostgreSQL has many features that simplify storage. Taking advantage of all the capabilities of a relational database are not the intent of this storage. The idea of a PostgreSQL storage has been attempted before, but at the time, PostgreSQL did not have the TOAST feature. TOAST simplifies and optimizes the task of storing binary objects in the database.

Multiple Zope instances can connect to the database, so ZEO is not needed when using this storage. ZODB caches are invalidated by polling the database at transaction boundaries. Is that an efficient choice? Maybe. If it's not, maybe the PostgreSQL "listen" and "notify" statements can help.

I think PGStorage has a chance of beating ZEO in scaleability, since PostgreSQL's networking layer is likely to be more efficient that ZEO. PGStorage has a fraction of the number of lines of code of FileStorage and ZEO, but it's certainly not as well tested yet.

I did end up fighting with ZODB somewhat. ZODB normally assumes that MVCC is implemented by the Connection object, but this storage prefers to depend on the MVCC capabilities already in PostgreSQL, so I had to override some methods in Connection. I didn't fight with ZODB in this package nearly as much as I did in Ape, though.

Try it out and see if it's any good. After the first round of testing, I plan to take advantage of the two-phase commit support in PostgreSQL 8.1.

Ape is an experiment in bridging transparent persistence to any medium. I am primarily interested in storing on a filesystem, but the Zope community seems interested only in Ape's RDBMS persistence. Well, here are some fundamental problems I've discovered that Ape has with RDBMS persistence:

1. Ape can't add support for complex queries while retaining transparence. Transparence would require all Ape gateways (such as filesystem support) to gain the ability to answer complex queries.
2. Ape sends data to the database only at the end of a transaction, and in random order. In high concurrency, this policy will lead to deadlocks.

Some more general problems with Ape:

1. Ape fights with ZODB to an extent. With every ZODB release, I've had to change Ape to override different methods and use different attributes of the Connection class.
2. The error messages produced by Ape (especially those involving UnmanagedJars) are hard to interpret. They can only be solved with deep study.
3. Projects typically become easier over time. Ape isn't getting easier.

So while I appreciate the work people have put into Ape, I no longer think it's the right idea for people to spend their skills on. However, I'm about to release a much simpler replacement. Stay tuned.