db functions future directions

Good handling of persistent data and related issues is Muq's greatest single strength, and future releases will continue to focus upon this.

Improvements which I expect to introduce roughly in the year 2000-2001 timeframe include:

• Supporting objects up through about 64Mbytes in size -- current limit is 64K.
• Supporting variable size objects -- currently size of an object is fixed at creation, at least in terms of the fundamental vm.t diskbase layer facilities.
• Mapping .muq files into memory. This will speed db open and close (particularly of uncompressed dbs) and somewhat speed up disk I/O on large objects, but primarily it will enable the following.
• Communicating with other processes via shared-memory access to .muq files. This will allow high-bandwidth communication with a variety of helper applications, such as lapack operations and sound, still and video I/O and transformations via large shared arrays.
• Mounting (as opposed to importing) .db files, in particular large, static read-only .db files. This would be an appropriate way of making large datasets such as MRI voxel datasets, gene sequence data, image libraries and such available to softcode without having to actually copy them into the master .muq file and duplicate their contents at each backup of the .muq file.
• More sophisticated garbage collection -- this is critical to working effectively with datasets larger than physical memory workingset. Options include going to a multi-generation garbage collection scheme, going to an incremental scheme -- likely based on Dijkstra's three-color algorithm -- or a combination of the two. Ideally, garbage collection pauses should become too short to be noticable to interactive users doing 3D graphics &tc.
• True incremental backup, so that (ideally, at least) the server continues to run without noticable pauses during backup operations.

In a slightly longer timeframe, it would be good to investigate input journalling, so as to be able to back up and reproduce a run exactly -- very handy for debugging, among other things.

Go to the first, previous, next, last section, table of contents.