[gambit-list] Mnesia
Christian Jaeger
christian at pflanze.mine.nu
Fri Jan 18 13:39:11 EST 2008
Note that I wrote on 2007/11/02:
> I'll
> now rather invest more time to get things right (the way I want them)
> than create a complete framework quickly.
Joe Hosteny wrote:
> I wanted to
> put a feeler out to see if anyone in the Scheme community was working
> on these things, or would like to.
Re: was working: yes, in the sense of having put thought into it
(pondered design choices) and doing experiments with creating records
that can be extended, with creating shared lowlevel hashtables in scheme
versus C code, starting to create a POSIX interface for the purpose of
using mmap, setuid, etc.
Re: would like to: yes, although there is also module systems stuff I'd
like to sort out first, and being busy by doing non-Scheme work atm.
> I haven't really dug into a lot of examples with Mnesia, but my
> understanding is that these are some of the main features.
>
> * Embedded
>
Yes, not some binding to some sql database... (although I'm not really
in the "in-memory-database" camp, and there's no issue running the
database in different processes than the processes making up your
applications--whether "process" means Unix (or, huh, that other OS)
processes or just Gambit threads is open, although once you start using
posix functionality directly there's not so much of a fit for Gambit
threads for it (not that that's a bad thing, doing the split of real
multiprocessing at the storage boundary is probably a pretty good
decision)).
> * Allows for replication
Sure, and not only unidirectional like most db systems, but allowing to
handle split-brain situations that happened to your servers. Means
offering merging capabilities (maybe coded by users so that it's
appropriate for their data structure).
> or distribution of tables
>
Yes, have data in different places (processes with their files) owned by
different users (unix users, or, in the case of Erlang maybe it's just
processes (they have that "must know uuid of a process to talk to it"
security)).
> * Transactions (I'm not sure about the full capabilities of these)
>
Of course. I intend to handle the data purely functional, meaning, no
hash tables (except for caching purposes), only trees. Database storage
files are append-only, stale data is pruned by garbage collection
passes, thanks to handling purely functional data, the GC can run in
constant memory (in the sense that a couple of tens of MB are enough for
garbage collecting hundreds of gigabytes), and thanks to fast linear
disk access it's efficient.
> * Native data structure storage
>
Sure. Well, of course the bytes in scheme objects are not stored 1:1 in
the database since pointers have to be turned into object id's, strings
can be stored more compactly (and on top of that you want to append
checksums and maybe compress blocks of the database) etc., so there's a
transformation layer involved (but pretty much straight-forward).
> * Live backups
>
Sure, through replication, or just copying the storage files (while they
are being written to their end..). (Nothing against Linux LVM.)
> * Live schema upgrades
>
Depends on what you mean with "schema". You say you want to store native
objects. Since Scheme is dynamically typed (and the database should thus
be as well), you can always modify a type and continue saving it into
the same tree ("table") where the old types have gone. This means,
you've got the old "schema" (or types) for old objects and the new one
for new objects. (Of course your code will have to be able to handle
both, unless the code is bound to the data by means of closures(*).) Now
of course you could hook a cleanup function into the database garbage
collector so that upon the next collection old objects are converted to
new ones (or initiate an immediate collection). One could automatically
create conversion functions (yes: you'll want to write the record
definition facility so that it can do that).
(*) I'm not sure: do we want to be able to save closures and
continuations into the DB? (I mean, not by using the Gambit serializer,
but by inspecting them and save every contained object as a database
object; so that data sharing is preserved, the same object can be
referenced by a closure or by some other means. I'm not sure where
mutation is becoming a problem; you could give mutated objects new
object id's in the database, but then they are not eq? anymore and
subsequent modifications are not affecting the original one. Is there a
way to solve that from the purely functional language world?)
> I don't really have a background in
> databases, but I do in filesystems and journaling.
>
My background in databases is using MySQL, Postgres, storing objects as
XML files, dumping objects as serialized blobs, etc., and maybe knowing
GIT also counts as some kind of database knowledge (ok, I've thought
through whether I'd want sha1 sums as object id's and choose "no"). My
background in journaling or rather replication is trying to manage
synchronized live web servers. I'm a programmer, not a "database" guru.
I know that "databases" don't fit my programming projects :)
BTW re journaling, I'm not sure how much that's just moot once you have
a purely functional storage (ok, when you're writing to several places
(different processes) and group those changes together into single
commits, then you need a journal of those; but then that's a distributed
computing problem and not a storage manager problem ;)).
I have to get chjmodule working...
Christian.
More information about the Gambit-list
mailing list