Hi all,
Apologies if this is the wrong place, but it is somewhat Termite specific.
Has anyone taken a close look at Mnesia, to the point of considering doing something like it using Termite? I'm not necessarily thinking of an architectural clone, but perhaps something similarly geared towards horizontal scalability that has native code queries?
Cheers,
--Joe
Afficher les réponses par date
On Jan 17, 2008 8:53 PM, Joe Hosteny jhosteny@gmail.com wrote:
Has anyone taken a close look at Mnesia, to the point of consideringdoing something like it using Termite? I'm not necessarily thinking of an architectural clone, but perhaps something similarly geared towards horizontal scalability that has native code queries?
Well there's a curious idea that I've been kicking around ever since the release of R6RS made me start to think through what I *really* thought needed fixing in Scheme - and that is the question of why it is that only strings can be interned? It seems to me that if you extend that concept to all the datatypes you have a very interesting-looking embedded database system. If you add a persistence manager it gets even better.
Obviously, the idea is a little trickier than what I can lay out in a five-sentence email, but I think there is a powerful notion rolling around in there somewhere. Termite seems like a fairly reasonable platform for it as well.
david rush
On 17-Jan-08, at 3:53 PM, Joe Hosteny wrote:
Hi all,
Apologies if this is the wrong place, but it is somewhat Termite specific.
Has anyone taken a close look at Mnesia, to the point of considering doing something like it using Termite? I'm not necessarily thinking of an architectural clone, but perhaps something similarly geared towards horizontal scalability that has native code queries?
Can you refresh my memory concerning the features of Mnesia? ;)
If I recall correctly it is a "in RAM memory" database, and you can dump or restore the database to/from the filesystem. Those operations are possible with Gambit's table datatype (hash tables) and the serialization/deserialization procedures. Here's a simple example.
(define db (make-table))
(define (dump obj fn) (let ((v (object->u8vector obj))) (with-output-to-file fn (lambda () (write-subu8vector v 0 (u8vector-length v))))))
(define (restore fn) (let ((x (call-with-input-file fn (lambda (p) (list->u8vector (read-all p read-u8)))))) (u8vector->object x)))
(define-type hacker id: 1A673423-70D6-4922-92A6-0C2D43DA5CC6 name language proc)
(table-set! db "Marc" (make-hacker "Marc" "Scheme" dump)) (table-set! db "James" (make-hacker "James" "Java" restore)) (table-set! db "Bjarne" (make-hacker "Bjarne" "C++" (lambda (x) x)))
(dump db "mydb")
(define db2 (restore "mydb"))
(define m (table-ref db2 "Marc"))
(pp (hacker-proc m)) ;; prints: (lambda (obj fn) (let ((v ...
Marc
P.S. While trying out this example I noticed a small bug in the deserialization of tables. You'll need to get the latest patch from the Gambit repository for this to work properly.
On Jan 17, 2008, at 10:02 PM, Marc Feeley wrote:
On 17-Jan-08, at 3:53 PM, Joe Hosteny wrote:
Hi all,
Apologies if this is the wrong place, but it is somewhat Termite specific.
Has anyone taken a close look at Mnesia, to the point of considering doing something like it using Termite? I'm not necessarily thinking of an architectural clone, but perhaps something similarly geared towards horizontal scalability that has native code queries?
Can you refresh my memory concerning the features of Mnesia? ;)
If I recall correctly it is a "in RAM memory" database, and you can dump or restore the database to/from the filesystem. Those operations are possible with Gambit's table datatype (hash tables) and the serialization/deserialization procedures. Here's a simple example.
(define db (make-table))
(define (dump obj fn) (let ((v (object->u8vector obj))) (with-output-to-file fn (lambda () (write-subu8vector v 0 (u8vector-length v))))))
(define (restore fn) (let ((x (call-with-input-file fn (lambda (p) (list->u8vector (read-all p read-u8)))))) (u8vector->object x)))
(define-type hacker id: 1A673423-70D6-4922-92A6-0C2D43DA5CC6 name language proc)
(table-set! db "Marc" (make-hacker "Marc" "Scheme" dump)) (table-set! db "James" (make-hacker "James" "Java" restore)) (table-set! db "Bjarne" (make-hacker "Bjarne" "C++" (lambda (x) x)))
(dump db "mydb")
(define db2 (restore "mydb"))
(define m (table-ref db2 "Marc"))
(pp (hacker-proc m)) ;; prints: (lambda (obj fn) (let ((v ...
Marc
P.S. While trying out this example I noticed a small bug in the deserialization of tables. You'll need to get the latest patch from the Gambit repository for this to work properly.
Marc,
I haven't really dug into a lot of examples with Mnesia, but my understanding is that these are some of the main features.
* Embedded * Allows for replication or distribution of tables * Transactions (I'm not sure about the full capabilities of these) * Native data structure storage * Live backups * Live schema upgrades
Here's an overview, published a few years back:
http://www.erlang.se/publications/mnesia_overview.pdf
I know there's been some traffic on the list about a Scheme web framework. I have some interest in this as well, but mainly just for applications that could use a distributed DB in a utility environment, e.g., on Amazon's EC2 (note: I cross posted to the Snow mailing list about work on an EC2 package). It seems to me that a framework could have value over something like Rails, though, if it addressed some difficulties it currently has, such as scalability of both the web server and back end (obviously, the back end is a separate issue from Rails). Incidentally, one of the pain points with EC2 currently is its lack of persistent storage. So, some method of DB replication, in addition to incremental backup, is really necessary.
I'm also interested in self-configuring compute clusters. I wanted to put a feeler out to see if anyone in the Scheme community was working on these things, or would like to. I don't really have a background in databases, but I do in filesystems and journaling.
Joe
P.S. Are there any Schemers on this list from Pittsburgh?
Note that I wrote on 2007/11/02:
I'll now rather invest more time to get things right (the way I want them) than create a complete framework quickly.
Joe Hosteny wrote:
I wanted to put a feeler out to see if anyone in the Scheme community was working on these things, or would like to.
Re: was working: yes, in the sense of having put thought into it (pondered design choices) and doing experiments with creating records that can be extended, with creating shared lowlevel hashtables in scheme versus C code, starting to create a POSIX interface for the purpose of using mmap, setuid, etc.
Re: would like to: yes, although there is also module systems stuff I'd like to sort out first, and being busy by doing non-Scheme work atm.
I haven't really dug into a lot of examples with Mnesia, but my understanding is that these are some of the main features.
- Embedded
Yes, not some binding to some sql database... (although I'm not really in the "in-memory-database" camp, and there's no issue running the database in different processes than the processes making up your applications--whether "process" means Unix (or, huh, that other OS) processes or just Gambit threads is open, although once you start using posix functionality directly there's not so much of a fit for Gambit threads for it (not that that's a bad thing, doing the split of real multiprocessing at the storage boundary is probably a pretty good decision)).
- Allows for replication
Sure, and not only unidirectional like most db systems, but allowing to handle split-brain situations that happened to your servers. Means offering merging capabilities (maybe coded by users so that it's appropriate for their data structure).
or distribution of tables
Yes, have data in different places (processes with their files) owned by different users (unix users, or, in the case of Erlang maybe it's just processes (they have that "must know uuid of a process to talk to it" security)).
- Transactions (I'm not sure about the full capabilities of these)
Of course. I intend to handle the data purely functional, meaning, no hash tables (except for caching purposes), only trees. Database storage files are append-only, stale data is pruned by garbage collection passes, thanks to handling purely functional data, the GC can run in constant memory (in the sense that a couple of tens of MB are enough for garbage collecting hundreds of gigabytes), and thanks to fast linear disk access it's efficient.
- Native data structure storage
Sure. Well, of course the bytes in scheme objects are not stored 1:1 in the database since pointers have to be turned into object id's, strings can be stored more compactly (and on top of that you want to append checksums and maybe compress blocks of the database) etc., so there's a transformation layer involved (but pretty much straight-forward).
- Live backups
Sure, through replication, or just copying the storage files (while they are being written to their end..). (Nothing against Linux LVM.)
- Live schema upgrades
Depends on what you mean with "schema". You say you want to store native objects. Since Scheme is dynamically typed (and the database should thus be as well), you can always modify a type and continue saving it into the same tree ("table") where the old types have gone. This means, you've got the old "schema" (or types) for old objects and the new one for new objects. (Of course your code will have to be able to handle both, unless the code is bound to the data by means of closures(*).) Now of course you could hook a cleanup function into the database garbage collector so that upon the next collection old objects are converted to new ones (or initiate an immediate collection). One could automatically create conversion functions (yes: you'll want to write the record definition facility so that it can do that).
(*) I'm not sure: do we want to be able to save closures and continuations into the DB? (I mean, not by using the Gambit serializer, but by inspecting them and save every contained object as a database object; so that data sharing is preserved, the same object can be referenced by a closure or by some other means. I'm not sure where mutation is becoming a problem; you could give mutated objects new object id's in the database, but then they are not eq? anymore and subsequent modifications are not affecting the original one. Is there a way to solve that from the purely functional language world?)
I don't really have a background in databases, but I do in filesystems and journaling.
My background in databases is using MySQL, Postgres, storing objects as XML files, dumping objects as serialized blobs, etc., and maybe knowing GIT also counts as some kind of database knowledge (ok, I've thought through whether I'd want sha1 sums as object id's and choose "no"). My background in journaling or rather replication is trying to manage synchronized live web servers. I'm a programmer, not a "database" guru. I know that "databases" don't fit my programming projects :)
BTW re journaling, I'm not sure how much that's just moot once you have a purely functional storage (ok, when you're writing to several places (different processes) and group those changes together into single commits, then you need a journal of those; but then that's a distributed computing problem and not a storage manager problem ;)).
I have to get chjmodule working...
Christian.
On Jan 18, 2008, at 1:39 PM, Christian Jaeger wrote:
Note that I wrote on 2007/11/02:
I'll now rather invest more time to get things right (the way I want them) than create a complete framework quickly.
Joe Hosteny wrote:
I wanted to put a feeler out to see if anyone in the Scheme community was working on these things, or would like to.
Re: was working: yes, in the sense of having put thought into it (pondered design choices) and doing experiments with creating records that can be extended, with creating shared lowlevel hashtables in scheme versus C code, starting to create a POSIX interface for the purpose of using mmap, setuid, etc.
Re: would like to: yes, although there is also module systems stuff I'd like to sort out first, and being busy by doing non-Scheme work atm.
Certainly let me know if you get some time and want to talk a bit more about this offline.
I haven't really dug into a lot of examples with Mnesia, but my understanding is that these are some of the main features.
- Embedded
Yes, not some binding to some sql database... (although I'm not really in the "in-memory-database" camp, and there's no issue running the database in different processes than the processes making up your applications--whether "process" means Unix (or, huh, that other OS) processes or just Gambit threads is open, although once you start using posix functionality directly there's not so much of a fit for Gambit threads for it (not that that's a bad thing, doing the split of real multiprocessing at the storage boundary is probably a pretty good decision)).
Yes, though perhaps it could be extended to support SQL if so desired? I was also thinking of something that had write back semantics for updates / deletes too (or at least some sort of backing aside from RAM).
- Allows for replication
Sure, and not only unidirectional like most db systems, but allowing to handle split-brain situations that happened to your servers. Means offering merging capabilities (maybe coded by users so that it's appropriate for their data structure).
Excuse my ignorance - how is it non-unidirectional? Do you mean at the database level, or the table level? Do you mean that individual records are not authoritative at a master location if they are replicated?
or distribution of tables
Yes, have data in different places (processes with their files) owned by different users (unix users, or, in the case of Erlang maybe it's just processes (they have that "must know uuid of a process to talk to it" security)).
- Transactions (I'm not sure about the full capabilities of these)
Of course. I intend to handle the data purely functional, meaning, no hash tables (except for caching purposes), only trees. Database storage files are append-only, stale data is pruned by garbage collection passes, thanks to handling purely functional data, the GC can run in constant memory (in the sense that a couple of tens of MB are enough for garbage collecting hundreds of gigabytes), and thanks to fast linear disk access it's efficient.
- Native data structure storage
Sure. Well, of course the bytes in scheme objects are not stored 1:1 in the database since pointers have to be turned into object id's, strings can be stored more compactly (and on top of that you want to append checksums and maybe compress blocks of the database) etc., so there's a transformation layer involved (but pretty much straight- forward).
- Live backups
Sure, through replication, or just copying the storage files (while they are being written to their end..). (Nothing against Linux LVM.)
Yes, or maybe even as PostgresQL does it, by writing the write-ahead log.
- Live schema upgrades
Depends on what you mean with "schema". You say you want to store native objects. Since Scheme is dynamically typed (and the database should thus be as well), you can always modify a type and continue saving it into the same tree ("table") where the old types have gone. This means, you've got the old "schema" (or types) for old objects and the new one for new objects. (Of course your code will have to be able to handle both, unless the code is bound to the data by means of closures(*).) Now of course you could hook a cleanup function into the database garbage collector so that upon the next collection old objects are converted to new ones (or initiate an immediate collection). One could automatically create conversion functions (yes: you'll want to write the record definition facility so that it can do that).
Well, I don't necessarily want to store native objects. I was just noting what I think Mnesia does. This is where my lack of database knowledge comes into play. I'm not sure what DB model is most appropriate.
(*) I'm not sure: do we want to be able to save closures and continuations into the DB? (I mean, not by using the Gambit serializer, but by inspecting them and save every contained object as a database object; so that data sharing is preserved, the same object can be referenced by a closure or by some other means. I'm not sure where mutation is becoming a problem; you could give mutated objects new object id's in the database, but then they are not eq? anymore and subsequent modifications are not affecting the original one. Is there a way to solve that from the purely functional language world?)
I don't really have a background in databases, but I do in filesystems and journaling.
My background in databases is using MySQL, Postgres, storing objects as XML files, dumping objects as serialized blobs, etc., and maybe knowing GIT also counts as some kind of database knowledge (ok, I've thought through whether I'd want sha1 sums as object id's and choose "no"). My background in journaling or rather replication is trying to manage synchronized live web servers. I'm a programmer, not a "database" guru. I know that "databases" don't fit my programming projects :)
I'm in roughly the same boat.
BTW re journaling, I'm not sure how much that's just moot once you have a purely functional storage (ok, when you're writing to several places (different processes) and group those changes together into single commits, then you need a journal of those; but then that's a distributed computing problem and not a storage manager problem ;)).
Well, even if you're writing in one place, depending on when your commit happens with respect to when RAM contents are actually persisted, I think you would need some kind of WAL.
I have to get chjmodule working...
Thanks for the thoughtful response. Again, feel free to contact me if you get some time and want to share your design thoughts and any current code you may have.
Christian.
--Joe
-
Christian Jaeger -
David Rush -
Joe Hosteny -
Marc Feeley