[gambit-list] Web Framework for Gambit

Christian Jaeger christian at pflanze.mine.nu
Thu Nov 1 20:42:16 EDT 2007


Rouan van Dalen wrote:
> I was wondering what your thoughts are on this and if something like
> this already exists.  I definitely need support for the latest MySQL
> database server and I am thinking of writing an HTTP server
> specifically for Gambit to form part of the web framework.
>   

Choosing Gambit for me has been the result of looking for the best 
language and implementation for rewriting an existing web application 
(and writing new ones).

The first thing that has been missing has been a module system which 
supports runtime (re)loading of code into the application. That's why I 
did write "chjmodule" (note my recent comment about me wanting to 
release a new version and give access to my repo soon).

I also took Marc's http server example and split it into several 
"chjmodule" modules.

Sadly, the customer whose web application I wanted to rewrite, choose to 
give it into other hands instead (not for technical reasons, as has been 
said explicitely). As a result of this, my work life shifted from 
writing web applications to other things. While I still would like to 
'finish' a web application framework (and would like to write a few 
small things in the next few months), the priorities have changed. I'll 
now rather invest more time to get things right (the way I want them) 
than create a complete framework quickly.

Actually we (me and a co-worker) wrote a tiny application using not the 
http server, but fastcgi instead, by going through Perl's fastcgi 
support (through the use of "gperl", my Gambit perl bindings).

I did spend quite some time thinking about how to design the web 
framework. My aim is to have an http server in one Gambit process (using 
Gambit's builtin threading), but do the actual processing in a slew of 
separate "worker" processes (possibly running under different unix 
uid's). The reasons are: not to block other clients while calling C 
code, not to affect other clients when a process crashes, to make use of 
multiple processors, and to increase security (by using a separate uid 
for each logged-in user, or switching off traceability under Linux). The 
http front end process would parse the request and decide upon the 
possibly contained user id and continuation id which worker process to 
dispatch it to. The worker processes are created by a worker supervisor 
(which is (possibly) running as root, forks and then changes uid to the 
respective user)). Communication occurs through a command pipe or a 
(unix/IP) socket (possibly using Termite, maybe with modifications), as 
a message (containing the request header) and possibly a separate body 
pipe/socket (containing the request body). The worker process will then 
create the response and feed it back to the front end either through a 
pipe/socket (if it's endless data) or through a temporary file (tmpfs) 
if the length is finite and small enough; the latter approach is better 
for freeing up the worker child quickly by being able to spool data 
faster than the client accepts (to keep the number of separate unix 
processes small), and optionally allows for caching the response at the 
same time; the http frontend would poll the file (or possibly use 
inotify/dnotify) for new chunks to be delivered, and get an 
end-of-request back from the child when it's finished, this is fast and 
allows the worker process to use normal IO routines (including charset 
conversions), without worrying about http chunking etc. (I actually also 
expect this to be rather more efficient than trying to pipeline page 
creation and delivery (e.g. using object pipes) in one process). BTW 
recently I've read that Paul Graham used about the same approach for 
Viaweb (using another clisp process for each client). This did actually 
surprise me because until now I thought of common lisp as "forget about 
the host OS, do everything in the CL system" way of doing things, and 
have got my worries about how I'd debug an application where processing 
is distributed across several OS processes (maybe even machines); seems 
Graham&Co. already solved that problem, and Erlang and Termite will have 
found or will find ways to solve that too.

I'm coming from an XML/XSLT background (AxKit); we did write a helper 
pseudo-XSLT library which we used to rewrite existing XSLT stylesheets 
to Scheme almost 1:1 (I've been aware of SXSLT library but writing that 
library ourselves seemed quicker than trying to understand SXSLT and was 
a good learning experience). I did also write an SXML serializer tuned 
for speed.

In the pseudo-XSLT helper library, we're making use of delayed 
evaluation (streams). This made me realize that in the longer term, I 
want lazy-by-default scheme modules (using a transformer as chjmodule 
plugin, with the ability to see and debug the resulting strict scheme code).

> 1) Is there any interest in something like this?
>   

Yes there is, albeit I've got my own ideas how things should work as 
pointed out above, but I'll always be glad if I can use some piece of 
existing software (if it doesn't mean it locks me in or requires big 
time investments to get to know without paying off).

Note that Chicken has got quite many eggs in the web app space recently. 
You may want to look at what's there (and maybe [help] turn them into 
Snow packages, or at least get an overview on what's going on). I also 
think Askemos is interesting (but I haven't gotten around looking at it 
closely yet).

> 2) Does MySQL5 bindings exist for Gambit? (if not I will gladly write them)
>   

You should be aware that in Gambit (as in many other language 
implementations), other Gambit threads aren't being scheduled while you 
run C code. But with my above described design that's no problem.

A more serious drawback of MySQL (except if they changed that in release 
5) for me is that once a query has been issued, no new query can be 
issued before the previous result set has been read completely (and 
there are (have been?) no cursors); if you want to use functional 
streams for representing result sets (lazy lists) like me, that's a 
no-go (there's no point using lazy evaluation if you have to read in 
everything anyway).

See also:

http://bendiken.net/2006/06/11/poor-mans-mysql-api-for-gambit

https://webmail.iro.umontreal.ca/pipermail/gambit-list/2006-August/000782.html 
and
http://blog.jonnay.net/archives/779-Version-0.3-of-the-MySQL-FFI-for-Gambit-released.html

(and I think I saw someone reimplementing the MySQL protocol in Gambit? 
Or maybe it was Postgres, by Dominique Boucher. Maybe I saw someone 
writing the MySQL protocol in another Scheme or language.)

> I am also interested in expanding Gambit's libraries, so there is no problem for
> me to write libraries/bindings for various things..
>   

(Honestly, I'm a bit hesitant when it comes to writing libraries. I've 
got lying around 150 "chjmodule" modules of my own writing right now. I 
really want to get module parametrization (and probably also 
lazy-by-default and other niceties like proper cross-module inlining 
declarations, stacked language layers and more) before I seriously 
continue writing modules, because if I do it the other way round, I'll 
have to rewrite more and more code once parametrization and lazyness is 
in place. At the same time that I think I need parametrized modules so 
soon, I hope R6RS doesn't catch on; at the same time that I'm tempted to 
say we should use a more powerful module system (and probably a common 
one, not one I'm just writing for my own use..) I hope that I (and maybe 
we all) get the chance of writing the right module system "ourselfes" 
(in the community). Or maybe rather, have a multitude of module styles 
but have them interoperate. Anyway it's time for me to make a more 
powerful modulesystem before I invest too much into writing more actual 
modules. I guess what I want to say is, go ahead, but be prepared that 
the way you write them now might not the final way (well, in any case 
Snow is probably a good starting point).)

Cheers,
Christian.




More information about the Gambit-list mailing list