Hello

So, as I've discussed on IRC, I want to get back into module system development. And as mentioned, I want to concentrate on "from-scratch matters" for now to help make existing module system development efforts' interoperation and further development possible.

What are my aims?

- having a base on which I can write own modularization approaches if I'd like to.

- being able to use code written for other module systems.

- study how that interoperation between the various approaches can be made to work.

- work together with all other interested parties; "as I see fit", at least. The only thing I can say is that I'll try to avoid to tie myself too much into particular usage perspectives; for me Scheme (or a subset thereof, we'll see) is a means to layer languages on, and I want to make this layering as easily workable as possible.

And I'm certainly going to focus development on Gambit as the underlying system, for the foreseeable future.

(What we've once written in the wiki is still true for me: http://dynamo.iro.umontreal.ca/~gambit/wiki/index.php/Module_System )

So what do I plan to work on at first?

I think that the most important thing right now is more flexibility/power in the compilation of code pieces. What we have now is |compile-file| (and of course batch compilation) and |load|, for compilation; and we have |eval| which is the combination of those two for interpretation.

|compile-file| is currently pretty much tied to a "an output file has an input file" philosophy. You can define macros before compiling a file, and add preludes, and even use the c#expand-source and ##expand-source hooks to get to transform the code completely on your own, but it still assumes that there is an input file, whose path you need to give to |compile-file| and which has to exist since tha procedure is going to read from it.

Sometimes you want to build an object file from code coming from multiple input files (in a more practical way than using macro definitions in the compiler environment, I think); and sometimes you want to build multiple object files from one input file, for example for module parametrization, or for splitting an input file containing test cases intermixed with normal code into two object files so that you can (build and) run the tests separately from the normal case.

Creating binary code in the filesystem (as does Gambit) instead of directly in memory has some advantages: the code blobs can be shared amongst processes, and they don't have to be rebuilt (the filesystem is automatically a cache). But we want that as transparent and easy as possible, too. |compile-file| and |load| is quite good already but those don't address the following things in their current form:

- recompilation only if needed - not depend on *input files* - recompilation of single procedures by entering them in the repl

I want a compile-expression procedure which takes an S-expression as input (maybe later on, we'll also want it a level deeper, after lexical parsing). Gambit does have it's internal form of location-annotated S-expressions already which is just fine; they can be read from files using ##read-expr-from-port, for example.

I also want that this procedure *only* depends on the input it is being fed; i.e. it's a pure function in the sense that the resulting object file only depends on it's inputs. It being pure, you can build a digest of it's inputs, and count on every output it generated being the same if the digest is the same. Thus you can use that digest string as object filename, and if that file already exists, it need not be generated (compiled), only (re)loaded.

I've started to look into implementing this, but it wasn't easy enough to do in the couple of partial days I've had to invest so far; I think especially the part of reloading object files will need some core work: an object file that has been loaded already doesn't need to be reloaded, just re-"linked", or whatever it's being called what Gambit does after loading an object file to make it's toplevel variables visible in the global symbol table. Reloading an object file which has already been loaded once means it should just relink those bindings.

Once those are there, you could just redefine |compile-file| and |load| in terms of those. But use digest strings as object file names instead, maybe, or whatever you'd like; or write |compile-file-and-load-if-changed|, or whatever you'd like to use in your module implementation, anyway. File dependency algorithms etc. can then even more easily be built on top.

And then the next level can be tackled: lexical parsing and macro implementation; of course Gambit has those already built in for the lowlevel kind of macros, and maybe we can use part of that code for the purpose; parsed trees could then, so I hope, be handled by hygienic code merging algorithms, and there we might get chances to make different approaches work together (i.e. code of one module system can use macros of another one); we'll see.

What I'm also thinking about is that for working with code, we may want a simpler version of Scheme, one which doesn't know about the various syntactical sugar forms (like, there is no let or let*, just define[2] and lambda; letrec may be unavoidable). This should make writing macro algorithms easier (or manually written lowlevel macros), right?, or at least other code analyzers (type inferencers, documentation extraction,..). Kind of an even lower language in a syntactic tower. I don't know what other Scheme systems do in this respect but I've read a little bit about GHC's 'core' language, I'm imagining something in this direction. (And at the same time that it is simpler than R5RS Scheme it would also have more core 'ops', namely unsafe ones -- meant to be used safely by using type inferencers/ systems. But from the point of the language, those would still just be procedures.)

([2] well actually I'm not sure yet whether define could/should be reduced to lambda and letrec's too, and some sort of more flexible handling than binding them to "toplevel" variables.)

That's it for now. I'll tell you once I've got some code written; in the mean time, I'm glad hearing your comments.

Christian.