I have made small changes to my strawman module system for Gambit to clean up the code, improve performance and allow multiple ##use in a body (I now use ##use instead of ##require). Take a look at the attachment.
Per: I checked out the module system you sent to the list (black hole?) and I'm unclear how it is meant to be used. There is no documentation and the 2 examples are really too simple to grasp the possibilities. I could read the code... but I feel it would be best if you gave a quick tour of how your system is supposed to be used. Do I understand correctly that it does not require a "module" special form to wrap the module or be put as a header?
An objective I have is to design the Gambit module system so that it can support several other module systems, such as black hole, Jazz, Chicken, Bigloo, PLT, etc. With this feature, a user could access in his code, libraries that come from different communities with their specific module conventions. That would be a great step forward for Gambit!
Marc
Afficher les réponses par date
Marc:
I don't have time right now to check this. I'm hoping that whatever you come up with integrates well with a hygienic macro system (specifically with syntax-case, that seems to be the most popular system). Perhaps this does, I didn't check.
Brad
On 26-Mar-09, at 8:06 PM, Bradley Lucier wrote:
Marc:
I don't have time right now to check this. I'm hoping that whatever you come up with integrates well with a hygienic macro system (specifically with syntax-case, that seems to be the most popular system). Perhaps this does, I didn't check.
Brad
Strawman v2 does not support hygiene "all the way down". That's because syntax-case.scm does not know anything about namespace declarations.
My short term goal is to design a basic module system that is sufficiently powerful to modularize the Gambit runtime system. Hygiene is not required for that. One thing that is missing is the ability to export record types.
Of course I don't want this basic module system to be incompatible with hygienic macros, since that is a worthwhile feature to support.
But let me repeat myself... strawman is meant mainly to kick around some ideas and elicit discussion.
The main question in my mind... what is lacking in this module system to implement other module systems (such as Scheme48's, PLT's, Chicken's, Bigloo's, Guile's). I would like people on this list (yes *you*) to chip in by picking one of these module systems and trying to implement it on top of strawman. It is very likely that v2 is insufficient, but at least we can have a clearer idea of what is missing, and this can feed the design of the next version.
Marc
I said earlier that it is theoretically possible to implement hygiene with just macros, but that doesn't mean it's feasible. Implementing hygiene on top of another module system would probably mean implementing another module system on top of that one.
Strawman v2 does not support hygiene "all the way down". That's because syntax-case.scm does not know anything about namespace declarations.
Yes, of course. That's because hygiene and modules are intermingled concepts. I think it is possible to implement a hygiene system, and then a module system above it, but that requires very careful design to let the module system access internals of the hygiene system in a well-defined way.
My short term goal is to design a basic module system that is sufficiently powerful to modularize the Gambit runtime system. Hygiene is not required for that. One thing that is missing is the ability to export record types.
Implementing ability to export record types in general requires that the module system does macro expansion of the code when analyzing modules' exports. Basically, the module system must be able to macro expand things.
If the module system was built on top of a hygiene system or had one itself, this is a rather trivial thing to do, because hygiene systems are in effect a macro-expand function.
Of course I don't want this basic module system to be incompatible with hygienic macros, since that is a worthwhile feature to support.
As I see it, a module system either has hygienic macros, or are in practise incompatible with them. It would be cool to see a counter- example to this, however.
But let me repeat myself... strawman is meant mainly to kick around some ideas and elicit discussion.
Yes. I think it's good at that.
The main question in my mind... what is lacking in this module system to implement other module systems (such as Scheme48's, PLT's, Chicken's, Bigloo's, Guile's)
I don't know all of these systems, but I can for sure say that it's impossible to implement PLT's modules (and R6RS libraries, which are similar) without many more features, hygiene in particular.
Incidentally, this is very similar to how black hole started its life: I wanted something quick that just worked, to be able to organize code. I have since then learned the hard way which features a module system has to have to be practical to use. I quickly realized that I wanted to support define-type records. So I hacked together something for that. After fixing all the bugs, I basically had a very mundane macro expansion function. Later on I realized that hygiene is a very good thing to have. And so on.
From personal experience, I can tell that the approach you are taking will require lots of extra work.
/Per
I second Brad's comment. Support for hygienic macros should be on the feature list (you can still provide unhygienic macros as well). This is important because it requires changes to the evaluator (I think), and is something that can't be added on.
Your module system looks cool, Marc. I built something on top of Gambit's namespaces a while ago as well. It uses Scheme48's module language however (which I am partial to). Whenever we get Gambit's low-level module system built the first thing I hope to do is write a scheme48-like module language on top of it!
I hit one major problem with namespaces, however, which your module system is also prone to since it's built on top of the namespaces. I posted a message to this list a while ago about it:
https://webmail.iro.umontreal.ca/pipermail/gambit-modules-list/2009-January/...
Basically, something's a little funky with how DEFINE expressions are evaluated. When the evaluator evaluates the identifier provided to DEFINE, it can evaluate to an identifier in a different namespace. See my post above for examples with the namespace mechanism. I haven't had time to look through Gambit's source to understand why this happens. Here are a few examples in your system:
EXAMPLE 1 (wrongly overrides another library):
(##module library1 gambit (export foo bar))
(define (foo) (display "[library1] foo'ing\n"))
(define (bar) (display "[library1] bar'ing\n") (foo))
******
(##module library2 gambit) (##use library1)
(define (foo) (display "[library2] foo'ing (this should never be called)\n"))
(display "[library2] calling out to BAR in library1\n") (bar)
% gsi -e '(include "modules.scm")' foo.scm [library2] calling out to BAR in library1 [library1] bar'ing [library2] foo'ing (this should never be called)
EXAMPLE 2 (dangerously overrides a global function):
(##module library1 gambit (export foo bar))
(define (integer->char i) (display "bar#integer->char\n"))
*******
(##module library2 gambit) (##use library1)
(integer->char 5)
% gsi -e '(include "modules.scm")' library2.scm bar#integer->char
- James
27 mar 2009 kl. 04.25 skrev James Long:
I second Brad's comment. Support for hygienic macros should be on the feature list (you can still provide unhygienic macros as well). This is important because it requires changes to the evaluator (I think), and is something that can't be added on.
Just to be a little picky: Actually no, it is sufficient to override lambda, let, let*, define and define-macro to implement hygiene. I did that first, but even though it worked, it felt slightly hackish, so now i use the ##expand-source and c#expand-source hooks. This is cleaner and more powerful as well. No changes to Gambit are required.
/Per
On Mar 27, 2009, at 4:21 AM, Per Eckerdal wrote:
27 mar 2009 kl. 04.25 skrev James Long:
I second Brad's comment. Support for hygienic macros should be on the feature list (you can still provide unhygienic macros as well). This is important because it requires changes to the evaluator (I think), and is something that can't be added on.
Just to be a little picky: Actually no, it is sufficient to override lambda, let, let*, define and define-macro to implement hygiene. I did that first, but even though it worked, it felt slightly hackish, so now i use the ##expand-source and c#expand-source hooks. This is cleaner and more powerful as well. No changes to Gambit are required.
True. I haven't look at your source code, but I suppose that would be a metacircular evaluator? I think what I meant to say was that hygiene should be part of the base module implementation, and shouldn't be left to higher-level module systems.
Just to be a little picky: Actually no, it is sufficient to override lambda, let, let*, define and define-macro to implement hygiene. I did that first, but even though it worked, it felt slightly hackish, so now i use the ##expand-source and c#expand- source hooks. This is cleaner and more powerful as well. No changes to Gambit are required.
True. I haven't look at your source code, but I suppose that would be a metacircular evaluator? I think what I meant to say was that hygiene should be part of the base module implementation, and shouldn't be left to higher-level module systems.
Re metacircular evaluator? I'm not sure what you mean by that, but I don't think so. I just redefined the scope-introducing forms, letting them expand into their ##* counterparts, while colouring all variable names. But yes, implementing hygiene is not something that I think should be left to higher-level module systems.
/Per
On 26-Mar-09, at 11:25 PM, James Long wrote:
I second Brad's comment. Support for hygienic macros should be on the feature list (you can still provide unhygienic macros as well). This is important because it requires changes to the evaluator (I think), and is something that can't be added on.
Your module system looks cool, Marc. I built something on top of Gambit's namespaces a while ago as well. It uses Scheme48's module language however (which I am partial to). Whenever we get Gambit's low-level module system built the first thing I hope to do is write a scheme48-like module language on top of it!
I hit one major problem with namespaces, however, which your module system is also prone to since it's built on top of the namespaces. I posted a message to this list a while ago about it:
https://webmail.iro.umontreal.ca/pipermail/gambit-modules-list/2009-January/...
Basically, something's a little funky with how DEFINE expressions are evaluated. When the evaluator evaluates the identifier provided to DEFINE, it can evaluate to an identifier in a different namespace. See my post above for examples with the namespace mechanism. I haven't had time to look through Gambit's source to understand why this happens. Here are a few examples in your system:
EXAMPLE 1 (wrongly overrides another library):
(##module library1 gambit (export foo bar))
(define (foo) (display "[library1] foo'ing\n"))
(define (bar) (display "[library1] bar'ing\n") (foo))
(##module library2 gambit) (##use library1)
(define (foo) (display "[library2] foo'ing (this should never be called)\n"))
(display "[library2] calling out to BAR in library1\n") (bar)
I don't view this as a bug. One interpretation of (define v ...) is (set! v ...) in an environment where v existed, and that's how Gambit has viewed "define" since the beginning. It has the nice feature that it fits well with the REPL (i.e. you can re-define a faulty procedure declaration). You are asking for "define" to do more than this. You want "define" to create a new variable even if it existed previously. Perhaps your example looks like a real bug to you, but a slight variant of it looks like a real problem to me:
(##module m1 gambit (export foo)) (define (foo) (print "m1#foo\n"))
(##module m2 gambit) (##use m1) (define (bar) (foo)) ;; refers to which foo? ;... lots of definitions ... (define (foo) (print "m2#foo\n"))
Perhaps issuing a warning is the right thing to do when defining a variable that is imported.
% gsi -e '(include "modules.scm")' foo.scm [library2] calling out to BAR in library1 [library1] bar'ing [library2] foo'ing (this should never be called)
EXAMPLE 2 (dangerously overrides a global function):
(##module library1 gambit (export foo bar))
(define (integer->char i) (display "bar#integer->char\n"))
(##module library2 gambit) (##use library1)
(integer->char 5)
% gsi -e '(include "modules.scm")' library2.scm bar#integer->char
Here too I don't see this as a bug due to my interpretation of "define" = "set!". Note that when in library1 you
(define (integer->char i) (display "bar#integer->char\n"))
you are really defining the global variable "integer->char" and *NOT* "library1#integer->char" (so obviously the message printed is misleading, beyond the typo... I assume you meant to print "library1#integer->char").
If you had exported integer->char from library1, i.e.
(##module library1 gambit (export foo bar integer->char))
(define (integer->char i) (display "bar#integer->char\n"))
then indeed you would have defined the global variable "library1#integer->char" (and the message printed, the same, would have been appropriate).
So my conclusion is that you don't like "define" = "set!" + allocation of global var if it doesn't exist. Right?
Marc
So my conclusion is that you don't like "define" = "set!" + allocation of global var if it doesn't exist. Right?
One approach to this, which in my experience leads to fewer odd bugs and makes the programmer's intention clearer is to do define it like define = "set!" if it exists in the current module, otherwise allocation of var in the current module. You then add special form to switch between modules, in black hole I simly call it "module":
Let module A contain a variable called test-a.
(In the repl:)
test-a => not defined (import A) test-a => refers to test-a in module A (define test-a 'lalala) test-a => refers to the test-a defined above, will evaluate to
'lalala
(module A) test-a => refers to test-a in module A (define test-a 'new-test-a-in-module-A) => redefines test-a in
module A
(module #f) => this means go back to the REPL pseudo-"module" test-a => refers to the test-a that has the value 'lalala
/Per
Marc Feeley wrote:
So my conclusion is that you don't like "define" = "set!" + allocation of global var if it doesn't exist. Right?
Thanks for the clarification. I like those semantics. The "global var" part becomes confusing when dealing with a module system. As you've said, it's really an issue with how to resolve identifiers.
Your example:
(##module m1 gambit (export foo)) (define (foo) (print "m1#foo\n"))
(##module m2 gambit) (##use m1) (define (bar) (foo)) ;; refers to which foo? ;... lots of definitions ... (define (foo) (print "m2#foo\n"))
That is right, that is ambiguous. I agree with Per that we should stay within our current module, always. I think the easiest and most efficient way of doing this, and if we want to keep the "define" = "set!" semantics, is to import all of the identifiers of your imported packages into your local environment (which is what chjmodule does I think). Gambit's builtin procedures would also be imported locally for every module.
So this would be the expanded source of your example above:
(##module m1 gambit (export foo)) (define (m1#print ##print) (define (m1#foo) (m1#print "m1#foo\n"))
(##module m2 gambit) (###load m1) ;; <- (##use m1) is expanded (define m2#foo m1#foo) ;; to both these lines
(define (m2#bar) (m2#foo)) ;... lots of definitions ... (define (m2#foo) (print "m2#foo\n"))
Now it would handle that situation the same way Gambit currently does with the global environment. This creates module-level scoping of all identifiers for each module. Note that the above example doesn't include all the primitive forms that would be imported, it would really look more like this (the ## prefix will be replaced whenever you modularize Gambit):
(##module m1 gambit (export foo)) (define m1#car ##car) (define m1#cdr ##cdr) (define m1#map ##map) (define m1#vector ##vector) ... (all of Gambit's builtins) ...
Now the global environment is still a problem. How do we reference the global environment in a module? I say don't let it, or keep the ## syntax.
The REPL is also interesting too. It's another kind of environment. I like the way Scheme48 does it: provide a set of builtin commands (Scheme48's "configuration language") which allow you to open, go inside, reload, etc. modules (akin to what Per did). When evaluating commands, it evaluates them in the environment of the current module. If you are not inside a module, you are inside an implicit "REPL" module, *not* the global environment.
You could effectively do away with the global environment. I don't see any use for it as long as you provide the REPL with a place to go when it's not in a module. We can make it backwards compatible with a few hacks.
Per, I don't know how your system resolves symbols. Fixing the DEFINE problem is one thing, but how do you resolve other identifiers?
The main question in my mind... what is lacking in this module system to implement other module systems (such as Scheme48's, PLT's, Chicken's, Bigloo's, Guile's). I would like people on this list (yes *you*) to chip in by picking one of these module systems and trying to implement it on top of strawman
Unfortunately I don't have the time to give you as full of a report as I'd like, but off the top of my head:
* Scoping correctness (ability to define "car" locally, for example) * Identifier renaming * Syntactic tower-like behavior (being able to import a module for macro expansion time)
Per wrote:
Incidentally, this is very similar to how black hole started its life: I wanted something quick that just worked, to be able to organize code. I have since then learned the hard way which features a module system has to have to be practical to use. I quickly realized that I wanted to support define-type records. So I hacked together something for that. After fixing all the bugs, I basically had a very mundane macro expansion function. Later on I realized that hygiene is a very good thing to have. And so on.
From personal experience, I can tell that the approach you are taking will require lots of extra work.
I have to agree with Per. It feels a bit overwhelming right now to support all kinds of different module system. We should figure out exactly what Gambit needs to just have basic module semantics, and implement them in a way that fits Gambit. Then we can talk about adding features or tweaking the module system to support other ones.
- James
Per, I don't know how your system resolves symbols. Fixing the DEFINE problem is one thing, but how do you resolve other identifiers?
I don't know exactly what you want me to explain here, but I'll give it a shot. First, a couple of basic premises: My implementation almost entirely bypasses Gambit namespaces. There are several reasons for it, I won't take them here. Basically it's more trouble than it's worth. Also, the system is designed in a way that doesn't let the user choose namespaces, it is seen completely as an implementation detail. The ns#id syntax is supported, and is used internally, but production code doesn't need it and shouldn't have it.
Every define expression expands into a fully qualified name before it reaches Gambit. In the current version, the REPL has the namespace ~, so if you just boot it up and write (define a #f) it expands into (##define ~#a #f)
The environment keeps track of all imported identifiers and their names. For instance, you could write
(import (prefix: (only: (std srfi/1) fold) 1/))
(Yes, the syntax gets a little bit unreadable when doing more advanced stuff, but I couldn't come up with any better. Plus, it's close to R6RS libraries, so I figure it's ok.)
After this, 1/fold will expand into 1_#fold (1# is a reserved namespace for coloring, so it's renamed to 1_#)
Is this what you were thinking of when asking the question?
The main question in my mind... what is lacking in this module system to implement other module systems (such as Scheme48's, PLT's, Chicken's, Bigloo's, Guile's). I would like people on this list (yes *you*) to chip in by picking one of these module systems and trying to implement it on top of strawman
Unfortunately I don't have the time to give you as full of a report as I'd like, but off the top of my head:
- Scoping correctness (ability to define "car" locally, for example)
- Identifier renaming
- Syntactic tower-like behavior (being able to import a module for
macro expansion time)
This week I have implemented all those three in black hole. (I'm not 100% sure that the syntactic tower thing is entirely conceptually correct though. I'd love to discuss that here) There is still a bug when having define-syntax forms directly inside let(rec)-syntax forms, but I have decided to leave that for now and give the problem some rest for my head to process it.
/Per
Per: I checked out the module system you sent to the list (black hole?) and I'm unclear how it is meant to be used. There is no documentation and the 2 examples are really too simple to grasp the possibilities. I could read the code... but I feel it would be best if you gave a quick tour of how your system is supposed to be used.
Cool. I will write some kind of documentation for black hole asap, hopefully quite soon.
Over the last week I have been working a lot with cleaning up the code and adding a couple of features, primarily being able to import only some things from a module, renaming, and proper separation between compilation phases. The last feature required a change in the fundamental data structures of the hygiene system, and I'm still fighting the last bugs that came out of that. I'd encourage anyone who is interested in reading the code of black hole to wait until that is done, hopefully today or tommorow.
If anyone is interested, I can make the current version public, but I don't think it'll help because some things are broken, and I think it will be difficult to understand how to use it then.
Do I understand correctly that it does not require a "module" special form to wrap the module or be put as a header?
Yes, black hole does not use a "module" wrapper special form. The reasons for this are that it allows complete code compatibility with plain R5RS code, which is good thing in my opinion, plus it gives a more lightweight "feel" to it, which I also think is nice.
An objective I have is to design the Gambit module system so that it can support several other module systems, such as black hole, Jazz, Chicken, Bigloo, PLT, etc. With this feature, a user could access in his code, libraries that come from different communities with their specific module conventions. That would be a great step forward for Gambit!
With the features i have implemented this week, only a syntax-case implementation is required to be roughly feature compatible with most module systems I know of. (I might very well have missed some important points though) This should be rather simple to implement, check out syntax-rules.scm, which is the syntax-rules implementation. It's 264 lines long, took a day to write and I have found only a couple of minor bugs in it since I wrote it.
Right now I think that the best way to implement compatibility with many different module systems is to make it explicit in the import statement which kind of code you're importing. This takes away any confusion about which kind of code it is. Also it's compatible module system where each module is more than one file, and it is possible to extend to fancier things like a pseudo module type that can act as ruby-gems, or something similar.
I'm thinking that the idea of http://catdancer.github.com/lib.html might fit in here, I love the simplicity and sheer decentralized power of it.
Of course, this can be combined with a "default" module kind, which is able to parse formats that are easy to distinguish.
/Per
gambit-modules-list@iro.umontreal.ca
-
Bradley Lucier -
James Long -
Marc Feeley -
Per Eckerdal