Andrew Lentvorski wrote:

Christian Jaeger wrote:

...

From what I've seen, the experiences of the GHC implementors working on a parallel system (and at first, iirc, a concurrent GC?) may be worthwhile readings.

How does Erlang handle this? The latest versions handle multicore/multiprocessor if I remember correctly.

Just to make clear about what we are talking: Erlang "processes" do not share state (in the form of mutations that could be directly visible in other processes), from the point of view of a user they are *not* shared memory threads. And, not only does the language not offer direct cross-process mutations, but their implementation traditionally did use a separate heaps for every process, even in their user-space scheduled implementation, so not even the *implementation* is/has been using shared memory (except probably for (immutable) program code). As long as they keep it that way, it's dead easy running multiple OS threads, you simply use the OS communication instruments for passing messages between the threads. Whether using OS threads is significantly faster than OS processes in this case is what I have been wondering about several times, numbers would be interesting here (I guess it would eliminate (or reduce) the costs of setting up MMU tables during context switches between OS threads of the same process, other than that, I see no reason for a speedup). Now Scheme is obviously different in that it offers mutation and if you want to make the mutations been passed over to other processes (like the heavy math vector calculation people would like), the shared-nothing approach doesn't work. (I've already suggested using separate shared memory for such tasks.) But on top of that, even if you're using Termite (which does not support mutation): even if you strip shared-memory mutation from Scheme completely (e.g. apply the Termite semantics to the underlying Scheme system), there is one thing left: Termite is so fast in passing big data structures between Termite "processes" (Scheme threads) because Termite "processes" are employing underlying shared memory storage. So no serialization/deserialization has to be done for local data transfers. Now you can't simply run multiple Termite "threads" (i.e. underlying Scheme threads) on different OS threads anymore, because you've got a heap that needs synchronization. You now need a concurrent GC etc., something like what non-green threaded Java virtual machines do (or something less scalable like the global interpreter lock of Python). Now that being said, Erlang people have been discussing moving to using shared heaps, just like Termite. Now the interesting question is: have they managed to efficiently implements heaps being shared *across OS threads*? Note that if they did not (e.g. one heap is only ever green-threaded), they are simply using the Gambit approach, but saving the above mentioned MMU overhead. This would be the approach of running multiple Gambit interpreters in separate pthreads in the same OS process. This should be quite easy (but, as mentioned, I wonder how much of a difference that will make?). Do you have pointers to corresponding Erlang posts/docs?

...

BTW the chicken people have also had some interesting discussion about GSoC participation in the last few days (don't be sad about the difference in volume), also including a few points about threading. Gambit does have a better chance of doing fancy stuff with garbage collection (because of it handling the continuation stack explicitely), of course.

Is there some documentation about this somewhere? I'd love to read it.

(I've got that knowledge gradually over time from experiments / mailing list posts / reading the sources. I suggest to start by reading the paper Marc has posted on March 5th.)

...

...

- web framework

Should I mentor on this? I've actually got some actual professional (well, that doesn't mean particularly successful) experience in this area (with an XML background). But I'd rather prefer having a module system done before continuing writing suboptimally modularized code as I did up to now.

I'm not sure a "web framework" is that great an idea. "Frameworks" work best when they are scratching a particular itch. In addition, any good "web framework" for Scheme is likely to be not very Gambit specific.

Well, of course I've got some ideas on how that framework should look like. It could be open to plug Termite in. Of course you've got a point in that you could most probably port it to other Scheme systems, and the same of course applies to frameworks developed on other systems, they could be ported to Gambit. But if there is no Scheme system agnostic organization out there willing to take on the task of implementing such libraries, then this is being left to the individual system development organizations. I see no reason for Gambit playing purely a library consumer, as opposed to contributor as everyone else. Of course creating such a framework could also leverage (by porting) *parts* of other projects. Two things to keep in mind are: (a) system specifics to make something efficient; (b) module system specifics to make something particularly elegant. Part (a) could help show off Gambit's performance, of course (but yeah, is there a user out there right now who wants this?), part (b) is why I'd prefer seeing module stuff done first. Christian.

Christian Jaeger <christian@pflanze.mine.nu> writes:

- study enough Gambit internals in the area of parsing/compilation stages, macro expander integration, repl handling, and debugger displays, symbol table handling, and possibly more, to be able to adapt and integrate any module system work cleanly.

- create a module system that's easy and practical for everyday work, meaning highly interactive, and at the same time aiming to enable optimizations which can lead to fast code without compromising code clarity (like e.g. cross module inlining, and specialization for parametrized modules).

I would be very interested in this kind of work, especially studying Gambit's internals. It would overlap some with my plans of prototyping a native threading strategy this summer.

- experiment also with the practical aspects on the filesystem side (filesystem layout, compilation caching strategies, C code compilation / some sort of make tools integration (integrate make tool calls into the module system and not vice versy!)).

Essentially a build system. It would be nice to have a build system on top of a module system. (I don't think make tool calls should be integrated with modules, but rather placed on top of.)

I would be happy if a student or other people would take part in this. So I would like to offer to be a mentor. I think we may not get done everything of the above during this summer, so if you want to participate (or also if you don't) help pick out what you think is important.

Count me in.

I wrote:

Cool!

James Long and I have met on the IRC and have now basically already started the project. We(*) have created a section on the wiki here: http://dynamo.iro.umontreal.ca/~gambit/wiki/index.php/Module_System (* well, I have written the pages, James may not yet have verified them) The system does not yet have a name because we want to wait with that until we know the team is complete. James is not a student (anymore), and we think we are open for another core contributer, which still leaves up a possibility for one student. We are currently wildly optimistic, and our project will get us enough building blocks in the way to make it fun and chances realistic that we hit a stumbling rock somewhere. We're out to learn the hard way all the non-working things that generations of scientists didn't have a chance of writing successful papers about. I hope it will advance Gambit nonetheless, and I'll keep an eye on the whole thing to at least stay working in a good basic version for daily usage. So, wish us good luck (and tell us when we go wrong)! Christian.

On Mar 14, 2008, at 9:57 AM, Joel Borggrén-Franck wrote:

"and have discussed why hygienic macro systems rename symbols and that when using namespaces in the symbols, this might be enough as "rename" means, thus providing nice integration with preexisting namespace-using Gambit code."

This seems interesting. Is there a chat-log, blog or paper describing this?

There's a wealth of information on hygienic renaming. Two examples: http://citeseer.ist.psu.edu/109510.html http://citeseer.ist.psu.edu/bawden88syntactic.html As far as the renaming scheme goes, it seems somewhat arbitrary, as long as you're able to recover the correct environment from the renamed identifier (so Gambit namespaces might suffice, but there's also an issue with hygienic "security"). Our discussions on this are scattered but you're welcome to read through our brainstorms on the scheme channel (I have since done some research and understand things better). http://ircbrowse.com/channel/scheme/20080312#hour20

Also I kind of assume you have read or at least know of "Composable and Compilable Macros" by Flatt a good read when you put together a "syntactic tower".

I have heard of it but have not read through it. I will look through it. - James

Date: Mon, 17 Mar 2008 20:33:15 +0000 From: "David Rush" <kumoyuki@gmail.com> On Fri, Mar 14, 2008 at 1:57 PM, Joel Borggr�n-Franck < joel.borggren.franck@gmail.com> wrote:

"and have discussed why hygienic macro systems rename symbols and that when using namespaces in the symbols, this might be enough as "rename" means, thus providing nice integration with preexisting namespace-using Gambit code."

Well, that's what I'd call a viable engineering solution, but I doubt that it truly solves the problem. With a suitable definition of `rename' it does. Macros can capture names even within a single namespace, can't they? However, by reducing the number of symbols in the namespace (since core Scheme has only one), you reduce the probability of capture - assuming that names are randomly distributed in the space of all possible identifiers. Which of course they're not. But that doesn't mean that ny controlling the boundaries of the problem that it doesn't become easier to write real code. I think that long experience is showing that hygienic macros are great, but they don't really serve as a practical substitute for the ability to break hygiene when necessary. And frankly, good old define-macro is hard to beat for that, no matter how groovy syntax-case appears to everyone. This is absurd. The ability to break hygiene when necessary is offered by every reasonable programmatic macro system out there, not merely SYNTAX-CASE, yet long experience has shown that this is almost always a mistake; and DEFINE-MACRO is at best an excellent source of insanely obscure bugs that will turn up when you least expect, because a macro's description failed to include a subtle detail about what names it expects not to be rebound. Such obtrusive implementation detail is a necessary part of an unhygienic macro's interface, just like the names of local variables are in a dynamically scoped language; this is a very real problem in engineering large systems.

Date: Mon, 17 Mar 2008 21:16:51 +0000 From: "David Rush" <kumoyuki@gmail.com> SYNTAX-CASE and explicit renaming (Larceny & S48, IIRC) are the main ones I'm aware of. Are there other major contenders? Explicit renaming is a special case of syntactic closures. I'm not sure whether Andre van Tonder's macro system (one of them high-numbered SRFIs) counts as another; in spirit it is explicit renaming with a dynamically scoped renaming procedure in the macro engine.

yet long experience has shown that this is almost always a mistake;

I never said anything different. But for the rare cases where hygiene breaking is desired, I find the semi-hygienic systems to be awfully cryptic. In local macros that you plan to use in only one place to abbreviate the construction of elaborate names, it is not necessary to concern yourself with hygiene, and you can write a macro transformer procedure with explicit renaming or syntactic closures that looks identical to the DEFINE-MACRO version. But for anything beyond purely local macros, if you don't concern yourself with hygiene, however cryptic it may be, you are bound to lose later on. Here is an exercise for the reader to illustrate the problem. Consider, for example, a DEFINE-ENUMERATION macro, defined so that (define-enumeration foo (bar baz quux)) defines FOO to be a dispatching macro such that (FOO BAR) expands to 0, (FOO BAZ) to 1, and (FOO QUUX) to 2; and defines FOO-ENUMERANDS to be the vector #(BAR BAZ QUUX). Now suppose we write a SYNTAX-RULES macro that uses DEFINE-ENUMERATION: (define-syntax mumble-frotz (syntax-rules () ((MUMBLE-FROTZ a b (c ...)) (BEGIN ... (DEFINE-ENUMERATION AN-ENUMERATION (c ...)) ... (FROB GROVEL AN-ENUMERATION-ENUMERANDS B))))). Let's suppose DEFINE-ENUMERATION inserts the literal symbol AN-ENUMERATION-ENUMERANDS into the output: (define-syntax define-enumeration (er-macro-transformer (lambda (form rename compare) ... (,(rename 'DEFINE) ,(string->symbol (string-concatenate (map symbol->string (list (name->symbol enumeration-name) '- 'ENUMERANDS)))) (,(rename 'QUOTE) ,(list->vector components))) ...))). (We could even strip the renaming if you want, and make it look more like the analogous DEFINE-MACRO.) Why won't MUMBLE-FROTZ work? Why won't AN-ENUMERATION-ENUMERANDS in the output of MUMBLE-FROTZ refer to the variable we want? And since I'm not interested in a religious advocacy war over hygiene, I'd like to know how you think module/namespace-based renaming is an adequate substitute for true hygiene in macros? Because I just don't see it. I'm not interested in religious advocacy either; what I say here is a purely technical judgement. Module-based renaming and macrotic renaming operate at sufficiently different levels of detail as to seem like rather different operators to me. Hygienic renaming does not necessarily just mean affixing to each name a tilde or octothorpe and a big or random number that we hope is unique. Hygienic renaming means associating with each name its meaning, which may for local variables imply the affixing of funny-looking suffixes, but for names whose meaning must persist uniquely throughout a large system of components, it more likely implies associating some description of the module from which the name originated, or a path by which to get there. There is no fundamental, conceptual distinction between `module-based renaming' and `macrotic renaming'. Take, for example, a DELAY macro, in some module that exports FORCE and DELAY and nothing else (perhaps throw EAGER and LAZY in to appease those of us who actually want to use laziness). We want to define the DELAY macro in terms of an internal MAKE-PROMISE: (define-syntax delay (syntax-rules () ((DELAY expression) (MAKE-PROMISE (LAMBDA () expression))))). But the relevant module does not export MAKE-PROMISE, so we can't just insert the literal symbol MAKE-PROMISE into the output, even if the user hasn't locally shadowed that name: if we insert the literal symbol in DELAY's output, it would be just as if the user had written MAKE-PROMISE in her own code, and she would encounter an undefined variable error. Hygiene guarantees that the *name* (not `symbol') in the output of DELAY refers to exactly the same meaning as the symbol MAKE-PROMISE meant in the environment where the DELAY macro was defined. Exactly how this name is represented is inessential to the point of hygiene. It could be a qualified reference directly to the LAZINESS module, if we have a global naming scheme for modules. In Scheme48, it will be a `generated' name, which remembers: the name used to invoke the macro transformer that generated it, and the name by which the transformer identified it. Both names may be generated names themselves, but if we follow them all the way to the end, by recursively looking up the macro transformer, and looking up the name in that transformer's environment, we shall eventually end up with the original meaning we sought. Whether this is the duty of the module system or of the macro system doesn't really matter -- that's an implementation detail, and requires only that the module system and the macro system cooperate. That it work is what matters.

You may have noticed already that Gambit has not been accepted for the GSoC 2008 program, sadly. What does this mean for our module system project? We want to proceed as planned, even if, certainly, a part of the impetus got lost. In the upcoming days, we're going to think about the inputs we've already got so far, and will report back again once we've got some news. We think there will still be enough place for more volunteers. Just talk to us if you're interested (for example on #gambit on irc.freenode.net); or just send us your thoughts. Thanks for caring & keep up with the interesting comments! James & Christian

Date: Wed, 12 Mar 2008 14:52:40 +0100 From: Christian Jaeger <christian@pflanze.mine.nu> - especially after the above infrastructure is well in place, study the existing approaches of modularization and syntax handling out there (Scheme48, MzScheme, Chicken's attempts at mixing syntax systems, riaxpander (http://mumble.net/~campbell/darcs/riaxpander/), R6RS, ERR5RS) A word or two of advice on this subject. `Mixing syntax systems' in the manner pioneered by Chicken is an excellent way to lose badly, and it would be a mistake to replicate this lossage elsewhere. Also, unless you think about macros in conjunction with modules from the beginning, you will not have an easy time tacking them on later. Riaxpander makes parts of this easier, but you'll still need to think about these issues well in advance.