Hi.
I have finally decided it's time to release the embryo of a module system that I have written. Some things worth noting:
License:
I don't know what license to distribute it under, so right now I have decided to not choose any formal license at all and just informally state that you're free to read and poke around with the code. The reason for this is a thread on the gambit mailing list a while ago that was about adding a BSD (or MIT) style license, but someone said that if that should be done there is little reason to keep the LGPL/ Apache. To me it seems like it's best if this is resolved before people actually start working on improving this piece of code. I'm not particularly biased towards any license, but I do think a simpler license scheme is better than a complex one all else being equal.
expr.scm:
The tarball contains a file called expr.scm, which is almost entirely written by Christian Jaeger. Other than that, all code is written by me.
What it is:
The initial goal for this was to provide a simple and intuitive way of organizing code in larger projects. It shares many similarities with the Java package system, in particular that the module's name is directly mapped to the file system. To be able to do this neatly with macros, it also contains a system for hygienic macros which is based on syntactic closures. On top of this there is a R5RS syntax-rules implementation. Even though most serious issues have been nailed down, there are still a couple quite basic problems left to be dealt with.
Some inherent limitation of this kind of design:
* It strictly replaces Gambit's current namespace functionality. You can load object files that are compiled with it, but that's about how far you get in terms of interoperability. IME Gambit's namespaces simply aren't nice to work with. * define-macro gets tossed away. There's a lot to say about this topic, so I'll leave it until later. This version of the code does have define-macro, but in practise, it doesn't work. * The namespace#identifier notation is more or less reserved for internal and debugging use. There are a number of reasons for this, I won't take them here.
Usage:
* Untar the package. * (optional) Alter conf.scm to taste. It shouldn't be required to test the basic functionality. The main things that are set here are compilation options and package resolvers. This configuration will get compiled into the module system object file, which kinda sucks, but I haven't wanted to fix it yet. * (optional) Compile the thing: Issue a "gsc build" * To actually use the system: do (load "build") from the REPL (with the appropriate path of course). I have set up a simple shellscript that does this for me.
To test things, you could for instance do this:
;; See the hygiene in action (expand-macro '(let ((a #t)) a))
;; syntax-rules support (define-syntax test (syntax-rules () ((test) #t))) (test)
;; Load a module (make sure to be in the same directory as build.scm and conf.scm) ;; You could also load tests/2/test but it will fail. (use tests/1/test)
;; Compile a module (again, this has to be done from the right directory) (use (build)) (module-compile! 'tests/1/test)
See tests/1/test.scm and tests/1/test2.scm for some clues about how to use the |use| macro. In its simplest form, it takes a symbol which in practise is a path to another .scm file (excluding the .scm suffix) relative to the current file, or pwd if in the REPL. The test- build.scm file is not very neat right now, but it contains a lot of tests for the hygienic system which might be useful to look at.
I guess that's it. Oh, the tarball can be found here:
http://mwaza.dyndns.org/apps/files/module.tar.gz
/Per
Afficher les réponses par date
Hi Per
Per Eckerdal wrote:
Hi.
I have finally decided it's time to release the embryo of a module system that I have written.
Cool, thanks for releasing.
I don't have time to really look at it right now, though; I'll be back coding in the beginning of february.
Some things worth noting:
License:
I don't know what license to distribute it under, so right now I have decided to not choose any formal license at all and just informally state that you're free to read and poke around with the code. The reason for this is a thread on the gambit mailing list a while ago that was about adding a BSD (or MIT) style license, but someone said that if that should be done there is little reason to keep the LGPL/Apache. To me it seems like it's best if this is resolved before people actually start working on improving this piece of code. I'm not particularly biased towards any license, but I do think a simpler license scheme is better than a complex one all else being equal.
I personally find it difficult to generally understand all the issues around licences; I don't feel dual licensing makes matters more complicated; neither for deciding upon a license for a project (since you have to understand what you want your users to be able to do anyway first) nor for a user of a project (since the user can simply choose whatever license he likes or knows, it may make matter even simpler for him); might it make a difference is for a user wanting to contribute to the project?
I'll be happy to contribute to a discussion about licensing when I'm back working (either here on the list or on IRC).
Anyway, as long as the contributors are few and stay reachable until the discussion has happened, it should be no problem just choosing a license now already (in the worst case you'll have to toss away a contribution and rewrite it yourself); I'd suggest the same licenses as Gambit's current ones, since that's what I'd be using for my own contributions too (so we can easily move or share parts).
- It strictly replaces Gambit's current namespace functionality. You
can load object files that are compiled with it, but that's about how far you get in terms of interoperability. IME Gambit's namespaces simply aren't nice to work with.
(I'll like to discuss this when I'm looking at it. As I've discussed with James Long some time (*), I wonder if renaming couldn't use Gambit's namespaces too and conversely code using Gambit's namespace feature couldn't be made accessible cleanly from automatically-renaming code.)
(*) http://dynamo.iro.umontreal.ca/~gambit/wiki/index.php/Module_System/Log
- define-macro gets tossed away. There's a lot to say about this
topic, so I'll leave it until later.
Yep, this will be getting at the interesting parts then.. ;-)
.. the tarball can be found here:
I suggest you check it into Git now; this will help against future confusion. If you don't know Git already:
tar xzf module.tar.gz cd module git init git add . git commit
Then you could either tar it up again and offer it instead of the current tar, but better:
log into your server cd /your_http_root_dir/apps/files/ git init mv -i .git modules.git # depending on git version: chmod +x modules.git/hooks/post-update # or mv -i modules.git/hooks/post-update.sample modules.git/hooks/post-update
and then from inside the git'ified module/ dir on your local machine:
git remote add publish ssh://user@your-server/your_http_root_dir/apps/files/modules.git git push publish master
Christian.
Christian Jaeger wrote:
tar xzf module.tar.gz cd module git init
And I should have said that at this point, you should write a .gitignore file which contains the paths, or shell glob patterns, to the files you don't want under version control, such as your .*.scm files probably (they seem to be generated). It makes sense to also commit the .gitignore file if it contains matches that concern files that are being generated on the users' machines too (exactly those autogenerated files).
git add . git commit
Christian.
And I should have said that at this point, you should write a .gitignore file which contains the paths, or shell glob patterns, to the files you don't want under version control, such as your .*.scm files probably (they seem to be generated). It makes sense to also commit the .gitignore file if it contains matches that concern files that are being generated on the users' machines too (exactly those autogenerated files).
I don't know where those ._*.scm files came from. They don't exist on my machine. I saw that they were there when I made the tarball. I must've done something wrong when issuing the tar command or it's just OSX playing games with me. Those files don't exist in the git repo anyways.
I did what you said and am now able to clone the repo with
git clone http://mwaza.dyndns.org/apps/files/modules.git
I decided to use the same license as Gambit for now. I don't know exactly how this should be done, I just copied the LGPL and Apache license texts to the repo and added a LICENSE file where I say that these licenses are what the code is under.
/Per
Per Eckerdal wrote:
I don't know where those ._*.scm files came from. They don't exist on my machine. I saw that they were there when I made the tarball. I must've done something wrong when issuing the tar command or it's just OSX playing games with me.
Aha those were probably HFS+ metainformation files.
I decided to use the same license as Gambit for now. I don't know exactly how this should be done, I just copied the LGPL and Apache license texts to the repo and added a LICENSE file where I say that these licenses are what the code is under.
As far as I know the general consensus is that you need to add a copyright (and licensing?) statement to every source file.
Gambit only carries copyright statements in its files, no licensing statement, so that might be enough.
I'm following up this mail with a patch for expr.scm (and a second one to add a note).
Christian.
Signed-off-by: Christian Jaeger christian@pflanze.mine.nu --- expr.scm | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-)
Note: save these emails as file and use "git am the-files" to apply them to your git repo.
diff --git a/expr.scm b/expr.scm index c006a9f..216b40e 100644 --- a/expr.scm +++ b/expr.scm @@ -1,4 +1,10 @@ -(define (map* fn obj #!optional (tail '())) +;; Copyright 2006-2009 by Christian Jaeger, <christian at pflanze mine nu> + +;; Published under the terms of either the GNU LGPL Version 2.1 or the +;; Apache License Version 2.0; you can choose whichever license you +;; prefer. + +(define (map* fn obj #!optional (tail '()))) (let rec ((obj obj)) (cond ((pair? obj) (cons (fn (car obj)) @@ -25,8 +31,6 @@ (else (fn obj)))))
-;; ------------------------------------------------------------ -
;; cj Sat, 01 Jul 2006 19:51:48 +0200 ;; moved some stuff from my unfinished cj-syntax module here, and added some more.
Signed-off-by: Christian Jaeger christian@pflanze.mine.nu --- expr.scm | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
diff --git a/expr.scm b/expr.scm index 216b40e..11ef4e7 100644 --- a/expr.scm +++ b/expr.scm @@ -39,6 +39,14 @@ ;; or "syntax" like before? or ?
+;; NOTE: most (if not all) of the functionality in this module is also +;; available as routines in Gambit core; when I wrote this code I +;; didn't know what was available where, so I wrote this from scratch, +;; but at some point I or someone else should look into replacing this +;; with the Gambit core routines and maybe merge what's left into +;; Gambit core. + + (define (expr? obj) ;; returns false for expressions without 'attached notes' (and (vector? obj) (##fixnum.= (##vector-length obj) 4)
Ok, this discussion doesn't feel extremely urgent, but it's not that hard, and it has to be done some time or later, so let's begin.
- It strictly replaces Gambit's current namespace functionality. You
can load object files that are compiled with it, but that's about how far you get in terms of interoperability. IME Gambit's namespaces simply aren't nice to work with.
(I'll like to discuss this when I'm looking at it. As I've discussed with James Long some time (*), I wonder if renaming couldn't use Gambit's namespaces too and conversely code using Gambit's namespace feature couldn't be made accessible cleanly from automatically- renaming code.)
(*) http://dynamo.iro.umontreal.ca/~gambit/wiki/index.php/Module_System/Log
When I think about it, it is not that hard to interface to "legacy" code, using the namespace system. The module system actually uses it internally. Only partially, and I'm hoping to remove it relatively soon, because as I said, it's not nice to work with, but the naming convention namespace#identifier is used and I don't see why that would change.
The module system (ehhh... we ought to find a good name for it) is built on top of the notion of packages, which is a rather powerful mechanism. It should be possible to write code for a package that uses namespace. (Package is the word I use for a collection of modules) The problems I see are these:
Other people might be thinking differently than me here, but I really think that namespaces shouldn't be picked by the user. The way to distinguish modules ought to be a more flexible one. It is really the same problem as prefixing in C, except that in this case there is a really clean solution IMO:
Because namespaces aren't explicitly shown to the user (except when debugging), the module system can choose a namespace for the module when compiling. (There must be a machine-global dictionary of taken namespace names or equivalent to ensure consistency.) This means I can make a CGI module, call it cgi.scm and release it. Someone else might make a different CGI module and call it cgi.scm. With this system, the modules can coexist, the first one installed will get the namespace cgi#, the second one might get cgi-2#.
To me, this is an important feature, both because it hides needless complexity from the user (choosing a namespace, which is redundant work to choosing the file's physical location) and it doesn't require ad-hoc prefixes. The current "solution" seems to be that I ought to call the file something like pe-cgi.scm. To me, this solution sucks, because to be able to make complex software, you need to use hundreds of modules and organise them in a tree structure and I don't see how that fits neatly together.
There are more reasons to why the module system should generate namespaces, mostly related to what a future ruby gems-equivalent tool would benefit from.
Anyways. The problem I see is simply this: All "normal" modules will have a machine-generated namespace, to avoid name clashes. This impossible to do with the old system, so namespace clashes might happen.
The second problem I see is the one with dependency tracking. The system is based on the fact that it is able to track and automatically load dependencies. I don't see how that is done with namespaces and the #.scm convention.
Well, these are the initial problems I can see. It's definitely possible to cook together a hack that is able to load most code, often without problems. A completely clean solution is more difficult.
/Per
Per Eckerdal wrote:
Ok, this discussion doesn't feel extremely urgent, but it's not that hard, and it has to be done some time or later, so let's begin.
(As I said I can't dig into the topic deeper right now, so this is probably my last mail until february.)
When I think about it, it is not that hard to interface to "legacy" code, using the namespace system. The module system actually uses it internally. Only partially, and I'm hoping to remove it relatively soon, because as I said, it's not nice to work with, but the naming convention namespace#identifier is used and I don't see why that would change.
You haven't said what isn't nice.
Using namespace prefixes for the automatic naming of identifiers deems me a good idea for these reasons:
- for debugging, you need some identifier syntax; going to some extent to keep those deterministic should make it easier for the user to type them in at the repl (i.e. something like foo:macrocall-bar#a would be deterministic, while a7481 isn't). - also for work on making different module systems interoperate it will be easier to have such readable representations.
You may have meant these when you said "debugging".
The module system (ehhh... we ought to find a good name for it)
If you'd like to have a name for what you have written, and intend to keep working (because you've got code written using it), now, then best choose a name yourself; could just be pe-modulesystem or so. I think we're going to be a number of people working on partly common, partly differing ideas for some time, so there's not going to be a single unified module system soon and I think it's best if each of the respective original authors keep responsibility for their code base and hence also for the name. Anyway, I can't tell before I've taken a deeper look.
is built on top of the notion of packages, which is a rather powerful mechanism. It should be possible to write code for a package that uses namespace. (Package is the word I use for a collection of modules) The problems I see are these:
Other people might be thinking differently than me here, but I really think that namespaces shouldn't be picked by the user.
- code always needs to specify a dependency somehow; so you will always have the problem of unique global naming in some way or another. - the idea to group together modules into packages, so that you can locally name the other modules and hence can use simple naming, looks sensible or at least interesting (although global naming for remote dependencies is not solved by this, and making two ways of dependency selection maybe adds some additional complexity). - systems which create code need to choose namespaces automatically, sure (for example for parametrization)
So, - a module developer will always have to pick *some* sort of *name*. (I did write you some suggestions in a private mail from 09/22/2008 to achieve this; may I quote that here?) - with plain old manual ##namespace handling, usually the namespace and the files (which are kind of the dependency name) are usually the same. - as mentioned above, for debugging (both system interoperation and at the repl) it would be nice to keep module name and namespace "close", like, module named "foo" in your module system might get namespace "pe-modsys:foo#".
I'm also thinking that a function in Gambit to list all identifiers in a particular namespace would be good to have (easy to write as a filter on all symbols, anyway; maybe it should be made scalable by using trees though).
The way to distinguish modules ought to be a more flexible one. It is really the same problem as prefixing in C, except that in this case there is a really clean solution IMO:
Because namespaces aren't explicitly shown to the user (except when debugging), the module system can choose a namespace for the module when compiling. (There must be a machine-global dictionary of taken namespace names or equivalent to ensure consistency.) This means I can make a CGI module, call it cgi.scm and release it. Someone else might make a different CGI module and call it cgi.scm. With this system, the modules can coexist, the first one installed will get the namespace cgi#, the second one might get cgi-2#.
To me, this is an important feature, both because it hides needless complexity from the user (choosing a namespace, which is redundant work to choosing the file's physical location)
As you've noticed already we're in agreement: I've never implied that you would use ##namespace directives in modules; chjmodule didn't, jazz doesn't, your system doesn't. But it's the suggested mechanism to keep to (a) make module systems work together and debug, (b) there is "legacy" code which would be nice to keep that way (it's like writing assembly, going to a lower level manually, and unsafe if you want to, might disappear with time, but useful for now).
and it doesn't require ad-hoc prefixes. The current "solution" seems to be that I ought to call the file something like pe-cgi.scm. To me, this solution sucks, because to be able to make complex software, you need to use hundreds of modules and organise them in a tree structure and I don't see how that fits neatly together.
What do you mean with "tree"? Filesystem tree? Would you take the filesystem tree 1:1 for the naming in dependencies? Then you still got the problem of conflicts, it's just going from pe-cgi to pe/cgi. (But you might have that "local naming" advantage as above--do you mean this? Are packages living in a "subfolder" for you? Or are they more akin to lexical scoping, where the outside world can't see the inside names?)
There are more reasons to why the module system should generate namespaces, mostly related to what a future ruby gems-equivalent tool would benefit from.
You should tell more.
Anyways. The problem I see is simply this: All "normal" modules will
(there is no "normal" for now--or, there are several definitions of "normal" for now)
have a machine-generated namespace, to avoid name clashes. This impossible to do with the old system, so namespace clashes might happen.
(Again, you didn't say how you avoid *name* clashes; but as mentioned, agreed on that the system chooses the namespace.)
The second problem I see is the one with dependency tracking. The system is based on the fact that it is able to track and automatically load dependencies. I don't see how that is done with namespaces and the #.scm convention.
(I don't think code using ##namespace manually will ever want to load a module of a module system; only the inverse. I.e. code using ##namespace will be at the leaves of the dependency tree. Usually anyway, maybe it's useful for hacking to be able to call module system loader functions from that lower level.)
Christian.
On Jan 14, 2009, at 9:27 AM, Christian Jaeger wrote:
- as mentioned above, for debugging (both system interoperation and at
the repl) it would be nice to keep module name and namespace "close", like, module named "foo" in your module system might get namespace "pe-modsys:foo#".
I'm also thinking that a function in Gambit to list all identifiers in a particular namespace would be good to have (easy to write as a filter on all symbols, anyway; maybe it should be made scalable by using trees though).
Indeed, it would be good for the module system to provide introspection utilities. However, I don't think the technical implementation of the system should be influenced by "debuggability". It should take whatever approach is easiest/sensible/appropriate, and provide debugging tools to interact with the system. So if I wanted to look up a symbol, I would do (module-ref foo-mod "bar") instead of foo#bar. My two cents anyway.
(Again, you didn't say how you avoid *name* clashes; but as mentioned, agreed on that the system chooses the namespace.)
I'd say the simplest way to achieve this is to provide a renaming mechanism, i.e. (import (foo-module (rename (bar bar1) (baz baz1))).
Can't wait to play around with this module system. I'm glad we're discussing this again; I haven't worked on modules for a while and this makes we want to get back into it.
- James
I'd say the simplest way to achieve this is to provide a renaming mechanism, i.e. (import (foo-module (rename (bar bar1) (baz baz1))).
My module system currently lacks this feature, but I'm planning to implement it sometime. The problem we were discussing is related to the naming of modules, not the things modules export.
The problem will arise for instance if you have two modules called "util". If both modules use the "util" namespace, and export the function "foo", the function will be called util#foo internally. If you load both modules into one Gambit process, one of the functions will overwrite the other. Renaming on import doesn't help here.
Can't wait to play around with this module system. I'm glad we're discussing this again; I haven't worked on modules for a while and this makes we want to get back into it.
Great! I'm currently working on taking some of the more general- purpose modules we use (SRFIs, xml handling etc.) and compile them into a "standard library". We'll see what role it gets, the primary purpose I have with doing it right now is to make it easier for other people to see how the system works in practise.
/Per
On 14-Jan-09, at 1:14 PM, Per Eckerdal wrote:
I'd say the simplest way to achieve this is to provide a renaming mechanism, i.e. (import (foo-module (rename (bar bar1) (baz baz1))).
My module system currently lacks this feature, but I'm planning to implement it sometime. The problem we were discussing is related to the naming of modules, not the things modules export.
The problem will arise for instance if you have two modules called "util". If both modules use the "util" namespace, and export the function "foo", the function will be called util#foo internally. If you load both modules into one Gambit process, one of the functions will overwrite the other. Renaming on import doesn't help here.
That's why namespace prefixes can be composed, i.e.
(##namespace ("eckerdal#utils#" append!)) (##namespace ("feeley#utils#" sort!))
It is just the usual hierarchical nesting idea.
Marc
That's why namespace prefixes can be composed, i.e.
(##namespace ("eckerdal#utils#" append!)) (##namespace ("feeley#utils#" sort!))
It is just the usual hierarchical nesting idea.
I think I will need to think up a better way of explaining the approach I am taking here. I'm avoiding having any global prefix at all, because prefixing like this is either prone to clashes (like C) or gives very long names (like Java)
/Per
(Sorry Christian, I forgot to Cc this mail to the list, so you'll get this mail twice)
Christian Jaeger wrote:
When I think about it, it is not that hard to interface to "legacy" code, using the namespace system. The module system actually uses it internally. Only partially, and I'm hoping to remove it relatively soon, because as I said, it's not nice to work with, but the naming convention namespace#identifier is used and I don't see why that would change.
You haven't said what isn't nice.
* When do I use ##namespace and when can I use just namespace? It's a little bit odd.
* The fact that (namespace ("foo")) and (namespace ("foo" bar)) does two completely different things is definitely not nice. (namespace ("foo")) ought to be something like (namespace "foo") instead. Automatic tools need to be careful not to insert an empty list of identifiers that belong to a namespace, since that is something completely different.
* I find (namespace ("foo")) quite difficult to tame. It's easy to mess up lots of things, including the repl when working with it.
* Maybe I should have been clearer in what I meant with the "namespace system": In my eyes the ##namespace directive is very closely tied to the concept of having separate #.scm and .scm files, and the way it is used in practise is broken in my opinion.
Using namespace prefixes for the automatic naming of identifiers deems me a good idea for these reasons:
- code always needs to specify a dependency somehow; so you will
always have the problem of unique global naming in some way or another.
Yes. My point was that using prefixes is a solution that is much inferior to other possible solutions.
- the idea to group together modules into packages, so that you can
locally name the other modules and hence can use simple naming, looks sensible or at least interesting (although global naming for remote dependencies is not solved by this, and making two ways of dependency selection maybe adds some additional complexity).
It does add additional complexity, but I don't think it adds too much. Especially since I am using the concepts of relative and absolute paths, which ought to be familiar to most Unix people.
I think if you spend some time checking out how this module system does it, it might be more clear how it approaches these problems.
The current "solution" seems to be that I ought to call the file something like pe-cgi.scm. To me, this solution sucks, because to be able to make complex software, you need to use hundreds of modules and organise them in a tree structure and I don't see how that fits neatly together.
What do you mean with "tree"? Filesystem tree? Would you take the filesystem tree 1:1 for the naming in dependencies? Then you still got the problem of conflicts, it's just going from pe-cgi to pe/cgi. (But you might have that "local naming" advantage as above--do you mean this? Are packages living in a "subfolder" for you? Or are they more akin to lexical scoping, where the outside world can't see the inside names?)
Every package has its own "filesystem". If you, from a module called foo, do (use bar), it searches for the module bar in the same directory as foo. If you do (use /foofoo/bar), it searches for the module /foofoo/bar in the package foo resides in.
To reach the package from outside, an entirely different nomenclature is used: (use ([package resolver name] [arguments ...]))
If the package resides in a CPAN-like repository, cpan is the package resolver for that repository, and the package name is foobar, you might write (use (cpan foobar /foofoo/bar)) to use the /foofoo/bar module in that package.
So to answer your question: A directory containing a file called Packagefile is considered a package. It has its own tree of modules, mapped 1:1 to the filesystem. For inter-package dependencies, package resolvers are used. Package resolvers can be configured either globally (standard library, CPAN) or per-package.
There are more reasons to why the module system should generate namespaces, mostly related to what a future ruby gems-equivalent tool would benefit from.
You should tell more.
See above. This actually completely removes the problem of name clashes because it is configurable per package.
The second problem I see is the one with dependency tracking. The system is based on the fact that it is able to track and automatically load dependencies. I don't see how that is done with namespaces and the #.scm convention.
(I don't think code using ##namespace manually will ever want to load a module of a module system; only the inverse. I.e. code using ##namespace will be at the leaves of the dependency tree. Usually anyway, maybe it's useful for hacking to be able to call module system loader functions from that lower level.)
What I meant was "legacy" module inter-dependencies. Of course, if there are none, this is a no-issue.
/Per
On 14-Jan-09, at 1:04 PM, Per Eckerdal wrote:
(Sorry Christian, I forgot to Cc this mail to the list, so you'll get this mail twice)
Christian Jaeger wrote:
When I think about it, it is not that hard to interface to "legacy" code, using the namespace system. The module system actually uses it internally. Only partially, and I'm hoping to remove it relatively soon, because as I said, it's not nice to work with, but the naming convention namespace#identifier is used and I don't see why that would change.
You haven't said what isn't nice.
- When do I use ##namespace and when can I use just namespace? It's a
little bit odd.
Not at all... remember that a fully qualified identifier (i.e. which contains a # sign) is an "absolute name". Names without a # sign are relative, and the current namespace scope decides what they actually resolve to.
So the code
(let ((a 10)) (namespace ("foo#")) (and a b))
is equivalent to
(let ((a 10)) (foo#and foo#a foo#b))
By the same logic,
(let ((a 10)) (namespace ("foo#")) (namespace ("bar#" a b)) (and a b))
is equivalent to
(let ((a 10)) (foo#namespace ("bar#" foo#a foo#b)) (foo#and foo#a foo#b))
It may be surprising that the second namespace was not treated as a namespace declaration, but it is consistent because the first namespace changed the meaning of the second reference to "namespace".
The code could be written
(let ((a 10)) (namespace ("foo#")) (##namespace ("bar#" a b)) ;; this is really a namespace declaration (and a b))
or use ##namespace everywhere or use:
(let ((a 10)) (namespace ("foo#") ("bar#" a b)) (and a b))
- The fact that (namespace ("foo")) and (namespace ("foo" bar)) does
two completely different things is definitely not nice. (namespace ("foo")) ought to be something like (namespace "foo") instead. Automatic tools need to be careful not to insert an empty list of identifiers that belong to a namespace, since that is something completely different.
Good suggestion.
- I find (namespace ("foo")) quite difficult to tame. It's easy to
mess up lots of things, including the repl when working with it.
- Maybe I should have been clearer in what I meant with the "namespace
system": In my eyes the ##namespace directive is very closely tied to the concept of having separate #.scm and .scm files, and the way it is used in practise is broken in my opinion.
I agree that the separation is awkward (very C like). But a module system can hide the ##namespace declarations so that users never have to see them.
Using namespace prefixes for the automatic naming of identifiers deems me a good idea for these reasons:
- code always needs to specify a dependency somehow; so you will
always have the problem of unique global naming in some way or another.
Yes. My point was that using prefixes is a solution that is much inferior to other possible solutions.
All solutions boil down to prefixing of one form of another. For example filesystem paths are really a prefixing solution. Same for domain names, URLs, etc.
Note: I'll have to look at your system before I make more comments.
Marc
This is a bit off-topic, but I've been itching to ask this question for a long time, and since we're discussing Gambit's namespacing mechanism...
You can't re-define identifiers that are imported. That is, when you do re-define them, they are changed in the original identifier's namespace instead of the current one. Is this suppose to happen?
--- test.scm ----
(namespace ("foo#")) (##include "~~/lib/gambit#.scm")
(define (run) (display "doing something important in namespace FOO") (newline))
(namespace ("bar#")) (##include "~~/lib/gambit#.scm") (namespace ("foo#" run))
(define (run) (display "doing something important in namespace BAR") (newline))
(namespace (""))
---------
Gambit v4.3.2
(include "test.scm") (run)
*** ERROR IN (console)@2.2 -- Unbound variable: run 1>
(foo#run)
doing something important in namespace BAR
(bar#run)
*** ERROR IN (console)@4.2 -- Unbound variable: bar#run 1>
More importantly, since all primitive functions are defined in the global namespace, there's no way of overriding primitive procedures locally for separate namespaces. For example,
--- test2.scm ---
(namespace ("foo#")) (##include "~~/lib/gambit#.scm")
(define (integer->char i) 'error)
(namespace (""))
------
(include "test2.scm") (foo#integer->char 5)
*** ERROR IN (console)@6.2 -- Unbound variable: foo#integer->char 1>
(integer->char 5)
error
This seems to violate a rather basic principle of namespaces and modules.
- James
James Long wrote:
This is a bit off-topic, but I've been itching to ask this question for a long time, and since we're discussing Gambit's namespacing mechanism...
You can't re-define identifiers that are imported. That is, when you do re-define them, they are changed in the original identifier's namespace instead of the current one. Is this suppose to happen?
Yes, and that's the reason for chjmodule doing copying instead of aliasing identifiers from the source packages directly. Unless you say (global ..) in the chjmodule export declaration, in which case it does the aliasing instead of the copying, which can be useful for things like global variables that should be settable from everywhere.
Christian.
You can't re-define identifiers that are imported. That is, when you do re-define them, they are changed in the original identifier's namespace instead of the current one. Is this suppose to happen?
Yes, and that's the reason for chjmodule doing copying instead of aliasing identifiers from the source packages directly.
The way I do it right now is to expand all defines into something like
(begin (namespace "foo#" bar) (define bar ...))
to ensure that no external symbol is overwritten by define. (This gives slightly different semantics to chjmodule's approach though. Don't know if it's good or not) This might be okay for module systems, but if I were to manually use the namespacing mechanism it would be a source of odd bugs, when an already taken and imported name is used.
Even though this is clearly obvious when you take the definition of ##namespace into consideration, this is definitely one aspect of it that makes me thinks it's odd to work with in practise.
Just like ##namespace vs. namespace, it is very well-defined, but the definition doesn't match the behavior I expect intuitively.
/Per
Marc Feeley wrote:
On 14-Jan-09, at 1:04 PM, Per Eckerdal wrote:
(Sorry Christian, I forgot to Cc this mail to the list, so you'll get this mail twice)
Actually I didn't get this second mail of you at all. I've now changed the mailman settings of my subscription to always send me list posts (which should really be the default, imho, but whatever).
My only comment atm is regarding:
If the package resides in a CPAN-like repository, cpan is the package resolver for that repository, and the package name is foobar, you might write (use (cpan foobar /foofoo/bar)) to use the /foofoo/bar module in that package.
A problem I see with this approach is if you move a module from one package to another. All users of that module will then have to adapt their dependency.
(I have to think about this more to see how bad it fares compared with other approaches. Anyway, the Perl CPAN does not do as you describe, but instead index every module, and then when you want to install one, it finds out to which package it currently belongs.)
You also still have a problem with package name conflicts. Maybe you think that's better because you'll only have to make package names "long and ugly". But you pay with the price of tieing modules to packages.
Christian.
If the package resides in a CPAN-like repository, cpan is the package resolver for that repository, and the package name is foobar, you might write (use (cpan foobar /foofoo/bar)) to use the /foofoo/bar module in that package.
A problem I see with this approach is if you move a module from one package to another. All users of that module will then have to adapt their dependency.
Can you give an example of this? You are correct in that all uses have to be updated, but I don't see that as a bad thing. Moving a module to another package ought to have the same effect as renaming a module, in the same way moving a file does.
IMO there has to be some kind of module hierarchy, all other things would be completely unusable if the module system is supposed to be used for a wide variety of applications, where many different libraries are needed and any combination must be supported. And as I see it, having hierarchy implies this "problem".
Am I getting you right?
You also still have a problem with package name conflicts.
No, I don't. In case of a central repository of packages, yes, of course there cannot be more than one package with a given name, but because it's central that's not a problem. When there is no central repository, the problem doesn't exist, because every package specifies its dependencies a configuration file.
For instance, say that there is a port of the Ogre 3d engine to Gambit. It isn't in any central repository. I am writing a game which uses the port. In the game's package I specify |ogre| to be a package resolver for the Ogre package path.
Anywhere in my game's package, I can issue an (use (ogre /whatever))
Of course, this may cause problems when moving a module from one package to another, possibly requiring a rename of package resolvers. But this is relatively easily done automatically, and it probably doesn't happen very often, because in reality most heavily shared code will be in a central repository anyways.
/Per
Per Eckerdal wrote:
If the package resides in a CPAN-like repository, cpan is the package resolver for that repository, and the package name is foobar, you might write (use (cpan foobar /foofoo/bar)) to use the /foofoo/bar module in that package.
A problem I see with this approach is if you move a module from one package to another. All users of that module will then have to adapt their dependency.
Can you give an example of this? You are correct in that all uses have to be updated, but I don't see that as a bad thing. Moving a module to another package ought to have the same effect as renaming a module, in the same way moving a file does.
I see moving a module happen if it fits better in another package, if another author takes over maintenance of it but not the rest of the original package.
Many modules on CPAN have changed maintainers many times.
Not sure about the moving a module ought to have the same effect as renaming.
IMO there has to be some kind of module hierarchy, all other things would be completely unusable if the module system is supposed to be used for a wide variety of applications, where many different libraries are needed and any combination must be supported.
Well, both Perl with CPAN and Java with it's namespaces are solving the approach by only relying on module names (and using a central registry, CPAN vs. DNS-registry), and there are a wide variety of applications in both, so you'll need to be more specific.
I'm not saying I think your idea is bad, just that I'm not convinced.
And as I see it, having hierarchy implies this "problem".
Am I getting you right?
You also still have a problem with package name conflicts.
No, I don't. In case of a central repository of packages, yes, of course there cannot be more than one package with a given name, but because it's central that's not a problem. When there is no central repository, the problem doesn't exist, because every package specifies its dependencies a configuration file.
So you let the user resolve the conflict.
For instance, say that there is a port of the Ogre 3d engine to Gambit. It isn't in any central repository. I am writing a game which uses the port. In the game's package I specify |ogre| to be a package resolver for the Ogre package path.
Anywhere in my game's package, I can issue an (use (ogre /whatever))
Of course, this may cause problems when moving a module from one package to another, possibly requiring a rename of package resolvers. But this is relatively easily done automatically, and it probably doesn't happen very often, because in reality most heavily shared code will be in a central repository anyways.
(In which case the problem can just as well be solved on a per-module basis.)
Christian.
Okay. Another attempt:
I have very dated experience with Java, but in my opinion Java packages seem to have two parts in practise: Take the class org.apache.cactus.internal.configuration.DefaultFilterConfiguration as an example. In my mind I split it into
org.apache.cactus || internal.configuration.DefaultFilterConfiguration
With bh, org.apache.cactus is the package, /internal/configuration/ DefaultFilterConfiguration is the module path and DefaultFilterConfiguration is the module name. /internal/configuration is there simply as a means of organizing the internals of cactus, org.apache.cactus is there as the global unique qualifier.
As I see it, there /must/ be a way of doing the right hand part of this; projects must be able to split themselves up into a filesystem like structure. I have never ever seen any other way that works. Moving /internal/configuration/DefaultFilterConfiguration to /internal/ DefaultFilterConfiguration should require a rename of all uses of the module, because this is a file system like thing.
org.apache.cactus is there as a means of guaranteeing the uniqueness of the java package. I don't like this solution, because it yields very long names which in practise makes it impossible to code Java without a fancy editor like Eclipse.
I don't think the CPAN way of having a central repository would work; This is Scheme. I don't think we'll be able to unite around one central system. Even if one system becomes standard in practise, people will want to have the power of being able in theory to make a better one.
The motivation behind this design is that it allows for a clean, non- name-clashing way of providing unique names to everyone's code without resorting to very long names or forcing one central thing on everyone. Comparing to the Java approach, this is like allowing local shorthand names (ie let cactus be an alias for org.apache.cactus) at the same time you explicitly state the path to the package. ("let cactus be an alias for the package at /home/per/prog/cactus")
Also, this can be combined with a (or more than one) central repository, which will remove the need of configuring where each dependency is located on the local computer.
One more advantage of this is that it is very easy to do simple hacks that won't go anywhere. If I create a file called foo.scm, it is by definition a module that can be loaded. I might then feel like adding one more file, bar.scm and use foo from it; no problem. Later on, I might realize that this is some great software that ought to be distributed. No problem; I just create a Packagefile in that directory and it is a package, ready for distribution. More people might like it and i might even decide to publish it to the central code repository.
It is easy to create code that other people can use, which IMO is one of the greatest problems with Gambit right now. Nowhere in this process will I have to rewrite anything to make it fit a bigger audience. No worries about namespaces or name clashes.
Another advantage of this kind of approach (it might not be unique to it, though) is that it's clear by the look of the use form what kind of import it is: (use (... ...)) refers to something external, (use /...) to something not very closely related to this module, (use ...) to something in the same directory as this file.
This is in contrast to ruby, where a 'require "json"' might refer to json anywhere in the system, it's impossible to tell.
Christian Jaeger wrote:
I see moving a module happen if it fits better in another package, if another author takes over maintenance of it but not the rest of the original package.
I still cannot see a real-world example of this. One package is not = one maintainer. One package is one functional entity, like HTTP utilities. When I want to load some cookie tools, i use /cookie-tools from the http-util package.
/Per
I must go into suspend mode now in this thread for 2½ weeks.
(thread-sleep! 1.8e6)
Hm ps. Marc, there seems something fishy with Gambit v4.3.2, ctl-c doesn't work anymore on the above.
I wrote:
Hm ps. Marc, there seems something fishy with Gambit v4.3.2, ctl-c doesn't work anymore on the above.
(Entered as http://www.iro.umontreal.ca/~gambit/bugzilla/show_bug.cgi?id=87)
gambit-modules-list@iro.umontreal.ca
-
Christian Jaeger -
James Long -
Marc Feeley -
Per Eckerdal