In relation with the conversation below on the use of Black Hole, here's a reflection on Gambit w Black Hole:
That the Scheme environment (i.e. Gambit) and the module system (i.e. Black Hole) are completely separate, appears to be a good abstraction to me.
Even if for instance R7RS introduces modules at the level of the language spec, still having that part of the language performed by a separate module system tool appears like a good idea to me.
(Disclaimer: I didn't study how R7RS does modules, and I'm in no way an expert on module system implementation)
This as there's so many things a module system could be set up to do - for instance, access module source code data in various ways such as retrieve packages from a remote host using who-knows what method to ensure that the respective package is up to date. Also I would guess that there are details about how the code expansion is done that the R7RS spec leaves open to the individual module system implementation to decide. Also as a sidenote, a module system brings a lot of complexity with it that you're best off distributing to code that's completely outside of the Scheme implementation's code, so you can develop, debug and study the both completely in separate from each other.
So while it's a realistic goal to make a 'perfect' Scheme implementation (in the sense that it that covers 'all' usecases and could be used for years and years with only very minor code updates), making a module system that's 'perfect' in this same sense is probably impossible.
Thus there could easily be the need for multiple module systems for a Scheme implementation.
So, I'd suppose the solution to the issue Per pointed out below about that it's hard to integrate a module system into Gambit in a reasonable way, is to make documentation and tools for the respective module system, that keeps it super-clear how the user uses the module system (import of modules, compilation etc.), and what parts of Gambit should not be used anymore by the user except for as part of intentionally breaking into the module system's setup during module system debugging or alike, now that the user has chosen to use the module system. Obviously a module system could also block access to the Gambit features that should no longer be used i.e. (load), (include), possibly the -exe argument etc., or overload them with corresponding features from the module system. The solution to the issue is not to make a single module system inside Gambit that's hardwired to it and that's it.
In all cases this is what I perceive as the proper path of development, based on seeing Gambit and Black Hole currently - how they interact and the outcome from this interaction.
Corrections, thoughts or suggestions are welcome. For instance, what ways do you see in which Gambit could be made even more suitable for implementing a module system atop of it?
Kind regards, Mikael
2011/6/8 Per Eckerdal per.eckerdal@gmail.com ..
Hehe.. The stuff you have been experiencing is a large reason to why I have pretty much given up working on Black Hole, it is so hard to integrate it into Gambit in a reasonable way.
It kindof works for me personally, for instance because I know that the warning messages you see are from Gambit and not BH, but there are so many small things that don't make sense unless you know the inner workings of BH (debugging from the REPL and attempting to access the hygienically renamed variables is a mess), and I haven't been able to get around it.
Afficher les réponses par date
On 9 Jun 2011, at 02:38, Mikael wrote:
In relation with the conversation below on the use of Black Hole, here's a reflection on Gambit w Black Hole:
That the Scheme environment (i.e. Gambit) and the module system (i.e. Black Hole) are completely separate, appears to be a good abstraction to me.
I largely agree, but I'll wade in with my thoughts so far…
• Per hinted that a module system needs to be able to monkey about with the implementation at a reasonably deep level. Perhaps an interface can be provided for this though, and this could then be used for other module systems (and perhaps other facilities too).
• I strongly feel that languages "do well" when they come with batteries included. The particular metric of do well I'm thinking of is getting widely adopted and used by loving developers. While being able to use other module systems or support an extensible module system would be lovely, I think Gambit should instal with a module system "out of the box" as soon as possible. Once that is achieved, the system can be refined further.
Thanks, Benjohn
On Monday, 13 June 2011 at 20:02, Benjohn Barnes wrote:
• I strongly feel that languages "do well" when they come with batteries included. The particular metric of do well I'm thinking of is getting widely adopted and used by loving developers.
I believe there are several key parts to the "batteries included" feel of a language ecosystem:
One of them is that it has to be easy to create a library and share it with others; otherwise a sufficiently large body of libraries won't be created. This has been the main motivation behind Black Hole: It could make it feasible to write and share libraries with non-trivial dependency graphs.
But I have realized that simply making it possible to share code easily isn't sufficient. Some important issues here are
1) R5RS, being a minimal language, simply doesn't provide enough syntax (ie macros). The special forms are so few that each developer has to choose between using vanilla R5RS, which is often painful (the syntax for multiple values for instance), or creating his own set of base macros, which leads to loads of separate and suboptimal designs of things that should be core language constructs.
This works well for isolated projects with few dependencies to other libraries, but it seriously hurts code readability when sharing libraries.
2) R5RS has a culture of not incorporating things into the language until it is known that the particular design is arguably the best possible. This works well when it is acceptable to have many different approaches to the same problems until they're solved, but it seriously hurts the ability to share code.
If one wants to make a "batteries included" system, this design philosophy simply doesn't work. It is necessary to incorporate things before they are perfectly worked out, and then improve things incrementally. (Of course this doesn't make careful design less important.)
The minimalistic, don't-add-until-perfect, way of things has left R5RS with several important (for projects that need "batteries included") holes in its core functionality, for instance a module system, unwind-protect, complete I/O, threads and even simple things like a basic set of list functions (fold, where are you?).
3) Most "batteries included" languages nowadays come with a relatively large standard library, and rarely encourage multiple language implementations like Scheme does. I think discouraging multiple implementations of the language is an important ingredient to make a batteries included language work.
There can be more than one language implementation, but there has to exist an obvious default. Ruby, Python, Java, PHP, .NET, and to some extent C/C++/ObjC all do this.
4) I have argued that making it easy to share code isn't enough to give the language a "batteries included" feel: To get there, we'd have to change the whole language philosophy to the extent that we'd have a new language, complete with its own syntax, semantics, feel and culture.
Yet I think it's important to remember that, for some applications, Scheme's simplicity and smallness are really nice. Examples of this are education and embedded apps.
The people who currently use Scheme now probably use it because it actually fits what they are trying to do. Changing the language this much would change the target audience of the language so much that I can't see how Scheme could evolve into a batteries included language.
----
Racket matches what I have described above fairly well, but it has some design choices that I haven't (yet?) been able to come to terms with.
I am investigating the possibility to create my own Lisp based on these ideas, but I don't know yet if it would just hurt things by segmenting the Lisp world even further, or could help by contributing some new ideas and thoughts. I think that I at least know enough about Scheme and Lisp to be able to avoid repeating most historical mistakes though.
/Per
2011/6/14 Per Eckerdal per.eckerdal@gmail.com:
[many interesting things]
Now that I know scheme somehow enough, I decided to use Gambit to get things done, instead of another implementation. I am fairly happy with the way Gambit respects the reports and SRFIs, and wish not to change implementation.
I would not mind at all Gambit going batteries included, and even not being the perfect gem that some people look for or try to craft during their whole life. I'd rather have an implement that is imperfect and that evolves. I do believe that this is a good thing in this situation, as I am not considering a random language, but a Scheme implementation from someone who has managed to build a very good system that is faithful to its Scheme roots.
Even though I do not see Scheme, the language, become a batteries included language, I think that Gambit, the implementation, may very well. Still, I would wait a little bit to see what R7RS has to offer in terms of API and design. Hopefully, it will be accepted by the community, and be used long enough so that Gambit can become a wonderful batteries included dialect of R7RS.
Cheers,
P!
On 14 Jun 2011, at 11:06, Per Eckerdal wrote:
I believe there are several key parts to the "batteries included" feel of a language ecosystem:
One of them is that it has to be easy to create a library and share it with others; otherwise a sufficiently large body of libraries won't be created. This has been the main motivation behind Black Hole: It could make it feasible to write and share libraries with non-trivial dependency graphs.
But I have realized that simply making it possible to share code easily isn't sufficient. Some important issues here are
- R5RS, being a minimal language, simply doesn't provide enough syntax (ie macros). The special forms are so few that each developer has to choose between using vanilla R5RS, which is often painful (the syntax for multiple values for instance), or creating his own set of base macros, which leads to loads of separate and suboptimal designs of things that should be core language constructs.
This works well for isolated projects with few dependencies to other libraries, but it seriously hurts code readability when sharing libraries.
- R5RS has a culture of not incorporating things into the language until it is known that the particular design is arguably the best possible. This works well when it is acceptable to have many different approaches to the same problems until they're solved, but it seriously hurts the ability to share code.
If one wants to make a "batteries included" system, this design philosophy simply doesn't work. It is necessary to incorporate things before they are perfectly worked out, and then improve things incrementally. (Of course this doesn't make careful design less important.)
The minimalistic, don't-add-until-perfect, way of things has left R5RS with several important (for projects that need "batteries included") holes in its core functionality, for instance a module system, unwind-protect, complete I/O, threads and even simple things like a basic set of list functions (fold, where are you?).
- Most "batteries included" languages nowadays come with a relatively large standard library, and rarely encourage multiple language implementations like Scheme does. I think discouraging multiple implementations of the language is an important ingredient to make a batteries included language work.
There can be more than one language implementation, but there has to exist an obvious default. Ruby, Python, Java, PHP, .NET, and to some extent C/C++/ObjC all do this.
Speaking as a scheme outsider until recently (so my view is probably not worth much), I think a big issue with Scheme is that the community has too many heros. Plenty of people are building clever implementations (my hat comes off and my thanks go out to them), but there seem to be relatively few undertaking the difficult but probably less glamorous and less instantly gratifying task, of making useful software development features accessible across as many implementations as possible. I'm not sure multiple implementations are a bad thing per se, but they do surely make it hard for a system of cross Scheme tooling to gain traction.
Doesn't a module system address your points 1, 2 & 3? –
1) Additional special forms living in modules could "compete" and grow. Perhaps one would finally become a de facto standard and be ratified by RnRS (or the best features of several libraries could form the basis of ratification, spinning off in to a new library that existing client libraries would develop towards). If RnRS doesn't want to include unnecessary details, it can just ignore this work, or an additional standards "track" could take it on.
2) Modules allow library and syntax to be developed without being formally included in the standard until they are ready. It seems to me this would complement the understandably gradual development of the language core very well.
3) A module system that can be hooked on to by existing Schemes (with some work on their part) allows them to develop in relative isolation, while the shared community also grows around the modules which act as a (de facto) standard library.
- I have argued that making it easy to share code isn't enough to give the language a "batteries included" feel: To get there, we'd have to change the whole language philosophy to the extent that we'd have a new language, complete with its own syntax, semantics, feel and culture.
Here I disagree, but as I said, I'm a newcomer.
Unification work isn't glamourous or attractive (or perhaps even a significant priority) for most of the people able to do it. It also probably has a significant "political" facet, rather than being purely technical. If I am right, this is a different from the underlying philosophy – it's more about people as a whole, rather than Scheme :-)
The existing syntax, semantics and feel are all pretty great from my point of view. They just have a few areas where more is needed. A widely supported module system should let this develop.
Yet I think it's important to remember that, for some applications, Scheme's simplicity and smallness are really nice. Examples of this are education and embedded apps.
The people who currently use Scheme now probably use it because it actually fits what they are trying to do. Changing the language this much would change the target audience of the language so much that I can't see how Scheme could evolve into a batteries included language.
As I've said above, I don't think such a big shift is needed. Assuming this is the case, the existing audience would be greatly augmented, rather than replaced. Existing implementers wouldn't need to hook their Scheme to a new module system. They could use the module system externally to pull down code (rather than having it available natively from within the language), or they could simply ignore it completely.
Racket matches what I have described above fairly well, but it has some design choices that I haven't (yet?) been able to come to terms with.
Jazz also seems to do this, and is derived from Gambit (I think), but I lost patience when I found I'd need to instal a pile of extra packages to get it working (which is certainly stupid of me and I should look again).
I am investigating the possibility to create my own Lisp based on these ideas, but I don't know yet if it would just hurt things by segmenting the Lisp world even further, or could help by contributing some new ideas and thoughts. I think that I at least know enough about Scheme and Lisp to be able to avoid repeating most historical mistakes though.
I don't think you'll hurt things, and perhaps you'll get the necessary traction for your implementation. Wouldn't it be easier (though much less exciting) to take an existing Scheme and patch it to support a minimal interface through which is can connect to an independently developing set of tools?
That said, I'm aware that to my mind, Black Hole with Gambit already seems to provide most of what would seem necessary to me. This must mean I've hopelessly misjudged the situation because were I correct, I think this would be evident to you too, and you would only consider your proposed course of action if you were an incorrigible glutton for punishment :-)
Thanks, Benjohn
On Tuesday, 14 June 2011 at 16:36, Benjohn Barnes wrote:
[...] I think a big issue with Scheme is that the community has too many heros. Plenty of people are building clever implementations (my hat comes off and my thanks go out to them)
There is for sure lots of incredible talent in the Scheme community. Many people have done truly amazing things. And I agree wholeheartedly that many of us Schemers are problematically anti-social about what we do. I have been trying to think of how to remedy this, but I'm a really bad offender myself. The first step for me is probably to get out of my closet and get to know more coders.
[...] there seem to be relatively few undertaking the difficult but probably less glamorous and less instantly gratifying task, of making useful software development features accessible across as many implementations as possible.
My personal reason for not bothering doing this kind of thing is that if the goal is to have a language with "batteries", I think the gains from having multiple implementations are so minor as to make it not worth it.
If, however, the different implementations are geared towards special application domains (Stalin for numerical and other performance sensitive code, Guile for scripting, SISC for embedding into the JVM, Scsh for shell scripting), the benefits to having multiple implementations are clear. But sharing code between, say, Stalin and Scsh isn't all that important, because applications written in one tool will have few common dependencies with applications written in another.
Moreover, none of the Scheme implementations I mentioned (except maybe Scsh) would benefit a whole lot from having a full-blown module system with package handling (as in ASDF, rubygems etc) and stuff like that. One main reason to why they fit their problem domain so well is that they are so simple.
Doesn't a module system address your points 1, 2 & 3? –
Not quite. An important aspect of the "batteries included" languages I have used is that because the batteries are included as standard, people tend to use a common tool set for most common tasks.
It is relatively easy for the average Pythonista to read most Python code, because he will recognize much of the basic syntax, for instance for defining classes, because everyone uses the same syntax for defining classes.
Because of this, I don't think it's enough to make it possible to create and share common macros. The benefit of having a sufficiently large core language that everyone uses is substantial.
An example of this is an ad hoc polymorphism facility that would make it possible to overload functions like + or fold. Sure, you could do it as a library, but the main benefit comes when all code, including the standard library, uses this. It would for instance make it very natural for users to define custom port types, to implement e.g. encrypting or compressing ports (by overloading read-char, write-char, close-port and friends). It would also encourage creation of a larger range of data structures, like balanced trees and persistent queues, because it would be much more natural to use non-builtin data structures if it would be possible to define basic operations like fold to work on custom data types as well as builtins.
- I have argued that making it easy to share code isn't enough to give the language a "batteries included" feel: To get there, we'd have to change the whole language philosophy to the extent that we'd have a new language, complete with its own syntax, semantics, feel and culture.
Here I disagree, but as I said, I'm a newcomer.
Even though I'm not a newcomer, I'm certainly not a veteran, and many of them seem to disagree with me on this one =)
I am investigating the possibility to create my own Lisp based on these ideas, ...
I don't think you'll hurt things, and perhaps you'll get the necessary traction for your implementation. Wouldn't it be easier (though much less exciting) to take an existing Scheme and patch it to support a minimal interface through which is can connect to an independently developing set of tools?
Perhaps. To be honest, I haven't really gotten that far yet. At the moment it is a simple big-step interpreter written in Gambit, which works well for experimentation and bootstrapping purposes. Using an existing Scheme implementation as a base would place some very particular requirements on that implementation, but I will investigate the possibility. Thanks for the advice.
That said, I'm aware that to my mind, Black Hole with Gambit already seems to provide most of what would seem necessary to me. This must mean I've hopelessly misjudged the situation because were I correct, I think this would be evident to you too, and you would only consider your proposed course of action if you were an incorrigible glutton for punishment :-)
I am probably overly pessimistic about the situation of Scheme, and biased towards creating something different because I have a couple of ideas that I would like to try that would involve changing rather fundamental things in the language.
From my point of view, there is still much work to be done before the workflow of Black Hole won't require the user to know about some internals of BH and some Gambit idiosyncrasies that become obvious because of how BH abuses the system.
Cheers,
Per