[...] I think a big issue with Scheme is that the community has too many heros. Plenty of people are building clever implementations (my hat comes off and my thanks go out to them)
There is for sure lots of incredible talent in the Scheme community. Many people have done truly amazing things. And I agree wholeheartedly that many of us Schemers are problematically anti-social about what we do. I have been trying to think of how to remedy this, but I'm a really bad offender myself. The first step for me is probably to get out of my closet and get to know more coders.
[...] there seem to be relatively few undertaking the difficult but probably less glamorous and less instantly gratifying task, of making useful software development features accessible across as many implementations as possible.
My personal reason for not bothering doing this kind of thing is that if the goal is to have a language with "batteries", I think the gains from having multiple implementations are so minor as to make it not worth it.
If, however, the different implementations are geared towards special application domains (Stalin for numerical and other performance sensitive code, Guile for scripting, SISC for embedding into the JVM, Scsh for shell scripting), the benefits to having multiple implementations are clear. But sharing code between, say, Stalin and Scsh isn't all that important, because applications written in one tool will have few common dependencies with applications written in another.
Moreover, none of the Scheme implementations I mentioned (except maybe Scsh) would benefit a whole lot from having a full-blown module system with package handling (as in ASDF, rubygems etc) and stuff like that. One main reason to why they fit their problem domain so well is that they are so simple.
Doesn't a module system address your points 1, 2 & 3? –
Not quite. An important aspect of the "batteries included" languages I have used is that because the batteries are included as standard, people tend to use a common tool set for most common tasks.
It is relatively easy for the average Pythonista to read most Python code, because he will recognize much of the basic syntax, for instance for defining classes, because everyone uses the same syntax for defining classes.
Because of this, I don't think it's enough to make it possible to create and share common macros. The benefit of having a sufficiently large core language that everyone uses is substantial.
An example of this is an ad hoc polymorphism facility that would make it possible to overload functions like + or fold. Sure, you could do it as a library, but the main benefit comes when all code, including the standard library, uses this. It would for instance make it very natural for users to define custom port types, to implement e.g. encrypting or compressing ports (by overloading read-char, write-char, close-port and friends). It would also encourage creation of a larger range of data structures, like balanced trees and persistent queues, because it would be much more natural to use non-builtin data structures if it would be possible to define basic operations like fold to work on custom data types as well as builtins.
4) I have argued that making it easy to share code isn't enough to give the language a "batteries included" feel: To get there, we'd have to change the whole language philosophy to the extent that we'd have a new language, complete with its own syntax, semantics, feel and culture.
Here I disagree, but as I said, I'm a newcomer.
Even though I'm not a newcomer, I'm certainly not a veteran, and many of them seem to disagree with me on this one =)
I am investigating the possibility to create my own Lisp based on these ideas, ...
I don't think you'll hurt things, and perhaps you'll get the necessary traction for your implementation. Wouldn't it be easier (though much less exciting) to take an existing Scheme and patch it to support a minimal interface through which is can connect to an independently developing set of tools?
Perhaps. To be honest, I haven't really gotten that far yet. At the moment it is a simple big-step interpreter written in Gambit, which works well for experimentation and bootstrapping purposes. Using an existing Scheme implementation as a base would place some very particular requirements on that implementation, but I will investigate the possibility. Thanks for the advice.
That said, I'm aware that to my mind, Black Hole with Gambit already seems to provide most of what would seem necessary to me. This must mean I've hopelessly misjudged the situation because were I correct, I think this would be evident to you too, and you would only consider your proposed course of action if you were an incorrigible glutton for punishment :-)
I am probably overly pessimistic about the situation of Scheme, and biased towards creating something different because I have a couple of ideas that I would like to try that would involve changing rather fundamental things in the language.
From my point of view, there is still much work to be done before the workflow of Black Hole won't require the user to know about some internals of BH and some Gambit idiosyncrasies that become obvious because of how BH abuses the system.
Cheers,
Per