Hi. Following some brainstorming with Christian Jaeger, Jérémie Lasalle-Ratelle, and Bradley Lucier last week I have returned to thinking about runtime system extensions that would be useful for modular programming.
Concurrently Guillaume Cartier was working on improving the startup time of Jedi. Guillaume expressed the need to statically link some of the modules of the Jazz system into the executable (to avoid the startup delay caused by accessing the filesystem for many small modules). He would like the modules to be linked into the executable, but *not initialized*. Instead the modules should only be initialized if and when they are actually used by the application.
One of the things that we (excluding Guillaume) did during the brainstorming was to look into some Chicken eggs to see how they are organized, and how they could be used within Gambit with little or no changes to the egg source code. It turns out that Gambit's namespace system provides most of what is needed (although the FFI may need small extensions for some eggs).
One thing that came up was the need to ensure that a module would be loaded at most once. This is because a given module may be required by multiple modules that are required by the main module/program. For example the main module A may require modules B and C, and both B and C require module D. Simply putting a (load "D") at the top of B and C will end up loading "D" twice. During our brainstorming I proposed a "load-once" procedure which would maintain a table of the previously loaded modules so that a module would only be loaded once. There are two issues with this:
1) How are modules identified? In other words what is the key that is used to access the previously loaded module table? One simple approach would be to use the filename as the key, but this does not work in a static linking model where the modules have been linked with the executable (and the executable is meant to be installed on a different machine). So what is a good way to identify modules unambiguously? A hash? A user supplied "module id"? Should modules have version numbers?
2) For interactive development there is a need to force the loading of modules, even if they have been loaded already. How should this be expressed by the user?
If you have some thoughts about these issues please tell!
Marc
Afficher les réponses par date
Hallo,
On Tue, Mar 24, 2009 at 4:38 PM, Marc Feeley feeley@iro.umontreal.ca wrote:
- How are modules identified? In other words what is the key that is
used to access the previously loaded module table? One simple approach would be to use the filename as the key, but this does not work in a static linking model where the modules have been linked with the executable (and the executable is meant to be installed on a different machine). So what is a good way to identify modules unambiguously? A hash? A user supplied "module id"? Should modules have version numbers?
Lua's REQUIRE takes a string as parameter. It first looks for the module in a "preload" table using the string as key, which is how statically linked modules register themselves. If not found there, it searches the module in the file system using the string as a file name. Here is a link for the relevant documentation:
http://www.lua.org/manual/5.1/manual.html#pdf-require
Cheers,
Concurrently Guillaume Cartier was working on improving the startup time of Jedi. Guillaume expressed the need to statically link some of the modules of the Jazz system into the executable (to avoid the startup delay caused by accessing the filesystem for many small modules). He would like the modules to be linked into the executable, but *not initialized*. Instead the modules should only be initialized if and when they are actually used by the application.
I think that's a great idea. It would be a generally useful tool, for module systems and some other applications as well.
I think an important part of making a module system work well is not to have a (load) function that "loads" a module, but insted have an import/use construct that declaratively states that this module needs this other module to work. It is then up to the module system when and how to load the modules. This makes it easier to analyze dependencies and, imho, yields clearer code.
- How are modules identified? In other words what is the key that is
used to access the previously loaded module table? One simple approach would be to use the filename as the key, but this does not work in a static linking model where the modules have been linked with the executable (and the executable is meant to be installed on a different machine). So what is a good way to identify modules unambiguously? A hash? A user supplied "module id"? Should modules have version numbers?
There are of course many possible solutions, but I would try to do it completely transparent for the user. If the static linking model is to work well, the user shouldn't have to think about it, and the same code should work regardless of how the code is linked.
It should be possible to build an abstraction that seems to work by filename, with some automatic trickery behind the scenes for other linking strategies.
- For interactive development there is a need to force the loading of
modules, even if they have been loaded already. How should this be expressed by the user?
In the system I have written this is done simply by re-importing the module from the REPL. The module system can see (with a hash and/or with timestamps) that the module is updated and reloads it. It has simply never been a problem.
/Per
On 24-Mar-09, at 2:25 PM, Per Eckerdal wrote:
Concurrently Guillaume Cartier was working on improving the startup time of Jedi. Guillaume expressed the need to statically link some of the modules of the Jazz system into the executable (to avoid the startup delay caused by accessing the filesystem for many small modules). He would like the modules to be linked into the executable, but *not initialized*. Instead the modules should only be initialized if and when they are actually used by the application.
I think that's a great idea. It would be a generally useful tool, for module systems and some other applications as well.
I think an important part of making a module system work well is not to have a (load) function that "loads" a module, but insted have an import/use construct that declaratively states that this module needs this other module to work. It is then up to the module system when and how to load the modules. This makes it easier to analyze dependencies and, imho, yields clearer code.
Yes I completely agree. However, at the lowest level the system must perform some operation which makes the module usable to the program. In other words, an "import" or "use" construct must rely on lower level operations to do its work. I want to brainstorm on what should be provided at the lowest level. Then this lower level could be used to implement a higher level module system on top.
- How are modules identified? In other words what is the key that
is used to access the previously loaded module table? One simple approach would be to use the filename as the key, but this does not work in a static linking model where the modules have been linked with the executable (and the executable is meant to be installed on a different machine). So what is a good way to identify modules unambiguously? A hash? A user supplied "module id"? Should modules have version numbers?
There are of course many possible solutions, but I would try to do it completely transparent for the user. If the static linking model is to work well, the user shouldn't have to think about it, and the same code should work regardless of how the code is linked.
It should be possible to build an abstraction that seems to work by filename, with some automatic trickery behind the scenes for other linking strategies.
I find filenames to be too brittle of an identifier for modules. It shouldn't matter where a module is located.
- For interactive development there is a need to force the loading
of modules, even if they have been loaded already. How should this be expressed by the user?
In the system I have written this is done simply by re-importing the module from the REPL. The module system can see (with a hash and/or with timestamps) that the module is updated and reloads it. It has simply never been a problem.
You mean because the "last modification" time of the source is newer than what was recorded when the module was loaded? What happens if module A requires B, and B has changed but A hasn't. Will your module system reload B if you ask to load A again? Moreover, often the initialisation of a module depends on the initialisation of the module it depends on. In my example, module B might create a unique key for a record type (i.e. generative records) which is then used by A's initialisation to instantiate some records of that type. So in this case it is necessary to reload A even if it hasn't changed. I believe the safest approach is to reload the whole dependence graph of the module to be reloaded.
Marc
Hallo,
On Tue, Mar 24, 2009 at 6:51 PM, Marc Feeley feeley@iro.umontreal.ca wrote:
It should be possible to build an abstraction that seems to work by filename, with some automatic trickery behind the scenes for other linking strategies.
I find filenames to be too brittle of an identifier for modules. It shouldn't matter where a module is located.
The module search path could be set using different primitives than the module loading ones. Chicken uses a combination of a compile-time string and environment variables, for instance. I've never seen chicken code require modules with full paths. And chicken supports static linking of modules, that you must require to use.
Cheers,
24 mar 2009 kl. 19.51 skrev Marc Feeley:
I think an important part of making a module system work well is not to have a (load) function that "loads" a module, but insted have an import/use construct that declaratively states that this module needs this other module to work. It is then up to the module system when and how to load the modules. This makes it easier to analyze dependencies and, imho, yields clearer code.
Yes I completely agree. However, at the lowest level the system must perform some operation which makes the module usable to the program. In other words, an "import" or "use" construct must rely on lower level operations to do its work. I want to brainstorm on what should be provided at the lowest level. Then this lower level could be used to implement a higher level module system on top.
Conceptually, I agree, but a load-once procedure is, depending on what you want it to do, not very difficult to do, or relies heavily on high- level constructs in the system, for instance dependencies, so I don't see how it can be implemented as a reusable low-level component in any useful way.
It should be possible to build an abstraction that seems to work by filename, with some automatic trickery behind the scenes for other linking strategies.
I find filenames to be too brittle of an identifier for modules. It shouldn't matter where a module is located.
I think we are talking about two different issues here: Firstly, how a potential load-once function would keep track of loaded modules internally. For this purpose I don't think filenames are a brittle solution at all. Somehow it must boil down to files and where they are, and these files probably won't move too much in the timeframe of the program execution.
From my point of view this is completely separated from the issue of how any import/use construct would look like for the end user.
On the other hand, I'm rather convinced that filenames are the opposite of a brittle solution, even when it comes to the user interface, but that's another discussion.
- For interactive development there is a need to force the
loading of modules, even if they have been loaded already. How should this be expressed by the user?
In the system I have written this is done simply by re-importing the module from the REPL. The module system can see (with a hash and/or with timestamps) that the module is updated and reloads it. It has simply never been a problem.
You mean because the "last modification" time of the source is newer than what was recorded when the module was loaded? What happens if module A requires B, and B has changed but A hasn't. Will your module system reload B if you ask to load A again?
First of all, this issue rarely arises, because interactive development is fastest accomplished by using the plain REPL and not reloading whole modules.
The current version of my module system would reload only B in the above situation. I explain why I don't think this is wrong below. Another correct behavior that would make sense would be to do nothing - when developing interactively I will always reload the module I have actually edited, no?
You could of course reload both A and B in that situation, that is, reload A's whole dependency graph. I haven't found that very practical in practise however, because that wipes all globals. Yes, it is bad practise to rely on globals but sometimes for hacks it's convenient. Reloading the whole graph renders those techniques pretty useless. (I know Christian Jaeger has written a cool hack for this, but it is just that - a hack) Moreover, as I describe below, reloading A's whole dependency graph doesn't even solve the problem.
Moreover, often the initialisation of a module depends on the initialisation of the module it depends on. In my example, module B might create a unique key for a record type (i.e. generative records) which is then used by A's initialisation to instantiate some records of that type. So in this case it is necessary to reload A even if it hasn't changed. I believe the safest approach is to reload the whole dependence graph of the module to be reloaded.
The problem you mention isn't actually that B will have to be reloaded when A is requested, even if A itself is unchanged. As I see it, the problem in your example is that A must be reloaded, when B is requested. Otherwise there might for instance be a function in A that creates the old version of the B's record type (this can happen if A has a closure from B that generates records).
One solution to this is to reload dependencies upwards in the hierarchy when a module is reloaded, but that doesn't seem very practical, especially when working with utilities that many modules use. But most of all, this approach doesn't make sense in combination with the REPL. What would happen if I re-evaluate the record definition in the REPL? Should A automatically be reloaded? Should that happen for every expression that is evaluated in B's environment?
The approach I have taken here is to simply state that for any module M, no module that depends on it should have to be reloaded when M is reloaded. This is means that you must not implement records in the way you described. (Which for that particular example I think is good; records in memory don't necessarily become incompatible with the new version just because the module is reloaded)
This assumption implies that it's possible to reload any module at any time, without worrying about dependencies in any direction. It is also the only approach that I'm aware of that makes it possible to really leverage the power of the REPL in the context of modules, so I think it's a good trade-off.
Those are interesting and difficult issues. PLT Scheme's module system has addressed those questions rather thoroughly, so it's probably a good idea to check how they are doing it.
/Per
gambit-modules-list@iro.umontreal.ca
-
Alex Queiroz -
Marc Feeley -
Per Eckerdal