Currently a Gambit executable program is a list of modules, with the "main" program as the last module. During the setup phase (function ___setup) these modules are initialized sequentially (as though they were "load"ed in turn). For example, the Gambit interpreter, gsi, consists of these modules ("_kernel" "_system" "_num" "_std" "_eval" "_io" "_nonstd" "_thread" "_repl" "_gsilib" "_gsi"). The module "_gsi" contains the code which starts the REPL.
This is fine in the context of a single Gambit VM instance per OS process. However, for the upcoming multithreaded Gambit, I want to support multiple VM instances per OS process, so the current linking model is not ideal. If the current linking model isn't changed, each VM instance will have to run the same program (requiring all VMs to initialize all the modules of the program), and the programmer will have to create custom logic in the "main" program to select the appropriate VM behaviour. This is clumsy.
So I'm considering changing the linker data structures to record module dependencies. Something like "before initializing module A, it is necessary to initialize modules B, C, and D". An executable program would thus be a set of modules with dependencies, and the name of the main module. The setup phase would do a depth first traversal of the dependency graph starting at the main module.
This would enable a few interesting features:
1) Allow the instantiation of a VM with a "main" module different from the other VMs, e.g. (start-new-vm "other-main").
2) Lazy initialization of modules (only the modules actually relevant to the main program need to be initialized).
3) Multiple modules could be bundled this way in a single executable program. The "load" procedure could be changed to lookup modules in the bundled set before going out to the file system to dynamically load a .o1 file. This would simplify the installation of multi-module applications and give better startup performance.
Note that for this to work, it will be necessary for the compiler to record module dependencies with each module and to output these dependencies in the linker data structures. A new special form, such as (depends-on <module>), will have to be added so that the programmer can express initialization dependencies in the source code. For consistency, the interpreter will also have to support this form.
Before going forward with this change, I would like to have some feedback. Do you see problems with this or suggestions?
Marc
Afficher les réponses par date
What is the context here:
That all the GVM:s share one and the same heap so objects are inter-accessible between GVM:s, but, while the set of globals in existence is shared between GVM:s too, each GVM has a unique variable slot for each global?
So then, this context is what is calling for this new dependency abstraction?
Then to understand you further, please correct my following attempt at filling out some blanks:
Will each module be required to be initialized once per GVM (so that the module's globals in the GVM will refer properly to the module's procedures etc. on the heap) with the exception of modules containing bootstrap code (such as, launching the REPL), and therefore the dependency information will also contain for each module an instruction about if it is intended to be initialized in one or every GVM?
When you create a new GVM, will its global variables slots be empty or a copy of those of the "parent" GVM?
2013/11/5 Marc Feeley feeley@iro.umontreal.ca
Currently a Gambit executable program is a list of modules, with the "main" program as the last module. During the setup phase (function ___setup) these modules are initialized sequentially (as though they were "load"ed in turn). For example, the Gambit interpreter, gsi, consists of these modules ("_kernel" "_system" "_num" "_std" "_eval" "_io" "_nonstd" "_thread" "_repl" "_gsilib" "_gsi"). The module "_gsi" contains the code which starts the REPL.
This is fine in the context of a single Gambit VM instance per OS process. However, for the upcoming multithreaded Gambit, I want to support multiple VM instances per OS process, so the current linking model is not ideal. If the current linking model isn't changed, each VM instance will have to run the same program (requiring all VMs to initialize all the modules of the program), and the programmer will have to create custom logic in the "main" program to select the appropriate VM behaviour. This is clumsy.
So I'm considering changing the linker data structures to record module dependencies. Something like "before initializing module A, it is necessary to initialize modules B, C, and D". An executable program would thus be a set of modules with dependencies, and the name of the main module. The setup phase would do a depth first traversal of the dependency graph starting at the main module.
This would enable a few interesting features:
- Allow the instantiation of a VM with a "main" module different from the
other VMs, e.g. (start-new-vm "other-main").
- Lazy initialization of modules (only the modules actually relevant to
the main program need to be initialized).
- Multiple modules could be bundled this way in a single executable
program. The "load" procedure could be changed to lookup modules in the bundled set before going out to the file system to dynamically load a .o1 file. This would simplify the installation of multi-module applications and give better startup performance.
Note that for this to work, it will be necessary for the compiler to record module dependencies with each module and to output these dependencies in the linker data structures. A new special form, such as (depends-on <module>), will have to be added so that the programmer can express initialization dependencies in the source code. For consistency, the interpreter will also have to support this form.
Before going forward with this change, I would like to have some feedback. Do you see problems with this or suggestions?
Marc
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Nov 5, 2013, at 3:30 PM, Mikael mikael.rcv@gmail.com wrote:
What is the context here:
That all the GVM:s share one and the same heap so objects are inter-accessible between GVM:s, but, while the set of globals in existence is shared between GVM:s too, each GVM has a unique variable slot for each global?
So then, this context is what is calling for this new dependency abstraction?
Then to understand you further, please correct my following attempt at filling out some blanks:
Will each module be required to be initialized once per GVM (so that the module's globals in the GVM will refer properly to the module's procedures etc. on the heap) with the exception of modules containing bootstrap code (such as, launching the REPL), and therefore the dependency information will also contain for each module an instruction about if it is intended to be initialized in one or every GVM?
When you create a new GVM, will its global variables slots be empty or a copy of those of the "parent" GVM?
Gambit's threading model will be two tiered. There's the concept of Gambit virtual machine (VM) and the concept of "processor". These conceptually correspond to the classical operating system abstractions of process and thread, but these abstractions are not a 1-to-1 mapping to the OS abstractions. In fact, each processor is an OS thread and a VM corresponds to a self contained address space.
So a VM has an independent set of global variables and a heap (where Scheme objects are allocated). A VM cannot access the global variables and the heap of another VM, at least not directly. Within a VM, there are multiple processors (OS threads) running. These processors can share objects and access the same global environment. So a VM is the natural choice for implementing Termite's concept of "process", whereas the shared-memory concurrency provided by processors is ideal for implementing futures.
Please reformulate your questions with this new information.
Marc
2013/11/6 Marc Feeley feeley@iro.umontreal.ca
Gambit's threading model will be two tiered. There's the concept of Gambit virtual machine (VM) and the concept of "processor". These conceptually correspond to the classical operating system abstractions of process and thread, but these abstractions are not a 1-to-1 mapping to the OS abstractions. In fact, each processor is an OS thread and a VM corresponds to a self contained address space.
So a VM has an independent set of global variables and a heap (where Scheme objects are allocated). A VM cannot access the global variables and the heap of another VM, at least not directly. Within a VM, there are multiple processors (OS threads) running. These processors can share objects and access the same global environment. So a VM is the natural choice for implementing Termite's concept of "process", whereas the shared-memory concurrency provided by processors is ideal for implementing futures.
Aha - this abstraction makes enormous sense.
Please reformulate your questions with this new information.
Am all clear now, thank you.
Marc
To your request for feedback regarding whether to change the linker data structures to record module dependencies,
I would propose you instead only implement a fundamental core functionality for the user to declare whether a linked-in module should also be executed on start.
This way, you leave dependency handling logics all to the user to perform himself programmatically, just as it is now - preliminarily I believe this would be for the win in the bigger picture, for instance considering the level of flexibility and control it gives.
This is my preliminary feedback.
Find my reasoning behind this below, and also four followup questions, highlighted.
Best regards, Mikael
-
*Context* So, the levels are
Gambit global level contains one or more Gambit VM:s. A VM is/represents/has a Scheme environment (global variables environment, address space, etc). Each Gambit VM contains one or more Gambit processors. A Gambit processor is running at max one CPU core at a time, and generally Gambit processor creation implies an OS thread creation.
And, each Gambit processor is running one or more green threads right? And, there is some way that the user can assign and reassign executing processor within the VM, for a green thread?
Ok context understood.
*Problem* So the conversation topic now is, that now that C linking/loading means a concurrent loading to the C level of all Gambit VM:s at the same time, then how should injection and execution of the loaded Scheme code be done into the VM:s, now that the user wants differentiated behavior between VM:s.
So problems that come with this are that for C code [modules] loaded, on the one hand * you need a way to define what code [modules] is actually executed in the primordial VM&processor as code not intended to be executed there on load can be linked in too, and, on the other hand * you need a way to specify what code [modules] should be executed in other VM:s.
This is the same problem as Unix and other OS:es face on boot: where to start execution, what's the first process and how to commence operations from there. The difference is just cosmetic in that a Unix system has the modules (the "init" program etc. and library files) in a filesystem while Gambit has them also on the heap already (the prelinked modules that this conversation are about).
Indeed, in Unix each module (executable & library) has a dependencies definition, and the OS loader is tasked to loads those deps.
*Possible solution (A): Gambit bundles dependency loading logic. |preload-module| + hook* Gambit can reuse this as |preload-module| (as you suggested) and due to its higher level of abstraction have such dep definitions not just per module but per lambda in a module (as you suggested).
For this to be fully satisfactory, (as you said) the user needs to be able to inject modules both in form of interpreted and compiled code on runtime.
This probably also means there needs to be some runtime hook for a module system to perform the actual dependency loading, at the most basic level meaning resolving what already-linked-in module is actually meant by a particular module name specified to |preload-module|.
*Possible solution (B): Gambit does not bundle dependency loading logic. (declare (not execute-on-load)) / (load module #!optional (execute-on-load? #t)) , |create-vm!|, |inject-module!|* So, the alternative would be something like vyzo suggested above: That
Gambit not contains recursive dependency loading logic per the suggestion above, but just
1) a way to specify which linked-in modules should be actually executed by the primordial VM&processor on start (all linked-in modules, executed on start or not, can be commanded to be executed by the interface described below) , and then
2) a programmatic interface to handle process creation and module execution, including operations to
* create a Gambit VM including specifying the module it should be started (injected and executed on its start) with - |create-vm!| , and
* a command to inject & execute a given module into the current VM - |inject-module!|
..and like in the suggestion above, modules here can either be interpreted or compiled, and linked in or loaded on runtime.
1. could be achieved with an argument to |load| and for when |load| happens on executable start due to C linking, using a declare form, like, (declare (not execute-on-load)) .
Both (A) and (B) require some way to enumerate modules, be it as a symbol name or using a first-class object representation.
In this moment given my current understanding, I'd vote rather for (B).
(B) is more basic and fundamental and programmatic.
Also, (A) can be implemented in terms of (B) anyhow, I would believe, adequately well?
Please share
* if you see any options beyond these two, and
* if you see any reason for Gambit to do (A) that I may not have understood right now.
On Nov 6, 2013, at 1:30 PM, Mikael mikael.rcv@gmail.com wrote:
2013/11/6 Marc Feeley feeley@iro.umontreal.ca Gambit's threading model will be two tiered. There's the concept of Gambit virtual machine (VM) and the concept of "processor". These conceptually correspond to the classical operating system abstractions of process and thread, but these abstractions are not a 1-to-1 mapping to the OS abstractions. In fact, each processor is an OS thread and a VM corresponds to a self contained address space.
So a VM has an independent set of global variables and a heap (where Scheme objects are allocated). A VM cannot access the global variables and the heap of another VM, at least not directly. Within a VM, there are multiple processors (OS threads) running. These processors can share objects and access the same global environment. So a VM is the natural choice for implementing Termite's concept of "process", whereas the shared-memory concurrency provided by processors is ideal for implementing futures.
Aha - this abstraction makes enormous sense.
Please reformulate your questions with this new information.
Am all clear now, thank you.
Marc
To your request for feedback regarding whether to change the linker data structures to record module dependencies,
I would propose you instead only implement a fundamental core functionality for the user to declare whether a linked-in module should also be executed on start.
This way, you leave dependency handling logics all to the user to perform himself programmatically, just as it is now - preliminarily I believe this would be for the win in the bigger picture, for instance considering the level of flexibility and control it gives.
This is my preliminary feedback.
Find my reasoning behind this below, and also four followup questions, highlighted.
Best regards, Mikael
Context So, the levels are
Gambit global level contains one or more Gambit VM:s. A VM is/represents/has a Scheme environment (global variables environment, address space, etc). Each Gambit VM contains one or more Gambit processors. A Gambit processor is running at max one CPU core at a time, and generally Gambit processor creation implies an OS thread creation.
And, each Gambit processor is running one or more green threads right? And, there is some way that the user can assign and reassign executing processor within the VM, for a green thread?
Ok context understood.
Problem So the conversation topic now is, that now that C linking/loading means a concurrent loading to the C level of all Gambit VM:s at the same time, then how should injection and execution of the loaded Scheme code be done into the VM:s, now that the user wants differentiated behavior between VM:s.
So problems that come with this are that for C code [modules] loaded, on the one hand
- you need a way to define what code [modules] is actually executed in the primordial VM&processor as code not intended to be executed there on load can be linked in too, and, on the other hand
- you need a way to specify what code [modules] should be executed in other VM:s.
This is the same problem as Unix and other OS:es face on boot: where to start execution, what's the first process and how to commence operations from there. The difference is just cosmetic in that a Unix system has the modules (the "init" program etc. and library files) in a filesystem while Gambit has them also on the heap already (the prelinked modules that this conversation are about).
Yes, it is a bootstraping problem. A large part of the Gambit runtime is written in Scheme, and the rest is in C. The C part has to be initialized first to provide support for running Scheme. The runtime written in Scheme must be initialized in a methodical way to create the infrastructure on which more complex Scheme features are implemented. For example, the _kernel module must be initialized first because it defines very basic stuff like interrupt handlers and memory allocation procedures (make-vector, make-string, etc) that other modules need. After that that _num module, which implements bignums and other numerical types, is initialized. Sometime later the interpreter is initialized (_eval module), the I/O system is initialized (_io module), the thread system is initialized (_thread module), and finally the REPL (_repl module). These modules build on top of previous modules.
Indeed, in Unix each module (executable & library) has a dependencies definition, and the OS loader is tasked to loads those deps.
Possible solution (A): Gambit bundles dependency loading logic. |preload-module| + hook Gambit can reuse this as |preload-module| (as you suggested) and due to its higher level of abstraction have such dep definitions not just per module but per lambda in a module (as you suggested).
For this to be fully satisfactory, (as you said) the user needs to be able to inject modules both in form of interpreted and compiled code on runtime.
I'm not sure what you mean by "inject modules"...
This probably also means there needs to be some runtime hook for a module system to perform the actual dependency loading, at the most basic level meaning resolving what already-linked-in module is actually meant by a particular module name specified to |preload-module|.
Possible solution (B): Gambit does not bundle dependency loading logic. (declare (not execute-on-load)) / (load module #!optional (execute-on-load? #t)) , |create-vm!|, |inject-module!|
The preload-module I was proposing is a *special form*. The extension to "load" you suggest can't work because "load" is a procedure that is only executed at run time. A compile time annotation is required. Perhaps I don't understand what you mean.
So, the alternative would be something like vyzo suggested above: That
Gambit not contains recursive dependency loading logic per the suggestion above, but just
a way to specify which linked-in modules should be actually executed by the primordial VM&processor on start (all linked-in modules, executed on start or not, can be commanded to be executed by the interface described below) , and then
a programmatic interface to handle process creation and module execution, including operations to
create a Gambit VM including specifying the module it should be started (injected and executed on its start) with - |create-vm!| , and
a command to inject & execute a given module into the current VM - |inject-module!|
..and like in the suggestion above, modules here can either be interpreted or compiled, and linked in or loaded on runtime.
- could be achieved with an argument to |load| and for when |load| happens on executable start due to C linking, using a declare form, like, (declare (not execute-on-load)) .
Both (A) and (B) require some way to enumerate modules, be it as a symbol name or using a first-class object representation.
In this moment given my current understanding, I'd vote rather for (B).
(B) is more basic and fundamental and programmatic.
Also, (A) can be implemented in terms of (B) anyhow, I would believe, adequately well?
Please share
if you see any options beyond these two, and
if you see any reason for Gambit to do (A) that I may not have understood right now.
I'm all in favor of implementing the dependency loading logic in Scheme code, with hooks to change how it is done if needed.
However, it would be problematic to implement an API where one VM can force another VM to load a module of code. A synchronization of the two VMs would be needed, and it is unclear what type of synchronization is needed (is the target VM interrupted? at any time or must we wait for the target VM to be "idle"? what is "idle"?). This is problematic given that VMs are supposed to be independent. It is also problematic in a system like Gambit where there are no blocking operations (at the lowest level). Currently, when an operation would block at the lowest level, the Gambit thread scheduler hands the CPU to a thread that is not blocked. Introducing a low-level synchronization would void the guarantees of liveness of the system.
That's why I prefer a model where the VM is in charge of its own initialization. When one VM creates a new VM, the source VM indicates the name of the main module of the new VM. A new "processor" is created for the new VM and it goes about initializing the modules it requires. The source VM is completely decoupled from the new VM. This is a simpler higher-level API and covers the use cases I can think of.
Another incentive for tracking module dependencies is that it simplifies linking of executable applications. Instead of having to provide the list of all the modules an application needs, it would be possible to only provide the name of the main module. The linker would figure out all the required modules automatically. Other modules could be specified, but these would only be loaded by an explicit request at run time (i.e. it would be for bundling multiple modules in a standalone executable).
Marc
|preload-module| builtin dependency handling logics can work out real well.
I'm asking myself if that logic needs to be there though -
I understand that Gambit's builtin modules require a particular order of introduction into the GVM, however that order is hardcoded today isn't it, so can't that just remain so;
Would the additional complexity of |preload-modules| really be worth it, considering that it works well for Gambit's built-in modules to have a hardcoded load order -could that be 15 lines of code- , and as soon as Gambit is running fully, it delivers well anyhow that the user does dependency loading programmatically instead of having that done by additional logics in Gambit and a new special form?
2013/11/6 Marc Feeley feeley@iro.umontreal.ca
On Nov 6, 2013, at 1:30 PM, Mikael mikael.rcv@gmail.com wrote:
Problem So the conversation topic now is, that now that C linking/loading means
a concurrent loading to the C level of all Gambit VM:s at the same time, then how should injection and execution of the loaded Scheme code be done into the VM:s, now that the user wants differentiated behavior between VM:s.
So problems that come with this are that for C code [modules] loaded, on
the one hand
- you need a way to define what code [modules] is actually executed in
the primordial VM&processor as code not intended to be executed there on load can be linked in too, and, on the other hand
- you need a way to specify what code [modules] should be executed in
other VM:s.
This is the same problem as Unix and other OS:es face on boot: where to
start execution, what's the first process and how to commence operations from there. The difference is just cosmetic in that a Unix system has the modules (the "init" program etc. and library files) in a filesystem while Gambit has them also on the heap already (the prelinked modules that this conversation are about).
Yes, it is a bootstraping problem. A large part of the Gambit runtime is written in Scheme, and the rest is in C. The C part has to be initialized first to provide support for running Scheme. The runtime written in Scheme must be initialized in a methodical way to create the infrastructure on which more complex Scheme features are implemented. For example, the _kernel module must be initialized first because it defines very basic stuff like interrupt handlers and memory allocation procedures (make-vector, make-string, etc) that other modules need. After that that _num module, which implements bignums and other numerical types, is initialized. Sometime later the interpreter is initialized (_eval module), the I/O system is initialized (_io module), the thread system is initialized (_thread module), and finally the REPL (_repl module). These modules build on top of previous modules.
Yep am completely with you.
Indeed, in Unix each module (executable & library) has a dependencies
definition, and the OS loader is tasked to loads those deps.
Possible solution (A): Gambit bundles dependency loading logic.
|preload-module| + hook
Gambit can reuse this as |preload-module| (as you suggested) and due to
its higher level of abstraction have such dep definitions not just per module but per lambda in a module (as you suggested).
For this to be fully satisfactory, (as you said) the user needs to be
able to inject modules both in form of interpreted and compiled code on runtime.
I'm not sure what you mean by "inject modules"...
Ah, by "inject" i just wanted to find a word to complement "execution" - say there's a module containing the code
(define x (+ y z))
(+ y z) is obviously "executed" in a VM, and i thought of injecting as a way to point out the introduction of the globals defined by a module (x here) into a VM. Anyhow same thing.
This injection/execution as a contrast to a module being linked in to a Gambit executable, which means it's only reachable for that but unlike now, not automatically does that in a GVM.
This probably also means there needs to be some runtime hook for a
module system to perform the actual dependency loading, at the most basic level meaning resolving what already-linked-in module is actually meant by a particular module name specified to |preload-module|.
Possible solution (B): Gambit does not bundle dependency loading logic.
(declare (not execute-on-load)) / (load module #!optional (execute-on-load? #t)) , |create-vm!|, |inject-module!|
The preload-module I was proposing is a *special form*. The extension to "load" you suggest can't work because "load" is a procedure that is only executed at run time. A compile time annotation is required. Perhaps I don't understand what you mean.
Exactly - I'm with you. Indeed load is for loading on runtime only and does not apply to the injection into/execution in the VM of any linked-in modules, be they Gambit-internal or of the user.
For linked-in modules, a form (declare ([not] execute-on-load)) could be provided, to tell if they should automatically be injected/executed into the primordial VM & processor or not.
All modules remain loadable programmatically using (load) or alike for later, for any VM.
I suggested this (declare ([not] execute-on-load)) as an alternative to preload-module only because it would mean an introduction of a lower amount of new complexity/features, that was all.
This probably also means there needs to be some runtime hook for a
module system to perform the actual dependency loading, at the most basic level meaning resolving what already-linked-in module is actually meant by a particular module name specified to |preload-module|.
I'm all in favor of implementing the dependency loading logic in Scheme code, with hooks to change how it is done if needed.
Ok
However, it would be problematic to implement an API where one VM can force another VM to load a module of code. A synchronization of the two VMs would be needed, and it is unclear what type of synchronization is needed (is the target VM interrupted? at any time or must we wait for the target VM to be "idle"? what is "idle"?). This is problematic given that VMs are supposed to be independent. It is also problematic in a system like Gambit where there are no blocking operations (at the lowest level). Currently, when an operation would block at the lowest level, the Gambit thread scheduler hands the CPU to a thread that is not blocked. Introducing a low-level synchronization would void the guarantees of liveness of the system.
Agreed.
That's why I prefer a model where the VM is in charge of its own initialization. When one VM creates a new VM, the source VM indicates the name of the main module of the new VM. A new "processor" is created for the new VM and it goes about initializing the modules it requires. The source VM is completely decoupled from the new VM. This is a simpler higher-level API and covers the use cases I can think of.
Super; yes!
So in this picture, starting a new VM would mean that it automatically injects/executes Gambit's built-in modules, and then a main module chosen by the user for that particular VM, that takes charge of all of its initialization.
And question is just what degree of dependency loading logics Gambit puts in place here - only something like a |load| used programmatically by the user as is now but upgraded with the ability to also load specified linked-in modules, or, a dependency loading mechanism.
Another incentive for tracking module dependencies is that it simplifies
linking of executable applications. Instead of having to provide the list of all the modules an application needs, it would be possible to only provide the name of the main module. The linker would figure out all the required modules automatically.
Right - so this is a design choice and either way would work.
Gambit without dependency loading/handling logics gives something more resembling GCC & C linker.
Gambit with dependency loading/handling logics would be something slightly more high-level than has been typical for Gambit until now?
For contrast, funny to see how Unix & C does this:
The OS loader indeed loads dependencies non-programmatically;
And the C linker and compiler have limited or no dependency tracking depending on how you define dependency tracking.
I wonder if Gambit's linker having built-in logics for tracking dependencies would be a non-leaky abstraction, in the sense that
there's all kinds of cases with user modules being available in different forms (compiled binary/interpreted sourcecode file, different source and input method e.g. disk, heap or network, and all kinds of sourcecode preprocessing by the user implying for instance that the module names that Gambit routines get will be mangled already) -
all of that would probably require many users to implement their own dependency handling logics, also with regard to the linking, from scratch, so why implement a special dependency handling logic that turns out to be used only for the initialization of Gambit's built-in modules anyhow?
Other modules could be specified, but these would only be loaded by an
explicit request at run time (i.e. it would be for bundling multiple modules in a standalone executable).
Sure!
Marc
Best regards, Mikael
Hm, interesting:
Independent of whether Gambit will not have or will have a dep loader built-in, it will be greatly practical that on creation of a new VM, the user can specify code that will be taken in use as main module for interpreted execution by the newly created VM i.e. something like
(create-vm! '(begin (load/preload-module/alike "otherdep.o1") (hello-world) etc.))
or in the future, perhaps a native backend-compiled binary image instead
On Nov 7, 2013, at 11:04 AM, Mikael mikael.rcv@gmail.com wrote:
Hm, interesting:
Independent of whether Gambit will not have or will have a dep loader built-in, it will be greatly practical that on creation of a new VM, the user can specify code that will be taken in use as main module for interpreted execution by the newly created VM i.e. something like
(create-vm! '(begin (load/preload-module/alike "otherdep.o1") (hello-world) etc.))
Communicating some information to a new VM would be nice. However, there are technical problems... the source VM and the target VM don't share the same address space, so it isn't possible to just pass a reference to the new VM (except for symbols, which I intend to make unique over all VMs). So some copying would be involved. But what to do about cyclical data, or data with sharing, or with unserializable data (continuations? foreign pointers?). Passing a string would be easy enough, but it is very limitative and doesn't feel schemey.
Marc
On 11/07/2013 12:57 PM, Marc Feeley wrote:
On Nov 7, 2013, at 11:04 AM, Mikael mikael.rcv@gmail.com wrote:
Hm, interesting:
Independent of whether Gambit will not have or will have a dep loader built-in, it will be greatly practical that on creation of a new VM, the user can specify code that will be taken in use as main module for interpreted execution by the newly created VM i.e. something like
(create-vm! '(begin (load/preload-module/alike "otherdep.o1") (hello-world) etc.))
Communicating some information to a new VM would be nice. However, there are technical problems... the source VM and the target VM don't share the same address space, so it isn't possible to just pass a reference to the new VM (except for symbols, which I intend to make unique over all VMs). So some copying would be involved. But what to do about cyclical data, or data with sharing, or with unserializable data (continuations? foreign pointers?). Passing a string would be easy enough, but it is very limitative and doesn't feel schemey.
What about serializing things? This is what my students do when hooking up Gambit and MPI.
Brad
On Nov 7, 2013, at 1:00 PM, Bradley Lucier lucier@math.purdue.edu wrote:
On 11/07/2013 12:57 PM, Marc Feeley wrote:
On Nov 7, 2013, at 11:04 AM, Mikael mikael.rcv@gmail.com wrote:
Hm, interesting:
Independent of whether Gambit will not have or will have a dep loader built-in, it will be greatly practical that on creation of a new VM, the user can specify code that will be taken in use as main module for interpreted execution by the newly created VM i.e. something like
(create-vm! '(begin (load/preload-module/alike "otherdep.o1") (hello-world) etc.))
Communicating some information to a new VM would be nice. However, there are technical problems... the source VM and the target VM don't share the same address space, so it isn't possible to just pass a reference to the new VM (except for symbols, which I intend to make unique over all VMs). So some copying would be involved. But what to do about cyclical data, or data with sharing, or with unserializable data (continuations? foreign pointers?). Passing a string would be easy enough, but it is very limitative and doesn't feel schemey.
What about serializing things? This is what my students do when hooking up Gambit and MPI.
Brad
Some things can't be serialized, for example foreign pointers. So in general, a port can't be serialized. Closures and continuations are also problematic, because they contain a pointer to code and the module which this code belongs to in the source VM may not be loaded in the new VM.
Marc
On Nov 7, 2013, at 10:32 AM, Mikael mikael.rcv@gmail.com wrote:
|preload-module| builtin dependency handling logics can work out real well.
I'm asking myself if that logic needs to be there though -
I understand that Gambit's builtin modules require a particular order of introduction into the GVM, however that order is hardcoded today isn't it, so can't that just remain so;
I'm not sure what you mean by "hardcoded order". When the Gambit linker is called to create the link file for the Gambit runtime library, the list of modules of the runtime library is passed to the linker (just like when linking an application) and the linker creates an array of module descriptors that will be iterated over to initialize each runtime library module.
Would the additional complexity of |preload-modules| really be worth it, considering that it works well for Gambit's built-in modules to have a hardcoded load order -could that be 15 lines of code- , and as soon as Gambit is running fully, it delivers well anyhow that the user does dependency loading programmatically instead of having that done by additional logics in Gambit and a new special form?
When the Gambit interpreter is built, the main module of the interpreter, "_gsi", is added to the array of module descriptors by the linker. This module is in charge of handling command-line options and starting a REPL. So this module is initialized in addition to the modules of the runtime library, and it is this initialization that starts the interpreter's REPL. Gambit's initialization code doesn't distinguish between the modules that are part of the runtime system and the modules of the application. Indeed, it is even possible to create a library that extends the base runtime library, and use this extended runtime library to link applications. The only module with a special status is "_kernel", because it contains essential procedures (such as the code that builds rest parameter lists). I can imagine scenarios where some programs may not need the modules "_eval" and "_repl" (implementing the interpreter and REPL), so it is wrong to make them "builtin" for all programs.
So, in a Gambit supporting multiple VMs, what should happen when the main VM starts a new VM? It would be wrong to automatically start a new REPL just because the main VM also started a REPL. To achieve this, we'd have to add to the end of "_gsi" some program logic like this:
(if (main-vm? (current-vm)) (start-REPL))
This clumsiness is due to the fact that the VM creation operation isn't explicit about what modules that VM should load. It is also a performance problem if the main VM has many modules, most of which are irrelevant to the new VM's behavior.
Marc
Is it really necessary?
Why not handle module loading in the primordial thread, which would be the first boot thread for global data structures and then the boot thread per VM for concurrent initialization.
The programmer can then load modules explicitly per VM. All that is needed is an API to select the target VM.
-- vyzo
PS: a performance related suggestion: use __thread for TLS with gcc/glibc if you are planning a 1-1 VM to OS thread mapping (instead of pthread tls). On Nov 5, 2013 9:11 AM, "Marc Feeley" feeley@iro.umontreal.ca wrote:
Currently a Gambit executable program is a list of modules, with the "main" program as the last module. During the setup phase (function ___setup) these modules are initialized sequentially (as though they were "load"ed in turn). For example, the Gambit interpreter, gsi, consists of these modules ("_kernel" "_system" "_num" "_std" "_eval" "_io" "_nonstd" "_thread" "_repl" "_gsilib" "_gsi"). The module "_gsi" contains the code which starts the REPL.
This is fine in the context of a single Gambit VM instance per OS process. However, for the upcoming multithreaded Gambit, I want to support multiple VM instances per OS process, so the current linking model is not ideal. If the current linking model isn't changed, each VM instance will have to run the same program (requiring all VMs to initialize all the modules of the program), and the programmer will have to create custom logic in the "main" program to select the appropriate VM behaviour. This is clumsy.
So I'm considering changing the linker data structures to record module dependencies. Something like "before initializing module A, it is necessary to initialize modules B, C, and D". An executable program would thus be a set of modules with dependencies, and the name of the main module. The setup phase would do a depth first traversal of the dependency graph starting at the main module.
This would enable a few interesting features:
- Allow the instantiation of a VM with a "main" module different from the
other VMs, e.g. (start-new-vm "other-main").
- Lazy initialization of modules (only the modules actually relevant to
the main program need to be initialized).
- Multiple modules could be bundled this way in a single executable
program. The "load" procedure could be changed to lookup modules in the bundled set before going out to the file system to dynamically load a .o1 file. This would simplify the installation of multi-module applications and give better startup performance.
Note that for this to work, it will be necessary for the compiler to record module dependencies with each module and to output these dependencies in the linker data structures. A new special form, such as (depends-on <module>), will have to be added so that the programmer can express initialization dependencies in the source code. For consistency, the interpreter will also have to support this form.
Before going forward with this change, I would like to have some feedback. Do you see problems with this or suggestions?
Marc
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Nov 5, 2013, at 11:58 PM, Dimitris Vyzovitis vyzo@hackzen.org wrote:
Is it really necessary?
As I said, the current linking model could be kept, but this would require a convoluted main program logic. Each VM would basically boot by "loading" the same set of modules, and the main program logic would have to do a dispatch on the VM identifier (small integer?) to select the appropriate behavior for that VM.
Why not handle module loading in the primordial thread, which would be the first boot thread for global data structures and then the boot thread per VM for concurrent initialization.
The programmer can then load modules explicitly per VM. All that is needed is an API to select the target VM.
I'm not against a Scheme level API to initialize the target VM. What I proposed in my message is to add module dependencies so that initialization can be driven by the specific needs of a VM's main module.
-- vyzo
PS: a performance related suggestion: use __thread for TLS with gcc/glibc if you are planning a 1-1 VM to OS thread mapping (instead of pthread tls).
Yes, that's already in place for accessing the context of a VM (when the C compiler supports it, such as gcc).
Marc
It would be like a module system without the namespaces. I think that this is a very nice step, because it's an important part of the functionality of Blackhole or Schemespheres, but with the new features that you can implement at a lower level. I'd actually like that this goes one step further an build a minimal module system (even if just select which variables to export per module). Such a simple module system (export / depends-on) would be very helpful for developing bigger applications with Gambit.
Best regards
On Tue, Nov 5, 2013 at 6:10 PM, Marc Feeley feeley@iro.umontreal.ca wrote:
Currently a Gambit executable program is a list of modules, with the "main" program as the last module. During the setup phase (function ___setup) these modules are initialized sequentially (as though they were "load"ed in turn). For example, the Gambit interpreter, gsi, consists of these modules ("_kernel" "_system" "_num" "_std" "_eval" "_io" "_nonstd" "_thread" "_repl" "_gsilib" "_gsi"). The module "_gsi" contains the code which starts the REPL.
This is fine in the context of a single Gambit VM instance per OS process. However, for the upcoming multithreaded Gambit, I want to support multiple VM instances per OS process, so the current linking model is not ideal. If the current linking model isn't changed, each VM instance will have to run the same program (requiring all VMs to initialize all the modules of the program), and the programmer will have to create custom logic in the "main" program to select the appropriate VM behaviour. This is clumsy.
So I'm considering changing the linker data structures to record module dependencies. Something like "before initializing module A, it is necessary to initialize modules B, C, and D". An executable program would thus be a set of modules with dependencies, and the name of the main module. The setup phase would do a depth first traversal of the dependency graph starting at the main module.
This would enable a few interesting features:
- Allow the instantiation of a VM with a "main" module different from the
other VMs, e.g. (start-new-vm "other-main").
- Lazy initialization of modules (only the modules actually relevant to
the main program need to be initialized).
- Multiple modules could be bundled this way in a single executable
program. The "load" procedure could be changed to lookup modules in the bundled set before going out to the file system to dynamically load a .o1 file. This would simplify the installation of multi-module applications and give better startup performance.
Note that for this to work, it will be necessary for the compiler to record module dependencies with each module and to output these dependencies in the linker data structures. A new special form, such as (depends-on <module>), will have to be added so that the programmer can express initialization dependencies in the source code. For consistency, the interpreter will also have to support this form.
Before going forward with this change, I would like to have some feedback. Do you see problems with this or suggestions?
Marc
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Nov 6, 2013, at 3:03 AM, Álvaro Castro-Castilla alvaro.castro.castilla@gmail.com wrote:
It would be like a module system without the namespaces.
Yes, that's a way to put it.
Note that my proposal only aims to provide a very basic mechanism for loading in a VM modules of code with their dependencies. The "depends-on" form could of course be used for building a (slightly) higher-level module system to ensuring that module dependencies are satisfied. By the way, I think this form is probably best called preload-module, so that a program could be written like this:
(define (f n) (preload-module math) (+ (math:cbrt n) (math:fact n)))
(define (g x) (preload-module postscript) (preload-module math) (postscript:circle (math:square x)))
The code would be roughly equivalent to:
(if (not (loaded? 'math)) (load 'math)) (if (not (loaded? 'postscript)) (load 'postscript))
(define (f n) (+ (math:cbrt n) (math:fact n)))
(define (g x) (postscript:circle (math:square x)))
In other words, the requests to preload modules are just annotations that are accumulated for the whole module. The set of requests causes these modules to be loaded at the very beginning of the module's execution.
I think that this is a very nice step, because it's an important part of the functionality of Blackhole or Schemespheres, but with the new features that you can implement at a lower level. I'd actually like that this goes one step further an build a minimal module system (even if just select which variables to export per module). Such a simple module system (export / depends-on) would be very helpful for developing bigger applications with Gambit.
I agree that a lightweight module system would be nice. This should be fairly simple to do once the linker is modified to support preload-module.
Marc
Marc,
I assume there is 1:1 correspondence between a prospective (preload-module foo) and a "x/y/z/foo.scm".
Do you have a plan for how you intend to map the preload-module identifier symbols to their corresponding pathed file name strings?
I foresee that *preload-module* can provide a syntax that is isomorphic to the chez scheme *visit,* and it can thus provide better syntax-case integration. With preload-module, sources may be able to specify a dependency on compiled syntax across gambit modules, using a syntax that is native to gambit.
It is also great to see module loading order removed from the ordering of objects on the gsc command-line, which if I'm not mistaken is how it is done today.
Does the existing gsc command-line functionality continue to provide an implicit module load order in cases where the new syntax is not present?
Is there a reasonable behavior in the absence of *preload-module *syntax, or use cases where *preload-module* is partially provided by a subset of sources*?*
Do you anticipate modules with symmetric or symmetric transitive dependence? In such a use case, I'd presume load order is irrelevant providing that linkage is complete, but the issue of ordering top-level initialization may remain. This could be problematic for a proposed syntax case usage, as it is easy to specify symmetrically dependent compiled syntax between two files: if cyclic *preload-module* is not permitted, then syntax-case presumably needs a separate syntax to specify syntax dependency between files, if symmetrically imported syntax is a warranted feature.
On Wed, Nov 6, 2013 at 7:45 AM, Marc Feeley feeley@iro.umontreal.ca wrote:
On Nov 6, 2013, at 3:03 AM, Álvaro Castro-Castilla < alvaro.castro.castilla@gmail.com> wrote:
It would be like a module system without the namespaces.
Yes, that's a way to put it.
Note that my proposal only aims to provide a very basic mechanism for loading in a VM modules of code with their dependencies. The "depends-on" form could of course be used for building a (slightly) higher-level module system to ensuring that module dependencies are satisfied. By the way, I think this form is probably best called preload-module, so that a program could be written like this:
(define (f n) (preload-module math) (+ (math:cbrt n) (math:fact n)))
(define (g x) (preload-module postscript) (preload-module math) (postscript:circle (math:square x)))
The code would be roughly equivalent to:
(if (not (loaded? 'math)) (load 'math)) (if (not (loaded? 'postscript)) (load 'postscript))
(define (f n) (+ (math:cbrt n) (math:fact n)))
(define (g x) (postscript:circle (math:square x)))
In other words, the requests to preload modules are just annotations that are accumulated for the whole module. The set of requests causes these modules to be loaded at the very beginning of the module's execution.
I think that this is a very nice step, because it's an important part of
the functionality of Blackhole or Schemespheres, but with the new features that you can implement at a lower level.
I'd actually like that this goes one step further an build a minimal
module system (even if just select which variables to export per module). Such a simple module system (export / depends-on) would be very helpful for developing bigger applications with Gambit.
I agree that a lightweight module system would be nice. This should be fairly simple to do once the linker is modified to support preload-module.
Marc
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
2013/11/8 Matthew Hastie matthastie@gmail.com
Marc,
I assume there is 1:1 correspondence between a prospective (preload-module foo) and a "x/y/z/foo.scm".
Do you have a plan for how you intend to map the preload-module identifier symbols to their corresponding pathed file name strings?
That's what the hook is for - resolving a module identifier to an actual module, be it linked-in, in a file, module evaled from s-expression, etc.
I foresee that *preload-module* can provide a syntax that is isomorphic to
the chez scheme *visit,* and it can thus provide better syntax-case integration. With preload-module, sources may be able to specify a dependency on compiled syntax across gambit modules, using a syntax that is native to gambit.
It is also great to see module loading order removed from the ordering of
objects on the gsc command-line, which if I'm not mistaken is how it is done today.
Does the existing gsc command-line functionality continue to provide an implicit module load order in cases where the new syntax is not present?
Is there a reasonable behavior in the absence of *preload-module *syntax, or use cases where *preload-module* is partially provided by a subset of sources*?*
Do you anticipate modules with symmetric or symmetric transitive dependence? In such a use case, I'd presume load order is irrelevant providing that linkage is complete, but the issue of ordering top-level initialization may remain. This could be problematic for a proposed syntax case usage, as it is easy to specify symmetrically dependent compiled syntax between two files: if cyclic *preload-module* is not permitted, then syntax-case presumably needs a separate syntax to specify syntax dependency between files, if symmetrically imported syntax is a warranted feature.
I guess this kind of questions will make custom module/macro/etc. system want to implement dependency handling itself, more or less from scratch?
-
Bradley Lucier -
Dimitris Vyzovitis -
Marc Feeley -
Matthew Hastie -
Mikael -
Álvaro Castro-Castilla