|preload-module| builtin dependency handling logics can work out real well.

I'm asking myself if that logic needs to be there though -

I understand that Gambit's builtin modules require a particular order of introduction into the GVM, however that order is hardcoded today isn't it, so can't that just remain so;

Would the additional complexity of |preload-modules| really be worth it, considering that it works well for Gambit's built-in modules to have a hardcoded load order -could that be 15 lines of code- , and as soon as Gambit is running fully, it delivers well anyhow that the user does dependency loading programmatically instead of having that done by additional logics in Gambit and a new special form?

2013/11/6 Marc Feeley <feeley@iro.umontreal.ca>

On Nov 6, 2013, at 1:30 PM, Mikael <mikael.rcv@gmail.com> wrote:

> Problem
> So the conversation topic now is, that now that C linking/loading means a concurrent loading to the C level of all Gambit VM:s at the same time, then how should injection and execution of the loaded Scheme code be done into the VM:s, now that the user wants differentiated behavior between VM:s.
>
> So problems that come with this are that for C code [modules] loaded, on the one hand
> * you need a way to define what code [modules] is actually executed in the primordial VM&processor as code not intended to be executed there on load can be linked in too, and, on the other hand
> * you need a way to specify what code [modules] should be executed in other VM:s.
>
> This is the same problem as Unix and other OS:es face on boot: where to start execution, what's the first process and how to commence operations from there. The difference is just cosmetic in that a Unix system has the modules (the "init" program etc. and library files) in a filesystem while Gambit has them also on the heap already (the prelinked modules that this conversation are about).

Yes, it is a bootstraping problem. A large part of the Gambit runtime is written in Scheme, and the rest is in C. The C part has to be initialized first to provide support for running Scheme. The runtime written in Scheme must be initialized in a methodical way to create the infrastructure on which more complex Scheme features are implemented. For example, the _kernel module must be initialized first because it defines very basic stuff like interrupt handlers and memory allocation procedures (make-vector, make-string, etc) that other modules need. After that that _num module, which implements bignums and other numerical types, is initialized. Sometime later the interpreter is initialized (_eval module), the I/O system is initialized (_io module), the thread system is initialized (_thread module), and finally the REPL (_repl module). These modules build on top of previous modules.

Yep am completely with you.

>
> Indeed, in Unix each module (executable & library) has a dependencies definition, and the OS loader is tasked to loads those deps.
>
> Possible solution (A): Gambit bundles dependency loading logic. |preload-module| + hook
> Gambit can reuse this as |preload-module| (as you suggested) and due to its higher level of abstraction have such dep definitions not just per module but per lambda in a module (as you suggested).
>
> For this to be fully satisfactory, (as you said) the user needs to be able to inject modules both in form of interpreted and compiled code on runtime.

I'm not sure what you mean by "inject modules"...

Ah, by "inject" i just wanted to find a word to complement "execution" - say there's a module containing the code

(define x (+ y z))

(+ y z) is obviously "executed" in a VM, and i thought of injecting as a way to point out the introduction of the globals defined by a module (x here) into a VM. Anyhow same thing.

This injection/execution as a contrast to a module being linked in to a Gambit executable, which means it's only reachable for that but unlike now, not automatically does that in a GVM.

>
> This probably also means there needs to be some runtime hook for a module system to perform the actual dependency loading, at the most basic level meaning resolving what already-linked-in module is actually meant by a particular module name specified to |preload-module|.
>

> Possible solution (B): Gambit does not bundle dependency loading logic. (declare (not execute-on-load)) / (load module #!optional (execute-on-load? #t)) , |create-vm!|, |inject-module!|

The preload-module I was proposing is a *special form*. The extension to "load" you suggest can't work because "load" is a procedure that is only executed at run time. A compile time annotation is required. Perhaps I don't understand what you mean.

Exactly - I'm with you. Indeed load is for loading on runtime only and does not apply to the injection into/execution in the VM of any linked-in modules, be they Gambit-internal or of the user.

For linked-in modules, a form (declare ([not] execute-on-load)) could be provided, to tell if they should automatically be injected/executed into the primordial VM & processor or not.

All modules remain loadable programmatically using (load) or alike for later, for any VM.

I suggested this (declare ([not] execute-on-load)) as an alternative to preload-module only because it would mean an introduction of a lower amount of new complexity/features, that was all.

> This probably also means there needs to be some runtime hook for a module system to perform the actual dependency loading, at the most basic level meaning resolving what already-linked-in module is actually meant by a particular module name specified to |preload-module|.

I'm all in favor of implementing the dependency loading logic in Scheme code, with hooks to change how it is done if needed.

However, it would be problematic to implement an API where one VM can force another VM to load a module of code. A synchronization of the two VMs would be needed, and it is unclear what type of synchronization is needed (is the target VM interrupted? at any time or must we wait for the target VM to be "idle"? what is "idle"?). This is problematic given that VMs are supposed to be independent. It is also problematic in a system like Gambit where there are no blocking operations (at the lowest level). Currently, when an operation would block at the lowest level, the Gambit thread scheduler hands the CPU to a thread that is not blocked. Introducing a low-level synchronization would void the guarantees of liveness of the system.

Agreed.

That's why I prefer a model where the VM is in charge of its own initialization. When one VM creates a new VM, the source VM indicates the name of the main module of the new VM. A new "processor" is created for the new VM and it goes about initializing the modules it requires. The source VM is completely decoupled from the new VM. This is a simpler higher-level API and covers the use cases I can think of.

Super; yes!

So in this picture, starting a new VM would mean that it automatically injects/executes Gambit's built-in modules, and then a main module chosen by the user for that particular VM, that takes charge of all of its initialization.

And question is just what degree of dependency loading logics Gambit puts in place here - only something like a |load| used programmatically by the user as is now but upgraded with the ability to also load specified linked-in modules, or, a dependency loading mechanism.

Another incentive for tracking module dependencies is that it simplifies linking of executable applications. Instead of having to provide the list of all the modules an application needs, it would be possible to only provide the name of the main module. The linker would figure out all the required modules automatically.

Right - so this is a design choice and either way would work.

Gambit without dependency loading/handling logics gives something more resembling GCC & C linker.

Gambit with dependency loading/handling logics would be something slightly more high-level than has been typical for Gambit until now?

For contrast, funny to see how Unix & C does this:

The OS loader indeed loads dependencies non-programmatically;

And the C linker and compiler have limited or no dependency tracking depending on how you define dependency tracking.

I wonder if Gambit's linker having built-in logics for tracking dependencies would be a non-leaky abstraction, in the sense that

there's all kinds of cases with user modules being available in different forms (compiled binary/interpreted sourcecode file, different source and input method e.g. disk, heap or network, and all kinds of sourcecode preprocessing by the user implying for instance that the module names that Gambit routines get will be mangled already) -

all of that would probably require many users to implement their own dependency handling logics, also with regard to the linking, from scratch, so why implement a special dependency handling logic that turns out to be used only for the initialization of Gambit's built-in modules anyhow?

Other modules could be specified, but these would only be loaded by an explicit request at run time (i.e. it would be for bundling multiple modules in a standalone executable).

Sure!

Marc

Best regards,

Mikael