[gambit-list] Gambit linking

Marc Feeley feeley at iro.umontreal.ca
Wed Nov 6 15:30:33 EST 2013


On Nov 6, 2013, at 1:30 PM, Mikael <mikael.rcv at gmail.com> wrote:

> 
> 
> 
> 2013/11/6 Marc Feeley <feeley at iro.umontreal.ca>
> Gambit's threading model will be two tiered.  There's the concept of Gambit virtual machine (VM) and the concept of "processor".  These conceptually correspond to the classical operating system abstractions of process and thread, but these abstractions are not a 1-to-1 mapping to the OS abstractions.  In fact, each processor is an OS thread and a VM corresponds to a self contained address space.
> 
> So a VM has an independent set of global variables and a heap (where Scheme objects are allocated).  A VM cannot access the global variables and the heap of another VM, at least not directly.  Within a VM, there are multiple processors (OS threads) running.  These processors can share objects and access the same global environment.  So a VM is the natural choice for implementing Termite's concept of "process", whereas the shared-memory concurrency provided by processors is ideal for implementing futures.
> 
> Aha - this abstraction makes enormous sense.
>  
> Please reformulate your questions with this new information.
> 
> Am all clear now, thank you.
>  
> Marc
> 
> 
> To your request for feedback regarding whether to change the linker data structures to record module dependencies,
> 
> I would propose you instead only implement a fundamental core functionality for the user to declare whether a linked-in module should also be executed on start.
> 
> This way, you leave dependency handling logics all to the user to perform himself programmatically, just as it is now - preliminarily I believe this would be for the win in the bigger picture, for instance considering the level of flexibility and control it gives.
> 
> 
> This is my preliminary feedback.
> 
> 
> Find my reasoning behind this below, and also four followup questions, highlighted.
> 
> 
> Best regards,
> Mikael
> 
>  - 
> 
> 
> 
> Context
> So, the levels are
> 
> Gambit global level contains one or more Gambit VM:s.
> A VM is/represents/has a Scheme environment (global variables environment, address space, etc).
> Each Gambit VM contains one or more Gambit processors.
> A Gambit processor is running at max one CPU core at a time, and generally Gambit processor creation implies an OS thread creation.
> 
> And, each Gambit processor is running one or more green threads right?
> And, there is some way that the user can assign and reassign executing processor within the VM, for a green thread?
> 
> 
> Ok context understood.
> 
> 
> Problem
> So the conversation topic now is, that now that C linking/loading means a concurrent loading to the C level of all Gambit VM:s at the same time, then how should injection and execution of the loaded Scheme code be done into the VM:s, now that the user wants differentiated behavior between VM:s.
> 
> So problems that come with this are that for C code [modules] loaded, on the one hand
>  * you need a way to define what code [modules] is actually executed in the primordial VM&processor as code not intended to be executed there on load can be linked in too, and, on the other hand
>  * you need a way to specify what code [modules] should be executed in other VM:s.
> 
> This is the same problem as Unix and other OS:es face on boot: where to start execution, what's the first process and how to commence operations from there. The difference is just cosmetic in that a Unix system has the modules (the "init" program etc. and library files) in a filesystem while Gambit has them also on the heap already (the prelinked modules that this conversation are about).

Yes, it is a bootstraping problem.  A large part of the Gambit runtime is written in Scheme, and the rest is in C.  The C part has to be initialized first to provide support for running Scheme.  The runtime written in Scheme must be initialized in a methodical way to create the infrastructure on which more complex Scheme features are implemented.  For example, the _kernel module must be initialized first because it defines very basic stuff like interrupt handlers and memory allocation procedures (make-vector, make-string, etc) that other modules need.  After that that _num module, which implements bignums and other numerical types, is initialized.  Sometime later the interpreter is initialized (_eval module), the I/O system is initialized (_io module), the thread system is initialized (_thread module), and finally the REPL (_repl module).  These modules build on top of previous modules.

> 
> Indeed, in Unix each module (executable & library) has a dependencies definition, and the OS loader is tasked to loads those deps.
> 
> Possible solution (A): Gambit bundles dependency loading logic. |preload-module| + hook
> Gambit can reuse this as |preload-module| (as you suggested) and due to its higher level of abstraction have such dep definitions not just per module but per lambda in a module (as you suggested).
> 
> For this to be fully satisfactory, (as you said) the user needs to be able to inject modules both in form of interpreted and compiled code on runtime.

I'm not sure what you mean by "inject modules"...

> 
> This probably also means there needs to be some runtime hook for a module system to perform the actual dependency loading, at the most basic level meaning resolving what already-linked-in module is actually meant by a particular module name specified to |preload-module|.
> 

> Possible solution (B): Gambit does not bundle dependency loading logic. (declare (not execute-on-load)) / (load module #!optional (execute-on-load? #t)) , |create-vm!|, |inject-module!|

The preload-module I was proposing is a *special form*.  The extension to "load" you suggest can't work because "load" is a procedure that is only executed at run time.  A compile time annotation is required.  Perhaps I don't understand what you mean.

> So, the alternative would be something like vyzo suggested above: That
> 
> Gambit not contains recursive dependency loading logic per the suggestion above, but just
> 
>  1) a way to specify which linked-in modules should be actually executed by the primordial VM&processor on start
>      (all linked-in modules, executed on start or not, can be commanded to be executed by the interface described below)
>       , and then
> 
>  2) a programmatic interface to handle process creation and module execution, including operations to
> 
>       * create a Gambit VM including specifying the module it should be started (injected and executed on its start) with - |create-vm!| , and
> 
>       * a command to inject & execute a given module into the current VM - |inject-module!|
> 
> ..and like in the suggestion above, modules here can either be interpreted or compiled, and linked in or loaded on runtime.
> 
> 1. could be achieved with an argument to |load| and for when |load| happens on executable start due to C linking, using a declare form, like, (declare (not execute-on-load)) .
> 
> 
> 
> Both (A) and (B) require some way to enumerate modules, be it as a symbol name or using a first-class object representation.
> 
> 
> 
> 
> In this moment given my current understanding, I'd vote rather for (B).
> 
> (B) is more basic and fundamental and programmatic.
> 
> Also, (A) can be implemented in terms of (B) anyhow, I would believe, adequately well?
> 
> 
> Please share
> 
>  * if you see any options beyond these two, and
> 
>  * if you see any reason for Gambit to do (A) that I may not have understood right now.
> 

I'm all in favor of implementing the dependency loading logic in Scheme code, with hooks to change how it is done if needed.

However, it would be problematic to implement an API where one VM can force another VM to load a module of code.  A synchronization of the two VMs would be needed, and it is unclear what type of synchronization is needed (is the target VM interrupted?  at any time or must we wait for the target VM to be "idle"?  what is "idle"?).  This is problematic given that VMs are supposed to be independent.  It is also problematic in a system like Gambit where there are no blocking operations (at the lowest level).  Currently, when an operation would block at the lowest level, the Gambit thread scheduler hands the CPU to a thread that is not blocked. Introducing a low-level synchronization would void the guarantees of liveness of the system.

That's why I prefer a model where the VM is in charge of its own initialization.  When one VM creates a new VM, the source VM indicates the name of the main module of the new VM.  A new "processor" is created for the new VM and it goes about initializing the modules it requires.  The source VM is completely decoupled from the new VM.  This is a simpler higher-level API and covers the use cases I can think of.

Another incentive for tracking module dependencies is that it simplifies linking of executable applications.  Instead of having to provide the list of all the modules an application needs, it would be possible to only provide the name of the main module.  The linker would figure out all the required modules automatically.  Other modules could be specified, but these would only be loaded by an explicit request at run time (i.e. it would be for bundling multiple modules in a standalone executable).

Marc




More information about the Gambit-list mailing list