Wow. This is a big day. Thank you.
2017-08-21 21:13 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca:
On Aug 21, 2017, at 8:11 AM, Adam adam.mlmb@gmail.com wrote:
Hi Marc,
Aha - so - the garbage collector can run in parallell as of today.
Here are some more in depth questions (also hopefully more in line with
your terminology):
Can objects/heap be shared between (processors in different) VM:s?
No, by design. Each VM has its independent GC and runtime system. Very few things are shared… the heartbeat timer (this is an OS constraint) and symbols.
How far has the work on multiple processors in one VM
(--enable-multiple-threaded-vms) gone? Specifically:
Can different processors which are running in one VM, share heap/objects
reliably?
Yes this is reliable. In other words, the GC is reliable on multiple threaded VMs.
Can multiple processors which are running in one VM execute [code] in
parallel reliably, presuming all but the root/first processor only utilizes some given subset of the runtime (e.g. math and list processing is fine but no IO as the IO not is multiprocessor-safe yet)?
Yes. The subset is very large… mainly stay away from thread priorities… for now.
Do routines to allocate/launch and stop/collect/deallocate (more)
processors in one VM exist & work reliably?
Yes. The only part that is not thoroughly tested is the handling of errors when the VM is resized (i.e. resizing the VM from 1 to 8 processors, what happens when the OS fails to allocate the OS thread for processor #4?). The handling code is there to detect this but it isn’t graceful (fatal error) and it hasn’t been tested much.
The call (##current-vm-resize ##startup-processor! N) where N is the new size will do this, but it is an internal function for implementing the VM that shoudn’t be used if you don’t know what you are doing…
Do routines to send messages between multiple processors in one VM exist
& work reliably?
Yes. Here’s a simple example with thread-send :
(declare (standard-bindings) (block))
(define (busy-sleep n) (if (> n 0) (busy-sleep (- n 1)) #f))
(define (short-delay) (busy-sleep 100000)) ;; about 100 microseconds
(define (go n)
(define (ring next-thread) (let loop () (let ((msg (thread-receive))) (thread-send next-thread (- msg 1)) (short-delay) (if (> msg 0) (loop)))))
(letrec ((t1 (make-thread (lambda () (ring t2)))) (t2 (make-thread (lambda () (ring t3)))) (t3 (make-thread (lambda () (ring t4)))) (t4 (make-thread (lambda () (ring t1)))))
(for-each thread-start! (list t1 t2 t3 t4)) (thread-send t1 n) (for-each thread-join! (list t1 t2 t3 t4))))
(time (go 1000000))
When this is run in a 4 processor VM, all the processors are kept working at a high percentage (~ 80%), so the speedup over a 1 processor VM is close to 4. See the parallelism profile generated by xactlog below.
Other interprocessor communication mechanisms also work (mutexes, condition variables, etc).
Do routines to facilitate work stealing between multiple processors in
one VM exist & work reliably?
Yes work stealing is implemented.
I presume all the above is in the smp branch. Given some common sense
testing for a particular git commit, can the smp branch be counted as reliable?
The SMP branch was combined with the master branch yesterday. The configure option --enable-smp will enable the SMP Scheme thread scheduler (but you have to “make;make bootclean;make” to activate it). You also need --enable-multiple-threaded-vms.
While I’m on this subject, --enable-multiple-threaded-vms will become the new default after the next release. So those who prefer to not use the parallel GC should start adding --disable-multiple-threaded-vms in their build process.
Also given common sense testing, can the main branch be counted as
reliable?
The master branch contains the latest development patches, so it should not be considered maximally reliable. It happens on occasion that a patch breaks some existing infrequently used feature. If you want the highest reliability use a release.
When in the future do you think the runtime including IO will be proofed
for use across multiple processors in one VM?
On my TODO over the next 6 months. The SMP Scheme thread scheduler is already in good shape. The time consuming part (yet to be done) is implementing thread priorities and fine tuning and testing the runtime system.
Thanks and best regards, Adam
Marc