I have just created the “smp” branch which contains a major refactoring of the thread scheduler to support parallel thread execution.
Please note that this is an experimental branch that omits some of the thread features of Gambit (thread priorities, thread interruption, etc). So you should avoid those features in your experiments, and don’t use the smp branch for production code!
I have done some quick testing on OS X and linux, on intel architecture. I’m interested in feedback on Windows and other processor architectures, ARM in particular. I have also only tried the GNU gcc C compiler.
There are special steps needed to build the smp branch. Here are the commands starting with a git clone:
git clone https://github.com/gambit/gambit.git cd gambit
git checkout master ./configure make -j4 latest-release ./configure --enable-single-host make -j4 from-scratch
git checkout smp cp gsc/_prims.scm gsc/_prims.scm-smp cp gsc/_t-c-2.scm gsc/_t-c-2.scm-smp
git checkout master cp gsc/_prims.scm-smp gsc/_prims.scm cp gsc/_t-c-2.scm-smp gsc/_t-c-2.scm
make -j4 make bootstrap git reset --hard
git checkout smp make bootclean ./configure --enable-single-host --enable-multiple-threaded-vms make -j4 make check
Preliminary performance testing on 4 and 64 core machines show good performance. Threaded fibonacci of 45, which creates 317810 threads and has a thread granularity of 0.2 ms, runs 7.96x faster when using 8 processors instead of 1, and it runs 30x faster when using 64 processors (see attached transcript).
So please give it a try and report any issues!
Marc
% gsc/gsc -:=. tfib.scm % gsi/gsi -:p1 tfib (time (tfib 45)) 69957 ms real time 69983 ms cpu time (69249 user, 733 system) 9 collections accounting for 1811 ms real time (1577 user, 235 system) 521655584 bytes allocated 215447 minor faults no major faults 1836311903 % gsi/gsi -:p8 tfib (time (tfib 45)) 8794 ms real time 70097 ms cpu time (69745 user, 353 system) 14 collections accounting for 200 ms real time (1307 user, 107 system) 104214752 bytes allocated 69552 minor faults no major faults 1836311903 % gsi/gsi -:p64 tfib (time (tfib 45)) 2333 ms real time 143035 ms cpu time (141703 user, 1332 system) 8 collections accounting for 395 ms real time (18796 user, 992 system) 10612584 bytes allocated 29126 minor faults no major faults 1836311903 % cat tfib.scm (declare (standard-bindings) (not safe) (fixnum) (block))
(define (tfib n) (if (< n 2) 1 (if (< n 20) (+ (tfib (- n 1)) (tfib (- n 2))) (let ((x (make-thread (lambda () (tfib (- n 1)))))) (thread-start! x) (let ((y (tfib (- n 2)))) (+ (thread-join! x) y))))))
(pp (time (tfib 45)))
Afficher les réponses par date
On 01/28/2017 08:16 PM, Marc Feeley wrote:
I have just created the “smp” branch which contains a major refactoring of the thread scheduler to support parallel thread execution.
Please note that this is an experimental branch that omits some of the thread features of Gambit (thread priorities, thread interruption, etc). So you should avoid those features in your experiments, and don’t use the smp branch for production code!
I have done some quick testing on OS X and linux, on intel architecture. I’m interested in feedback on Windows and other processor architectures, ARM in particular. I have also only tried the GNU gcc C compiler.
There are special steps needed to build the smp branch. Here are the commands starting with a git clone:
git clone https://github.com/gambit/gambit.git cd gambit git checkout master ./configure make -j4 latest-release ./configure --enable-single-host make -j4 from-scratch git checkout smp cp gsc/_prims.scm gsc/_prims.scm-smp cp gsc/_t-c-2.scm gsc/_t-c-2.scm-smp git checkout master cp gsc/_prims.scm-smp gsc/_prims.scm cp gsc/_t-c-2.scm-smp gsc/_t-c-2.scm make -j4 make bootstrap git reset --hard git checkout smp make bootclean ./configure --enable-single-host --enable-multiple-threaded-vms make -j4 make check
I get a hang right here on
------------ TEST 11 (run unit tests) make ut make[2]: Entering directory '/home/lucier/programs/gambit/gambit/tests' LD_LIBRARY_PATH=../lib:../gsi:../gsc:/usr/local/libimobiledevice/lib:/pkgs/chicken-4.10.0/lib:/usr/local/Gambit/current/lib:/usr/local/netpbm10/lib: ../gsi/gsi -:p1,tl,~~bin=../bin,~~lib=../lib,~~include=../include -f ./run-unit-tests.scm [ 31| 0| 0] 25% ##########................................ .3s Suspended firefly:~/programs/gambit/gambit> kill -9 %+ firefly:~/WWW/615-2016/software> uname -a Linux firefly 4.8.0-34-generic #36-Ubuntu SMP Wed Dec 21 17:24:18 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux firefly:~/programs/gambit/gambit> gsi -v v4.8.7 20170129000735 x86_64-unknown-linux-gnu "./configure 'CC=gcc -march=native -D___CAN_IMPORT_CLIB_DYNAMICALLY' '--enable-single-host' '--enable-shared' '--enable-multiple-versions' '--enable-multiple-threaded-vms'"
It doesn't hang the next "make check" then it hangs with
make[2]: Entering directory '/home/lucier/programs/gambit/gambit/tests' LD_LIBRARY_PATH=../lib:../gsi:../gsc:/usr/local/libimobiledevice/lib:/pkgs/chicken-4.10.0/lib:/usr/local/Gambit/current/lib:/usr/local/netpbm10/lib: ../gsi/gsi -:p1,tl,~~bin=../bin,~~lib=../lib,~~include=../include -f ./run-unit-tests.scm [ 74| 0| 0] 60% #########################................. .8s
There's no CPU running anything.
Brad
Would be good if you could narrow it down some more…
I couldn’t reproduce that on the linux machine I have. What are the specs of yours?
Marc
On Jan 28, 2017, at 11:01 PM, Bradley Lucier lucier@math.purdue.edu wrote:
On 01/28/2017 08:16 PM, Marc Feeley wrote:
I have just created the “smp” branch which contains a major refactoring of the thread scheduler to support parallel thread execution.
Please note that this is an experimental branch that omits some of the thread features of Gambit (thread priorities, thread interruption, etc). So you should avoid those features in your experiments, and don’t use the smp branch for production code!
I have done some quick testing on OS X and linux, on intel architecture. I’m interested in feedback on Windows and other processor architectures, ARM in particular. I have also only tried the GNU gcc C compiler.
There are special steps needed to build the smp branch. Here are the commands starting with a git clone:
git clone https://github.com/gambit/gambit.git cd gambit
git checkout master ./configure make -j4 latest-release ./configure --enable-single-host make -j4 from-scratch
git checkout smp cp gsc/_prims.scm gsc/_prims.scm-smp cp gsc/_t-c-2.scm gsc/_t-c-2.scm-smp
git checkout master cp gsc/_prims.scm-smp gsc/_prims.scm cp gsc/_t-c-2.scm-smp gsc/_t-c-2.scm
make -j4 make bootstrap git reset --hard
git checkout smp make bootclean ./configure --enable-single-host --enable-multiple-threaded-vms make -j4 make check
I get a hang right here on
------------ TEST 11 (run unit tests) make ut make[2]: Entering directory '/home/lucier/programs/gambit/gambit/tests' LD_LIBRARY_PATH=../lib:../gsi:../gsc:/usr/local/libimobiledevice/lib:/pkgs/chicken-4.10.0/lib:/usr/local/Gambit/current/lib:/usr/local/netpbm10/lib: ../gsi/gsi -:p1,tl,~~bin=../bin,~~lib=../lib,~~include=../include -f ./run-unit-tests.scm [ 31| 0| 0] 25% ##########................................ .3s Suspended firefly:~/programs/gambit/gambit> kill -9 %+ firefly:~/WWW/615-2016/software> uname -a Linux firefly 4.8.0-34-generic #36-Ubuntu SMP Wed Dec 21 17:24:18 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux firefly:~/programs/gambit/gambit> gsi -v v4.8.7 20170129000735 x86_64-unknown-linux-gnu "./configure 'CC=gcc -march=native -D___CAN_IMPORT_CLIB_DYNAMICALLY' '--enable-single-host' '--enable-shared' '--enable-multiple-versions' '--enable-multiple-threaded-vms'"
It doesn't hang the next "make check" then it hangs with
make[2]: Entering directory '/home/lucier/programs/gambit/gambit/tests' LD_LIBRARY_PATH=../lib:../gsi:../gsc:/usr/local/libimobiledevice/lib:/pkgs/chicken-4.10.0/lib:/usr/local/Gambit/current/lib:/usr/local/netpbm10/lib: ../gsi/gsi -:p1,tl,~~bin=../bin,~~lib=../lib,~~include=../include -f ./run-unit-tests.scm [ 74| 0| 0] 60% #########################................. .8s
There's no CPU running anything.
Brad
On 01/28/2017 11:10 PM, Marc Feeley wrote:
Would be good if you could narrow it down some more…
I couldn’t reproduce that on the linux machine I have. What are the specs of yours?
It's running Ubuntu 16.10, processor
model name : Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
so four cores with eight threads, compiled with the shipping gcc:
firefly:~/WWW/615-2016/software> gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/6/lto-wrapper Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 6.2.0-5ubuntu12' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12)
I believe I followed your instructions.
Brad
I’ve now tried “make ut” several times on two different linux SMP machines (one with 12 cores and the other 64 cores), and can’t reproduce the issue…
Maybe it is a race condition that becomes more of an issue due to the fast clock of your CPU, or perhaps a difference in memory consistency model.
Marc
On Jan 28, 2017, at 11:11 PM, Bradley Lucier lucier@math.purdue.edu wrote:
On 01/28/2017 11:10 PM, Marc Feeley wrote:
Would be good if you could narrow it down some more…
I couldn’t reproduce that on the linux machine I have. What are the specs of yours?
It's running Ubuntu 16.10, processor
model name : Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
so four cores with eight threads, compiled with the shipping gcc:
firefly:~/WWW/615-2016/software> gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/6/lto-wrapper Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 6.2.0-5ubuntu12' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12)
I believe I followed your instructions.
Brad
OK… on the smaller machine (which actually has 24 processors), after *many* “make ut” I finally see a deadlock of some sort, so I will investigate. It seems to coincide with starting to do something else on the machine (like typing in emacs) but not consistently. It takes several hundred “make ut” to get a deadlock. The CPUs are “Intel(R) Xeon(R) X5650 @ 2.67GHz”.
Can you reproduce the issue more consistently on your machine? Does it happen every second or so “make ut”?
Marc
On Jan 29, 2017, at 10:54 AM, Marc Feeley feeley@iro.umontreal.ca wrote:
I’ve now tried “make ut” several times on two different linux SMP machines (one with 12 cores and the other 64 cores), and can’t reproduce the issue…
Maybe it is a race condition that becomes more of an issue due to the fast clock of your CPU, or perhaps a difference in memory consistency model.
Marc
On Jan 28, 2017, at 11:11 PM, Bradley Lucier lucier@math.purdue.edu wrote:
On 01/28/2017 11:10 PM, Marc Feeley wrote:
Would be good if you could narrow it down some more…
I couldn’t reproduce that on the linux machine I have. What are the specs of yours?
It's running Ubuntu 16.10, processor
model name : Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
so four cores with eight threads, compiled with the shipping gcc:
firefly:~/WWW/615-2016/software> gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/6/lto-wrapper Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 6.2.0-5ubuntu12' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12)
I believe I followed your instructions.
Brad
Here's an idea for debugging this on your end. Use the --enable-debug-c-backtrace and --enable-debug-ctrl-flow-history configure options to generate a C backtrace when the Scheme program crashes. So rebuild Gambit with:
make clean ./configure --enable-single-host --enable-multiple-threaded-vms --enable-debug-c-backtrace --enable-debug-ctrl-flow-history make
Then run the unit tests, and when it hangs, send a termination signal to the hanging Scheme process:
kill -s SIGTERM <process_id>
You should get on stdout the C backtrace (which might be more informative without the --enable-single-host) and at the tail of gambit.log you will have the most recent control-flow history of each processor (the last entries of each processor is where each processor was executing when the termination signal was received).
Marc
On Jan 29, 2017, at 11:18 AM, Marc Feeley feeley@iro.umontreal.ca wrote:
OK… on the smaller machine (which actually has 24 processors), after *many* “make ut” I finally see a deadlock of some sort, so I will investigate. It seems to coincide with starting to do something else on the machine (like typing in emacs) but not consistently. It takes several hundred “make ut” to get a deadlock. The CPUs are “Intel(R) Xeon(R) X5650 @ 2.67GHz”.
Can you reproduce the issue more consistently on your machine? Does it happen every second or so “make ut”?
Marc
On Jan 29, 2017, at 10:54 AM, Marc Feeley feeley@iro.umontreal.ca wrote:
I’ve now tried “make ut” several times on two different linux SMP machines (one with 12 cores and the other 64 cores), and can’t reproduce the issue…
Maybe it is a race condition that becomes more of an issue due to the fast clock of your CPU, or perhaps a difference in memory consistency model.
Marc
On Jan 28, 2017, at 11:11 PM, Bradley Lucier lucier@math.purdue.edu wrote:
On 01/28/2017 11:10 PM, Marc Feeley wrote:
Would be good if you could narrow it down some more…
I couldn’t reproduce that on the linux machine I have. What are the specs of yours?
It's running Ubuntu 16.10, processor
model name : Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
so four cores with eight threads, compiled with the shipping gcc:
firefly:~/WWW/615-2016/software> gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/6/lto-wrapper Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 6.2.0-5ubuntu12' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12)
I believe I followed your instructions.
Brad
On 01/29/2017 11:18 AM, Marc Feeley wrote:
OK… on the smaller machine (which actually has 24 processors), after *many* “make ut” I finally see a deadlock of some sort, so I will investigate. It seems to coincide with starting to do something else on the machine (like typing in emacs) but not consistently. It takes several hundred “make ut” to get a deadlock. The CPUs are “Intel(R) Xeon(R) X5650 @ 2.67GHz”.
Can you reproduce the issue more consistently on your machine? Does it happen every second or so “make ut”?
I updated and rebooted my linux box at work (so nothing else was running) and made Gambit with your new instructions. Ran make ut a dozen or so times and got nothing.
But after I started an emacs window it hung with
LD_LIBRARY_PATH=../lib:../gsi:../gsc:/usr/local/libimobiledevice/lib:/usr/local/Gambit/current/lib:/usr/local/netpbm10/lib: ../gsi/gsi -:p1,tl,~~bin=../bin,~~lib=../lib,~~include=../include -f ./run-unit-tests.scm
first time! That was with
./configure 'CC=gcc -march=native -D___CAN_IMPORT_CLIB_DYNAMICALLY' '--enable-single-host' '--enable-multiple-versions' '--enable-shared' --enable-multiple-threaded-vms
On my home box (with a lot of other things running) it didn't finish the "make -j 8 from-scratch", hanging at
LD_LIBRARY_PATH=..:/usr/local/libimobiledevice/lib:/pkgs/chicken-4.10.0/lib:/usr/local/Gambit/current/lib:/usr/local/netpbm10/lib: ../gsc-boot -:~~bin=../bin,~~lib=../lib,~~include=../include -f -warnings -link -flat -o _gambit.c -preload _kernel.c _system.c _num.c _std.c _eval.c _io.c _nonstd.c _thread.c _repl.c
That was configured with
./configure 'CC=gcc -march=native -D___CAN_IMPORT_CLIB_DYNAMICALLY' '--enable-single-host' '--enable-shared' '--enable-multiple-versions' '--enable-multiple-threaded-vms'
I'll look at your suggestions for debugging this.
Brad
Another debugging idea is to add CC="gcc -D___DEBUG_TICKETLOCK" to the configure options.
This will add a deadlock check to the ticket lock implementation, so if it busy waits too long in a “lock” operation it will show the location of the lock operation and the other lock operation that locked the lock.
Marc
On Jan 29, 2017, at 12:32 PM, Bradley Lucier lucier@math.purdue.edu wrote:
On 01/29/2017 11:18 AM, Marc Feeley wrote:
OK… on the smaller machine (which actually has 24 processors), after *many* “make ut” I finally see a deadlock of some sort, so I will investigate. It seems to coincide with starting to do something else on the machine (like typing in emacs) but not consistently. It takes several hundred “make ut” to get a deadlock. The CPUs are “Intel(R) Xeon(R) X5650 @ 2.67GHz”.
Can you reproduce the issue more consistently on your machine? Does it happen every second or so “make ut”?
I updated and rebooted my linux box at work (so nothing else was running) and made Gambit with your new instructions. Ran make ut a dozen or so times and got nothing.
But after I started an emacs window it hung with
LD_LIBRARY_PATH=../lib:../gsi:../gsc:/usr/local/libimobiledevice/lib:/usr/local/Gambit/current/lib:/usr/local/netpbm10/lib: ../gsi/gsi -:p1,tl,~~bin=../bin,~~lib=../lib,~~include=../include -f ./run-unit-tests.scm
first time! That was with
./configure 'CC=gcc -march=native -D___CAN_IMPORT_CLIB_DYNAMICALLY' '--enable-single-host' '--enable-multiple-versions' '--enable-shared' --enable-multiple-threaded-vms
On my home box (with a lot of other things running) it didn't finish the "make -j 8 from-scratch", hanging at
LD_LIBRARY_PATH=..:/usr/local/libimobiledevice/lib:/pkgs/chicken-4.10.0/lib:/usr/local/Gambit/current/lib:/usr/local/netpbm10/lib: ../gsc-boot -:~~bin=../bin,~~lib=../lib,~~include=../include -f -warnings -link -flat -o _gambit.c -preload _kernel.c _system.c _num.c _std.c _eval.c _io.c _nonstd.c _thread.c _repl.c
That was configured with
./configure 'CC=gcc -march=native -D___CAN_IMPORT_CLIB_DYNAMICALLY' '--enable-single-host' '--enable-shared' '--enable-multiple-versions' '--enable-multiple-threaded-vms'
I'll look at your suggestions for debugging this.
Brad
I found the cause of this issue. There is a deadlock possible in the program termination logic (which is non-trivial because all OS threads have to be terminated cleanly and memory deallocated). The problem disappears when a “exit(0);” is added at line 4577 of lib/setup.c (but of course this does not perform the usual cleanup).
I’ll investigate later but I think this is a sufficiently infrequent issue that should not block experimentation of the smp branch.
Marc
On Jan 29, 2017, at 1:28 PM, Marc Feeley feeley@iro.umontreal.ca wrote:
Another debugging idea is to add CC="gcc -D___DEBUG_TICKETLOCK" to the configure options.
This will add a deadlock check to the ticket lock implementation, so if it busy waits too long in a “lock” operation it will show the location of the lock operation and the other lock operation that locked the lock.
Marc
On Jan 29, 2017, at 12:32 PM, Bradley Lucier lucier@math.purdue.edu wrote:
On 01/29/2017 11:18 AM, Marc Feeley wrote:
OK… on the smaller machine (which actually has 24 processors), after *many* “make ut” I finally see a deadlock of some sort, so I will investigate. It seems to coincide with starting to do something else on the machine (like typing in emacs) but not consistently. It takes several hundred “make ut” to get a deadlock. The CPUs are “Intel(R) Xeon(R) X5650 @ 2.67GHz”.
Can you reproduce the issue more consistently on your machine? Does it happen every second or so “make ut”?
I updated and rebooted my linux box at work (so nothing else was running) and made Gambit with your new instructions. Ran make ut a dozen or so times and got nothing.
But after I started an emacs window it hung with
LD_LIBRARY_PATH=../lib:../gsi:../gsc:/usr/local/libimobiledevice/lib:/usr/local/Gambit/current/lib:/usr/local/netpbm10/lib: ../gsi/gsi -:p1,tl,~~bin=../bin,~~lib=../lib,~~include=../include -f ./run-unit-tests.scm
first time! That was with
./configure 'CC=gcc -march=native -D___CAN_IMPORT_CLIB_DYNAMICALLY' '--enable-single-host' '--enable-multiple-versions' '--enable-shared' --enable-multiple-threaded-vms
On my home box (with a lot of other things running) it didn't finish the "make -j 8 from-scratch", hanging at
LD_LIBRARY_PATH=..:/usr/local/libimobiledevice/lib:/pkgs/chicken-4.10.0/lib:/usr/local/Gambit/current/lib:/usr/local/netpbm10/lib: ../gsc-boot -:~~bin=../bin,~~lib=../lib,~~include=../include -f -warnings -link -flat -o _gambit.c -preload _kernel.c _system.c _num.c _std.c _eval.c _io.c _nonstd.c _thread.c _repl.c
That was configured with
./configure 'CC=gcc -march=native -D___CAN_IMPORT_CLIB_DYNAMICALLY' '--enable-single-host' '--enable-shared' '--enable-multiple-versions' '--enable-multiple-threaded-vms'
I'll look at your suggestions for debugging this.
Brad
I have changed the build procedure to make it more robust to changes in the compiler and to allow the smp branch to be built more easily.
So now the "current-gsc-boot" make target must be used instead of "latest-release":
git clone https://github.com/gambit/gambit.git cd gambit ./configure make -j4 current-gsc-boot ./configure --enable-single-host make -j4 from-scratch make check
To build the smp branch, do a "git checkout smp" before the first invocation of configure and add the --enable-multiple-threaded-vms option to the second invocation of configure:
git clone https://github.com/gambit/gambit.git cd gambit git checkout smp ./configure make -j4 current-gsc-boot ./configure --enable-single-host --enable-multiple-threaded-vms make -j4 from-scratch make check
Marc
On Jan 28, 2017, at 8:16 PM, Marc Feeley feeley@iro.umontreal.ca wrote:
I have just created the “smp” branch which contains a major refactoring of the thread scheduler to support parallel thread execution.
Please note that this is an experimental branch that omits some of the thread features of Gambit (thread priorities, thread interruption, etc). So you should avoid those features in your experiments, and don’t use the smp branch for production code!
I have done some quick testing on OS X and linux, on intel architecture. I’m interested in feedback on Windows and other processor architectures, ARM in particular. I have also only tried the GNU gcc C compiler.
There are special steps needed to build the smp branch. Here are the commands starting with a git clone:
git clone https://github.com/gambit/gambit.git cd gambit
git checkout master ./configure make -j4 latest-release ./configure --enable-single-host make -j4 from-scratch
git checkout smp cp gsc/_prims.scm gsc/_prims.scm-smp cp gsc/_t-c-2.scm gsc/_t-c-2.scm-smp
git checkout master cp gsc/_prims.scm-smp gsc/_prims.scm cp gsc/_t-c-2.scm-smp gsc/_t-c-2.scm
make -j4 make bootstrap git reset --hard
git checkout smp make bootclean ./configure --enable-single-host --enable-multiple-threaded-vms make -j4 make check
Preliminary performance testing on 4 and 64 core machines show good performance. Threaded fibonacci of 45, which creates 317810 threads and has a thread granularity of 0.2 ms, runs 7.96x faster when using 8 processors instead of 1, and it runs 30x faster when using 64 processors (see attached transcript).
So please give it a try and report any issues!
Marc
% gsc/gsc -:=. tfib.scm % gsi/gsi -:p1 tfib (time (tfib 45)) 69957 ms real time 69983 ms cpu time (69249 user, 733 system) 9 collections accounting for 1811 ms real time (1577 user, 235 system) 521655584 bytes allocated 215447 minor faults no major faults 1836311903 % gsi/gsi -:p8 tfib (time (tfib 45)) 8794 ms real time 70097 ms cpu time (69745 user, 353 system) 14 collections accounting for 200 ms real time (1307 user, 107 system) 104214752 bytes allocated 69552 minor faults no major faults 1836311903 % gsi/gsi -:p64 tfib (time (tfib 45)) 2333 ms real time 143035 ms cpu time (141703 user, 1332 system) 8 collections accounting for 395 ms real time (18796 user, 992 system) 10612584 bytes allocated 29126 minor faults no major faults 1836311903 % cat tfib.scm (declare (standard-bindings) (not safe) (fixnum) (block))
(define (tfib n) (if (< n 2) 1 (if (< n 20) (+ (tfib (- n 1)) (tfib (- n 2))) (let ((x (make-thread (lambda () (tfib (- n 1)))))) (thread-start! x) (let ((y (tfib (- n 2)))) (+ (thread-join! x) y))))))
(pp (time (tfib 45)))
-
Bradley Lucier -
Marc Feeley