[gambit-list] Information about the Gambit-C Javascript Backend
Paulo Silva Filho
paulosfilho at gmail.com
Sat Jul 20 20:23:42 EDT 2019
Sires,
I got the Gambit Javascript Backend working suiting my needs yesterday. It
was somewhat difficult and a little overworking, but I got my needed
results.
Since the job to do so is not trivial, I'm posting my steps, here, to let
it documented. Maybe, as my work advances, I can create an article on
Medium, to show everybody how to create small, efficient and compiled
scheme applications in the web.
My objectives were:
1 - To compile scheme scripts in Javascript, to import them from Javascript
projects or from any language running on Javascript or compiling to
Javascript. So, interpreters, like BiwaScheme, or Chibi-Scheme on WebASM
wouldn't suit. If I couldn't achieve this objective, I would try a
workaround with some Scheme interpreter in Javascript.
2 - These compiled scripts be small, or very small, since the algorithms I
need to compile to Javascript are simple, only to make some numeric
analysis, and User Interface manipulation in real time. Javascript has a
very good performance to do these tasks, but the language is cumbersome to
program these kinds of programs. I know this is a matter of opinion, but,
in my opinion, scheme is better than Javascript to do this due scheme
consistency and clarity.
3 - I would need to call to and from javascript native calls, to enable
interoperability. Usually, scheme environments and native environment are
orthogonal, due Scheme specificities (like call/cc and required TCO), but
this orthogonality is no problem to me, since I can wrap the compiled
scheme code in a consistent Javascript interface.
4 - The language to be compiled needed to be TRUE SCHEME. No Common-Lisp,
nor Scheme-Like, nor Clojure. Clojure has a very competent Javascript
Compiler, but, as a language, it's not suitable to be, due its complexity
and compromises, regarding TCO and continuations.
Considering these objectives, I started to look for scheme->javascript
COMPILERS, not interpreters. As a matter of fact, I've been using
BiwaScheme as a Scheme interpreter in Javascript, and the only thing I'm
really missing is syntax-case and syntax-rules macros, since BiwaScheme
doesn't support define-syntax clauses. I've had some performance problems
with BiwaScheme, but I've realized, during the experiments I was perfoming
with Biwa and other compiled Schemes, that these performance issues are due
my misuse of the interpreter, not due BiwaScheme properties. BiwaScheme has
an excellent call/cc and TCO implementation, and if the algorithms use
these, avoiding populating the stack (which access in BiwaScheme is slow),
this Scheme implementation performs really well.
So, I decided to try these Scheme->Javascript compilers:
1 - Spock, from Chicken scheme.
2 - Gambit-C Javascript Compiler.
3 - Whalesong or RacketScript, from the Racket Project.
4 - Scheme2JS
I thought Whalesong and RacketScript somewhat low documented, although
belonging to a huge project like Racket. So I decided to pass it. And I
didn't want to deal with the huge complexity of Racket - I needed a more
focused Scheme project to deal, since these smaller projects, by the rule,
have more approachable people to deal with. My choice on Gambit has just
proved this point. Other point, is that I would need to hack any
Scheme->Javascript compiler, and in a smaller, more focused, project, it's
easier to do.
Scheme2JS uses the GPL license, and, since I'll need to distribute my
software in a proprietary format, although I think I could (and I will)
open-source the entire Javascript Scheme Stack I'm developing, I discarded
Scheme2JS as as an option.
The other two options were Spock and Gambit-C. After reading this article:
http://www.schemeworkshop.org/2012/papers/thivierge-feeley-paper-sfp12.pdf,
I decided to give Gambit-C a try.
Since I'm using windows as operating system, I installed Gambit from
Chocolatey and tried to use the information I've gotten from the Gambit
mail group to compile some Javascript code. No success with default Gambit
installation from Chocolatey (version 4.6.6).
Then I tried a more up to date version, from Gambit web site (version
4.9.3). No success at all. So, I contacted the newsgroup by myself, with my
doubts, and I realized that Javascript Backend is an experimental one, so I
decided to use a very up to date version of Gambit, from source code
repository, in Github.
No success too. So I decided to hack the Javascript created by Gambit, and
I realized that this javascript, although complex, is very, very readable -
and I discovered that the generated code has some undeclared objects
(mainly the Gambit object), with procedural functions replacing their
previous usage. So I decided to hack this generated code and, voilà, the
script worked suitably in Node.js and in Web browser (Firefox and Chrome).
This process has been a little cumbersome and overworking, but the results
were beautiful.
I started the process installing MSYS2 from Chocolatey:
;; Windows CMD Shell:
> choco install -y msys2
Then, after installing msys2, I loaded the bash shell, and installed the
development environment:
;; MSYS2 Bash Shell:
$ pacman -Syu
$ pacman -S base-devel gcc git
Then, my computer was ready to build Gambit-C. The next steps can be done
in a Linux and MacOS X machines, as well, since they have the correct tools
installed.
I created a directory to receive the Gambit code, then I download its
source from Github, compiled it and installed it in a specific directory on
my computer, to avoid clashes with the default installed Gambit
distribution:
;; MSYS2 Bash Shell:
$ git clone git at github.com:gambit/gambit.git
$ cd gambit
$ git log # To show the EXACT version of the repository, if you want to
repeat this process I'm presenting here, without any surprise:
commit 55a0b75ed9391c6e27475b251bd2bb9babfd9aab (HEAD -> master,
origin/master, origin/HEAD)
Author: Marc Feeley <feeley at iro.umontreal.ca>
Date: Tue Jul 16 08:46:48 2019 -0400
Don't ignore the settings parameter of ##open-predefined
...
Other commit information...
...
...
$ ./configure --prefix=/c/tools/gambit-c --enable-cplusplus --enable-c-opt
--enable-gcc-opts --enable-shared --enable-single-host
configure: loading site script /etc/config.site
checking build system type... x86_64-pc-msys
checking host system type... x86_64-pc-msys
checking target system type... x86_64-pc-msys
checking for gcc... gcc
checking whether the C compiler works... yes
...
Lots of output!!!
...
...
$ make
making all in include
make[1]: Entering directory '/c/Users/.../gambit/include'
make[1]: Leaving directory '/c/Users/.../gambit/include'
making all in lib
make[1]: Entering directory '/c/Users/.../gambit/lib'
making deselect-gen-for-commit in gambit
...
Lots of output!!!
...
...
$ make install
...
...
Lots of output!!!
...
...
On you need to update suitably the next environment variables:
* LIBRARY_PATH: add "/tools/gambit-c/lib directory or the correspondent
directory you installed gambit."
* INCLUDE: add "/tools/gambit-c/include directory or the correspondent
directory you installed gambit."
* LIB: add "/tools/gambit-c/lib directory or the correspondent directory
you installed gambit."
* CPATH: add "/tools/gambit-c/lib directory or the correspondent directory
you installed gambit."
* PATH: : add "/tools/gambit-c/lib and /tools/gambit/bin directories or the
correspondent directories you installed gambit."
After that, you need to return to your gambit source code folder and build
the Gambit javascript library:
;; MSYS2 Bash Shell, going to the gambit source, downloaded from Git:
$ cd gambit
Then, go to the lib directory, and build the Javascript runtime:
;; MSYS2 Bash Shell:
$ cd lib
$ make _gambit.js # This command finally build the needed Gambit Javascript
runtime
LD_LIBRARY_PATH=..: ../gsc/gsc
-:~~bin=../bin,~~lib=../lib,~~include=../include -f -target js -prelude
"(define-cond-expand-feature|enable-type-checking|)(define-cond-expand-feature|disable-auto-forcing|)(define-cond-expand-feature|enable-sharp-dot|)(define-cond-expand-feature|enable-bignum|)(define-cond-expand-feature|enable-ratnum|)(define-cond-expand-feature|enable-cpxnum|)(define-cond-expand-feature|disable-smp|)(##include\"../lib/header.scm\")"
-o _gambit.js ../lib/_univlib.scm
$ cp _gambit.js /c/tools/gambit-c/lib/_gambit.js # This command publish the
Javascript runtime to the installed Gambit-C.
$
At this point, the Gambit-C Javascript Backend is fully installed. Maintain
Gambit-C source code in this state, in the case of any future updates,
which can be common, since the Javascript backend is experimental.
Now, we need to build and execute a sample. You can build such sample with
the next instructions.
First, in a suitable place, create a directory:
;; MSYS2 bash (assuming the new installed Gambit is in PATH, and all other
environment variables are set)
$ mkdir test
$ cd test
Initialize the folder as an NPM project, to use Uglify-JS to minify a
little the generated files. The minification isn't very efficient, but
improve the file sizes. You need to install npm and put it on path:
;; Windows CMD Shell, with Administrative roles:
> choco install -y npm
Then, in the created test directory, perform the next proceedings:
;; MSYS2 bash
$ npm init
... Accept all given options, or customize them in according to your needs.
$ npm i uglify-js
Then, now, create the files below:
;; FILE app.scm
(declare (standard-bindings) (extended-bindings) (not safe) (not
run-time-bindings))
(println ">>> app.scm")
(define (fib n)
(fib-iter 1 0 n))
(define (fib-iter a b count)
(if (= count 0)
b
(fib-iter (+ a b) a (- count 1))))
(for-each
(lambda (x)
(println (fib x)))
(append '(1 2 3 4 5 6 7 8 9 10 11 12 13 14 15)
'(16 17 18 19 20 21 22 23 24 25 26 27 28 29 30)))
(let* ((start (current-milliseconds))
(result (fib 35))
(end (current-milliseconds)))
(js-alert (fx- end start))
(js-alert result))
(console.log 123456)
(console.log "This is a scheme string!!!")
(console.log 22.22)
(console.log '(1 2 3 4))
(console.log '(1 . 2))
(js-global "teste"
(lambda () (println "Function test")))
(js-global "fib" ; Javascript exported calls can't be recursive!!!!!
(lambda (n) (fib n)))
;; FILE lib.scm
(declare (standard-bindings) (extended-bindings) (not safe) (not
run-time-bindings))
(println ">>> lib.scm")
(define (for-each f lst)
(if (pair? lst)
(begin
(f (car lst))
(for-each f (cdr lst)))))
(define (append lst1 lst2)
(if (pair? lst1)
(cons (car lst1) (append (cdr lst1) lst2))
lst2))
(define (current-milliseconds)
(##inline-host-expression "Date.now()"))
(define (js-alert obj)
(##inline-host-statement "console.log(g_scm2js(@1@));" obj))
(define (console.log obj)
(##inline-host-statement "console.log(g_scm2js(@1@));" obj))
(define (js-global name obj)
(##inline-host-statement "window[g_scm2js(@1@)]=g_scm2js(@2@);" name obj))
(app#)
;; FILE makefile
GSC=gsc -target js
all: run
lib.js: lib.scm
$(GSC) -c lib.scm
app.js: app.scm
$(GSC) -c app.scm
app_.js: lib.js app.js
$(GSC) -link -l lib app.js
linked_app.js: lib.js app.js app_.js
$(GSC) -c lib.scm
cat app_.js lib.js app.js > linked_app.js
wc *.js
run: linked_app.js
node linked_app.js
clean:
rm -f lib.js app.js app_.js linked_app.js
;; FILE patch.sh
#!/usr/bin/env bash
sed -i 's/Gambit\.sp/g_sp/g' linked_app.js
sed -i 's/Gambit\.stack/g_stack/g' linked_app.js
sed -i 's/Gambit\.void_obj/void 0/g' linked_app.js
sed -i 's/Gambit\.nargs/g_nargs/g' linked_app.js
sed -i 's/Gambit\.stack/g_stack/g' linked_app.js
sed -i 's/Gambit\.underflow/g_underflow/g' linked_app.js
sed -i 's/Gambit\.trampoline/g_trampoline/g' linked_app.js
sed -i 's/Gambit\.scm2js/g_scm2js/g' linked_app.js
sed -i 's/Gambit\.r/g_r/g' linked_app.js
sed -i 's/Gambit\.js2scm_call/g_js2scm_call/g' linked_app.js
sed -i 's/Gambit\.scm2js/g_scm2js/g' linked_app.js
sed -i 's/Gambit\.String/G_ScmString/g' linked_app.js
sed -i 's/Gambit\.Flonum/G_Flonum/g' linked_app.js
sed -i 's/Gambit\.Pair/G_Pair/g' linked_app.js
sed -i 's/Gambit\.Structure/G_Structure/g' linked_app.js
sed -i 's/Gambit\./g_/g' linked_app.js
sed -i 's/var fs = require\(.*\);//g' linked_app.js
sed -i 's/var g_argv;/var g_argv = ["web-browser"];\/\*/g' linked_app.js
sed -i 's/g_argv = arguments;/g_argv = arguments;*\//g' linked_app.js
npx uglifyjs --compress --mangle --output linked_app.min.js -- linked_app.js
;; FILE index.html
<!DOCTYPE html>
<html>
<head>
<title>fib</title>
<script type="text/javascript">
function print(x) { console.log(x); }
</script>
<script type="text/javascript" src="linked_app.min.js"></script>
</head>
<body></body>
</html>
After creating all these files, you can create the project with the next
proceedings:
;; MSYS2 bash shell:
$ chmod ugo+x patch.sh # Turn the patch script executable
$ make linked_app.js
gsc -target js -link -l lib app.js
gsc -target js -c lib.scm
cat app_.js lib.js app.js > linked_app.js
wc *.js
889 2224 20534 app.js
722 2048 16061 app_.js
427 1025 10221 lib.js
2038 5297 46816 linked_app.js
4076 10594 93632 total
If you only run Make, the scripts will be compiled, but the patches aren't
applied, so the process will fail.
After you created the linked_app.js file, patch it:
;; MSYS2 bash shell:
$ ./patch.sh
After that, the script, compatible with web browser, is ready.
You can load it using your default browser:
$ explorer index.html
And, in the browser (Firefox or Chrome), you can hit <F12>, and select the
console. And you'll see the output:
>>> lib.scm
>>> app.scm
1
1
2
3
5
8
13
21
34
55
89
144
233
377
610
987
1597
2584
4181
6765
10946
17711
28657
46368
75025
121393
196418
317811
514229
832040
2
9227465
123456
This is a scheme string!!!
22.22
Object(4) [ 1, 2, 3, 4 ]
Object [ 1 ]
The generated script isn't compatible with node.js, because the function
js-global assume the existence of the "window" object, which is absent in
Node.js. But, if in this function you replace "window" by "global", the
script will work.
So:
;; MSYS2 bash shell:
$ rm *.js
$ sed -i 's/window/global/g' lib.scm
$ make linked_app.js
$ ./patch.sh
$ node linked_app.min.js
1
1
2
3
5
8
13
21
34
55
89
144
233
377
610
987
1597
2584
4181
6765
10946
17711
28657
46368
75025
121393
196418
317811
514229
832040
0
9227465
123456
This is a scheme string!!!
22.22
{ '0': 1, '1': 2, '2': 3, '3': 4 }
{ '0': 1 }
But one of the most intersting things is to call Scheme code from
javascript:
;; MSYS2 bash shell:
$ node
> .load linked_app.min.js
> fib(42)
267914296
Strangely, fib(42) is the biggest value I can load using the fib function.
Some problem load the numerical tower. But, the entire process is very,
very interesting and well suitable to my needs to generate reusable
javascript libraries using Scheme. The performance is very good.
It's possible to load the entire Gambit runtime in a javascript file, which
can be suitable to create Scheme Node.js programs. First, create the next
files:
;; NodeApp.scm
(declare
(standard-bindings)
(extended-bindings)
(not safe)
(fixnum)
(block)
)
;; All function, variables and other definitions before any execution:
(define (console.log obj)
(##inline-host-statement "console.log(g_scm2js(@1@));" obj))
(define (js-global name obj)
(##inline-host-statement "global[g_scm2js(@1@)]=g_scm2js(@2@);" name obj))
(define (fib n)
(fib-iter 1 0 n))
(define (fib-iter a b count)
(if (= count 0)
b
(fib-iter (+ a b) a (- count 1))))
(js-global "teste"
(lambda () (console.log "Function test")))
(js-global "fib" ; Javascript exported calls can't be recursive!!!!!
(lambda (n) (fib n)))
; Then some commands executing things:
(for-each
(lambda (x)
(console.log (fib x)))
(append '(1 2 3 4 5 6 7 8 9 10 11 12 13 14 15)
'(16 17 18 19 20 21 22 23 24 25 26 27 28 29 30)
'(31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 300)))
(console.log 123456)
(console.log "This is a scheme string!!!")
(console.log 22.22)
(console.log '(1 2 3 4))
(console.log '(1 . 2))
;; patch-app.sh
#!/usr/bin/env bash
sed -i 's/Gambit\.sp/g_sp/g' NodeApp
sed -i 's/Gambit\.stack/g_stack/g' NodeApp
sed -i 's/Gambit\.void_obj/void 0/g' NodeApp
sed -i 's/Gambit\.nargs/g_nargs/g' NodeApp
sed -i 's/Gambit\.stack/g_stack/g' NodeApp
sed -i 's/Gambit\.underflow/g_underflow/g' NodeApp
sed -i 's/Gambit\.trampoline/g_trampoline/g' NodeApp
sed -i 's/Gambit\.scm2js/g_scm2js/g' NodeApp
sed -i 's/Gambit\.r/g_r/g' NodeApp
sed -i 's/Gambit\.js2scm_call/g_js2scm_call/g' NodeApp
sed -i 's/Gambit\.scm2js/g_scm2js/g' NodeApp
sed -i 's/Gambit\.String/G_ScmString/g' NodeApp
sed -i 's/Gambit\.Flonum/G_Flonum/g' NodeApp
sed -i 's/Gambit\.Pair/G_Pair/g' NodeApp
sed -i 's/Gambit\.Structure/G_Structure/g' NodeApp
sed -i 's/Gambit\./g_/g' NodeApp
sed -i 's/var fs = require\(.*\);//g' NodeApp
sed -i 's/var g_argv;/var g_argv = ["web-browser"];\/\*/g' NodeApp
sed -i 's/g_argv = arguments;/g_argv = arguments;*\//g' NodeApp
npx uglifyjs --compress --mangle --output Min-NodeApp -- NodeApp
mv Min-NodeApp NodeApp
Then make the executable:
;; MSYS2 bash shell:
$ chmod ugo+x patch-app.sh
$ gsc -target js -exe NodeApp.scm
$ ./patch-app.sh # This will take a while!
The NodeApp executable is 15MB heavy. It is a very big file. But it
contains the ENTIRE Gambit-C runtime - it is a COMPLETE SCHEME, which can
be ran in Node.JS platform! And, although very large, NodeApp load and run
very fast:
;; MSYS2 bash shell:
$ node NodeApp
1
1
2
3
5
8
13
21
34
55
89
144
233
377
610
987
1597
2584
4181
6765
10946
17711
28657
46368
75025
121393
196418
317811
514229
832040
1346269
2178309
3524578
5702887
9227465
14930352
24157817
39088169
63245986
102334155
165580141
267914296
433494437
701408733
1134903170
2.2223224462942035e+62
123456
This is a scheme string!!!
22.22
{ '0': 1, '1': 2, '2': 3, '3': 4 }
{ '0': 1 }
##thread-end-with-uncaught-exception! not implemented yet
I don't know what is this error ##thread-end-with-uncaught-exception! not
implemented yet, but it didn't bothered me, at this moment. In the future I
can ask why this problem is happening and how to solve it, and I'm happy,
because it seems the numerical tower is accessible from Javascript.
Unfortunately, infinite precision integers aren't available - but this is
no problem, since a BigNum library can be implemented.
The error ##thread-end-with-uncaught-exception thrown by this app hindered
me to turn the generated NodeApp application into a Node.js library, which
can be reused from Javascript; so you can help me to solve this problem,
and we can turn these libraries accessible from Javascript. Loading the app
from the Web Browser is possible, and I can use this entire file as a
library, but from Node.JS I couldn't. I'll look for the samples sent to me
to know if this problem is already solved, but, if no, can you help me to
solve this exception, or to know how to avoid it?
After all this work, I thank you very much by the help.
The next steps are discovering how to solve the
##thread-end-with-uncaught-exception error, to learn how to use
define-syntax/syntax-rules macros on Javascript Backend, and how to use
define-macro in Javascript backend (if these macros can be used, at all),
solve the numerical tower problem in the small scripts generated using
makefiles, and how to import and define the runtime library for small
javascript libraries, created using makefiles. I'll need to hack into the
Gambit libraries, to discover how to extract from them the default library
functions and definitions for my web scheme runtime.
After I got the suitable gambit-scheme libraries working, without
exceptions, and modularized, I can write a Medium article, to spread that
Clojure isn't the only advanced Lisp usable to create Javascript libraries
and programs, in Node.js and in Web. Any help in my endeavor is welcome.
Thank you very much, best regards and bye,
Paulo Silva-Filho.
Em sex, 19 de jul de 2019 às 15:23, Paulo Silva Filho <paulosfilho at gmail.com>
escreveu:
> Sires,
>
> I've realized that the examples seemed to be built using a development
> version of Gambit-C. I tried on a very stable version (4.6.6), which is
> from six years ago, and I've tried from a more recent version, 4.9.3, from
> the default package installers, in Windows and Linux.
>
> I've looked for the some hints about which version of Gambit is being used
> on all examples, and the features and files shown (like _univlib.scm,
> present in the development version of Gambit - directly from Git, on
> https://github.com/gambit/gambit/blob/master/lib/_univlib.scm) and I
> realized that the compiler version used need to be an experimental one.
>
> So I'll try all the samples, again, but from a version of gambit compiled
> directly from the source code, downloaded from the main repository.
>
> Thank you from all answers. I'll keep you in touch.
>
> Em sex, 19 de jul de 2019 às 12:37, Marc Feeley <feeley at iro.umontreal.ca>
> escreveu:
>
>> Like so:
>>
>> % cd gambit
>> % cd lib
>> % make _gambit.js
>> ../gsc/gsc -:~~bin=../bin,~~lib=../lib,~~include=../include -f -target
>> js -prelude
>> "(define-cond-expand-feature|enable-type-checking|)(define-cond-expand-feature|disable-auto-forcing|)(define-cond-expand-feature|enable-sharp-dot|)(define-cond-expand-feature|enable-bignum|)(define-cond-expand-feature|enable-ratnum|)(define-cond-expand-feature|enable-cpxnum|)(define-cond-expand-feature|disable-smp|)(##include\"../lib/header.scm\")"
>> -o _gambit.js ../lib/_univlib.scm
>> % cd ..
>> % gsc/gsc -:=. -target js -exe app.scm
>> % ./app
>> 1
>> 256
>> 65536
>> 16777216
>> 4294967296
>> 1099511627776
>> 281474976710656
>> 72057594037927936
>> 18446744073709551616
>> 4722366482869645213696
>> 1208925819614629174706176
>> 309485009821345068724781056
>> 79228162514264337593543950336
>> 20282409603651670423947251286016
>> 5192296858534827628530496329220096
>> 1329227995784915872903807060280344576
>> 340282366920938463463374607431768211456
>> 87112285931760246646623899502532662132736
>> 22300745198530623141535718272648361505980416
>> 5708990770823839524233143877797980545530986496
>> % cat app.scm
>> (define n 20)
>> (for-each (lambda (i)
>> (println (object->string (expt 2 (* i 8)))))
>> (iota n))
>> (exit)
>>
>> This should work for the other backends too. Note that there’s some bit
>> rot of lib/univlib.scm, so there are issues with some parts of the Gambit
>> library (for example write and pretty-print are broken, (exit) is needed to
>> avoid an exception on exit, etc). You can revert to previous recent
>> releases of Gambit to get these working correctly.
>>
>> Sorry but lots of things are in flux right now…
>>
>> Marc
>>
>>
>>
>> > On Jul 19, 2019, at 11:08 AM, Amirouche Boubekki <
>> amirouche.boubekki at gmail.com> wrote:
>> >
>> > Sorry for the multiple emails!
>> >
>> > Le ven. 19 juil. 2019 à 13:38, Marc Feeley <feeley at iro.umontreal.ca> a
>> écrit :
>> > You should look at this recent thread:
>> https://mailman.iro.umontreal.ca/pipermail/gambit-list/2019-July/009103.html
>> >
>> > In the above mail thread, it is described how to compile a scheme file
>> into javascript WITHOUT "Gambit runtime".
>> >
>> > Q: How can I bundle the Gambit runtime into the compiled JavaScript
>> file?
>> >
>> >
>> > Thanks in advance!
>>
>>
>>
>
> --
> Paulo Silva Filho
>
>
--
Paulo Silva Filho
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.iro.umontreal.ca/pipermail/gambit-list/attachments/20190720/b97f6188/attachment.htm>
More information about the Gambit-list
mailing list