[gambit-list] Size of the universal libraries (eg. lib/_gambit.php)
feeley at iro.umontreal.ca
Fri Jun 9 16:18:08 EDT 2017
Hello Ben. With the help of Nicolas and Abdelhakim, I am currently working on improving the compactness of the programs generated by the universal backend. In the previous linking model, which is the only available currently, the Gambit runtime library was compiled separately into a file (e.g. lib/_gambit.php) and then the main program was compiled and linked with this file. “Linking” is basically just concatenating the code generated for the main program and the Gambit runtime library. This means that all the functionality of the library is included even if it is not used, yielding enormous files. This is OK for the C backend because the runtime library is about 7 MB of machine code, and that’s not a big problem for the operating system to load (about 3 ms of system time on my machine to load the interpreter for example).
The hidden constants are quite different for the targets of the universal backend, and in the particular case of PHP, there is so much code (more than 400 KLOC) that it exceeds some of the PHP interpreter's limits. I haven't investigated whether the limits can be raised or the code compacted (for example identifiers are not currently minimized), because really a smarter linker is needed.
But a smart linker is not easy to implement because it is hard to parcel out, for a given module, which parts are required and which ones aren’t. The smartest linker is actually the compiler itself, which can do a control-flow analysis to determine which functions are reachable from the program’s “main”.
So we currently are exploring the use of the declaration “optimize-dead-definitions” to minimize the size of the generated code. The idea is to prefix the main program with a (declare (optimize-dead-definitions)) and (include "lib/_univlib.scm") and then let the Gambit compiler eliminate all definitions that are not relevant to the main program.
In principle this should give the best results, and in some cases a mere 10-20 KB is generated for programs requiring few features. But there is an issue. Some functionalities have an indirect dependency on eval, and as soon as eval is “required” pretty much all of the library gets dragged in. Think of exceptions… Normally when an exception is raised the exception handler is called, but if the exception isn’t handled, the fallback is to start a REPL to debug the error, and this drags in eval and read and write. A similar situation happens for arithmetic… (+ i 1) might overflow into bignums, dragging in the bignum support. Or number->string (used by write) might have to convert a bignum, so it needs bignum division which uses a complex FFT multiplication algorithm (more importantly lots of code) in certain cases.
So part of the work is to reorganize the Gambit runtime library to remove gratuitous dependencies (already a few uses of eval have been replaced by simpler functionality). Another avenue we are exploring is the use of compiler flags to manually “pick and choose” the features that are required of the Gambit library. For instance, we may add some kind of flag to say “don’t start a REPL on uncaught exceptions… just exit with an error code” and “don’t use complex bignum algorithms” and “don’t support eval”. We don’t yet know what flags are needed to give the developer enough control to minimize the generated code.
Nicolas and Abdelhakim will be looking into this, so if you have any suggestions please contact them or me or the mailing list.
> On Jun 8, 2017, at 12:03 PM, Nicolas Hurtubise <nicolas.k.hurtubise at gmail.com> wrote:
> Hi !
> We're currently working on improving the universal backend, there are a
> few opened issues on this subject : https://github.com/gambit/gambit/issues
>> 1. AFAIK you have to cat 3 files code.php (compiled) code_.php (linked)
>> and lib/_gambit.php (precompiled) into a standalone script. The step
>> where you compile to code.php (gsc -:=. -target php code.scm) gives me a
>> code.o1 file. Is this a bug?
> You can now do :
> gsc -target php -exe code.scm
> Which will cat the required files in a single executable php script.
> (You still need to precompile lib/_gambit.php)
>> 2. The precompiled library (_gambit.php) is 15 Mb of size. So the first
>> thing I've tried to do is to reduce the size of this file.
> Marc is currently working on this issue :
> To make a long story short, there already is a procedure in the
> compilation process that can drastically shrink the size of the compiled
> programs by including only what's required, but as soon as you include
> (eval), you have no choice but to compile everything...
> For various reasons, (eval) tends to be included more often than you'd
> With a few tweaks, it is possible to get small executables for small
> programs (~0.01M for a Hello World instead of 15M), but it's not yet
> convenient to do. Maybe Marc can tell you more about that if you are
> Gambit-list mailing list
> Gambit-list at iro.umontreal.ca
More information about the Gambit-list