[gambit-list] Scheme in a Real-World Application

Jürgen Geßwein juergen at gesswein.net
Thu Apr 28 10:19:23 EDT 2011


This post gives some experience on how Scheme is used in the commercial
application FMDiff.  Some problems we faced when using and integrating
Gambit-C are given.  In this post I concentrate on the problems we had.  The
main development of FMDiff is done on a Mac OS X system, so some remarks may
be specific to that system.

Background

FMDiff is an application for both Mac OS X and Windows showing the differences
between FileMaker databases.  It is written in C++ and Scheme where C++
provides the “main” application.  The module system used for Scheme is Snow. 
FMDiff is produced using Xcode 3.2.6 on Mac OS X (Universal application) and
Visual C++ 2010 Express on Windows and is linked statically.  The application
is a 32-bit application.

Currently FMDiff uses Gambit-C as distributed.  Gambit-C is not recompiled for
FMDiff, it is just linked to the application.  Some problems might be solved
by “embedding” Gambit-C into FMDiff.

FileMaker databases are read directly by FMDiff i.e. there is no need for an
intermediate database representation (a.k.a. “database design report”). 
FileMaker databases are well suited to Scheme on the logical level.  The
physical format of a FileMaker database is rather complex and involves a lot
of bit-shuffling.  This part is handled by C++ for historical reasons.

There are unit-tests for some parts written in both C++ and Scheme.  All C and
Scheme unit tests are executed in a single application.  This auxiliary
application (a CLI tool) also runs on both Mac OS X and Windows.  Test classes
for the CppUnit framework for executing Scheme tests are generated from the
test sources.

What does Scheme do?

FMDiff uses Scheme to decode database elements, find differences between them,
and convert elements into readable text.  The database elements are described
in a small DSL easily implemented in Scheme.  In the first attempt TinyScheme
was used.  It turned out that it is easy to integrate (especially creating
data structures from C++ for Scheme is simple), but execution of Scheme is not
that fast.

Later on we switched to Gambit-C in order to get better performance as far as
executing Scheme code is concerned.  With TinyScheme formatting a report
sometimes took more time than reading the database.  This could be solved
rather easily with Gambit-C.  Integrating C++ and Gambit-C seemed easy as well
as the interface between Gambit-C and C is documented.

General problems

There is a standard for Scheme, but as far as a module system is concerned
there is no such thing.  R6RS is not officially supported by Gambit (I had no
time to look into the available implementation).  This makes it hard to test
Scheme code on other systems (especially DrRacket).  Snow does not work with
recent MzScheme versions.

FMDiff’s main module is written in C++ mainly due to the fact that there is no
GUI interface for Scheme.  I know about JazzScheme, but on Mac OS X it does
use X-Windows which is not an option as we want to use the native controls. 
One could wrap wxWidgets used for FMDiff’s interface into a Gambit module, but
wxWidgets is quite big and this seems to be lots of work.

Support for multi-core systems in Scheme is problematic.  Major systems
support threading, but all Scheme-threads usually run within a single
OS-thread.  If an application would benefit from multiple cores, this is hard
to accomplish in Scheme.  For FMDiff this is currently not a big problem.

Problems with Gambit-C

Gambit-C provides a good interface for interfacing C and Scheme.  When
integrating C++ and Scheme sometimes more information is needed than is
available readily from the manual.  For instance how to handle class
destructors is found in a post in the mailing list but not in the manual. 
Details about the C-Scheme-interface are however complex and would benefit
from a more detailed description in the manual.  One example for this is
Gambit's handling of types and the macros used for procedure-entry and -exit.

The Gambit-C user manual is pretty good but as indicated needs some further
improvement.  Information found in the mailing list (e.g. define-type) should
be added to the manual.  Routines that are documented elsewhere (e.g. threads)
should be documented in the manual or there should be an appropriate
reference.  The good thing is that one can find good information already but
it is inconvenient.

Integrating Gambit-C with the IDEs used is somewhat cumbersome.  The “main”
file needs to be compiled twice: once normally and once with the “-link”
option.  This is rather hard to set up within an IDE.  We use makefiles to
compile Scheme to C because this allows also to handle module dependencies in
a decent manner.  On Mac OS X Gambit’s libraries contain only a single
architecture and need to be combined for a Universal application.

Compilation of Scheme is a two-step process: Gambit produces C sources that
are handled by the C compiler.  Compiling the C source code produced by Gambit
might take quite some time especially for a release version that has all
options set to “fast”.  A backend creating object files directly would speed
up building considerably.

Debugging FMDiff is problematic as GDB (debugger on Mac OS X) does know
nothing about structures and data managed by Scheme.  When entering the Scheme
world there is no stack-trace of C++ anymore.  Checking variable values is
cumbersome (use display and recompile).  The C code produced by Gambit is
unreadable—at least for me.  Stepping through Scheme code in GDB does not
work. (I am aware of JazzScheme that allows single-stepping through Scheme
code, but on my machine I was not able to get it to work.)

The garbage collector is tricky to handle.  If something is allocated in C++
one has to be careful so that objects do not get relocated while building the
structure (use still objects).  Bugs in this area are rather hard to find as
they might materialise at some different point in the code.

While the interface between Scheme and C seems to be rather easy from the
documentation, it is not so easy to get it right as far as performance is
concerned.  The interface converts parameters (e.g. strings to character
arrays) on the fly for each call and these conversions add up for frequently
called routines.  It is best to have a decent granularity for C-functions
called by Scheme as too fine-grained functions cause considerable overhead. 
However to find the sweet spot is hard (and still an issue).

The size of objects in Gambit is somewhat limited.  For instance a string on a
32-bit system is limited to about 4 M characters (16 MB, one character in
Gambit has 4 bytes).  Very long strings therefore are a problem.  This could
be changed/improved by compiling Gambit appropriately.

Conclusion

Despite the problems listed above we were able to write a cross-platform
application partially in Scheme whose performance is quite acceptable. 
Gambit-C is a decent system to work with and it produces fast code.  Gaps in
the documentation can usually be filled by going to Gambit’s mailing list.  In
fact we want to move code from C++ to Scheme so that more development can be
done using the REPL.

In order to benefit most from Scheme it is important to have a decent set of
libraries (written in C++) with the right granularity.  With Gambit it is
possible to have some of these libraries written in Scheme because generated
code is really fast.  For an existing code base it is sometimes hard to
implement this set of libraries for historical reasons/accidents.

Regards, Jürgen


More information about the Gambit-list mailing list