This post gives some experience on how Scheme is used in the commercial application FMDiff. Some problems we faced when using and integrating Gambit-C are given. In this post I concentrate on the problems we had. The main development of FMDiff is done on a Mac OS X system, so some remarks may be specific to that system.
Background
FMDiff is an application for both Mac OS X and Windows showing the differences between FileMaker databases. It is written in C++ and Scheme where C++ provides the “main” application. The module system used for Scheme is Snow. FMDiff is produced using Xcode 3.2.6 on Mac OS X (Universal application) and Visual C++ 2010 Express on Windows and is linked statically. The application is a 32-bit application.
Currently FMDiff uses Gambit-C as distributed. Gambit-C is not recompiled for FMDiff, it is just linked to the application. Some problems might be solved by “embedding” Gambit-C into FMDiff.
FileMaker databases are read directly by FMDiff i.e. there is no need for an intermediate database representation (a.k.a. “database design report”). FileMaker databases are well suited to Scheme on the logical level. The physical format of a FileMaker database is rather complex and involves a lot of bit-shuffling. This part is handled by C++ for historical reasons.
There are unit-tests for some parts written in both C++ and Scheme. All C and Scheme unit tests are executed in a single application. This auxiliary application (a CLI tool) also runs on both Mac OS X and Windows. Test classes for the CppUnit framework for executing Scheme tests are generated from the test sources.
What does Scheme do?
FMDiff uses Scheme to decode database elements, find differences between them, and convert elements into readable text. The database elements are described in a small DSL easily implemented in Scheme. In the first attempt TinyScheme was used. It turned out that it is easy to integrate (especially creating data structures from C++ for Scheme is simple), but execution of Scheme is not that fast.
Later on we switched to Gambit-C in order to get better performance as far as executing Scheme code is concerned. With TinyScheme formatting a report sometimes took more time than reading the database. This could be solved rather easily with Gambit-C. Integrating C++ and Gambit-C seemed easy as well as the interface between Gambit-C and C is documented.
General problems
There is a standard for Scheme, but as far as a module system is concerned there is no such thing. R6RS is not officially supported by Gambit (I had no time to look into the available implementation). This makes it hard to test Scheme code on other systems (especially DrRacket). Snow does not work with recent MzScheme versions.
FMDiff’s main module is written in C++ mainly due to the fact that there is no GUI interface for Scheme. I know about JazzScheme, but on Mac OS X it does use X-Windows which is not an option as we want to use the native controls. One could wrap wxWidgets used for FMDiff’s interface into a Gambit module, but wxWidgets is quite big and this seems to be lots of work.
Support for multi-core systems in Scheme is problematic. Major systems support threading, but all Scheme-threads usually run within a single OS-thread. If an application would benefit from multiple cores, this is hard to accomplish in Scheme. For FMDiff this is currently not a big problem.
Problems with Gambit-C
Gambit-C provides a good interface for interfacing C and Scheme. When integrating C++ and Scheme sometimes more information is needed than is available readily from the manual. For instance how to handle class destructors is found in a post in the mailing list but not in the manual. Details about the C-Scheme-interface are however complex and would benefit from a more detailed description in the manual. One example for this is Gambit's handling of types and the macros used for procedure-entry and -exit.
The Gambit-C user manual is pretty good but as indicated needs some further improvement. Information found in the mailing list (e.g. define-type) should be added to the manual. Routines that are documented elsewhere (e.g. threads) should be documented in the manual or there should be an appropriate reference. The good thing is that one can find good information already but it is inconvenient.
Integrating Gambit-C with the IDEs used is somewhat cumbersome. The “main” file needs to be compiled twice: once normally and once with the “-link” option. This is rather hard to set up within an IDE. We use makefiles to compile Scheme to C because this allows also to handle module dependencies in a decent manner. On Mac OS X Gambit’s libraries contain only a single architecture and need to be combined for a Universal application.
Compilation of Scheme is a two-step process: Gambit produces C sources that are handled by the C compiler. Compiling the C source code produced by Gambit might take quite some time especially for a release version that has all options set to “fast”. A backend creating object files directly would speed up building considerably.
Debugging FMDiff is problematic as GDB (debugger on Mac OS X) does know nothing about structures and data managed by Scheme. When entering the Scheme world there is no stack-trace of C++ anymore. Checking variable values is cumbersome (use display and recompile). The C code produced by Gambit is unreadable—at least for me. Stepping through Scheme code in GDB does not work. (I am aware of JazzScheme that allows single-stepping through Scheme code, but on my machine I was not able to get it to work.)
The garbage collector is tricky to handle. If something is allocated in C++ one has to be careful so that objects do not get relocated while building the structure (use still objects). Bugs in this area are rather hard to find as they might materialise at some different point in the code.
While the interface between Scheme and C seems to be rather easy from the documentation, it is not so easy to get it right as far as performance is concerned. The interface converts parameters (e.g. strings to character arrays) on the fly for each call and these conversions add up for frequently called routines. It is best to have a decent granularity for C-functions called by Scheme as too fine-grained functions cause considerable overhead. However to find the sweet spot is hard (and still an issue).
The size of objects in Gambit is somewhat limited. For instance a string on a 32-bit system is limited to about 4 M characters (16 MB, one character in Gambit has 4 bytes). Very long strings therefore are a problem. This could be changed/improved by compiling Gambit appropriately.
Conclusion
Despite the problems listed above we were able to write a cross-platform application partially in Scheme whose performance is quite acceptable. Gambit-C is a decent system to work with and it produces fast code. Gaps in the documentation can usually be filled by going to Gambit’s mailing list. In fact we want to move code from C++ to Scheme so that more development can be done using the REPL.
In order to benefit most from Scheme it is important to have a decent set of libraries (written in C++) with the right granularity. With Gambit it is possible to have some of these libraries written in Scheme because generated code is really fast. For an existing code base it is sometimes hard to implement this set of libraries for historical reasons/accidents.
Regards, Jürgen