[gambit-list] More work on compiled syntax and modules

Matt Hastie matthastie at gmail.com
Tue Oct 22 01:46:34 EDT 2013

Gambit people,

After formerly providing a patch that provides rudimentary support for compiled syntax-case modules with Gambit scheme, I'd like to share some recent thoughts that have emerged since I first posted.

It was not long after the patch was included in the head, when user Sven Hartrumpf reported that the change regressed compiler functionality on 9/21. Since that time, I have been examining ways to make compilation seamless, which I think means:

1. Compilation should regress, wherein one may use gsc -:s -o <output> -exe <file1> <file2> ... without seeing additional warnings, and correctly run the resultant output. This is true for all other forms of compilation, which include both incremental and flat linking. Anything that is 'standards compliant' should compile correctly with just -:s specified, and ultimately link correctly when presented with other -:s generated output. The syntax-case file should never need to be loaded or included for standards compliant mode, other than internally by Gambit via -:s.

2. Executables and their dependent libraries should not depend on $sc-put-cte, or inclusion/loading of any syntax-case code. syntax-case is an expander and is not needed as a dependency.

3. Flat link files that are loaded into the REPL need $sc-put-cte content present, to effect functionality that was enabled by the former patch. The syntactic environment needs extension as objects are loaded.

In the meantime, I went looking in the Chez manual to better identify how it uses syntax-case. I found the definitions for the visit and revisit primitives useful. Dybvig states:

"visit reads the named file, which must contain compiled object code compatible with the current machine type and version, and it runs those portions of the compiled object code that establish compile-time information or correspond to expressions identified as "visit" time by eval-when forms contained in the original source file."

"revisit reads the named file, which must contain compiled object code compatible with the current machine type and version, and it runs those portions of the compiled object code that compute run-time values or correspond to expressions identified as "revisit" time by eval-when forms contained in the original source file."

(from http://www.scheme.com/csug8/system.html)

Thus, from a Gambit schemer's perspective: The visit function is that which can "generate" what we conventionally place in a compilable <file>#.scm file from a source of mixed syntax definitions and program. The revisit function provides the program content, with syntax definitions and module metadata stripped.

When one examines psyntax73.ss, there are two comments provided on the definitions of build-visit-only and build-revisit-only, the two syntaxes that result in the creation of the respective residuals for visit and revisit:

(define-syntax build-visit-only
  ; should mark the result as "visit only" for compile-file
  ; in implementations that support visit/revisit
   (syntax-rules ()
    ((_ exp) exp)))

 (define-syntax build-revisit-only
  ; should mark the result as "revisit only" for compile-file,
  ; in implementations that support visit/revisit
   (syntax-rules ()
    ((_ exp) exp)))

Although it is clear to me that Gambit needs credible implementations for these, my most sophisticated patch to date involves writing two files at compile time, one that contains visit content, and another (the main expander output) that contains the revisit content. I believe that it is necessary to generate both of these files in a single pass, and the generate-id implementation ensures they are uniquely paired.

I foresee the following immediate consequences of this implementation:

1. The generation of files, especially the scheme visit content, that were formerly not compiler outputs is undesirable behavior.

2. The c#expand-source is not capable of compiling a second scheme file as a side-effect of compilation. (... or is re-entrant in the general case?!) To proceed, one would have to wrap the compiler with a pre-processor to expand the visit content and perform separate compilations of both visit and revisit content.

3. In a current patch I'm working on without #2, to support symmetric syntax use between to compile units, one must duplicate source files in a compiler command line, just as needs to feed a worse-is-better UNIX ld implementation e.g.

 gsc -:s -c a b a

The first listing of a generates the visit information for compile unit b, whereas the second listing of a provides a correct revisit of a with the visited definitions of b present.

I suspect that it is thus necessary for a reasonable implementation of visit to transitively visit all dependencies until a graph of visits is closed, prior to compiling a revisit for a compilation unit.

As an aside, all this dogma makes me very happy of the basic choice of Gambit's define-macro, namespaces, and <file>#.scm design. It certainly makes for a beautifully simple scheme implementation, and cuts a lot of needless complexity. Perhaps the original bundling of syntax-case by Marc was an excellent local optima, use of ##include to propagate standards compliant syntax, and use of continued ##namespace for module support.

Moving forwards, with the inadequacies of the dual file implementation, I'd like to consider the possibility of building both visit and revisit information into a single object file. To effect this, I've studied the structure of the generated C, which has ripped by brain apart, and I've also looked at other possible mechanisms, like, for example injecting visit information into the linker info structure. I believe the practice of coupling visit + revisit content in a single object will simplify the compiled syntax-case experience for users, and also provide gsc contract continuance as described above, so that existing gambit codebases don't regress. In terms of Dybvig's comments above: How does the community best think "marking of visit/revisit" should occur so that full syntax-case integration can be achieved, assuming this is indeed a desirable goal!

Kind regards,
Matt Hastie.

More information about the Gambit-list mailing list