[gambit-list] Using Gambit Modules with BlackHole

Benjohn Barnes benjohn at fysh.org
Fri Oct 14 04:43:51 EDT 2011


These are two emails Per Eckerdal sent to me that I found very useful for using BlackHole. I proposed turning them in to a document but then didn't, and now I don't remember the details well enough to do that. I think the major confusion I had was understanding about how interactive development made things different from compiling up a whole project and running that.

I hope they'll be useful.

Many thanks,
	Benjohn

Begin forwarded message:
> From: Per Eckerdal <per.eckerdal at gmail.com>
> Date: 15 June 2011 23:18:38 GMT+01:00
> To: Benjohn Barnes <benjohn at fysh.org>
> Subject: Re: Black hole on OS X
> 
> On Wednesday, 15 June 2011 at 16:08, Benjohn Barnes wrote:
> 
>> Ah, module-compile-to-standalone doesn't leave the intermediate .o files in place? Is there an equivalent to it that does the same thing, but leaves the intermediate files available to speed up future builds?
> No, although it shouldn't be too hard to implement if you'd want it.
> 
> But if you ask this question, you're probably not using Black Hole in the way it's intended to be used. module-compile-to-standalone is intended to be used only when shipping a binary, it's not necessary to use it when developing.
> 
> A normal BH workflow is to compile all modules separately to .o1 files. When this is done, firing up the application is as easy as opening the REPL and (import)ing the main module.
> 
> When you decide that you want to change something, you remove the .o1 file for the module to work on (module-clean! does this).
> 
> One possible way forward would be to make the necessary changes, and then re-(import) the module. The module you're working on now will now be run in interpreted mode, but that's okay (unless that module uses the FFI), because the interpreter gives better debugging info. You could then change the module file again and re-(import) it as many times as is needed.
> 
> When you are done with modifying the module, you can re-compile it and start working somewhere else. When I use BH, I usually make sure to clean all of the modules I might be working on, leaving only parts that I won't be touching compiled.
> 
> Another useful feature is the (module) form. To use it, you type (module [module identifier]) into the REPL. This "enters" that module, meaning that if you evaluate a function or macro definition in the REPL, it will be run in the context of that module. To get back to the top level, issue (module #f) (or just (module), I don't remember)
> 
> (module) enables interactive development: You could open a particular module, change a function, evaluate its definition (C-c C-e if you use Emacs) and then use it immediately. That way you can update the system instantly, without ever having to compile or even load entire files. (Although in practise, you will probably want to restart Gambit every once in a while to make sure that the code still works on startup, and is not dependent on some old lingering things that has been removed from the source code.)
> 
> Hope this helps.
> 
> /Per


Begin forwarded message:
> From: Per Eckerdal <per.eckerdal at gmail.com>
> Date: 16 June 2011 22:58:03 GMT+01:00
> To: Benjohn Barnes <benjohn at fysh.org>
> Subject: Re: Black hole on OS X
> 
> On Wednesday, 15 June 2011 at 11:10, Benjohn Barnes wrote:
> 
>> Ah, module-compile-to-standalone doesn't build any intermediate .o files. Here's some quoted bsc interaction with comments from me…
>> 
>> [snippets from a REPL session]
> Ahh.. I see what you mean.
> 
> Yes, this is expected behavior. The two main motivations behind this is:
> 
> 1) .o files are not the same as the .o[n] files produced by Gambit:
> 
> .o files are (in Gambit and in a typical Unix C programming environment) "object files", files that contain compiled code, that needs linking (by a tool like ld) before use.
> 
> The .o[n] files produced by Gambit are not object files, they are dynamic libraries. I don't know why they are called .o[n]; the normal thing to call this type of file is .so in Unix, or .dll in Windows. So the .o[n] files are loaded into Gambit directly with dlopen.
> 
> There is (afaik) no way to "link" .o[n] files into one executable.
> 
> In theory, it might be possible to use the same .c files to compile to .o1 files and to a standalone executable, but that would be rather complicated to do, and it wouldn't save you much time. (Most of the time compiling is spent by GCC).
> 
> 2) The purpose of the .o[n] files and the compiled standalone is quite different:
> 
> .o[n] files are meant to be used in interactive development, and are also suitable for deploying an app when there is no problem to require the production environment to install Gambit and Black Hole. Servers are probably the main example of this use case.
> 
> module-compile-to-standalone is the implementation of the fact that a Black Hole module, by design, can be compiled, along with all of its dependencies /and then be stripped of all information and overhead that Black Hole imposes/. A standalone executable produced by module-compile-to-standalone doesn't contain any macros, and Black Hole doesn't need to be shipped with it in any form.
> 
> 
> There is also (I think) a function called something like module-compile-bunch, that compiles more than one module at once into one big .o1 file, and leaves references along the .scm files that are compiled into it. This might be what you want to do. The main use case for that function is to be able to compile one library (eg a web framework or a standard library) into one compiled file instead of lots of small ones. This reduces code size (I think) and loading times.
> 
> These quirks (module-compile-to-standalone vs module-compile! vs module-compile-bunch! vs interpreted mode, combined with the confusion around where the .o1 files are and why) are a side effect partially because of the suboptimal integration with Gambit, but mainly because it's difficult to hide the complexity of the compilation process (source code -[BH]-> macro expanded code -[Gambit]-> C code -[GCC]-> .o file -[ld]-> dynamic library or executable file). It would be possible to do this whole thing much nicer if Gambit supported incremental compilation like Ikarus or Chez.
> 
> In fact, one important reason to why I am considering to just roll my own Lisp instead of building something atop Gambit is that I have spent so much time trying to streamline this complicated build process, so much time that I think that if I would have started writing an incremental compiler from scratch instead, I would have gotten considerably further than where I am now.
> 
> I can't get away from the thought that building a multi-KLOC system on top of a compiler just to try to hide the limitations of its compilation process is a masochistically pointless exercise: Using appropriate tools (LLVM, libeio, libev etc), and taking the time to read scientific papers on how to implement specific parts of the system, it should be possible to write an entire compiler from scratch in a very small multiple of the code size of Black Hole. And then it would be possible to do it right in the first place instead of endlessly working around strange limitations and constraints.
> 
> That being said, BH as it is now is sufficiently good to write fairly large software systems. I have a friend who uses BH as an important foundation of a technology startup. He's worked on that startup for a while now, so I assume that the code base is at least some multiple of 10KLOC. (I say at least, because the previous system he wrote is >3MB (!) of Java code) He rarely has problems with BH, even though he uses a relatively old version of it.
> 
> Cheers,
> 
> Per

-- 
benjohn at fysh.org - Twitter @benjohnbarnes - Skype benjohnbarnes - Mobile +44 (0) 7968 851 636




More information about the Gambit-list mailing list