quoth Christian:
In our case (parallel gcc), it's problematic since running right below the process limit will make gcc fail when it tries to run subprocesses itself.
quite.
Will you be able to deduce gcc's reason of failure from it's exit status or will you have to parse it's output? Will you patch it to make it take an option telling it to retry forking his own children? Maybe we need a new GNU system to implement this idea right everywhere?
well, it'd be nice. I get your point, but that's exactly what you're doing, right? deducing gcc's limit from it's failure, and tweaking a parameter until you've manually constrained scheme to the point where it runs correctly. It's a fine solution for your problem, but it would scale better (up & down) with a restarting kind of approach.
The semaphore approach has at least the advantage of being efficient in the sense that new tasks are started as soon as older ones finish (no retry/polling needed, so no phases of inactivity).
that's true.
In the end, it's really just a question whether the limit should be in the OS or in the app. Lispers think "the lisp image is the system", and Gambit goes that route insofar as it implements such things as code reloading or threading. So it's reasonable to accept the limit is in this sub-OS, too.
That makes sense. I'll see where I get with my train of thought on this, and keep an eye on your objections. I'm still in the midst of moving from DJB-style systems to a lisp style, so it'll probably take a few attempts to reconcile what I want.
Heh, interesting approach; turning an imperative (mutating state) generator into a stream, right?
Correct. The generator in my program makes an SQL call. I like the metaphor, and I've used it a few times now for ports, etc.
Actually seems to make sense if you've got an imperative generator. (But even on Gambit, call/cc costs a little bit more than a cons or two, so rewriting that into a straight functional stream generator may be a little bit more efficient.)
(BTW you should probably rename |fn| to something else here as well.)
So, something like this:
(define (make-lazy proc) (letrec ((nothing (gensym)) (exit (lambda () nothing)) (me (lambda (proc) (let ((value (proc exit))) (if (eqv? value nothing) '() (cons value (delay (me)))))) )) (delay (me))))
That's just the one call to gensym, which vaguely seems like the most expensive part of doing it this way, and should be as correct. Could write a generator to break it, but I don't suppose it'd happen accidently.
So the order of the values in the output is not the same as in the stream, right? So it's not a "parallel map" really.
Indeed. Sorry about that, I should have mentioned it. The endpoint of all this is a for-each loop that inserts results through mutation. It's SQL, again. So, for my purposes, I wanted it to send back results in any order, but as quickly as possible.
Thanks for the feedback, and sorry for getting off-topic in this thread.
Lang