On 14-Oct-06, at 9:59 PM, Bradley Lucier wrote:
On Oct 4, 2006, at 12:07 PM, Marc Feeley wrote:
P.P.S. I wrote a genetic algorithm to figure out which combination of options gives the best results. Its pretty cool! Here's the code:
Marc:
Yes, this is cool. I'm wondering, can you write this as a distributed app that I can run on my 12 opteron cores over two systems? Most of the time nothing else is running (well, except for Folding@Home), so it would be cool to run something like this. Also, I see you used a population of 25 in the code you sent; when I was doing genetic programming (where you evolve programs) for image processing, I used a population of at least 2,000. So I'd like to try it with a larger population, too.
Brad
Here's the program. You have to run it in the "bench" subdirectory. If you want to parallelize it, you will have to change the procedure "evaluate-group" so that it performs a "parallel map":
(define (evaluate-group pop) (map evaluate-individual pop))
You also have to change the "evaluate" script (in the "bench" subdirectory) so that it doesn't clobber files that are generated by the concurrent invocations of the "evaluate" script. Finally, don't forget to modify the bin/gsc-cc-o script in the installation directory so that only a minimal set of GCC options are used (otherwise the options that the optimization program thinks it is using and the actual ones used by GCC will be different).
Marc
Afficher les réponses par date
I see you've CC'ed this to the Gambit list.
I was hoping you'd have some Termite-like "OK, execute this function on that node and return the result" infrastructure lying around that you could use for this.
Brad
At 14:56 Uhr -0400 15.10.2006, Bradley Lucier wrote:
I see you've CC'ed this to the Gambit list.
I was hoping you'd have some Termite-like "OK, execute this function on that node and return the result" infrastructure lying around that you could use for this.
If you only want to run this on one machine, I'd say there is no need for something like Termite: Gambit threads are enough since the costly work is being done in external tools.
To proove my point, I've put together a parallel map implementation (see attachment; semaphore implementation based on example from the manual). I haven't tested it on Marc's code, since I don't have a multiprocessor machine.
I suggest you increase parallelity-limit to about 12 for your machine.
(BTW I'm unsure how to handle interrupts in such code: if you want to interrupt some waiting (value #XX) call you have entered in the repl, chances are that ctl-c will break in another thread, and I don't know how to interrupt (not terminate) the primordial thread from there and let continue the former thread.)
Christian.
Seems my attachment has been sent with Mac newlines. If you don't want to fiddle with those, you can also fetch it from http://scheme.mine.nu/gambit/scratch/parallel-map/
To proove
prove
parallelity-limit
a better name might be concurrency-limit
Christian.
quoth Christian:
To proove my point, I've put together a parallel map implementation (see attachment; semaphore implementation based on example from the manual). I haven't tested it on Marc's code, since I don't have a multiprocessor machine.
I just made a parallel map myself the other day, but mine's pretty different. So, for the sake of posterity, I've attached it.
0) it's actually a for-each here. Moreover, it's a poorly named for-each that returns the number of iterations.
0.1) Several of these maps operate on streams, I defined those myself, in terms of (fold-anything cons car cdr null? nil list).
1) it's still a bit incomplete. There's no parallel limit -- I'm going to capture out of memory exceptions instead. I don't like the idea of a limit -- I want to go until I run out of juice.
2) Some of the names are bit fanciful and/or incorrect -- it's a weakness of mine. "Incorrect" is a result of being a new scheme programmer, and new to formal computing terms in general; "fanciful" is exhibited by the name "kermit", because it's reporter, you see, and I was wearing a sesame street t-shirt at coding time.
3) I have tested this, and it's performance was entirely satisfactory.
4) the reporting thread exits when it's received as many messages as the count stored in it's specific field. the main loop joins the reporting thread with a timeout, which is my el-cheapo method for dropping pending operations on the floor, since they will errors after some point in time for my purposes.
Criticism welcome.
Lang
At 14:57 Uhr -0400 16.10.2006, Lang Martin wrote:
- it's still a bit incomplete. There's no parallel limit -- I'm going to capture out of memory exceptions instead. I don't like the idea of a limit -- I want to go until I run out of juice.
(Note that the "limit" is not actually limiting any calculation (assuming that futures do terminate); it's just making other futures wait until earlier ones are done. You'll want this when starting external tools, otherwise you'll make your machine's responsitivity suffer.)
- Some of the names are bit fanciful and/or incorrect -- it's a weakness of mine. "Incorrect" is a result of being a new scheme programmer, and new to formal computing terms in general; "fanciful" is exhibited by the name "kermit", because it's reporter, you see, and I was wearing a sesame street t-shirt at coding time.
I'd rename reporter-total! -> reporter-total-set!
The name |fn!| in (define (make-reporting-thread fn! . timeout) ..) looks funny: fn is normally used for "function", and (pure) functions don't have side effects. |proc| might be a better name. But actually I'm not sure what it does. (Well I see that it's return value isn't used at all.)
I have tested this, and it's performance was entirely satisfactory.
the reporting thread exits when it's received as many messages as the count stored in it's specific field. the main loop joins the reporting thread with a timeout, which is my el-cheapo method for dropping pending operations on the floor, since they will errors after some point in time for my purposes.
Criticism welcome.
What does make-lazy do? It can't just be an alternative implementation of a make-promise, since those would take a thunk, not an 1-ary function.
All in all I don't get what it does.
Christian.
quoth Christian:
(Note that the "limit" is not actually limiting any calculation (assuming that futures do terminate); it's just making other futures wait until earlier ones are done. You'll want this when starting external tools, otherwise you'll make your machine's responsitivity suffer.)
yeah, I'm not actually calling external processes. However, my instinct is to still have gambit catch the appropriate exceptions, and go until it runs out of memory or filehandles or processes, or whatever. Typically I set the limits from the outside with a ulimit command before I start the process, giving me a unified interface, and confidence that the limits will be respected. Also makes gambit graceful if you need to scale it down, and run in an evironment with few resources.
From a unix perspective, I think external limits and graceful failure
are the right thing to do.
The name |fn!| in (define (make-reporting-thread fn! . timeout) ..) looks funny: fn is normally used for "function", and (pure) functions don't have side effects. |proc| might be a better name. But actually I'm not sure what it does. (Well I see that it's return value isn't used at all.)
That's it. it's passed to for-each, meaning it's "called for it's side-effects", as R5RS says.
What does make-lazy do? It can't just be an alternative implementation of a make-promise, since those would take a thunk, not an 1-ary function.
ah, right, it does this:
(define (make-lazy fn) (define (me) (delay (call/cc (lambda (exit) (cons (fn (lambda () (exit '()))) (me)))))) (me))
i.e., the fn creates the car part of a nearly infinite series, and can conditionally call the exit thunk to escape the cons & terminate the stream.
That seemed like the right way to do it.
So, overall, it folds over the stream, creating a thread for each element which sends it's result to the mailbox of the reporting thread. The fold counts the number of elements in the stream, and assigns the reporter's specific field that total number. the reporter loops over it's mailbox until it's counted as high as it's specific field, and then it exits.
the primoridal thread join!'s the reporter with a timeout, so that if a thread or two hang for a long period of time, I'll ignore them and quit anyway.
Lang