I'm trying to make a largish number of DNS MX type queries, thread-safely. I made a djbdns FFI, that works great. It's not thread safe -- the library internally uses global state variables.
So.
Started working on res_query, which seems like it's probably thread safe. However, res_query returns it's answer as a buffer of some size of u_char's.
What do I do with those? There's the type conversion thing in the manual for ECDIC characters which is incomplete, and it seems as though the unicode support in Gambit is implemented through byte ports.
Should I create a port? If so, can someone point me to a bit of the source code that'll get me started with that?
Should I convert the answer string to some other format? I'd like to preserve any unicode characters that return from DNS.
Should I return a byte vector from the function and run some scheme-land conversion on it to create a string? that'd be nice, I'm not really a C programmer.
If someone can light my way, I'd appreciate it.
Lang
Afficher les réponses par date
If you want to have multiple scheme threads call blocking C library functions concurrently, not only the C library itself has to be thread safe, but you also have to create a setup to dispatch the scheme threads to OS threads (pthreads). Scheme threads are all living within one OS threads, and calling into C without any further help will block other scheme threads for the duration of the call. So unless you're going to build such a dispatcher, there's no point in making the C library reentrant, since everything is actually running in one single OS thread.
So if you want to run DNS queries from multiple scheme threads in parallel, you have these options:
- create said dispatcher, using either pthreads underneath (and probably pipes for synchronisation), or pipes and fork. (At least for the first variant I suggest using the cj-posix library I've announced here some time ago (ask me for the newest version if you want to use it, it now has more functionality). For the second you can also use open-process and an external binary.) Thread/child pooling will be necessary to make it efficient. - look at what the Gambit host-info.. builtins are missing, and try to improve on that by patching Gambit. - port the DNS client library from "scheme underground" to Gambit (http://www.scsh.net/resources/sunet.html)
The second option is probably the best for us all :). The third may not be easy since the sunet libraries generally have many dependancies on other scsh stuff (but it would be nice to have a port of most of that stuff on Gambit some time). The first one may be the easiest if you don't like to delve into too many lowlevel details.
I've recently tried to start playing with creating/accessing pthreads from Gambit, but had to stop because of other work; I may return to this sooner or later.
What do I do with those? There's the type conversion thing in the manual for ECDIC characters which is incomplete,
I guess you mean EBCDIC. Why would this play a role with DNS, is DNS using EBCDIC, or are you on a mainframe?
and it seems as though the unicode support in Gambit is implemented through byte ports.
Hm, I'd say this is "wrong"; Gambit represents chars in the scheme world as 4-byte (or optionally (at compile time) 2-byte (which restricts it to a sub-range of current unicode)) values, and has to convert to other encodings for port I/O.
From looking at the sources, I'd say there's no EBCDIC conversion in the Gambit runtime except that hooks are ready for it (macros in lib/os.h).
Should I create a port? If so, can someone point me to a bit of the source code that'll get me started with that?
("No". But you may need ports to communicate readyness (and the address of some struct) to another pthread when implementing concurrency with pthreads.)
Should I convert the answer string to some other format? I'd like to preserve any unicode characters that return from DNS.
For converting between UTF-8 encoded byte arrays and Scheme strings just use the |utf8-string| type with c-lambda.
Christian.
quoth Christian:
and calling into C without any further help will block other scheme threads for the duration of the call.
ah. That sucks. I think I was under the impression that since you can call/cc around C functions, the thread scheduler would be able to do the same trick it does to interrupt scheme continuations.
So unless you're going to build such a dispatcher, there's no point in making the C library reentrant, since everything is actually running in one single OS thread.
Well, Bernsteins library has an asychronous mode -- his preferred way of dealing w/ threads.
- create said dispatcher, using either pthreads underneath (and
probably pipes for synchronisation), or pipes and fork.
The pthreads thing kind of scares me -- the folks who grok unix best (DJB being close to foremost) avoid them in favor of processes and pipes. My temporary solution does (run-process ...).
- look at what the Gambit host-info.. builtins are missing, and try
to improve on that by patching Gambit.
So far seems to only use gethostbyname(), which is a level of abstraction too high.
- port the DNS client library from "scheme underground" to Gambit (http://www.scsh.net/resources/sunet.html)
The second option is probably the best for us all :).
I might argue that this way is best.
Actually, I'd probably argue that we should do an s-expr C syntax, and compile down the FFI stuff. And then make new backends for other schemes. And then tell the python folks to kiss it.
The first one may be the easiest if you don't like to delve into too many lowlevel details.
Alas, I might punt, and do just that.
I guess you mean EBCDIC. Why would this play a role with DNS, is DNS using EBCDIC, or are you on a mainframe?
I do, and I thought that the OS 10.4 res_query might return it. Not working on a mainframe, unless you count the macbook, which would be cheeky. Wikipedia to the rescue! Whoa, I must have been looking at entirely the wrong page.
Should I create a port? If so, can someone point me to a bit of the source code that'll get me started with that?
("No". But you may need ports to communicate readyness (and the address of some struct) to another pthread when implementing concurrency with pthreads.)
seems to be a good interface if I go for the DJB async query. Can someone point me to the contruction of a port?
For converting between UTF-8 encoded byte arrays and Scheme strings just use the |utf8-string| type with c-lambda.
brilliant. Thanks.
Thanks very much, that was a thoughtful response.
At 12:39 Uhr -0400 13.10.2006, Lang Martin wrote:
Well, Bernsteins library has an asychronous mode -- his preferred way of dealing w/ threads.
Ah, I didn't know that. This makes things appear in a different light; the question will now be how to get notified when the result is ready, and make the scheme thread continue timely (probably requires hooking the library into the Gambit event loop (someone has made available a patch for this already on this list), and somehow into the Gambit scheduler to make it instant). The easiest solution may be polling with an exponentially increasing sleep interval, that way it's still scalable in the number of threads, but the answer will be delayed a little bit. (I need to take a look at djbdns.)
Should I create a port? If so, can someone point me to a bit of the source code that'll get me started with that?
("No". But you may need ports to communicate readyness (and the address of some struct) to another pthread when implementing concurrency with pthreads.)
seems to be a good interface if I go for the DJB async query.
I'm not sure that helps you anything; pipes are nice if you're waiting for another process or pthread (which really runs concurrently) to reply. But with an async query you'll have to "run the other end yourself", so there's no point using a pipe.
Can someone point me to the contruction of a port?
cj-posix has it (or look at the discussion from July 29 till Aug 2). Maybe I should do a new release.
Thanks very much, that was a thoughtful response.
You're welcome
Christian.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 13-Oct-06, at 12:39 PM, Lang Martin wrote:
Actually, I'd probably argue that we should do an s-expr C syntax, and compile down the FFI stuff. And then make new backends for other schemes. And then tell the python folks to kiss it.
That's already builtin!
% gsi Gambit Version 4.0 beta 20
\ for (int i=1; i<=10; i*=2) pp(i);
1 2 4 8
'\ for (int i=1; i<=10; i*=2) pp(i);
(six.for (six.define-variable (six.identifier i) int () (six.literal 1)) (six.x<=y (six.identifier i) (six.literal 10)) (six.x*=y (six.identifier i) (six.literal 2)) (six.call (six.identifier pp) (six.identifier i)))
Marc