Hi,
Quite some time ago I managed to successfully interface with a C library using FFI. Now I'm having another go, but this time I want to connect to it using native Scheme code. The C library is the interface to a select based TCP socket server, so I figure I should be able to connect to it directly using open-tcp-client and forgo the whole FFI thing.
The way the C library works is that a C struct is sent to the receiving socket first so that it knows the length of the message buffer that will follow and some other information specific to the socket server's operation. This is immediately followed up by the message buffer which is in binary format.
So far... I think I've managed to work out that I should be using u8vector data structure for both the C struct and the message buffer, although I don't yet understand what kind of encoding is going on in there. That will come after I've looked at some examples.
My plan is to extend the existing web-server.scm example to produce a web app that talks to it's relational database using native Scheme. I'm going down this route because I want to extend web-server.scm using the GSI. If I use the FFI I'm restricted to the GSC which prevents the whole interactive development scenario I want where I can keep the browser running, the database running and just stop and start the GSI as I hack code.
What I want to know first though is if I use open-tcp-client inside Gambit's threads will the native Scheme calls out to the external socket server block? If they don't then am I on the right track to use u8vector?
Thanks,
Rob.
Afficher les réponses par date
On Jul 7, 2014 5:54 PM, "Rob Kellock" sales@creditscore.co.nz wrote:
Hi,
Quite some time ago I managed to successfully interface with a C library using FFI. Now I'm having another go, but this time I want to connect to it using native Scheme code. The C library is the interface to a select based TCP socket server, so I figure I should be able to connect to it directly using open-tcp-client and forgo the whole FFI thing.
The way the C library works is that a C struct is sent to the receiving socket first so that it knows the length of the message buffer that will follow and some other information specific to the socket server's operation. This is immediately followed up by the message buffer which is in binary format.
So far... I think I've managed to work out that I should be using u8vector data structure for both the C struct and the message buffer, although I don't yet understand what kind of encoding is going on in there. That will come after I've looked at some examples.
My plan is to extend the existing web-server.scm example to produce a web app that talks to it's relational database using native Scheme. I'm going down this route because I want to extend web-server.scm using the GSI. If I use the FFI I'm restricted to the GSC which prevents the whole interactive development scenario I want where I can keep the browser running, the database running and just stop and start the GSI as I hack code.
What I want to know first though is if I use open-tcp-client inside Gambit's threads will the native Scheme calls out to the external socket server block? If they don't then am I on the right track to use u8vector?
Yes, using u8vector is the way to go imho.
I find it strange -- and a misfeature -- that the TCP server accepts a C struct, because depending on what architecture the server is running on and even what compiler was used to build it, the length, endianness, and padding of the fields in the struct can vary. So you will have to take those into account when packing a u8vector full of bytes to send to this server.
And you can certainly use FFI functions from within gsi. Simply compile just the FFI wrapper with gsc into a loadable module, then load it with LOAD into gsi while you develop the upper level bits in Scheme.
Thanks Jeff,
Still gotta ask though, if I use a loadable module, will it block? I had a bad experience a number of years ago where I developed an application using Zope, a Python web server, calling out to this same C database socket library using a Python-C bridge called a *.pyd Everything was going swimmingly until I put it into production and discovered that the Python-C bridge blocked Zope. Problem there was the Python GIL, there was no work around and the project had to be cancelled. Ever since then I've been very wary of threads.
Here too in Gambit aren't the threads green threads rather than OS threads? I don't understand continuations yet, so I'm unsure whether they are being used somehow in Gambit to keep the green threads from being blocked, but I am concerned that as soon as the threads try to communicate with the outside world, Gambit will block.
The TCP server, uses htonl, htons and ntohl, ntohs on the C struct, so I guess that covers the endianess etc you mentioned.
Cheers,
Rob.
On 8/07/2014 10:24 a.m., Jeff Read wrote:
On Jul 7, 2014 5:54 PM, "Rob Kellock" <sales@creditscore.co.nz mailto:sales@creditscore.co.nz> wrote:
Hi,
Quite some time ago I managed to successfully interface with a C library using FFI. Now I'm having another go, but this time I want to connect to it using native Scheme code. The C library is the interface to a select based TCP socket server, so I figure I should be able to connect to it directly using open-tcp-client and forgo the whole FFI thing.
The way the C library works is that a C struct is sent to the receiving socket first so that it knows the length of the message buffer that will follow and some other information specific to the socket server's operation. This is immediately followed up by the message buffer which is in binary format.
So far... I think I've managed to work out that I should be using u8vector data structure for both the C struct and the message buffer, although I don't yet understand what kind of encoding is going on in there. That will come after I've looked at some examples.
My plan is to extend the existing web-server.scm example to produce a web app that talks to it's relational database using native Scheme. I'm going down this route because I want to extend web-server.scm using the GSI. If I use the FFI I'm restricted to the GSC which prevents the whole interactive development scenario I want where I can keep the browser running, the database running and just stop and start the GSI as I hack code.
What I want to know first though is if I use open-tcp-client inside Gambit's threads will the native Scheme calls out to the external socket server block? If they don't then am I on the right track to use
u8vector?
Yes, using u8vector is the way to go imho.
I find it strange -- and a misfeature -- that the TCP server accepts a C struct, because depending on what architecture the server is running on and even what compiler was used to build it, the length, endianness, and padding of the fields in the struct can vary. So you will have to take those into account when packing a u8vector full of bytes to send to this server.
And you can certainly use FFI functions from within gsi. Simply compile just the FFI wrapper with gsc into a loadable module, then load it with LOAD into gsi while you develop the upper level bits in Scheme.
On Mon, Jul 7, 2014 at 10:15 PM, Rob Kellock sales@creditscore.co.nz wrote:
Thanks Jeff,
Still gotta ask though, if I use a loadable module, will it block? I had a bad experience a number of years ago where I developed an application using Zope, a Python web server, calling out to this same C database socket library using a Python-C bridge called a *.pyd Everything was going swimmingly until I put it into production and discovered that the Python-C bridge blocked Zope. Problem there was the Python GIL, there was no work around and the project had to be cancelled. Ever since then I've been very wary of threads.
Here too in Gambit aren't the threads green threads rather than OS threads? I don't understand continuations yet, so I'm unsure whether they are being used somehow in Gambit to keep the green threads from being blocked, but I am concerned that as soon as the threads try to communicate with the outside world, Gambit will block.
So far as I know, synchronous I/O with the server will block Gambit because yes -- Gambit uses green threads and switching between them is done inside the select loop. If the library provides the capability to do asynchronous I/O with callbacks, then you can get around the blocking that way.
The TCP server, uses htonl, htons and ntohl, ntohs on the C struct, so I guess that covers the endianess etc you mentioned.
Not really. Doesn't specify the padding. Also, there's the fact that an 'int' can be of different sizes depending on CPU architecture *and compiler*.
The *right* way to specify wire protocols and file formats with aggregate blocks of data is to explicitly specify the format down to the byte level, including endianness for multi-byte integer and floating-point values.
That people still do *not* do this is quite alarming.
Rob, I don't know if this contributes to answering your question but, (1) you can get the OS FD:s out of the Gambit TCP server and client ports, (2) note that (at least the client) ports' FD:s are set up in asynchronous mode at the level of OS FD so your reads & writes will be nonblocking unless you change that setting and supposedly you don't want to do that, and (3) you can have Gambit do typical select() work for you on port objects using the ##wait-for-input! & -output! procedures i.e. your Gambit and C code can cooperate this way i.e. Gambit code waits for FD readiness and then your C code does the IO. Last, (4) there's some procedure in Gambit to import an OS FD and make a port object out of it. Mikael
2014-07-07 23:32 GMT+02:00 Rob Kellock sales@creditscore.co.nz:
Hi,
Quite some time ago I managed to successfully interface with a C library using FFI. Now I'm having another go, but this time I want to connect to it using native Scheme code. The C library is the interface to a select based TCP socket server, so I figure I should be able to connect to it directly using open-tcp-client and forgo the whole FFI thing.
The way the C library works is that a C struct is sent to the receiving socket first so that it knows the length of the message buffer that will follow and some other information specific to the socket server's operation. This is immediately followed up by the message buffer which is in binary format.
So far... I think I've managed to work out that I should be using u8vector data structure for both the C struct and the message buffer, although I don't yet understand what kind of encoding is going on in there. That will come after I've looked at some examples.
My plan is to extend the existing web-server.scm example to produce a web app that talks to it's relational database using native Scheme. I'm going down this route because I want to extend web-server.scm using the GSI. If I use the FFI I'm restricted to the GSC which prevents the whole interactive development scenario I want where I can keep the browser running, the database running and just stop and start the GSI as I hack code.
What I want to know first though is if I use open-tcp-client inside Gambit's threads will the native Scheme calls out to the external socket server block? If they don't then am I on the right track to use u8vector?
Thanks,
Rob. _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list