OpenSSL SSL channels module pre-release and HTTP client early pre-release, attached
Hi guys,
For anyone interested here's the pre-release version of the OpenSSL SSL channels ("openssl") module and an early pre-release of the HTTP client.
(If you read this after May 2013 the attachment is most probably outdated and you better look at https://github.com/gambit-community or https://github.com/m-i-k-a-e-l/ .)
*openssl module* The openssl module works extremely well and is mature.
Relative to the amount of functionality provided, extreme effort has been put into its making. This is because OpenSSL's documentation on nonblocking use is incorrect in several parts, so study of other user implementations and practical testing has been required to make it work. Over the course of a couple of years, in the range three man weeks has gone into it this way.
Due to that this is the nature of the problem addressed by this module, I hope that this openssl module will be the main one for Gambit and that any more effort into OpenSSL SSL channel modules in Gambit will go into making this one even better.
The current version however relies on a version-unsafe hack for extracting the respective TCP port's socket, which possibly would be a feature to be included in Gambit, please see the "Next steps for release" section below. OpenSSL is let to write and read from the OS socket directly. This saves lots of roundtrips and buffering.
*http-client module* The HTTP client supports the passing of custom method, headers and request body, and supports retrieval of status code, headers and request body, and it does both these with support for all of HTTP 1.0 and 1.1, so this is all there is to the HTTP protocol in itself.
Exports:
(simple-raw-http-request uri #!optional request-body (headers '())) => req
(define-macro (with-http-client-all-response all-response . code) `(apply (lambda (reached-uri result stage status-code headers get-header response) ,@code) ,all-response))
(define (simple-raw-http-request-resp uri #!optional request-body) (with-http-client-all-response (simple-raw-http-request uri request-body) response))
Use:
(simple-raw-http-request "http://www.iro.umontreal.ca/~gambit/")
(#<uri #13 scheme: "http" userinfo: #f host: "www.iro.umontreal.ca" port: #f path: "/~gambit/" query: () fragment: #f> #t done 200 (("connection" . "close") ("content-type" . "text/html") ("content-length" . "115") ("accept-ranges" . "bytes") ("etag" . ""1c93d75-73-4443e1da730c0"") ("last-modified" . "Mon, 21 Jan 2008 16:38:03 GMT") ("server" . "Apache/2.0.54 (Fedora)") ("date" . "Wed, 20 Mar 2013 19:20:12 GMT")) #<procedure #14 26#get-header> "<HTML>\n<HEAD>\n<META HTTP-EQUIV=REFRESH CONTENT=0;URL= http://dynamo.iro.umontreal.ca/~gambit/wiki/%3E%5Cn</HEAD>\n</HTML>\n")
(simple-raw-http-request-resp "http://www.iro.umontreal.ca/~gambit/")
"<HTML>\n<HEAD>\n<META HTTP-EQUIV=REFRESH CONTENT=0;URL= http://dynamo.iro.umontreal.ca/~gambit/wiki/%3E%5Cn</HEAD>\n</HTML>\n"
Currently it supports single threaded execution i.e. it opens HTTP connection, makes request, retrieves response and closes the connection in a sequence and there's no connection reuse - reuse and multiplexing across threads as well as following redirects will be implemented. Also invocation of several declared callbacks is not done currently, and will be trivial to implement.
The DSL uses Sack's idea of providing all protocol functionality in a 'core' form. It can extremely easily be wrapped to a XMLHTTPRequest object.
This http-client module replaces Sack's previous http-client module, which unlike Sack's web server module which works great, had terrible problems, for more info see the header section in http-client.scm .
*Next steps for release* I'll make a "real" release of these two modules when the IO exception handling question discussed on the list now has got an answer and maybe extraction of TCP port socket can be done with a Gambit-builtin procedure. These two things currently bring version-unsafe hacks into the code, more specifically into tcpip.scm and sack/src/gamgambit-noexceptions-io.scm , so this is why I call this a pre-release.
(Note that port-wait-for-input and port-wait-for-output of the tcpip module are not needed anymore as the firstmentioned has been in Gambit all the time and the latter was recently included in Gambit, ##-prefixed. Will remove them when I doublechecked they're in the newest Gambit version's namespace. As it looks the tcpip module will be removed.)
*Future development* I'll be happy to see when current Black Hole will have package management functionality so that installation of these packages will be easy.
Currently the OpenSSL module exports its functionality as a set of IO primitive closures, and, the HTTP server and client uses such IO primitive closures for interaction with its underlying IO layer.
When there will be good support for application-layer port implementations in Gambit - I hope that will happen the next 1.5 years or so, we'll see - the HTTP server and client will use Gambit's IO primitives directly instead of these custom primitives, which will provide for a cleaner and more general code, and,
The OpenSSL module and I suppose the HTTP-client (and probably HTTP server too) will export Gambit ports for use something like
(open-ssl-channel-port settings) => port (open-http-client-port settings (lambda (request-body-port) (display .. request-body-port))) => response-body-port
Thoughts, feedback, patches much welcome.
Best regards, Mikael
Afficher les réponses par date
Update: HTTP client now has HTTPS support, and some minor fix.
(import openssl-sack) enables the HTTPS support in http-client.
SOCKS support essentially implemented and to be released someday.
When the IO error handling currently discussed is resolved, multithreaded HTTP connection reuse can be implemented in a robust, holistic, clean way. Will be great as by that Gambit finally has a complete HTTP client.
Mikael
2013/3/20 Mikael mikael.rcv@gmail.com
Hi guys,
For anyone interested here's the pre-release version of the OpenSSL SSL channels ("openssl") module and an early pre-release of the HTTP client.
(If you read this after May 2013 the attachment is most probably outdated and you better look at https://github.com/gambit-community or https://github.com/m-i-k-a-e-l/ .)
*openssl module* The openssl module works extremely well and is mature.
Relative to the amount of functionality provided, extreme effort has been put into its making. This is because OpenSSL's documentation on nonblocking use is incorrect in several parts, so study of other user implementations and practical testing has been required to make it work. Over the course of a couple of years, in the range three man weeks has gone into it this way.
Due to that this is the nature of the problem addressed by this module, I hope that this openssl module will be the main one for Gambit and that any more effort into OpenSSL SSL channel modules in Gambit will go into making this one even better.
The current version however relies on a version-unsafe hack for extracting the respective TCP port's socket, which possibly would be a feature to be included in Gambit, please see the "Next steps for release" section below. OpenSSL is let to write and read from the OS socket directly. This saves lots of roundtrips and buffering.
*http-client module* The HTTP client supports the passing of custom method, headers and request body, and supports retrieval of status code, headers and request body, and it does both these with support for all of HTTP 1.0 and 1.1, so this is all there is to the HTTP protocol in itself.
Exports:
(simple-raw-http-request uri #!optional request-body (headers '())) => req
(define-macro (with-http-client-all-response all-response . code) `(apply (lambda (reached-uri result stage status-code headers get-header response) ,@code) ,all-response))
(define (simple-raw-http-request-resp uri #!optional request-body) (with-http-client-all-response (simple-raw-http-request uri request-body) response))
Use:
(simple-raw-http-request "http://www.iro.umontreal.ca/~gambit/")
(#<uri #13 scheme: "http" userinfo: #f host: "www.iro.umontreal.ca" port: #f path: "/~gambit/" query: () fragment: #f> #t done 200 (("connection" . "close") ("content-type" . "text/html") ("content-length" . "115") ("accept-ranges" . "bytes") ("etag" . ""1c93d75-73-4443e1da730c0"") ("last-modified" . "Mon, 21 Jan 2008 16:38:03 GMT") ("server" . "Apache/2.0.54 (Fedora)") ("date" . "Wed, 20 Mar 2013 19:20:12 GMT")) #<procedure #14 26#get-header> "<HTML>\n<HEAD>\n<META HTTP-EQUIV=REFRESH CONTENT=0;URL= http://dynamo.iro.umontreal.ca/~gambit/wiki/%3E%5Cn</HEAD>\n</HTML>\n")
(simple-raw-http-request-resp "http://www.iro.umontreal.ca/~gambit/")
"<HTML>\n<HEAD>\n<META HTTP-EQUIV=REFRESH CONTENT=0;URL= http://dynamo.iro.umontreal.ca/~gambit/wiki/%3E%5Cn</HEAD>\n</HTML>\n"
Currently it supports single threaded execution i.e. it opens HTTP connection, makes request, retrieves response and closes the connection in a sequence and there's no connection reuse - reuse and multiplexing across threads as well as following redirects will be implemented. Also invocation of several declared callbacks is not done currently, and will be trivial to implement.
The DSL uses Sack's idea of providing all protocol functionality in a 'core' form. It can extremely easily be wrapped to a XMLHTTPRequest object.
This http-client module replaces Sack's previous http-client module, which unlike Sack's web server module which works great, had terrible problems, for more info see the header section in http-client.scm .
*Next steps for release* I'll make a "real" release of these two modules when the IO exception handling question discussed on the list now has got an answer and maybe extraction of TCP port socket can be done with a Gambit-builtin procedure. These two things currently bring version-unsafe hacks into the code, more specifically into tcpip.scm and sack/src/gamgambit-noexceptions-io.scm , so this is why I call this a pre-release.
(Note that port-wait-for-input and port-wait-for-output of the tcpipmodule are not needed anymore as the firstmentioned has been in Gambit all the time and the latter was recently included in Gambit, ##-prefixed. Will remove them when I doublechecked they're in the newest Gambit version's namespace. As it looks the tcpip module will be removed.)
*Future development* I'll be happy to see when current Black Hole will have package management functionality so that installation of these packages will be easy.
Currently the OpenSSL module exports its functionality as a set of IO primitive closures, and, the HTTP server and client uses such IO primitive closures for interaction with its underlying IO layer.
When there will be good support for application-layer port implementations in Gambit - I hope that will happen the next 1.5 years or so, we'll see - the HTTP server and client will use Gambit's IO primitives directly instead of these custom primitives, which will provide for a cleaner and more general code, and,
The OpenSSL module and I suppose the HTTP-client (and probably HTTP server too) will export Gambit ports for use something like
(open-ssl-channel-port settings) => port (open-http-client-port settings (lambda (request-body-port) (display .. request-body-port))) => response-body-port
Thoughts, feedback, patches much welcome.
Best regards, Mikael
-
Mikael