[gambit-list] How to bind future-based API (FoundationDB)

Marc Feeley feeley at iro.umontreal.ca
Fri Apr 30 08:26:22 EDT 2021


An asynchronous API can easily be mapped to Gambit threads.  The idea is to start a new Gambit thread for each asynchronous operation (using the procedures `thread` or `make-thread`) and to use the `thread-join!` procedure to wait for the result or exception.  I attach sample code below.

Note that this is a reasonable approach because Gambit threads are very cheap.  On my machine with the following program I measured that it takes less than 1 microsecond to create a thread and synchronize with that thread.

 (declare (standard-bindings))
 (time
  (let loop ((prev (thread list)) (n 1000000))
    (if (> n 0)
        (loop (thread (lambda () (thread-join! prev))) (- n 1))
        (pp (thread-join! prev)))))

So it is unlikely the thread creation/synchronization will represent much overhead over the network operations needed to interact with the database.

Marc


  (define (some-async-op param1 param2)
    (thread (lambda ()
              (thread-sleep! param1) ;; simulate a long operation
              (if (string? param2)   ;; simulate a possible exception/error
                  (error param2)
                  param2))))

  (define await-async-op thread-join!)

  ;; test it

  (define loops 10000000)

  (define (waste-time)
    (let loop ((n loops)) (if (> n 0) (loop (- n 1)) #f)))

  (define x (some-async-op 1 42))
  (waste-time)
  (pp (real-time))
  (pp (await-async-op x))
  (pp (real-time))

  (define y (some-async-op 1 "simulated error 1"))
  (waste-time)
  (pp (real-time))
  (pp (await-async-op y 0.1 "wait exceeded 0.1 secs"))

  (define z (some-async-op 1 "simulated error 2"))
  (waste-time)
  (pp (real-time))
  (with-exception-catcher
   (lambda (exc)
     (if (uncaught-exception? exc)
         (begin
           (display "uncaught-exception: ")
           (display-exception (uncaught-exception-reason exc)))
         (raise exc)))
   (lambda ()
     (pp (await-async-op z))))


> On Apr 30, 2021, at 2:32 AM, Amirouche Boubekki <amirouche.boubekki at gmail.com> wrote:
> 
> I want to bind Gambit with the future-based API of FoundationDB client driver.
> 
> FoundationDB is a database, most of the time distributed, supported by
> an OKVS interface, that I used as inspiration for SRFI-167.
> 
> Client side, there is a C library that implements the network
> protocol, that exposes a Future based API. There is really two things
> to that happens in chronological order at runtime in a FoundationDB
> application:
> 
> # select API version
> found.api_version(620)
> # Create the database handle
> db = await found.open()
> 
> Inside `open()` some POSIX threads trickery happens, it will run the
> foundationdb client network thread with something like:
> 
> class NetworkThread(threading.Thread):
>    def run(self):
>        check(lib.fdb_run_network())
> 
> Where fdb_run_network is bound to the C function [0] with the same
> name. Its documentation says:
> 
>> Must be called after fdb_setup_network() before any asynchronous functions in this API can be expected to complete. Unless your program is entirely event-driven based on results of asynchronous functions in this API and has no event loop of its own, you will want to invoke this function on an auxiliary thread (which it is your responsibility to create).
> 
>> This function will not return until fdb_stop_network() is called by you or a serious error occurs. It is not possible to run more than one network thread, and the network thread cannot be restarted once it has been stopped. This means that once fdb_run_network has been called, it is not legal to call it again for the lifetime of the running program.
> 
> [0] https://apple.github.io/foundationdb/api-c.html#c.fdb_run_network
> 
> In particular I am not in the case described with the following sentence:
> 
>> Unless your program is entirely event-driven based on results of asynchronous functions in this API and has no event loop of its own
> 
> Because my program is also a HTTP server.
> 
> Once the network is setup, and running, one can use the provided
> functions to access the database such as:
> 
> [1] https://github.com/amirouche/asyncio-foundationdb/blob/85cdb183f92af421d9c3fa1d947be786da4ca2bc/found/base.py#L282-L291
> 
> and the callback is here:
> 
> [2] https://github.com/amirouche/asyncio-foundationdb/blob/85cdb183f92af421d9c3fa1d947be786da4ca2bc/found/base.py#L160-L185
> 
> To summarize how I implemented it with Python asyncio (inspired from
> the official bindings):
> 
> - Call the C function that returns a FDB Future
> - Create a python asyncio future,
> - Pass the Python future as the data of the FDB future callback
> - When the FDB Future is complete, the future will trigger the above
> callback defined at [2]
> - The callback at [2], is executed in FDB network thread, will
> retrieve the result, and resolve the Python future with it, in a way
> that is POSIX thread safe
> - The code that is waiting on the Python Future, will proceed at [3]
> 
> [3] https://github.com/amirouche/asyncio-foundationdb/blob/85cdb183f92af421d9c3fa1d947be786da4ca2bc/found/base.py#L290
> 
> That is a description of the async approach. The FDB Future API, also
> allow a synchronous approach that still involves a network thread:
> 
> [4] https://apple.github.io/foundationdb/api-c.html#future
> 
> I think the easiest path is using the FDB Future synchronous API aka.
> blocking API, but that would still require to spin a network thread in
> a POSIX thread.
> 
> Any help will be greatly appreciated.
> 
> ref: https://github.com/apple/foundationdb
> 
> _______________________________________________
> Gambit-list mailing list
> Gambit-list at iro.umontreal.ca
> https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list
> 




More information about the Gambit-list mailing list