[gambit-list] How to bind future-based API (FoundationDB)

Amirouche Boubekki amirouche.boubekki at gmail.com
Fri Apr 30 02:32:44 EDT 2021


I want to bind Gambit with the future-based API of FoundationDB client driver.

FoundationDB is a database, most of the time distributed, supported by
an OKVS interface, that I used as inspiration for SRFI-167.

Client side, there is a C library that implements the network
protocol, that exposes a Future based API. There is really two things
to that happens in chronological order at runtime in a FoundationDB
application:

# select API version
found.api_version(620)
# Create the database handle
db = await found.open()

Inside `open()` some POSIX threads trickery happens, it will run the
foundationdb client network thread with something like:

class NetworkThread(threading.Thread):
    def run(self):
        check(lib.fdb_run_network())

Where fdb_run_network is bound to the C function [0] with the same
name. Its documentation says:

> Must be called after fdb_setup_network() before any asynchronous functions in this API can be expected to complete. Unless your program is entirely event-driven based on results of asynchronous functions in this API and has no event loop of its own, you will want to invoke this function on an auxiliary thread (which it is your responsibility to create).

> This function will not return until fdb_stop_network() is called by you or a serious error occurs. It is not possible to run more than one network thread, and the network thread cannot be restarted once it has been stopped. This means that once fdb_run_network has been called, it is not legal to call it again for the lifetime of the running program.

[0] https://apple.github.io/foundationdb/api-c.html#c.fdb_run_network

In particular I am not in the case described with the following sentence:

> Unless your program is entirely event-driven based on results of asynchronous functions in this API and has no event loop of its own

Because my program is also a HTTP server.

Once the network is setup, and running, one can use the provided
functions to access the database such as:

[1] https://github.com/amirouche/asyncio-foundationdb/blob/85cdb183f92af421d9c3fa1d947be786da4ca2bc/found/base.py#L282-L291

and the callback is here:

[2] https://github.com/amirouche/asyncio-foundationdb/blob/85cdb183f92af421d9c3fa1d947be786da4ca2bc/found/base.py#L160-L185

To summarize how I implemented it with Python asyncio (inspired from
the official bindings):

- Call the C function that returns a FDB Future
- Create a python asyncio future,
- Pass the Python future as the data of the FDB future callback
- When the FDB Future is complete, the future will trigger the above
callback defined at [2]
- The callback at [2], is executed in FDB network thread, will
retrieve the result, and resolve the Python future with it, in a way
that is POSIX thread safe
- The code that is waiting on the Python Future, will proceed at [3]

[3] https://github.com/amirouche/asyncio-foundationdb/blob/85cdb183f92af421d9c3fa1d947be786da4ca2bc/found/base.py#L290

That is a description of the async approach. The FDB Future API, also
allow a synchronous approach that still involves a network thread:

[4] https://apple.github.io/foundationdb/api-c.html#future

I think the easiest path is using the FDB Future synchronous API aka.
blocking API, but that would still require to spin a network thread in
a POSIX thread.

Any help will be greatly appreciated.

ref: https://github.com/apple/foundationdb



More information about the Gambit-list mailing list