Am Sun, 8 Nov 2020 21:10:31 -0500 schrieb John Cowan cowan@ccil.org:

On Sun, Nov 8, 2020 at 6:55 AM Jörg F. Wittenberger < Joerg.Wittenberger@softeyes.net> wrote:

...

(define (make-pipe)
  (open-u8vector-pipe '(buffering: #f) '(buffering: #f)))

This means that there is no buffering inside Gambit, which is almost always what you want.

That's been the intention.

(receive (in out) (make-pipe)

...

  (display "foo" out) ;; I'd expect this to block,
  (force-output out) ;; but even this does not.
  "too bad")

The force-output has no effect because it flushes the Gambit buffers, but you have already disabled them.

That's been my - obviously wrong - assumption.

However, Posix pipes also have buffers inside, and you can write up to PIPE_BUF bytes atomically before you can be blocked. The value of PIPE_BUF depends on your OS, but on Linux it is 64K by default. This is actually harmless, though. Unlike a stdio buffer, as soon as there are any bytes in the pipe the receiving end can pull them out, so there is no need to ever flush the pipe itself.

Sure, I could get some flow control from putting a OS pipe in between. But that unreasonably complicates the (supposed to be portable) code.

My guess was that I might have overlooked something in the manual. Or the hope some would know how to mess with the internals to add it.

thank you Marc!

Summary: a) issue solved b) discovered more surprising behavior (bug?) b) actually I'm now interested in something related: a different way to create pipes.

# (a) Trick Done

If you want to limit the buffering done by the in-memory FIFO, for example if you want to avoid a form of memory leak when the consumer is slower than the producer, then you can set a buffering limit with “macro-u8vector-port-buffering-limit-set!” like this:

(include "~~lib/_gambit#.scm")

(define (make-pipe) (open-u8vector-pipe '(buffering: #f) '(buffering: #f)))

(receive (in out) (make-pipe)

 ;; set buffering limit... actual limit = (* 64 (+ 2 (quotient

lim 64))) (let ((lim 1000)) (macro-u8vector-port-buffering-limit-set! out lim))

 (let ((i 0))
   (with-exception-catcher
    (lambda (e)
      (if (deadlock-exception? e)
          (pp `(deadlock after writing ,i bytes))
          (raise e)))
    (lambda ()
      (let loop ()
        (if (< i 1000000)
            (begin
              (write-u8 42 out)
              (set! i (+ i 1))
              (loop))
            (pp `(wrote ,i bytes without deadlock))))))))

This helped A LOT!

Thanks soo much!

# (b) Unexpected: successfully writing to closed ports!

Example:

> (define in) > (define out (receive (a b) (set! i a) b) > (close-output-port out) > (write-subu8vector '#u8(1 2 3 4) 0 2 out) 2 >

(Unhelpful when detection closed connections. :-/)

# (c) WANTED

The context I'm working on is essentially a proxy dispatching connections between ports; be them tcp, processes, files or - important in the context - custom I/O via Gambits C interface.

Using these u8vector-pipes - with these buffering limits - it does now what it is functionally supposed to do.

But: Most of the time the processing is like SOCKS or HTTP/S proxy work: handle the first (few) protocol dependent things and then connect the ports. Currently the latter is implemented using two threads copying one directions each. Plus some overhead:

Note that the actual limit that you get is really (* 64 (+ 2 (quotient limit 64))) because the in-memory FIFO is represented by a list of 64 byte chunks.

In fact I do actually not need that pipe, I feel.

Would there be a way to connect two (bi-directional) ports "from now until closed" and return. (I assume that there should be at least one thread to be joined in order to learn when the connection is done. Alternative hints, like callbacks etc. welcome.)

To be clear: I'd like to replace my procedure:

(ports-connect! in1 out1 in2 out2)

Which spawns a second thread to copy inA to outB and vice versa and returns once both threads terminated.

I feel gambit can do much better than what I have so far.

Just how? Once the connections are dispatched, all overhead is evil. ;-)

Thanks

/Jörg