[gambit-list] How do I create a pipeline of processes, as in shell?

Marc Feeley feeley at iro.umontreal.ca
Mon Jan 28 16:28:09 EST 2013

On 2013-01-27, at 4:48 PM, Jussi Piitulainen <jpiitula at ling.helsinki.fi> wrote:

> In shell it is easy to run a program, and it is easy to chain programs
> into a pipeline where each process writes to the following process.
> In Gambit-C it appears to be easy to run a program and write to the
> process and read from the process. I failed to find a way to specify
> that the output of one process should be connected to the input of
> another. I have found open-process, process-status. I have found some
> discussions about running single processes.
> How does one implement the following lines in Gambit-C?
> $ sort infile | uniq -c | sort -nr
> $ sort < infile | uniq -c | sort -nr > outfile
> Is this non-trivial? I'm looking for something like the following, so
> that those processes would wait for their input and proceed when it
> becomes available.
> (let ((p (pipe '(path: "sort" arguments: ("infile"))
>               '(path: "uniq" arguments: ("-c"))
>               '(path: "sort" arguments: ("-nr")))))
>   (let* ((r1 (read-line p)) ...) ...))
> (parameterize ((current-input-port (open-input-file "infile"))
>               (current-output-port (open-output-file "outfile")))
>   (process-status
>      (pipe "sort" '(path: "uniq" arguments: ("-c")) ...)))
> I'm _not_ looking for (string-append ...).

The code below will allow the creation of a pipeline of processes.  The output of one process in the pipeline is transferred to the input of the next process.  Each data transfer from one process to the next is achieved by a Gambit thread which reads the output of a process and writes it to the input of the next process.  Although it "works", it is not as efficient as having the operating system do this through file descriptors.  But that could be achieved by spawning a shell and asking it to create the pipeline, i.e. (open-process '(path: "/bin/sh" arguments: ("-c" "sort infile | uniq -c | sort -nr"))).  I assume you are interested in Gambit level piping.


(define (port-copy in-port out-port)
  (let ((buf (make-u8vector 4096)))
    (let loop ()
      (let ((n (read-subu8vector buf 0 (u8vector-length buf) in-port 1)))
        ;;(pp n)
        (if (= n 0)
            (close-output-port out-port)
              (write-subu8vector buf 0 n out-port)

(define (port-copy-in-tread in-port out-port)
  (thread-start! (make-thread (lambda () (port-copy in-port out-port)))))

(define (process args)
  (open-process (append args '(stderr-redirection: #t))))

(define (pipe-processes processes)
  (let ((in-out-port (car processes)))
    (values in-out-port
            (if (null? (cdr processes))
                (receive (out-port in-port)
                         (pipe-processes (cdr processes))
                  (port-copy-in-tread in-out-port out-port)

(define (pipe . processes)
  (pipe-processes processes))

(define (go)
  (receive (out-port in-port)
           (pipe (process (list path: "du" arguments: '()))
                 (process (list path: "sort" arguments: '("-n" "-r")))
                 (process (list path: "head" arguments: '("-10"))))
    (port-copy in-port (current-output-port))))


More information about the Gambit-list mailing list