How to save s-expressions
Good morning everybody, Forgive my ignorance, I am just learning. In order to embrace the "code is data" philosophy [1], I want to load and save data as s-expressions within files. I figured the load using (read (open-input-file "data.scm")). How can I save? Thanks in advance. Richard [1]: I just learnt that it is called http://en.wikipedia.org/wiki/Homoiconicity
Afficher les réponses par date
(write the-data (open-output-file "data.scm")) ? =) Generally though, you're better off with with-input-file and with-output-file , as the return of the thunk communicated explicitly to Gambit's IO system, that the port is no longer used, i.e. (with-input-from-file "data.scm" read) / (with-output-to-file "data.scm" (lambda (p) (write your-data p))). Also of course you could do (let* ((p (open-input-file "data.scm")) (d (read p))) (close-port p) d) , but you see it's less concise. (Also, what if read would throw an exception, clearly this variant does not come with any finalization of p - would with-intput-from/to-file finalize them? - anyone is free to clarify this.) Just dropping the file handle will wait for it to garbage-collect away, which generally is really fine but potentially could bring problems (say that you'd unintentionally do 10 000 file operations like that prior to a GC, that would generally cause the OS to run out of file descriptors and you'd get more or less weird error messages). Also note that with this kind of very high level resources, you cannot expect them to be automatically GC:ed away. Thread objects for instance, at least if they're running, are not GC:ed away. With file ports, I'd believe they are. With TCP ports, not clear. Anyone is free to bring clarity to this one too. Best regards 2012/10/27 Richard Prescott <rdprescott@gmail.com>
Good morning everybody,
Forgive my ignorance, I am just learning.
In order to embrace the "code is data" philosophy [1], I want to load and save data as s-expressions within files. I figured the load using (read (open-input-file "data.scm")).
How can I save?
Thanks in advance.
Richard
[1]: I just learnt that it is called http://en.wikipedia.org/wiki/Homoiconicity _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Le 2012-10-27 à 4:31 AM, Richard Prescott <rdprescott@gmail.com> a écrit :
Good morning everybody,
Forgive my ignorance, I am just learning.
In order to embrace the "code is data" philosophy [1], I want to load and save data as s-expressions within files. I figured the load using (read (open-input-file "data.scm")).
How can I save?
Thanks in advance.
Richard
While it does work, your code does not close the input-port and relies on the garbage collector to do this, which is a bad idea (because you may run out of file descriptors before a garbage collection is triggered). It is better to use call-with-input-file which closes the port. Here are better save and restore functions: ;; File: save-restore.scm (define (save-to-file filename obj) (call-with-output-file filename (lambda (port) (write obj port)))) (define (restore-from-file filename) (call-with-input-file filename (lambda (port) (read port)))) (save-to-file "data.txt" '#(1 2 3)) (pp (restore-from-file "data.txt")) Marc
Ah Richard, note that all of these four file output/input routines mentioned now, can take a settings structure argument in the place where you passed filenames up to now. Using those settings, you can specify things like character encoding to use for the read/written file. 2012/10/27 Marc Feeley <feeley@iro.umontreal.ca>
Le 2012-10-27 à 4:31 AM, Richard Prescott <rdprescott@gmail.com> a écrit :
Good morning everybody,
Forgive my ignorance, I am just learning.
In order to embrace the "code is data" philosophy [1], I want to load and save data as s-expressions within files. I figured the load using (read (open-input-file "data.scm")).
How can I save?
Thanks in advance.
Richard
While it does work, your code does not close the input-port and relies on the garbage collector to do this, which is a bad idea (because you may run out of file descriptors before a garbage collection is triggered). It is better to use call-with-input-file which closes the port.
Here are better save and restore functions:
;; File: save-restore.scm
(define (save-to-file filename obj) (call-with-output-file filename (lambda (port) (write obj port))))
(define (restore-from-file filename) (call-with-input-file filename (lambda (port) (read port))))
(save-to-file "data.txt" '#(1 2 3))
(pp (restore-from-file "data.txt"))
Marc
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Marc, Do all ports GC? (am thinking especially about file and tcp client and server ports) I suppose that the fact that with-*-port closes the port on finalization cannot be relied on to always work, because what if there's an exception during the thunk's execution and it's handled in a way that never leaves control back, or the thread is thread-terminate!:d (could happen a dynamic-wind finalizer would not be trigged right)? Mikael 2012/10/27 Marc Feeley <feeley@iro.umontreal.ca>
Le 2012-10-27 à 4:31 AM, Richard Prescott <rdprescott@gmail.com> a écrit :
Good morning everybody,
Forgive my ignorance, I am just learning.
In order to embrace the "code is data" philosophy [1], I want to load and save data as s-expressions within files. I figured the load using (read (open-input-file "data.scm")).
How can I save?
Thanks in advance.
Richard
While it does work, your code does not close the input-port and relies on the garbage collector to do this, which is a bad idea (because you may run out of file descriptors before a garbage collection is triggered). It is better to use call-with-input-file which closes the port.
Here are better save and restore functions:
;; File: save-restore.scm
(define (save-to-file filename obj) (call-with-output-file filename (lambda (port) (write obj port))))
(define (restore-from-file filename) (call-with-input-file filename (lambda (port) (read port))))
(save-to-file "data.txt" '#(1 2 3))
(pp (restore-from-file "data.txt"))
Marc
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Le 2012-10-27 à 8:11 PM, Mikael <mikael.rcv@gmail.com> a écrit :
Marc,
Do all ports GC? (am thinking especially about file and tcp client and server ports)
Yes. For all ports, the OS ressources which are attached to a port are freed (including closing file descriptors) when the port is GCed.
I suppose that the fact that with-*-port closes the port on finalization cannot be relied on to always work, because what if there's an exception during the thunk's execution and it's handled in a way that never leaves control back, or the thread is thread-terminate!:d (could happen a dynamic-wind finalizer would not be trigged right)?
Yes. If you need to force closing a file descriptor then you need to catch exceptions and explicitly close the port. Here's a small program you can use to test limits on your particular OS. ;; File: fd-autoclose.scm (define (on-gc thunk) (make-will (cons 1 2) ;; garbage (lambda (obj) (thunk) (on-gc thunk)))) (define counter 0) (on-gc (lambda () (pp counter) (set! counter 0))) (with-exception-catcher (lambda (e) (##gc) (display-exception e)) (lambda () (let loop ((n 5000)) (if (> n 0) (begin (set! counter (+ counter 1)) (let ((port (open-tcp-client "localhost:22"))) (loop (- n 1)))))))) It will try to open 5000 TCP connections in a loop, relying on the GC to close the connections. The program displays the number of connections that were open (and closed) since the last garbage collection. If the heap size is small, then the file descriptors never run out. But if a large heap is used, the program will reach the OS limit on the number of open file descriptors. Here's a test on an OS X machine: % gsi -:d2 fd-autoclose.scm *** GC: 0 ms, 180K alloc, 203K heap, 37.1K live (18% 18528+19448) 14 *** GC: 0 ms, 347K alloc, 209K heap, 60.4K live (29% 36752+25140) 19 *** GC: 1 ms, 495K alloc, 209K heap, 60.4K live (29% 36752+25140) 19 *** GC: 1 ms, 644K alloc, 209K heap, 60.4K live (29% 36752+25140) 19 *** GC: 5 ms, 792K alloc, 209K heap, 60.4K live (29% 36752+25140) 19 ...etc, until the normal end of the program (note the small 209 KB heap) and with a 19 MB heap the program works properly: % gsi -:d2,m19000 fd-autoclose.scm 2502 *** GC: 78 ms, 19.0M alloc, 18.6M heap, 63.4K live (0% 37696+27248) 2470 *** GC: 45 ms, 37.5M alloc, 18.6M heap, 63.4K live (0% 37696+27248) but with a 20 MB heap the program terminates with an error: % gsi -:d2,m20000 fd-autoclose.scm 2556 *** GC: 73 ms, 19.4M alloc, 19.9M heap, 60.3K live (0% 36600+25140) Too many open files (open-tcp-client "localhost:22") It would seem that the maximum number of open file descriptor on this machine is about 2560 (considering a few file descriptors are open for stdin, stdout, ...). Marc
Thanks guys. I am slowly learning. Richard. On 2012-10-28, at 9:11 AM, Marc Feeley <feeley@iro.umontreal.ca> wrote:
Le 2012-10-27 à 8:11 PM, Mikael <mikael.rcv@gmail.com> a écrit :
Marc,
Do all ports GC? (am thinking especially about file and tcp client and server ports)
Yes. For all ports, the OS ressources which are attached to a port are freed (including closing file descriptors) when the port is GCed.
I suppose that the fact that with-*-port closes the port on finalization cannot be relied on to always work, because what if there's an exception during the thunk's execution and it's handled in a way that never leaves control back, or the thread is thread-terminate!:d (could happen a dynamic-wind finalizer would not be trigged right)?
Yes. If you need to force closing a file descriptor then you need to catch exceptions and explicitly close the port.
Here's a small program you can use to test limits on your particular OS.
;; File: fd-autoclose.scm
(define (on-gc thunk) (make-will (cons 1 2) ;; garbage (lambda (obj) (thunk) (on-gc thunk))))
(define counter 0)
(on-gc (lambda () (pp counter) (set! counter 0)))
(with-exception-catcher (lambda (e) (##gc) (display-exception e)) (lambda () (let loop ((n 5000)) (if (> n 0) (begin (set! counter (+ counter 1)) (let ((port (open-tcp-client "localhost:22"))) (loop (- n 1))))))))
It will try to open 5000 TCP connections in a loop, relying on the GC to close the connections. The program displays the number of connections that were open (and closed) since the last garbage collection. If the heap size is small, then the file descriptors never run out. But if a large heap is used, the program will reach the OS limit on the number of open file descriptors.
Here's a test on an OS X machine:
% gsi -:d2 fd-autoclose.scm *** GC: 0 ms, 180K alloc, 203K heap, 37.1K live (18% 18528+19448) 14 *** GC: 0 ms, 347K alloc, 209K heap, 60.4K live (29% 36752+25140) 19 *** GC: 1 ms, 495K alloc, 209K heap, 60.4K live (29% 36752+25140) 19 *** GC: 1 ms, 644K alloc, 209K heap, 60.4K live (29% 36752+25140) 19 *** GC: 5 ms, 792K alloc, 209K heap, 60.4K live (29% 36752+25140) 19 ...etc, until the normal end of the program (note the small 209 KB heap)
and with a 19 MB heap the program works properly:
% gsi -:d2,m19000 fd-autoclose.scm 2502 *** GC: 78 ms, 19.0M alloc, 18.6M heap, 63.4K live (0% 37696+27248) 2470 *** GC: 45 ms, 37.5M alloc, 18.6M heap, 63.4K live (0% 37696+27248)
but with a 20 MB heap the program terminates with an error:
% gsi -:d2,m20000 fd-autoclose.scm 2556 *** GC: 73 ms, 19.4M alloc, 19.9M heap, 60.3K live (0% 36600+25140) Too many open files (open-tcp-client "localhost:22")
It would seem that the maximum number of open file descriptor on this machine is about 2560 (considering a few file descriptors are open for stdin, stdout, ...).
Marc
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Le 2012-10-27 à 8:44 AM, Marc Feeley <feeley@iro.umontreal.ca> a écrit :
Le 2012-10-27 à 4:31 AM, Richard Prescott <rdprescott@gmail.com> a écrit :
Good morning everybody,
Forgive my ignorance, I am just learning.
In order to embrace the "code is data" philosophy [1], I want to load and save data as s-expressions within files. I figured the load using (read (open-input-file "data.scm")).
How can I save?
Thanks in advance.
Richard
While it does work, your code does not close the input-port and relies on the garbage collector to do this, which is a bad idea (because you may run out of file descriptors before a garbage collection is triggered). It is better to use call-with-input-file which closes the port.
Here are better save and restore functions:
;; File: save-restore.scm
(define (save-to-file filename obj) (call-with-output-file filename (lambda (port) (write obj port))))
(define (restore-from-file filename) (call-with-input-file filename (lambda (port) (read port))))
(save-to-file "data.txt" '#(1 2 3))
(pp (restore-from-file "data.txt"))
Marc
Here's an improved version of those functions which allows any serializable object, including functions, to be saved/restored. The code uses the functions object->u8vector and u8vector->object which encode/decode any object to/from a vector of bytes. In the case of interpreted functions, the source code locations are saved in the encoding (so that any error in a restored function will give an error message which points to the correct source code location). See the example below. Marc ;; File: save-restore.scm (define (write-file filename obj) (call-with-output-file filename (lambda (port) (write obj port)))) (define (read-file filename) (call-with-input-file filename (lambda (port) (read port)))) (define (save-to-file filename obj) (write-file filename (object->u8vector obj))) (define (restore-from-file filename) (u8vector->object (read-file filename))) ;; test it (save-to-file "code.txt" (lambda (x) (/ 1.0 (* x x)))) (define inverse-square (restore-from-file "code.txt")) (pp (inverse-square 10)) (pp (inverse-square "foo")) ;; gives: ;; ;; .01 ;; *** ERROR IN inverse-square, "save-restore.scm"@21.45 -- (Argument 1) NUMBER expected ;; (* "foo" "foo")
participants (3)
-
Marc Feeley -
Mikael -
Richard Prescott