Hello:
Is there an easy way in Gambit to read binary files? I have got binary files created on my desktop Linux machine with the help of IDL. The binary files are created in Fortran 77 binary format and BIG_ENDIAN mode (IDL has such a feature). I read the files from a data base with IDL and store it in big_endian Fortran 77.
I'd like to read it on my Macintosh. However, the gdl clone of idl has some problems with that particular files (I scp it from my desktop machine).
The first 3 numbers in the file are Fortran integer numbers which describe the 3d dimension of the array and the following numbers in the binary file are the content of the array.
Or do I need a C-binding?
Thanks very much, Siegfried Gonzi
Afficher les réponses par date
Siegfried Gonzi wrote:
Hello:
Hello
Is there an easy way in Gambit to read binary files? I have got binary files created on my desktop Linux machine with the help of IDL. The binary files are created in Fortran 77 binary format and BIG_ENDIAN mode (IDL has such a feature). I read the files from a data base with IDL and store it in big_endian Fortran 77.
I'd like to read it on my Macintosh. However, the gdl clone of idl has some problems with that particular files (I scp it from my desktop machine).
The first 3 numbers in the file are Fortran integer numbers which describe the 3d dimension of the array and the following numbers in the binary file are the content of the array.
Or do I need a C-binding?
I guess that Fortran's integers have a fixed length. Therefore, you should be able to read integers byte after byte, and reorder them. It's no fun, but should be easy to do. read-char, eof-object?, char->integer... However, it might be harder, depending on the kind of data of your array.
Thanks very much, Siegfried Gonzi
Adrien.
Siegfried Gonzi wrote:
The first 3 numbers in the file are Fortran integer numbers which describe the 3d dimension of the array and the following numbers in the binary file are the content of the array.
Depends on whether you're for portability or speed.
A pure-scheme solution (you'll have to adapt if the integers are signed, or if the matrix doesn't contain integers but floats or so):
(define int-width-in-bytes 4)
(define (u8vector-bigendian-unsigned-number-ref vec pos) (let lp ((i 0) (res 0)) (if (< i int-width-in-bytes) (lp (+ i 1) (+ res (arithmetic-shift (u8vector-ref vec (+ pos i)) (arithmetic-shift i 3)))) res)))
(define (read-idl-matrix port) (let* ((dimensions-len (* 3 int-width-in-bytes)) (dimensions (make-u8vector dimensions-len))) (if (= (read-subu8vector dimensions 0 dimensions-len port) dimensions-len) (let ((x (u8vector-bigendian-unsigned-number-ref dimensions 0)) (y (u8vector-bigendian-unsigned-number-ref dimensions int-width-in-bytes)) (z (u8vector-bigendian-unsigned-number-ref dimensions (* 2 int-width-in-bytes)))) (let* ((matrix-len (* int-width-in-bytes x y z)) (matrix (make-u8vector matrix-len))) (if (= (read-subu8vector matrix 0 matrix-len port) matrix-len) (list x y z matrix) (error "file too short (2)")))) (error "file too short (1)"))))
If you're after speed and on a bigendian architecture (or are reading standard floats or doubles) and don't mind giving up portability, you could just create a u32vector, s32vector, f64vector or whatever you need and then fill that in with the unsafe variant of read-subu8vector, ##read-subu8vector. Just be careful you calculate the needed length correctly.
Christian.
Siegfried Gonzi wrote:
Hello:
Is there an easy way in Gambit to read binary files? I have got binary files created on my desktop Linux machine Thanks very much, Siegfried Gonzi
Hello, following is a routine I tried.. It works but is probably slow since it includes byte-by-byte I/O reads.
Why not use a C library and the FFI? It would give you an abstraction boundary with separation from hardware or architecture concerns. And could be designed to bind with any high-level language.
regards, Bob -----------------------------------------
(define (integer-in ) (set! *filesize* (- *filesize* 4)) (bitwise-ior (bitwise-and (arithmetic-shift (read-u8 INP) 24) #xff000000) (bitwise-and (arithmetic-shift (read-u8 INP) 16) #xff0000) (bitwise-and (arithmetic-shift (read-u8 INP) 8) #xff00) (bitwise-and (read-u8 INP) #xff)))
On Dec 18, 2007, at 10:31 PM, Bob McIsaac wrote:
(define (integer-in ) (set! *filesize* (- *filesize* 4)) (bitwise-ior (bitwise-and (arithmetic-shift (read-u8 INP) 24) #xff000000) (bitwise-and (arithmetic-shift (read-u8 INP) 16) #xff0000) (bitwise-and (arithmetic-shift (read-u8 INP) 8) #xff00) (bitwise-and (read-u8 INP) #xff)))
This code assumes that arguments to functions are evaluated left-to- right.
Do
(let* ((byte-1 (read-u8 INP)) (byte-2 (read-u8 INP))
etc.
Bradley Lucier wrote:
On Dec 18, 2007, at 10:31 PM, Bob McIsaac wrote:
(define (integer-in ) (set! *filesize* (- *filesize* 4)) (bitwise-ior (bitwise-and (arithmetic-shift (read-u8 INP) 24) #xff000000) (bitwise-and (arithmetic-shift (read-u8 INP) 16) #xff0000) (bitwise-and (arithmetic-shift (read-u8 INP) 8) #xff00) (bitwise-and (read-u8 INP) #xff)))
This code assumes that arguments to functions are evaluated left-to-right.
Do
(let* ((byte-1 (read-u8 INP)) (byte-2 (read-u8 INP))
etc.
Well, it's a ponderous solution that needs refactoring but I don't understand how order of evalution comes into play. Since LISP is said not to need parsing, there is no tree of values and operations to prioritize. Instead, there are only linked lists to follow. That's my primitive understanding so far.
Bob McIsaac wrote:
Instead, there are only linked lists to follow.
But what about the order in which those linked lists are being followed..? That's not specified in the language and implementation dependent.
(PS. of course nobody says that an implementation can't turn the linked lists into some other representation (like assembler). Not specifying the order of execution of function arguments was meant as to leave implementors more freedom to choose the best evaluation strategy.)
Christian Jaeger wrote:
Bob McIsaac wrote:
Instead, there are only linked lists to follow.
But what about the order in which those linked lists are being followed..? That's not specified in the language and implementation dependent.
(PS. of course nobody says that an implementation can't turn the linked lists into some other representation (like assembler). Not specifying the order of execution of function arguments was meant as to leave implementors more freedom to choose the best evaluation strategy.)
The puzzling comment to my humble form said : "This code assumes that arguments to functions are evaluated left-to-right." But what of it? There is a stream of text to be evaluated. Choices are 1. left-right evaluation in one pass; 2. parsing the stream into tokens and sorting the tokens into a tree according to the rules of the grammar ... not Lispy. Choice 1 means typing parentheses so that the evaluator can be simple and recursive. ( I assume that evaluation results in a linked list)
Cheers, -Bob
Bob McIsaac wrote:
The puzzling comment to my humble form said : "This code assumes that arguments to functions are evaluated left-to-right." But what of it? There is a stream of text to be evaluated. Choices are 1. left-right evaluation in one pass; 2. parsing the stream into tokens and sorting the tokens into a tree according to the rules of the grammar ... not Lispy. Choice 1 means typing parentheses so that the evaluator can be simple and recursive. ( I assume that evaluation results in a linked list)
Hm, we're not talking about nested expressions here. We're talking about multiple expressions given to one function. And at that point, regardless of whether your internal representation is lists or something else, the evaluator could be programmed to evaluate function arguments in different orders.
The problematic code is:
(bitwise-ior (bitwise-and (arithmetic-shift (read-u8 INP) 24) #xff000000) (bitwise-and (arithmetic-shift (read-u8 INP) 16) #xff0000) (bitwise-and (arithmetic-shift (read-u8 INP) 8) #xff00) (bitwise-and (read-u8 INP) #xff)))
It contains four (read-u8 INP) forms; the order in which they are evaluated matters (because reading from a port advances the position in the underlying stream as a side effect).
The problem is that those forms are indirectly positioned as arguments to bitwise-ior. bitwise-ior is a function. Scheme dictates that the arguments to functions are evaluated before the function is called (meaning before the body of the function definition is evaluated). So you're guaranteed that all four (read-u8 INP) forms have been evaluated before the bitwise-ior call. *But* Scheme does not specify in which order the four arguments of bitwise-ior are evaluated. So it's very well possible that some Scheme implementation first evaluates (bitwise-and (read-u8 INP) #xff)), then (bitwise-and (arithmetic-shift (read-u8 INP) 8) #xff00), then (bitwise-and (arithmetic-shift (read-u8 INP) 16) #xff0000), and at last (bitwise-and (arithmetic-shift (read-u8 INP) 24) #xff000000). Or the other way round. Or even in some other order or even in parallel. So the bytes coming from INP will end up at unspecified and possibly random places.
OTOH, special forms like let and let* do specify the order of evaluation (they are not functions). So writing
(let* ((a (bitwise-and (arithmetic-shift (read-u8 INP) 24) #xff000000)) (b (bitwise-and (arithmetic-shift (read-u8 INP) 16) #xff0000)) (c (bitwise-and (arithmetic-shift (read-u8 INP) 8) #xff00)) (d (bitwise-and (read-u8 INP) #xff))) (bitwise-ior a b c d))
makes the order of evaluation explicit (first a, then b, then c, then d) and thus portable (and deterministic).
Christian.
I wrote:
OTOH, special forms like let and let* do specify the order of evaluation (they are not functions).
Sorry, let (as opposed to let*) does not specify the order of evaluation. What I had in mind was nested let's (with every let statement only binding one variable); but one let with several bindings are the same as lambdas / functions, e.g. unspecified.
Christian.
Hello:
Thank for all the reply related to reading binary data. I apprehend each help. However, I am still figuring out some stuff. I posted the following also on the Chicken list because now as year long Bigloo user been somehow confused:
== Hello:
I am still evaluating the case with reading binary data files. However, I do not understand the following srfi-56 comment:
== The reference implementation has been tested with the following Schemes: Bigloo, Chez, Chicken, Gambit, Gauche, Guile, Kawa, KSI, MIT-Scheme, MzScheme, RScheme, Scheme48, SISC and Stklos. The *-float64 code turns out to be a very rigorous stress test for an implementation's numeric code. At time of writing, Chicken 2.0 (with the optional numbers egg), KSI 3.4.2 and MzScheme (both 200 and 299 versions) are currently the only implementations to pass all tests. Petite Chez 6.0a is the next most complete failing 6, followed by Gambit4b14 failing 8. Any Scheme that implements floating point numbers internally as C floats rather than doubles will be fundamentally unable to pass all *-float64 tests. ==
Does that mean Bigloo implements numbers as C float type rather than double precision? Same for Gambit-C?
Chicken implements numbers as double precision?
I always thought there are two types for most of the 'modern languages' (e.g. Clean which I first learnt and Scheme compilers): long integer, and double precision C type?
Anyone any experience with srfi-56 on Gambit. I think it never went through the peer review process and has been withdrawn. But anyway.
Thanks, Siegfried ==
Siegfried Gonzi wrote:
I always thought there are two types for most of the 'modern languages' (e.g. Clean which I first learnt and Scheme compilers): long integer, and double precision C type?
Maybe some partial answer:
Scheme systems implementing the full numeric tower (like Gambit) have bignums, "much better" than "long integer". But Gambit also supports various fixed-size number vectors, see the documentation.
Regarding floating point numbers, you should be aware that x86 processors have 80-bit wide floating point numbers which are not standard; IEEE standardized 64-bit floating point numbers. The latter can be chosen by using the "double" type in C, but usually, compiled C code on PC's does internal calculations in the 80-bit types.
In contrast to this, AFAICT, Gambit always works with clean 64-bit numbers in it's flonum handling. Of course when using the FFI, C code usually uses the 80-bit numbers for intermediates.
(I haven't looked at the code you're referring to.)
Christian.
On Dec 20, 2007, at 12:10 PM, Siegfried Gonzi wrote:
Petite Chez 6.0a is the next most complete failing 6, followed by Gambit4b14 failing 8.
Anyone any experience with srfi-56 on Gambit.
I just downloaded srfi-56 reference implementation and the test suite; the final result with Gambit 4.1.2 was
463 tests passed. 0 tests failed.
There were lots of individual tests that were reported as "close- enough"; I don't yet know what that means.
Brad
-------- Original-Nachricht --------
Datum: Thu, 20 Dec 2007 13:42:43 -0500 Von: Bradley Lucier lucier@math.purdue.edu An: Siegfried Gonzi siegfried.gonzi@gmx.at CC: Bradley Lucier lucier@math.purdue.edu, Christian Jaeger christian@pflanze.mine.nu, gambit-list@iro.umontreal.ca Betreff: Re: [gambit-list] SRFI-56(Re: Reading binary files)
On Dec 20, 2007, at 12:10 PM, Siegfried Gonzi wrote:
Petite Chez 6.0a is the next most complete failing 6, followed by Gambit4b14 failing 8.
Anyone any experience with srfi-56 on Gambit.
I just downloaded srfi-56 reference implementation and the test suite; the final result with Gambit 4.1.2 was
463 tests passed. 0 tests failed.
There were lots of individual tests that were reported as "close- enough"; I don't yet know what that means.
I haven't tested srfi-56 yet (I am still hoping Bigloo can be installed on my machine. But any way Gambit and Chicken are very mature distributions).
What I do not understand: srfi-56 seems to be a very useful library. I browsed through the srfis and haven't yet found an srfi which can be used for reading binary data (floats, doubles, and big endian, small endian).
I think it is a pity that it is being withdrawn.
Regards, Siegfried Btw: I checked up the Bigloo manual and there it is said: flonums are C double precision numbers and fixnums are 32-2 bits.
Christian Jaeger wrote:
Bob McIsaac wrote:
The puzzling comment to my humble form said : "This code assumes that arguments to functions are evaluated left-to-right." But what of it? There is a stream of text to be evaluated. Choices are 1. left-right evaluation in one pass; 2. parsing the stream into tokens and sorting the tokens into a tree according to the rules of the grammar ... not Lispy. Choice 1 means typing parentheses so that the evaluator can be simple and recursive. ( I assume that evaluation results in a linked list)
Hm, we're not talking about nested expressions here. We're talking about multiple expressions given to one function. And at that point, regardless of whether your internal representation is lists or something else, the evaluator could be programmed to evaluate function arguments in different orders.
The problematic code is:
(bitwise-ior (bitwise-and (arithmetic-shift (read-u8 INP) 24) #xff000000) (bitwise-and (arithmetic-shift (read-u8 INP) 16) #xff0000) (bitwise-and (arithmetic-shift (read-u8 INP) 8) #xff00) (bitwise-and (read-u8 INP) #xff)))
It contains four (read-u8 INP) forms; the order in which they are evaluated matters (because reading from a port advances the position in the underlying stream as a side effect).
The problem is that those forms are indirectly positioned as arguments to bitwise-ior. bitwise-ior is a function. Scheme dictates that the arguments to functions are evaluated before the function is called (meaning before the body of the function definition is evaluated). So you're guaranteed that all four (read-u8 INP) forms have been evaluated before the bitwise-ior call. *But* Scheme does not specify in which order the four arguments of bitwise-ior are evaluated. So it's very well possible that some Scheme implementation first evaluates (bitwise-and (read-u8 INP) #xff)), then (bitwise-and (arithmetic-shift (read-u8 INP) 8) #xff00), then (bitwise-and (arithmetic-shift (read-u8 INP) 16) #xff0000), and at last (bitwise-and (arithmetic-shift (read-u8 INP) 24) #xff000000). Or the other way round. Or even in some other order or even in parallel. So the bytes coming from INP will end up at unspecified and possibly random places.
OTOH, special forms like let and let* do specify the order of evaluation (they are not functions). So writing
(let* ((a (bitwise-and (arithmetic-shift (read-u8 INP) 24) #xff000000)) (b (bitwise-and (arithmetic-shift (read-u8 INP) 16) #xff0000)) (c (bitwise-and (arithmetic-shift (read-u8 INP) 8) #xff00)) (d (bitwise-and (read-u8 INP) #xff))) (bitwise-ior a b c d))
makes the order of evaluation explicit (first a, then b, then c, then d) and thus portable (and deterministic).
Christian.
Thank you for taking time to explain.
-Bob-
-
Adrien Pierard -
Bob McIsaac -
Bradley Lucier -
Christian Jaeger -
Siegfried Gonzi