Hi Gambit community,
If I understood Gambit's current behavior right, then there's no way to (compile-file "irrelevant.scm" cc-options: "$(SYSTEM EXPLOIT CODE HERE)") .
So Gambit's approach for this argument is similar to a SQL injection attack safety.
This is not a big deal as compile-file probably never is exported to any untrusted user.
However, for symmetry, it makes sense - so spontaneously, while it was a small surprise to me that the conveniency feature of cc-options: "$(pkg-config --libs libjpeg")" wouldn't work, on second thought I think the fact that it does not work is rather a feature than a bug, so I'm positive about the current behavior.
Does anyone have an opinion?
Thanks.
2016-07-13 13:56 GMT+08:00 Adam adam.mlmb@gmail.com:
Dear Marc,
I'm in a nasty environment where I not can know what exact "-I" and "-l" arguments the C compiler and linker need.
For this reason, I need the pkg-config shell tool to figure it out for me!
It would have seemed logical to me that |compile-file|'s |cc-options:| and |ld-options:| would be evaluated by the system shell by the gambc-cc script, so that this would work:
echo '(print "Hello world\n")' > test.scm
GAMBC_CC_VERBOSE=yes gsc
(compile-file "test.scm" cc-options: "$(pkg-config --cflags "libjpeg")" ld-options: "$(pkg-config --libs "libjpeg")" )
However, it does not - but instead, the "$(p... strings are passed on verbatim to the C compiler, leading to this output:
gcc [...] -o "test.o1" $(pkg-config --cflags "libjpeg") test.c $(pkg-config --libs "libjpeg") gcc: error: $(pkg-config: No such file or directory gcc: error: "libjpeg"): No such file or directory gcc: error: $(pkg-config: No such file or directory gcc: error: "libjpeg"): No such file or directory gcc: error: unrecognized command line option ‘--cflags’ gcc: error: unrecognized command line option ‘--libs’ *** ERROR IN (console)@2.1 -- C compilation or link failed while compiling "test.scm"
I.e. GCC actually gets a "$(pkg-config" argument, a "--cflags" argument and a ""libjpeg"" argument, etc. .
So this test is a total catastrophe.
The question then comes, is this a bug or a feature?
I can totally see that it is your intended design of gsc+gambc-cc that those arguments should be passed exactly verbatim all the way to the C compiler, it makes sense, for instance as a correctness and a security measure.
So what I am asking here is if you have any thoughts about the convenience factor, or if you have any design thought here or this just design choice just was arbitrary.
Anyhow in the absence of shell-evaluation of pkg-config anywhere else, I need to add it explicitly, meaning then that the compile-file command should be:
(compile-file "test.scm" cc-options: (myshellrun "pkg-config --cflags "libjpeg"") ld-options: (myshellrun "pkg-config --libs "libjpeg"") )
where myshellrun is a procedure that involves open-process and reads its output to a string.
Please confirm that this indeed is the intended best practice for solving this problem.
Thanks!
Afficher les réponses par date
Yes that’s one argument (the avoidance of system exploits through some data that is passed to the shell).
There’s also a portability argument. Shell escaping varies between platforms (Unix/Windows), and also from one shell to another.
Moreover, manual shell escaping has always been a nightmare (how many backslashes to add?) that it is best to avoid it completely by letting the implementation do it automatically. I wouldn’t like a system where the user has to worry about the possibility of arguments containing “$” and “\” and double/single quotes.
So for your use-case I would suggest:
(define (pkg-config-lib name)
(call-with-input-process (list path: "pkg-config" arguments: (list "--libs" name)) read-line))
(pkg-config-lib "zlib")
"-lz"
(compile-file "test.scm" ld-options: (pkg-config-lib "libjpeg"))
Marc
On Jul 13, 2016, at 3:09 AM, Adam adam.mlmb@gmail.com wrote:
Hi Gambit community,
If I understood Gambit's current behavior right, then there's no way to (compile-file "irrelevant.scm" cc-options: "$(SYSTEM EXPLOIT CODE HERE)") .
So Gambit's approach for this argument is similar to a SQL injection attack safety.
This is not a big deal as compile-file probably never is exported to any untrusted user.
However, for symmetry, it makes sense - so spontaneously, while it was a small surprise to me that the conveniency feature of cc-options: "$(pkg-config --libs libjpeg")" wouldn't work, on second thought I think the fact that it does not work is rather a feature than a bug, so I'm positive about the current behavior.
Does anyone have an opinion?
Thanks.
2016-07-13 13:56 GMT+08:00 Adam adam.mlmb@gmail.com: Dear Marc,
I'm in a nasty environment where I not can know what exact "-I" and "-l" arguments the C compiler and linker need.
For this reason, I need the pkg-config shell tool to figure it out for me!
It would have seemed logical to me that |compile-file|'s |cc-options:| and |ld-options:| would be evaluated by the system shell by the gambc-cc script, so that this would work:
echo '(print "Hello world\n")' > test.scm
GAMBC_CC_VERBOSE=yes gsc
(compile-file "test.scm" cc-options: "$(pkg-config --cflags "libjpeg")" ld-options: "$(pkg-config --libs "libjpeg")" )
However, it does not - but instead, the "$(p... strings are passed on verbatim to the C compiler, leading to this output:
gcc [...] -o "test.o1" $(pkg-config --cflags "libjpeg") test.c $(pkg-config --libs "libjpeg") gcc: error: $(pkg-config: No such file or directory gcc: error: "libjpeg"): No such file or directory gcc: error: $(pkg-config: No such file or directory gcc: error: "libjpeg"): No such file or directory gcc: error: unrecognized command line option ‘--cflags’ gcc: error: unrecognized command line option ‘--libs’ *** ERROR IN (console)@2.1 -- C compilation or link failed while compiling "test.scm"
I.e. GCC actually gets a "$(pkg-config" argument, a "--cflags" argument and a ""libjpeg"" argument, etc. .
So this test is a total catastrophe.
The question then comes, is this a bug or a feature?
I can totally see that it is your intended design of gsc+gambc-cc that those arguments should be passed exactly verbatim all the way to the C compiler, it makes sense, for instance as a correctness and a security measure.
So what I am asking here is if you have any thoughts about the convenience factor, or if you have any design thought here or this just design choice just was arbitrary.
Anyhow in the absence of shell-evaluation of pkg-config anywhere else, I need to add it explicitly, meaning then that the compile-file command should be:
(compile-file "test.scm" cc-options: (myshellrun "pkg-config --cflags "libjpeg"") ld-options: (myshellrun "pkg-config --libs "libjpeg"") )
where myshellrun is a procedure that involves open-process and reads its output to a string.
Please confirm that this indeed is the intended best practice for solving this problem.
Thanks!
2016-07-13 21:45 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca:
Yes that’s one argument (the avoidance of system exploits through some data that is passed to the shell).
There’s also a portability argument. Shell escaping varies between platforms (Unix/Windows), and also from one shell to another.
Moreover, manual shell escaping has always been a nightmare (how many backslashes to add?) that it is best to avoid it completely by letting the implementation do it automatically. I wouldn’t like a system where the user has to worry about the possibility of arguments containing “$” and “\” and double/single quotes.
Ah those are fantastic arguments for it to be like it is now too.
So for your use-case I would suggest:
(define (pkg-config-lib name)
(call-with-input-process (list path: "pkg-config" arguments: (list "--libs" name)) read-line))
(pkg-config-lib "zlib")
"-lz"
(compile-file "test.scm" ld-options: (pkg-config-lib "libjpeg"))
Yup perfect.
Thanks!
Best regards
2016-07-13 22:47 GMT+08:00 Adam adam.mlmb@gmail.com:
2016-07-13 21:45 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca:
Yes that’s one argument (the avoidance of system exploits through some data that is passed to the shell).
There’s also a portability argument. Shell escaping varies between platforms (Unix/Windows), and also from one shell to another.
Moreover, manual shell escaping has always been a nightmare (how many backslashes to add?) that it is best to avoid it completely by letting the implementation do it automatically. I wouldn’t like a system where the user has to worry about the possibility of arguments containing “$” and “\” and double/single quotes.
Ah those are fantastic arguments for it to be like it is now too.
So for your use-case I would suggest:
(define (pkg-config-lib name)
(call-with-input-process (list path: "pkg-config" arguments: (list "--libs" name)) read-line))
(pkg-config-lib "zlib")
"-lz"
(compile-file "test.scm" ld-options: (pkg-config-lib "libjpeg"))
Yup perfect.
Thanks!
Best regards
2016-07-14 5:48 GMT+08:00 Gorges, Matthias MGorges@cw.bc.ca:
Dear Adam,
What made me run into the question of the cc- and ld-options argument to compile-file should work, was that I for the first time ever needed to link to a library that has different names on different platforms.
I am sorry, but I am not sure I fully understand your request. Thus far I have only encountered one package which has different linking names for different platforms: libpcap (which is libpcap.a on most platforms other than Win32, where it is libwpcap.a). My solution was to copy the library file to make it match, which solved this for me. - I don't know enough about Gambit-internals to properly comment on a more sensible approach but feel free to mention my comment when it does get discussed on the Gambit mailing list.
Hi Matthias,
Thanks for sharing your thoughts on this one - I think it's all clarified on the ML but I'll just explain it as translated to your usecase, for everyone's reference.
My usecase is *almost* the same as yours.
My only presumption is that you would be compiling your particular Gambit module that uses libpcap using |compile-file| from within a Gambit-based build script, as long as that is true then you would run into the following reasoning.
I used to do it the way you did (rename a library) but I got to a place where I don't want to, but instead harmonize with more platforms by somehow automatically adapting to them.
The reasons for this could be multiple - maybe you just want it for whatever "cleanliness" reason you have, or your build script won't have administrative privileges and that would be needed, or maybe you want to harmonize with some packaging system or otherwise plug in to the system-provided library file.
Your first trial would be trying (compile-file "yourmodule.scm" ld-options-prelude: "-lpcap -lwpcap") to see if you can find one command that would deliver on all platforms, and the outcome is that it fails, as when libpcap is there the absence of libwpcap would crash the linking, and when libwpcap is there the absence of libpcap would crash the linking.
So the learning is that you need |ld-options-prelude| linking (-l) arguments that are specific to the particular platform.
In my case, the "pkg-config" tool was available on all target platforms, so I realized that "pkg-config --libs pcap" would give me the right arguments on every intended target platform, and so my question became, how to run "pkg-config --libs pcap" and channelize its output into the |ld-options-prelude| argument, "the right way".
And so I tried (compile-file "yourmodule.scm" ld-options-prelude: "$(pkg-config --libs pcap)"), and it didn't work, because the gambc-cc script which |compile-file| employs, does *not* shell-evaluate cc-options nor ld-options-prelude, and hence the C compiler would get the arguments "$(pkg-config", "--libs", and "pcap)", which are all obviously totally meaningless to it and only will cause compilation failure.
And then, my question to the ML was, is this behavior of compile-file's cc-&ld-options not being shell-evaluated, *intended* and *desired*.
And what I personally find is that the ML conversation and in particular Marc's clarification (above) is that, it is perfectly desired.
And hece the proper behavior for my usecase is (compile-file "yourmodule.scm" ld-options-prelude: (some-procedure-that-runs-pkg-config-and-returns-all-its-stdout-output-as-a-string with-arguments: "--libs pcap")) .
If by any reason pcap would be linked to by "-lABCpcap" on Debian and "-lDEFpcap" on Redhat, and "-lGHIpcap" on whatever other Unix, this would solve the linking for those all those platforms.
And if you would need the same and not have pkg-config on your target platform e.g. Windows, you'd need additional logics e.g. (compile-file "yourmodule.scm" ld-options-prelude: (if windows-target? "-l wpcap" (some-procedure-that-runs-pkg-config -and-returns-all-its-stdout-output-as-a-string with-arguments: "--libs pcap")))) etc. .
Best regards.