I have been looking into the issue of bootstrapping Gambit directly from the sources on github. Currently this is not possible. The latest .tar.gz release (obtained from the Gambit wiki) needs to be built, and then the latest commits must be pulled and compiled in a final "make". This process confuses novice users and is generally a pain. It would be much better if a clone of the github repository gave all the required pieces (as is the case for most other projects on github).
This "direct bootstrap" is not possible because the .c files generated by the Gambit compiler from the .scm files when compiling itself are not stored on the github repository. The generated .c files are very large and including all versions would make the repository considerably larger.
I am proposing to include the generated .c files in the repo, but only commit changes occasionally. Basically, every time a new release would be published (i.e. a .tar.gz with a new version number) the new .c files would be committed to the repository.
I have put on github a mockup of the sources to demonstrate how this would work. The repo is at https://github.com/feeley/test9 .
To build the mockup from github, start with these commands:
git clone git@github.com:feeley/test9.git cd test9 make bootstrap
This will create the gsc-boot executable (the Gambit compiler which can be used to recompile the Gambit .scm files from scratch should they be changed, or new versions pulled from the repo). Now we can remove the (possibly stale) generated .c files obtained from the repo and regenerate the correct .c files with the new gsc-boot:
make bootclean make
When commiting changes to the repo, two make targets can be used:
make commit
when the generated .c files should not be added to the repository, and
make commit-major
when the generated .c files should be added to the repository (typically when a new release is created).
I would be interested in having some feedback on this change before going ahead with them on the actual Gambit source code.
Marc
Afficher les réponses par date
While I believe you considered this already - so the issue with uploading these c files all the time, is that the total repo, including any clones of it made, would grow in size very fast, because a great number of versions of those .c files would be in the version history.
Does Git perhaps have some kind of "permanent unlimited erase" feature where you can actually remove file commits completely (from all the repo incl its history)? I'd suppose it does.
Though, for this to be meaningful, Git would require to propagate those erases (to github from your local machine and so on) on git pull/push. This could or could not exist, at least it's by far not an unreasonable feature to exist, I suppose?
If all this exists, you could make a script that would go through all versions of the entire git repo and erase all of those .c files in that fashion, except for the very most recent version (so cloners always get the newest set) and those that belong to very major Gambit versions (like, N.N.0:s or so) (so people can always have old Gambit versions readily compilable).
Checked on #git, they said this can be done with git log filename and filter-branch , and said there's clear examples in man filter-branch .
Brgds
Den 12 april 2012 22:46 skrev Marc Feeley feeley@iro.umontreal.ca:
I have been looking into the issue of bootstrapping Gambit directly from the sources on github. Currently this is not possible. The latest .tar.gz release (obtained from the Gambit wiki) needs to be built, and then the latest commits must be pulled and compiled in a final "make". This process confuses novice users and is generally a pain. It would be much better if a clone of the github repository gave all the required pieces (as is the case for most other projects on github).
This "direct bootstrap" is not possible because the .c files generated by the Gambit compiler from the .scm files when compiling itself are not stored on the github repository. The generated .c files are very large and including all versions would make the repository considerably larger.
I am proposing to include the generated .c files in the repo, but only commit changes occasionally. Basically, every time a new release would be published (i.e. a .tar.gz with a new version number) the new .c files would be committed to the repository.
I have put on github a mockup of the sources to demonstrate how this would work. The repo is at https://github.com/feeley/test9 .
To build the mockup from github, start with these commands:
git clone git@github.com:feeley/test9.git cd test9 make bootstrap
This will create the gsc-boot executable (the Gambit compiler which can be used to recompile the Gambit .scm files from scratch should they be changed, or new versions pulled from the repo). Now we can remove the (possibly stale) generated .c files obtained from the repo and regenerate the correct .c files with the new gsc-boot:
make bootclean make
When commiting changes to the repo, two make targets can be used:
make commit
when the generated .c files should not be added to the repository, and
make commit-major
when the generated .c files should be added to the repository (typically when a new release is created).
I would be interested in having some feedback on this change before going ahead with them on the actual Gambit source code.
Marc
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On 2012-04-12, at 4:53 PM, Mikael wrote:
While I believe you considered this already - so the issue with uploading these c files all the time, is that the total repo, including any clones of it made, would grow in size very fast, because a great number of versions of those .c files would be in the version history.
Does Git perhaps have some kind of "permanent unlimited erase" feature where you can actually remove file commits completely (from all the repo incl its history)? I'd suppose it does.
Though, for this to be meaningful, Git would require to propagate those erases (to github from your local machine and so on) on git pull/push. This could or could not exist, at least it's by far not an unreasonable feature to exist, I suppose?
If all this exists, you could make a script that would go through all versions of the entire git repo and erase all of those .c files in that fashion, except for the very most recent version (so cloners always get the newest set) and those that belong to very major Gambit versions (like, N.N.0:s or so) (so people can always have old Gambit versions readily compilable).
Checked on #git, they said this can be done with git log filename and filter-branch , and said there's clear examples in man filter-branch .
Brgds
But what would be the advantage of this over a plain old .tar.gz which contains exactly what is needed to build the system? These .tar.gz files, which I call "releases", are available for all releases of Gambit that were made (since 3.0 at least).
The allure of a git repo is that you can easily keep up to date with the most recent changes to the repo, and possibly contribute changes to the head of the repo. It is nice also to be able to move back and forth in the commit history to determine when bugs were introduced.
Marc
Den 13 april 2012 03:30 skrev Marc Feeley feeley@iro.umontreal.ca:
On 2012-04-12, at 4:53 PM, Mikael wrote:
..
If all this exists, you could make a script that would go through all
versions of the entire git repo and erase all of those .c files in that fashion, except for the very most recent version (so cloners always get the newest set) and those that belong to very major Gambit versions (like, N.N.0:s or so) (so people can always have old Gambit versions readily compilable).
..
But what would be the advantage of this over a plain old .tar.gz which contains exactly what is needed to build the system? These .tar.gz files, which I call "releases", are available for all releases of Gambit that were made (since 3.0 at least).
The allure of a git repo is that you can easily keep up to date with the most recent changes to the repo, and possibly contribute changes to the head of the repo. It is nice also to be able to move back and forth in the commit history to determine when bugs were introduced.
Marc
What about purposing the git repo, to be a code repo for all commits ever made to Gambit, but, to only provide "out of the box" compilability for the very latest version i.e. the one you get when you git clone the repo['s main branch]?
For anyone who wants compilable older versions, there's the .tar.bz2 files archived on Gambit's homepage to download.
What do you say, does it make sense?
(Btw, nowadays, do the .tar.bz:s contain a copy of the git repo files too as it was at that version?)
Mikael
On Thu, 12 Apr 2012 15:46:21 EDT Marc Feeley feeley@iro.umontreal.ca wrote:
This "direct bootstrap" is not possible because the .c files generated by the Gambit compiler from the .scm files when compiling itself are not stored on the github repository. The generated .c files are very large and including a ll versions would make the repository considerably larger.
I am proposing to include the generated .c files in the repo, but only commit changes occasionally. Basically, every time a new release would be published (i.e. a .tar.gz with a new version number) the new .c files would be commited to the repository.
I think the key idea is that bootstrap .c files must not be in the same place as generated .c files otherwise If/when you make local changes there will be conficts when you later do a git pull. I'v seen a similar problem in OS projects (where you need an initial set of binaries to get anything done but later you want to install locally built binaries in the same place).
Separating them also helps when you don't catch your mistake until after the buggy compiler generates a new set of .c file (and you have already installed it)!
How about if `make bootstrap' fetches .c files from a gambit-bootstrap repo strictly for bootstrapping?
"make commit-major" can then copy generated files to the separate gambit-bootstrap repo and commit them there. I like a separate repo because you can then blow it away once bootstrapped -- typical users won't want it beyond the initial bootstrap. Also note that what you really want is an initial gambit binary -- generated .c files is just one way to get it. I'd be happy to just start with a signed binary for my platform!
On 2012-04-12, at 4:55 PM, Bakul Shah wrote:
On Thu, 12 Apr 2012 15:46:21 EDT Marc Feeley feeley@iro.umontreal.ca wrote:
This "direct bootstrap" is not possible because the .c files generated by the Gambit compiler from the .scm files when compiling itself are not stored on the github repository. The generated .c files are very large and including a ll versions would make the repository considerably larger.
I am proposing to include the generated .c files in the repo, but only commit changes occasionally. Basically, every time a new release would be published (i.e. a .tar.gz with a new version number) the new .c files would be commited to the repository.
I think the key idea is that bootstrap .c files must not be in the same place as generated .c files otherwise If/when you make local changes there will be conficts when you later do a git pull. I'v seen a similar problem in OS projects (where you need an initial set of binaries to get anything done but later you want to install locally built binaries in the same place).
Can you explain this more? Do you mean manual changes to the generated .c files? That should *never* be done! The .c files are not meant for human consumption.
Separating them also helps when you don't catch your mistake until after the buggy compiler generates a new set of .c file (and you have already installed it)!
That's not a problem because when you do a "make" it builds the Gambit compiler executable in gsc/gsc . The compiler that is used to generate the .c files is ./gsc-comp (which will be renamed gsc-boot in the near future) which is the bootstrapped compiler. The executable gsc/gsc is copied to gsc-boot by a "make bootstrap", but this is done infrequently (basically when a new release is created). Note that because we are using git, it is easy to checkout the previous working version, build it and copy it to gsc-boot, and move back to the head to fix things (and only do a "make bootstrap" after proper testing).
How about if `make bootstrap' fetches .c files from a gambit-bootstrap repo strictly for bootstrapping?
"make commit-major" can then copy generated files to the separate gambit-bootstrap repo and commit them there. I like a separate repo because you can then blow it away once bootstrapped -- typical users won't want it beyond the initial bootstrap. Also note that what you really want is an initial gambit binary -- generated .c files is just one way to get it. I'd be happy to just start with a signed binary for my platform!
Binaries are not a good idea since they are not portable (and it is a pain to maintain a fairly complete set). The closest to that model, but completely portable, is what we have currently, i.e. the .tar.gz releases that are self-contained (you grab the latest one, then "make", then pull the latest sources from the repo, and "make" again). But that's not the model people expect. I want a "clone + make" model.
Marc
On Thu, Apr 12, 2012 at 03:46:21PM -0400, Marc Feeley wrote:
I am proposing to include the generated .c files in the repo, but only commit changes occasionally. Basically, every time a new release would be published (i.e. a .tar.gz with a new version number) the new .c files would be committed to the repository.
The .c bloat in the repository would be greatly reduced if gambit's output were to be a continuous function of the input -- that small changes in the gambit source code would usually produce small changes in the .c files. I'm already assuming, of course, that only the part of Gambit that's absolutely necessary for the bootstrap is to be committed as .c files.
-- hendrik
On 2012-04-13, at 10:40 AM, Hendrik Boom wrote:
On Thu, Apr 12, 2012 at 03:46:21PM -0400, Marc Feeley wrote:
I am proposing to include the generated .c files in the repo, but only commit changes occasionally. Basically, every time a new release would be published (i.e. a .tar.gz with a new version number) the new .c files would be committed to the repository.
The .c bloat in the repository would be greatly reduced if gambit's output were to be a continuous function of the input -- that small changes in the gambit source code would usually produce small changes in the .c files.
That is not the case. Think of the .c file as assembly code with lots of numbered labels for various things (code labels, global variable labels, etc). Just adding a single code label (because an "if" was added) will renumber many of the remaining code labels.
Marc
On 2012-04-13, at 11:00 AM, Marc Feeley wrote:
On 2012-04-13, at 10:40 AM, Hendrik Boom wrote:
On Thu, Apr 12, 2012 at 03:46:21PM -0400, Marc Feeley wrote:
I am proposing to include the generated .c files in the repo, but only commit changes occasionally. Basically, every time a new release would be published (i.e. a .tar.gz with a new version number) the new .c files would be committed to the repository.
The .c bloat in the repository would be greatly reduced if gambit's output were to be a continuous function of the input -- that small changes in the gambit source code would usually produce small changes in the .c files.
That is not the case. Think of the .c file as assembly code with lots of numbered labels for various things (code labels, global variable labels, etc). Just adding a single code label (because an "if" was added) will renumber many of the remaining code labels.
And also, as far as I know, git stores the *whole* file as soon as there is a change in the file (whether it is one byte or all the bytes).
Marc
Hallo,
On Fri, Apr 13, 2012 at 5:14 PM, Marc Feeley feeley@iro.umontreal.ca wrote:
And also, as far as I know, git stores the *whole* file as soon as there is a change in the file (whether it is one byte or all the bytes).
If Git did that, it would be woefully inefficient as a VCS.
http://progit.org/book/ch9-4.html
Cheers,
On Fri, Apr 13, 2012 at 11:14:45AM -0400, Marc Feeley wrote:
And also, as far as I know, git stores the *whole* file as soon as there is a change in the file (whether it is one byte or all the bytes).
Even the ancient RCS didn't do that. As far as I know, all source-code versioning systems store something like difference files to save space. Some of them get wuite sophisticated.
-- hendrik
On 2012-04-13, at 12:49 PM, Hendrik Boom wrote:
On Fri, Apr 13, 2012 at 11:14:45AM -0400, Marc Feeley wrote:
And also, as far as I know, git stores the *whole* file as soon as there is a change in the file (whether it is one byte or all the bytes).
Even the ancient RCS didn't do that. As far as I know, all source-code versioning systems store something like difference files to save space. Some of them get wuite sophisticated.
I thought I read somewhere that Linus had decided to store whole files because this was faster. But I may be confusing this with some other aspect of git. In any case, the generated .c files change so much for minor changes of the .scm files that there's no advantage to store a diff for them instead of a copy of the .c file (which is about 6 to 10 times larger than the .scm file) . For the .scm files there is certainly an advantage to using a diff.
Marc
This is probably wandering off topic, but git really does store snapshots of whole files, not patch lists or stacks of diffs. http://learn.github.com/p/intro.html http://eagain.net/articles/git-for-computer-scientists/
The git gc command and packing will compress stored objects and generate deltas and such, but absent that, your files are there byte for byte. http://metalinguist.wordpress.com/2007/12/06/the-woes-of-git-gc-aggressive-a...
-Fred
On Sat, Apr 14, 2012 at 12:58 AM, Marc Feeley feeley@iro.umontreal.ca wrote:
On 2012-04-13, at 12:49 PM, Hendrik Boom wrote:
On Fri, Apr 13, 2012 at 11:14:45AM -0400, Marc Feeley wrote:
And also, as far as I know, git stores the *whole* file as soon as there is a change in the file (whether it is one byte or all the bytes).
Even the ancient RCS didn't do that. As far as I know, all source-code versioning systems store something like difference files to save space. Some of them get wuite sophisticated.
I thought I read somewhere that Linus had decided to store whole files because this was faster. But I may be confusing this with some other aspect of git. In any case, the generated .c files change so much for minor changes of the .scm files that there's no advantage to store a diff for them instead of a copy of the .c file (which is about 6 to 10 times larger than the .scm file) . For the .scm files there is certainly an advantage to using a diff.
Marc
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Sat, Apr 14, 2012 at 01:46:27AM +0800, Frederick LeMaster wrote:
This is probably wandering off topic, but git really does store snapshots of whole files, not patch lists or stacks of diffs. http://learn.github.com/p/intro.html http://eagain.net/articles/git-for-computer-scientists/
The git gc command and packing will compress stored objects and generate deltas and such, but absent that, your files are there byte for byte. http://metalinguist.wordpress.com/2007/12/06/the-woes-of-git-gc-aggressive-a...
Apparently it does use deltas of some sort to save on storage -- it's just a bit clever about how it organises them so you don't end up with enormously long chains of them. You're right that these deltas have no conceptual part of the data base -- that's quite independent. It's just a data storage technique. But it's effective enough that you don't have the complete text of many separate files taking up lots and lots of disk space, and that's what the original discussion was about, I think.
I don't think they are the usual diff deltas, though, but there's very little information there about just what it *does* do.
-- hendrik
On Fri, Apr 13, 2012 at 11:00:40AM -0400, Marc Feeley wrote:
On 2012-04-13, at 10:40 AM, Hendrik Boom wrote:
On Thu, Apr 12, 2012 at 03:46:21PM -0400, Marc Feeley wrote:
I am proposing to include the generated .c files in the repo, but only commit changes occasionally. Basically, every time a new release would be published (i.e. a .tar.gz with a new version number) the new .c files would be committed to the repository.
The .c bloat in the repository would be greatly reduced if gambit's output were to be a continuous function of the input -- that small changes in the gambit source code would usually produce small changes in the .c files.
That is not the case. Think of the .c file as assembly code with lots of numbered labels for various things (code labels, global variable labels, etc). Just adding a single code label (because an "if" was added) will renumber many of the remaining code labels.
That's usually the case with generated C code. It takes a lot of effort to do otherwise, by making all generated names depend in a relatively deterministic way on relevant names in the original source code.
I tried that once. I never really succeeded, but it did mean that when I fixed code generation bugs, diffs between old and new output were somewhat easier to read.
-- hendrik
-
Alex Queiroz -
Bakul Shah -
Frederick LeMaster -
Hendrik Boom -
Marc Feeley -
Mikael