Optimizing the parser is not the solution. Even if we rewrite it to allocate very little memory, the other parts of the compiler will be allocating lots more memory.
If just parsing fib takes over 256MB of heap space, there is obviously something we can do to optimize that. Furthermore, it's not necessarily the parser we should tweak here, rather the string functions.
Recall that the parsing takes a small fraction of the total execution time, and the execution time is a good measure of the allocation rate, so the rest of the compiler probably allocates a lot.
Allocations are currently very fast, and as I explained, our string code is very wasteful. I would bet the IR code allocates less than the parser, even though it's several times slower. Obviously, that's still no guarantee we can bootstrap without a GC.
We need a global optimization called "garbage collection".
I agree that we need a GC. Either that or we need to make Tachyon run in 64-bit mode so it can run on that server of yours with 48GBs of RAM ;)
P.S. are we sure the problem is not that repeated array operations, such as series of pushes, generate quadratic space usage?
Array pushes append elements at the end. If the arrays do not have enough capacity to push an element, their capacity is doubled.
- Maxime