-------- Message original --------
Dear Erick Lavoie,
The Runtime Environments, Systems, Layering and Virtualized Environments
(resolve_2013) program committee is delighted to inform you that your paper
#3 has been accepted for presentation at the workshop. We look forward to
seeing you in Houston.
Title: Harnessing Performance for Flexibility in Instrumenting a
Virtual Machine for JavaScript through Metacircularity
Authors: Erick Lavoie (Université de Montréal)
Bruno Dufour (Université de Montréal)
Marc Feeley (Université de Montréal)
Paper site: http://svr-hotcrp.cl.cam.ac.uk/hotcrp/resolve_2013/paper.php/3?cap=03aKUmFsHqrWBc
Your paper was one of 6 accepted out of 7 submissions. Congratulations!
Reviews and comments on your paper are appended to this email. The
submissions site also has the paper's reviews and comments, as well as more
information about review scores. Please read the reviews and comments and
revise your paper and presentation accordingly.
Contact the site administrator, Robert Watson <Robert.Watson@cl.cam.ac.uk>,
with any questions or concerns.
- resolve_2013 Submissions
===========================================================================
resolve_2013 Review #3A
Updated Monday 21 Jan 2013 12:04:51am UTC
---------------------------------------------------------------------------
Paper #3: Harnessing Performance for Flexibility in Instrumenting a
Virtual Machine for JavaScript through Metacircularity
---------------------------------------------------------------------------
Overall merit: 3. Weak accept
Reviewer expertise: 3. Knowledgeable
===== Paper summary =====
The paper describes instrumentation of JavaScript programs by, in
effect, building a metacircular VM implementation.
===== Comments for author =====
I am OK accepting this paper for the workshop -- it is on-topic, and I
expect people will be interested in JavaScript instrumentation and
debugging.
That said, I do not think the presentation of the work is very
effective, and (as the paper says) distinction with source-to-source
instrumentation is relatively slight.
In terms of presentation, I would suggest adding an early running
example. In general it is better to start with a simple example that
the reasder can understand, and then to move on to the general case.
The current paper introduces a lot of abstract detail before getting
on to a relatively complex example. I would add (1) a diagram showing
the layers and components of the system, (2) a worked example of a
single method call and the associated caches.
Detailed comments:
* "regular semantics of JS", suggest avoiding the word "regular" (it
can mean "ordinary", or "orthogonal", or "at fixed periods", so is
often ambiguous).
* Section 2, I was unclear what level the call, get, send operations
are working at -- and if these are actual implementations, or just
meant as pseudo-code. Again, I think having a simple example before
these details would help explain this better.
* Memoized methods -- I was lost at this point. I was unsure if this
was a feature of the input JS programs, or an implenentation
technique being used by the VM.
* The paper should be more candid about perf in the abstract and
intro: it is 2x slower than an interpreter, and much slower than a
JIT implementation (50x). It would be interesting to look in more
detail at the source of the perf costs -- are there particular kinds
of instrumentation that are costly? Is there evidence that a
smarter underlying VM could strip away un-used instrumentation?
===========================================================================
resolve_2013 Review #3B
Updated Friday 25 Jan 2013 12:18:29pm UTC
---------------------------------------------------------------------------
Paper #3: Harnessing Performance for Flexibility in Instrumenting a
Virtual Machine for JavaScript through Metacircularity
---------------------------------------------------------------------------
Overall merit: 4. Accept
Reviewer expertise: 3. Knowledgeable
===== Paper summary =====
This paper discusses the creation of a metacircular JavaScript virtual machine as a means of instrumenting JavaScript code without having to maintain an instrumentation framework alongside a complex performance oriented JIT VM. The authors propose using message sending semantics as the means of building this VM, which, when executed on top of a state-of-the-art JIT based javascript VM, provides acceptable performance in comparison to a modern interpreter based JavaScript VM.
===== Comments for author =====
This is an on-topic paper which should be of interest to attendees of the RESoLVE workshop. I think the message sending based techniques used to build the metacircular VM is interesting, as are the insights that they can perform code cacheing using the dynamically specializable abilities of Javascript.
However, the paper's presentation makes this work difficult to follow. The paper dives right into the low-level techniques used to build the VM, with no real discussion of how these techniques relate to the "compilation" of user-level code. Then, in section 4 a compilation example is presented, but this example is quite complex and it's difficult to follow how the previously discussed techniques are used in it. I think the paper could be improved by having a simple running example which permeates the implementation discussions to help describe them, then put all these running examples together in Section 4 to show how the approaches work together.
I also struggled to clearly understand when and where the optimizations to the code cache were happening. For example, how does the __memoization__ tag interact with the virtual machine. This might be clearer if the authors presented the transformations for a simple compilation example and described where the VM interacts with the source-to-source translated user code in order to carry out these transformations.
Also, the use of the word memoization seemed a bit misleading, since it would imply that the function's result is memoized meaning the function doesn't need to be called at all, whereas what is being discussed in the paper is the memorization of the message send operation, while the user-function is still being called.
Finally, it would have been interesting to see how instrumentation is inserted into the compiled code - there is very little information given on the implementation of Pn-spl or Pn-fast presented in the evaluation. I would have been interested to discover how the two approaches differ in more detail as well as some discussion on how tied Pn-fast is to the current VM implementation, and whether changes in the VMs implementation would require significant work to ensure the instrumentation features still worked correctly.
Overall though, I think this paper would be a worthwhile addition to the RESoLVE workshop.
===========================================================================
resolve_2013 Review #3C
Updated Sunday 27 Jan 2013 4:59:40am UTC
---------------------------------------------------------------------------
Paper #3: Harnessing Performance for Flexibility in Instrumenting a
Virtual Machine for JavaScript through Metacircularity
---------------------------------------------------------------------------
Overall merit: 2. Weak reject
Reviewer expertise: 2. Some familiarity
===== Paper summary =====
This paper describes the a Javascript instrumentation framework that relies on meta-circular
evaluation to simplify its construction. The paper describes a few optimizations that reduce the
overhead of instrumentation. It finally describes preliminary performance results comparing the new
Javascript VM (Photon) with a interpreter (SpiderMonkey) and a JIT-based (V8) VM.
===== Comments for author =====
This was a difficult paper to read. Perhaps I would be more positive about the work if I had an
easier time grasping some of the concepts presented. Unfortunately, I still have several basic
questions that I was unable to answer. So, more than anything, I'd encourage the authors to improve
the presentation of the work so that readers can fully understand its technical value.
A few suggestions:
* The Introduction does not clearly describe what problem this work addresses, and how it proposes
to do so. It is important to describe what type of instrumentation this work attempts to perform
(and why). I did not find this anywhere in the paper. Furthermore, why is it important to make
instrumentation a separate entity (is instrumentation targeted to writers of Javascript VMs or
applications writers?).
Certain phrases, such as, "Our approach relies on an object representation based on proxies that
exploit a previously unexploited constraint: the prototype chain of the proxies mirrors the
prototype chain of the objects" are difficult to begin to parse without the context of the
remainder of the paper. Even then, I find it challenging.
* Closed-world vs Open-world assumptions. First, please define these terms more carefully in your
context. The terms I knew were from the area of formal logic, and it is not clear how they apply
in your context.
More importantly, why does it matter which type of system you build? Do these provide the same
functionality to your target instrumentation?
* A premise in this work is that meta-circular evaluation will make building VMs with
instrumentation easier. Do you believe that new Narcissus VM, also based on meta-circular
evaluation, will make instrumentation JS code easier? Or is it simply a function that the VM
proposed in this work is small and highly customized to provide instrumentation hooks?
* Because I did not understand the introductory questions above, I was left wondering if going
through the trouble of building a new VM based on multi-circular evaluation is worth the
trouble. For example, there are numerous projects that perform compilation to Javascript (see:
http://altjs.org/).
Also, given that I'm not sure about the level of instrumentation desired, it seemed that this work
was largely replicating the concepts of Aspect-Oriented Programming, where the instrumentation is
designed as aspects for specific JS objects. And a simple plugin to JQuery allows one to extend JS
object behavior using AOP (which just a couple hundred lines of JS code; also the overhead of this
is expected to be minimal and only dependent on the instrumentation itself):
https://code.google.com/p/jquery-aop/
* About the messaging. Please first explain, at a high-level, why you decided to base your system on
messaging, and how it is used in your system. Second 2 started discussing messaging as if I had
knowledge of its purpose in the overall system design. As a consequence, all of Section 2 was very
confusing.
What's the difference between messages and a simple method invocation? Why is it necessary to wrap
method calls as messages?
* It seems that a fundamental Javascript feature that this work leverages is the ability to evaluate
code at run-time (eval() function). It would be helpful to clearly explain this in the paper
(maybe in a section describing the high-level concepts of Photon, before describing the messaging
layer, for example).
* In the experimental section, we see that Photon reduces the performance of V8 by 200x. This is
quite significant, and would make be weary to use such a heavy-weight instrumentation
framework. For example, for identifying hot-spots, it is not clear that this instrumentation
framework would not significantly alter the performance behavior of execution, making any
conclusions from this framework unreliable.
* A minor point, also in the experimental section, one data point stuck out. In Table 9, Pn-fast
receives a 25% performance boost over Pn for DeltaBlue. The only justification is that these are
"natural variations of results". I find this to be a non-trivial variation in execution, and would
urge future versions of this work to attempt to improve (or quantify) the statistical significance
of the results.
===========================================================================
resolve_2013 Review #3D
Updated Monday 28 Jan 2013 10:04:57am UTC
---------------------------------------------------------------------------
Paper #3: Harnessing Performance for Flexibility in Instrumenting a
Virtual Machine for JavaScript through Metacircularity
---------------------------------------------------------------------------
Overall merit: 3. Weak accept
Reviewer expertise: 2. Some familiarity
===== Paper summary =====
The authors use a hierarchy of proxy objects to "wrap" an application's own object hierarchy and use this to instrument the application. The same wrapping mechanism is used to also support inline-caching optimizations to reduce the overhead of instrumentation. In all cases, the approach exploits the proxy feature which will be coming in future versions of JavaScript.
===== Comments for author =====
What if application already uses proxy objects? How are these instrumented by the additional hooks? What are the (circularity) issues here and how do different attempts at instrumentation compose?
Some questions about the performance section:
+ explain what the baseline is better at the start of section 6 (indirection through proxies, on V8)
+ what is the impact of conservative assumptions for simpler invalidation of caches, etc.?
+ in tables 7 and 9: what does "v8 score" mean? isn't this geo mean?
+ the tables would read better as graphs
Have the authors looked at frameworks like the Spring (aspect-oriented) framework that does something like this in a less general way to build large Java applications?
Minor comments:
Introduction: "running of the VM" --> "running on the VM"
Table 3: "Lookup method then call." --> "Look up method, then call."
Table 4: "has his method being updated" --> "has its method updated"
Section 3.2: "requires to specialize" --> "requires us to specialize"
Section 4: "syntax tree nodes as preserved." --> "syntax tree nodes are preserved."
Section 5: "is in most cases, is also" --> ", in most cases, is also"
Section 6: "they have the same name" --> "they have the same name."
"should the prototype of an object changes" --> "should the prototype of an object change"
===========================================================================
resolve_2013 Review #3E
Updated Wednesday 30 Jan 2013 8:12:23pm UTC
---------------------------------------------------------------------------
Paper #3: Harnessing Performance for Flexibility in Instrumenting a
Virtual Machine for JavaScript through Metacircularity
---------------------------------------------------------------------------
Overall merit: 3. Weak accept
Reviewer expertise: 3. Knowledgeable
===== Paper summary =====
This paper focuses on the low-level details of instrumenting OO accessors and method calls in Javascript by transforming programs into a proxy-based message-sending representation. The paper gives details of the model, the code transformation required including consideration of various corner cases, and an empirical performance analysis.
The work exists in a vacuum - in that little motivation is given for instrumenting Javascript programs. Further there is only limited discussion (in opening paragraphs of sec 1, closing paras of sec 5) about alternative approaches to Javascript instrumentation.
In summary - a detailed technical paper, which could be of interest to RESOLVE workshop attendees. Hence weak accept.
===== Comments for author =====
Generally clear where the paper is going - although given the amount of technical details sometimes it is difficult to see the wood for the trees. I would appreciate more high-level motivation personally.
Sect 5.5 - related work - make this a new top-level section, rather than a subsection.
You show how this proxy-based technique can be useful for whole-program instrumentation - e.g. for counting number of method calls etc. Could it also be a good approach to dynamic aspect weaving for prototype-based languages?