Commentaires sur l'article RESoLVE. Erick -------- Message original -------- Sujet: [resolve_2013] Accepted paper #3 "Harnessing Performance for Flexibility..." Date : Fri, 01 Feb 2013 03:57:07 +0000 De : Robert.Watson@cl.cam.ac.uk Pour : Erick Lavoie <acm@ericklavoie.com> Copie à : Robert Watson <Robert.Watson@cl.cam.ac.uk> Dear Erick Lavoie, The Runtime Environments, Systems, Layering and Virtualized Environments (resolve_2013) program committee is delighted to inform you that your paper #3 has been accepted for presentation at the workshop. We look forward to seeing you in Houston. Title: Harnessing Performance for Flexibility in Instrumenting a Virtual Machine for JavaScript through Metacircularity Authors: Erick Lavoie (Université de Montréal) Bruno Dufour (Université de Montréal) Marc Feeley (Université de Montréal) Paper site: http://svr-hotcrp.cl.cam.ac.uk/hotcrp/resolve_2013/paper.php/3?cap=03aKUmFsH... Your paper was one of 6 accepted out of 7 submissions. Congratulations! Reviews and comments on your paper are appended to this email. The submissions site also has the paper's reviews and comments, as well as more information about review scores. Please read the reviews and comments and revise your paper and presentation accordingly. Contact the site administrator, Robert Watson <Robert.Watson@cl.cam.ac.uk>, with any questions or concerns. - resolve_2013 Submissions =========================================================================== resolve_2013 Review #3A Updated Monday 21 Jan 2013 12:04:51am UTC --------------------------------------------------------------------------- Paper #3: Harnessing Performance for Flexibility in Instrumenting a Virtual Machine for JavaScript through Metacircularity --------------------------------------------------------------------------- Overall merit: 3. Weak accept Reviewer expertise: 3. Knowledgeable ===== Paper summary ===== The paper describes instrumentation of JavaScript programs by, in effect, building a metacircular VM implementation. ===== Comments for author ===== I am OK accepting this paper for the workshop -- it is on-topic, and I expect people will be interested in JavaScript instrumentation and debugging. That said, I do not think the presentation of the work is very effective, and (as the paper says) distinction with source-to-source instrumentation is relatively slight. In terms of presentation, I would suggest adding an early running example. In general it is better to start with a simple example that the reasder can understand, and then to move on to the general case. The current paper introduces a lot of abstract detail before getting on to a relatively complex example. I would add (1) a diagram showing the layers and components of the system, (2) a worked example of a single method call and the associated caches. Detailed comments: * "regular semantics of JS", suggest avoiding the word "regular" (it can mean "ordinary", or "orthogonal", or "at fixed periods", so is often ambiguous). * Section 2, I was unclear what level the call, get, send operations are working at -- and if these are actual implementations, or just meant as pseudo-code. Again, I think having a simple example before these details would help explain this better. * Memoized methods -- I was lost at this point. I was unsure if this was a feature of the input JS programs, or an implenentation technique being used by the VM. * The paper should be more candid about perf in the abstract and intro: it is 2x slower than an interpreter, and much slower than a JIT implementation (50x). It would be interesting to look in more detail at the source of the perf costs -- are there particular kinds of instrumentation that are costly? Is there evidence that a smarter underlying VM could strip away un-used instrumentation? =========================================================================== resolve_2013 Review #3B Updated Friday 25 Jan 2013 12:18:29pm UTC --------------------------------------------------------------------------- Paper #3: Harnessing Performance for Flexibility in Instrumenting a Virtual Machine for JavaScript through Metacircularity --------------------------------------------------------------------------- Overall merit: 4. Accept Reviewer expertise: 3. Knowledgeable ===== Paper summary ===== This paper discusses the creation of a metacircular JavaScript virtual machine as a means of instrumenting JavaScript code without having to maintain an instrumentation framework alongside a complex performance oriented JIT VM. The authors propose using message sending semantics as the means of building this VM, which, when executed on top of a state-of-the-art JIT based javascript VM, provides acceptable performance in comparison to a modern interpreter based JavaScript VM. ===== Comments for author ===== This is an on-topic paper which should be of interest to attendees of the RESoLVE workshop. I think the message sending based techniques used to build the metacircular VM is interesting, as are the insights that they can perform code cacheing using the dynamically specializable abilities of Javascript. However, the paper's presentation makes this work difficult to follow. The paper dives right into the low-level techniques used to build the VM, with no real discussion of how these techniques relate to the "compilation" of user-level code. Then, in section 4 a compilation example is presented, but this example is quite complex and it's difficult to follow how the previously discussed techniques are used in it. I think the paper could be improved by having a simple running example which permeates the implementation discussions to help describe them, then put all these running examples together in Section 4 to show how the approaches work together. I also struggled to clearly understand when and where the optimizations to the code cache were happening. For example, how does the __memoization__ tag interact with the virtual machine. This might be clearer if the authors presented the transformations for a simple compilation example and described where the VM interacts with the source-to-source translated user code in order to carry out these transformations. Also, the use of the word memoization seemed a bit misleading, since it would imply that the function's result is memoized meaning the function doesn't need to be called at all, whereas what is being discussed in the paper is the memorization of the message send operation, while the user-function is still being called. Finally, it would have been interesting to see how instrumentation is inserted into the compiled code - there is very little information given on the implementation of Pn-spl or Pn-fast presented in the evaluation. I would have been interested to discover how the two approaches differ in more detail as well as some discussion on how tied Pn-fast is to the current VM implementation, and whether changes in the VMs implementation would require significant work to ensure the instrumentation features still worked correctly. Overall though, I think this paper would be a worthwhile addition to the RESoLVE workshop. =========================================================================== resolve_2013 Review #3C Updated Sunday 27 Jan 2013 4:59:40am UTC --------------------------------------------------------------------------- Paper #3: Harnessing Performance for Flexibility in Instrumenting a Virtual Machine for JavaScript through Metacircularity --------------------------------------------------------------------------- Overall merit: 2. Weak reject Reviewer expertise: 2. Some familiarity ===== Paper summary ===== This paper describes the a Javascript instrumentation framework that relies on meta-circular evaluation to simplify its construction. The paper describes a few optimizations that reduce the overhead of instrumentation. It finally describes preliminary performance results comparing the new Javascript VM (Photon) with a interpreter (SpiderMonkey) and a JIT-based (V8) VM. ===== Comments for author ===== This was a difficult paper to read. Perhaps I would be more positive about the work if I had an easier time grasping some of the concepts presented. Unfortunately, I still have several basic questions that I was unable to answer. So, more than anything, I'd encourage the authors to improve the presentation of the work so that readers can fully understand its technical value. A few suggestions: * The Introduction does not clearly describe what problem this work addresses, and how it proposes to do so. It is important to describe what type of instrumentation this work attempts to perform (and why). I did not find this anywhere in the paper. Furthermore, why is it important to make instrumentation a separate entity (is instrumentation targeted to writers of Javascript VMs or applications writers?). Certain phrases, such as, "Our approach relies on an object representation based on proxies that exploit a previously unexploited constraint: the prototype chain of the proxies mirrors the prototype chain of the objects" are difficult to begin to parse without the context of the remainder of the paper. Even then, I find it challenging. * Closed-world vs Open-world assumptions. First, please define these terms more carefully in your context. The terms I knew were from the area of formal logic, and it is not clear how they apply in your context. More importantly, why does it matter which type of system you build? Do these provide the same functionality to your target instrumentation? * A premise in this work is that meta-circular evaluation will make building VMs with instrumentation easier. Do you believe that new Narcissus VM, also based on meta-circular evaluation, will make instrumentation JS code easier? Or is it simply a function that the VM proposed in this work is small and highly customized to provide instrumentation hooks? * Because I did not understand the introductory questions above, I was left wondering if going through the trouble of building a new VM based on multi-circular evaluation is worth the trouble. For example, there are numerous projects that perform compilation to Javascript (see: http://altjs.org/). Also, given that I'm not sure about the level of instrumentation desired, it seemed that this work was largely replicating the concepts of Aspect-Oriented Programming, where the instrumentation is designed as aspects for specific JS objects. And a simple plugin to JQuery allows one to extend JS object behavior using AOP (which just a couple hundred lines of JS code; also the overhead of this is expected to be minimal and only dependent on the instrumentation itself): https://code.google.com/p/jquery-aop/ * About the messaging. Please first explain, at a high-level, why you decided to base your system on messaging, and how it is used in your system. Second 2 started discussing messaging as if I had knowledge of its purpose in the overall system design. As a consequence, all of Section 2 was very confusing. What's the difference between messages and a simple method invocation? Why is it necessary to wrap method calls as messages? * It seems that a fundamental Javascript feature that this work leverages is the ability to evaluate code at run-time (eval() function). It would be helpful to clearly explain this in the paper (maybe in a section describing the high-level concepts of Photon, before describing the messaging layer, for example). * In the experimental section, we see that Photon reduces the performance of V8 by 200x. This is quite significant, and would make be weary to use such a heavy-weight instrumentation framework. For example, for identifying hot-spots, it is not clear that this instrumentation framework would not significantly alter the performance behavior of execution, making any conclusions from this framework unreliable. * A minor point, also in the experimental section, one data point stuck out. In Table 9, Pn-fast receives a 25% performance boost over Pn for DeltaBlue. The only justification is that these are "natural variations of results". I find this to be a non-trivial variation in execution, and would urge future versions of this work to attempt to improve (or quantify) the statistical significance of the results. =========================================================================== resolve_2013 Review #3D Updated Monday 28 Jan 2013 10:04:57am UTC --------------------------------------------------------------------------- Paper #3: Harnessing Performance for Flexibility in Instrumenting a Virtual Machine for JavaScript through Metacircularity --------------------------------------------------------------------------- Overall merit: 3. Weak accept Reviewer expertise: 2. Some familiarity ===== Paper summary ===== The authors use a hierarchy of proxy objects to "wrap" an application's own object hierarchy and use this to instrument the application. The same wrapping mechanism is used to also support inline-caching optimizations to reduce the overhead of instrumentation. In all cases, the approach exploits the proxy feature which will be coming in future versions of JavaScript. ===== Comments for author ===== What if application already uses proxy objects? How are these instrumented by the additional hooks? What are the (circularity) issues here and how do different attempts at instrumentation compose? Some questions about the performance section: + explain what the baseline is better at the start of section 6 (indirection through proxies, on V8) + what is the impact of conservative assumptions for simpler invalidation of caches, etc.? + in tables 7 and 9: what does "v8 score" mean? isn't this geo mean? + the tables would read better as graphs Have the authors looked at frameworks like the Spring (aspect-oriented) framework that does something like this in a less general way to build large Java applications? Minor comments: Introduction: "running of the VM" --> "running on the VM" Table 3: "Lookup method then call." --> "Look up method, then call." Table 4: "has his method being updated" --> "has its method updated" Section 3.2: "requires to specialize" --> "requires us to specialize" Section 4: "syntax tree nodes as preserved." --> "syntax tree nodes are preserved." Section 5: "is in most cases, is also" --> ", in most cases, is also" Section 6: "they have the same name" --> "they have the same name." "should the prototype of an object changes" --> "should the prototype of an object change" =========================================================================== resolve_2013 Review #3E Updated Wednesday 30 Jan 2013 8:12:23pm UTC --------------------------------------------------------------------------- Paper #3: Harnessing Performance for Flexibility in Instrumenting a Virtual Machine for JavaScript through Metacircularity --------------------------------------------------------------------------- Overall merit: 3. Weak accept Reviewer expertise: 3. Knowledgeable ===== Paper summary ===== This paper focuses on the low-level details of instrumenting OO accessors and method calls in Javascript by transforming programs into a proxy-based message-sending representation. The paper gives details of the model, the code transformation required including consideration of various corner cases, and an empirical performance analysis. The work exists in a vacuum - in that little motivation is given for instrumenting Javascript programs. Further there is only limited discussion (in opening paragraphs of sec 1, closing paras of sec 5) about alternative approaches to Javascript instrumentation. In summary - a detailed technical paper, which could be of interest to RESOLVE workshop attendees. Hence weak accept. ===== Comments for author ===== Generally clear where the paper is going - although given the amount of technical details sometimes it is difficult to see the wood for the trees. I would appreciate more high-level motivation personally. Sect 5.5 - related work - make this a new top-level section, rather than a subsection. You show how this proxy-based technique can be useful for whole-program instrumentation - e.g. for counting number of method calls etc. Could it also be a good approach to dynamic aspect weaving for prototype-based languages?