Re: [Lisa_seminaires] [Lisa_labo] Fwd: Connectionists: The ReScience journal
To what extent can this be remedied by using aws or similar? On Sep 5, 2015 7:00 PM, "David Warde-Farley" <d.warde.farley@gmail.com> wrote:
Very encouraging to see this happening and that other people are concerned about it.
I would add that reproducibility in machine learning looks simple compared to other scientific domains, but looks are deceiving. Every "simple Python script" is built upon a broad and deep tower of library dependencies, leading to an exponential number of ways that your computing environment can conspire against you (nevermind hardware differences...).
On Sat, Sep 5, 2015 at 6:35 PM, Yoshua Bengio <yoshua.umontreal@gmail.com> wrote:
Very interesting! Reproducibility is VERY weak in the machine learning community, and
needs to
be improved.
---------- Forwarded message ---------- From: Nicolas P. Rougier <Nicolas.Rougier@inria.fr> Date: 2015-09-03 8:57 GMT-04:00 Subject: Connectionists: The ReScience journal To: Connectionists group <connectionists@cs.cmu.edu>
It's our great pleasure to announce the creation of "ReScience" which is a peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research is reproducible.
To achieve such a goal, the whole editing chain is radically different from any other traditional scientific journal. ReScience lives on GitHub where each new implementation is made available together with comments, explanations and tests. Each submission takes the form of a pull request that is publicly reviewed and tested in order to guarantee that any researcher can re-use it.
Students are strongly encourage to submit to ReScience. Even if the publishing model is a bit different from other academic journals, this will give them a first experience at peer-reviewed publishing where they have to use a rigorous and scientific approach.
• More on the journal website: https://github.com/ReScience/ReScience/wiki • Current issue: https://github.com/ReScience/ReScience/wiki/Current-Issue • FAQ: https://github.com/ReScience/ReScience/wiki/Frequently-Asked-Questions • Follow us on twitter (@ReScienceEds): https://twitter.com/rescienceeds
And if you're familiar with Git and GitHub, you can also become a reviewer: just contact us.
Konrad Hinsen & Nicolas Rougier
_______________________________________________ Lisa_labo mailing list Lisa_labo@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo
_______________________________________________ Lisa_labo mailing list Lisa_labo@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo
Afficher les réponses par date
A lot of the papers that I've read recently use a private training set and a public testing set (for example, the Google and Facebook face recognition papers). In areas where the largest datasets are privately held, research is probably going to continue to be impossible to fully reproduce. On Sat, Sep 5, 2015 at 7:03 PM, David Krueger <david.scott.krueger@gmail.com
wrote:
To what extent can this be remedied by using aws or similar? On Sep 5, 2015 7:00 PM, "David Warde-Farley" <d.warde.farley@gmail.com> wrote:
Very encouraging to see this happening and that other people are concerned about it.
I would add that reproducibility in machine learning looks simple compared to other scientific domains, but looks are deceiving. Every "simple Python script" is built upon a broad and deep tower of library dependencies, leading to an exponential number of ways that your computing environment can conspire against you (nevermind hardware differences...).
On Sat, Sep 5, 2015 at 6:35 PM, Yoshua Bengio <yoshua.umontreal@gmail.com> wrote:
Very interesting! Reproducibility is VERY weak in the machine learning community, and
needs to
be improved.
---------- Forwarded message ---------- From: Nicolas P. Rougier <Nicolas.Rougier@inria.fr> Date: 2015-09-03 8:57 GMT-04:00 Subject: Connectionists: The ReScience journal To: Connectionists group <connectionists@cs.cmu.edu>
It's our great pleasure to announce the creation of "ReScience" which is a peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research is reproducible.
To achieve such a goal, the whole editing chain is radically different from any other traditional scientific journal. ReScience lives on GitHub where each new implementation is made available together with comments, explanations and tests. Each submission takes the form of a pull request that is publicly reviewed and tested in order to guarantee that any researcher can re-use it.
Students are strongly encourage to submit to ReScience. Even if the publishing model is a bit different from other academic journals, this will give them a first experience at peer-reviewed publishing where they have to use a rigorous and scientific approach.
• More on the journal website: https://github.com/ReScience/ReScience/wiki • Current issue: https://github.com/ReScience/ReScience/wiki/Current-Issue • FAQ: https://github.com/ReScience/ReScience/wiki/Frequently-Asked-Questions • Follow us on twitter (@ReScienceEds): https://twitter.com/rescienceeds
And if you're familiar with Git and GitHub, you can also become a reviewer: just contact us.
Konrad Hinsen & Nicolas Rougier
_______________________________________________ Lisa_labo mailing list Lisa_labo@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo
_______________________________________________ Lisa_labo mailing list Lisa_labo@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo
_______________________________________________ Lisa_labo mailing list Lisa_labo@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo
Yes. It's sad. 2015-09-05 19:44 GMT-04:00 Alex Lamb <alex6200@gmail.com>:
A lot of the papers that I've read recently use a private training set and a public testing set (for example, the Google and Facebook face recognition papers). In areas where the largest datasets are privately held, research is probably going to continue to be impossible to fully reproduce.
On Sat, Sep 5, 2015 at 7:03 PM, David Krueger < david.scott.krueger@gmail.com> wrote:
To what extent can this be remedied by using aws or similar? On Sep 5, 2015 7:00 PM, "David Warde-Farley" <d.warde.farley@gmail.com> wrote:
Very encouraging to see this happening and that other people are concerned about it.
I would add that reproducibility in machine learning looks simple compared to other scientific domains, but looks are deceiving. Every "simple Python script" is built upon a broad and deep tower of library dependencies, leading to an exponential number of ways that your computing environment can conspire against you (nevermind hardware differences...).
On Sat, Sep 5, 2015 at 6:35 PM, Yoshua Bengio <yoshua.umontreal@gmail.com> wrote:
Very interesting! Reproducibility is VERY weak in the machine learning community, and
needs to
be improved.
---------- Forwarded message ---------- From: Nicolas P. Rougier <Nicolas.Rougier@inria.fr> Date: 2015-09-03 8:57 GMT-04:00 Subject: Connectionists: The ReScience journal To: Connectionists group <connectionists@cs.cmu.edu>
It's our great pleasure to announce the creation of "ReScience" which is a peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research is reproducible.
To achieve such a goal, the whole editing chain is radically different from any other traditional scientific journal. ReScience lives on GitHub where each new implementation is made available together with comments, explanations and tests. Each submission takes the form of a pull request that is publicly reviewed and tested in order to guarantee that any researcher can re-use it.
Students are strongly encourage to submit to ReScience. Even if the publishing model is a bit different from other academic journals, this will give them a first experience at peer-reviewed publishing where they have to use a rigorous and scientific approach.
• More on the journal website: https://github.com/ReScience/ReScience/wiki • Current issue: https://github.com/ReScience/ReScience/wiki/Current-Issue • FAQ: https://github.com/ReScience/ReScience/wiki/Frequently-Asked-Questions • Follow us on twitter (@ReScienceEds): https://twitter.com/rescienceeds
And if you're familiar with Git and GitHub, you can also become a reviewer: just contact us.
Konrad Hinsen & Nicolas Rougier
_______________________________________________ Lisa_labo mailing list Lisa_labo@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo
_______________________________________________ Lisa_labo mailing list Lisa_labo@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo
_______________________________________________ Lisa_labo mailing list Lisa_labo@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo
_______________________________________________ Lisa_labo mailing list Lisa_labo@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo
participants (3)
-
Alex Lamb -
David Krueger -
Yoshua Bengio