[Lisa_seminaires] [Lisa_labo] Fwd: Connectionists: The ReScience journal

Kyle Kastner kastnerkyle at gmail.com
Dim 6 Sep 01:45:21 EDT 2015


Important to distinguish replication and reproduction. Replication is (in
my opinion) when experiments in similar or same environments work.
Reproduction is more independent - implementing the same ideas in a very
different setting. This uncovers much more subtle (but important) issues
than basic replication. And we need both in ML - but replication doubly so.

Also dwf's point is a strong reason to minimize dependencies in research
and library development. Its a personal opinion but one that has helped
greatly in the past.
On Sep 5, 2015 7:18 PM, "Jörg Bornschein" <bornj at iro.umontreal.ca> wrote:

>
> I agree and I was basically thinking the same thing (+that Docker or other
> container virtualization techniques could ease the pain [*]).
>
> But for reproducability in the scientific sense it is already a huge win
> when an independent implementation ran only once or twice on the machines
> of the reproducing authors (and the editors). From that point of view it
> would not be devastating if the code was not in a runnable state a few
> months later.
>
>
>    j
>
> [*] I actually wonder why they did't make the dependencies more explicit
> in their submission format.
>
>
>
>
> On Sat, Sep 5, 2015 at 7:00 PM David Warde-Farley <
> d.warde.farley at gmail.com> wrote:
>
>> Very encouraging to see this happening and that other people are
>> concerned about it.
>>
>> I would add that reproducibility in machine learning looks simple
>> compared to other scientific domains, but looks are deceiving. Every
>> "simple Python script" is built upon a broad and deep tower of library
>> dependencies, leading to an exponential number of ways that your
>> computing environment can conspire against you (nevermind hardware
>> differences...).
>>
>> On Sat, Sep 5, 2015 at 6:35 PM, Yoshua Bengio
>> <yoshua.umontreal at gmail.com> wrote:
>> >
>> > Very interesting!
>> > Reproducibility is VERY weak in the machine learning community, and
>> needs to
>> > be improved.
>> >
>> > ---------- Forwarded message ----------
>> > From: Nicolas P. Rougier <Nicolas.Rougier at inria.fr>
>> > Date: 2015-09-03 8:57 GMT-04:00
>> > Subject: Connectionists: The ReScience journal
>> > To: Connectionists group <connectionists at cs.cmu.edu>
>> >
>> >
>> >
>> > It's our great pleasure to announce the creation of "ReScience" which
>> is a
>> > peer-reviewed journal that targets computational research and
>> encourages the
>> > explicit replication of already published research, promoting new and
>> > open-source implementations in order to ensure that the original
>> research is
>> > reproducible.
>> >
>> > To achieve such a goal, the whole editing chain is radically different
>> from
>> > any other traditional scientific journal. ReScience lives on GitHub
>> where
>> > each new implementation is made available together with comments,
>> > explanations and tests. Each submission takes the form of a pull request
>> > that is publicly reviewed and tested in order to guarantee that any
>> > researcher can re-use it.
>> >
>> > Students are strongly encourage to submit to ReScience. Even if the
>> > publishing model is a bit different from other academic journals, this
>> will
>> > give them a first experience at peer-reviewed publishing where they
>> have to
>> > use a rigorous and scientific approach.
>> >
>> >         • More on the journal website:
>> > https://github.com/ReScience/ReScience/wiki
>> >         • Current issue:
>> > https://github.com/ReScience/ReScience/wiki/Current-Issue
>> >         • FAQ:
>> > https://github.com/ReScience/ReScience/wiki/Frequently-Asked-Questions
>> >         • Follow us on twitter (@ReScienceEds):
>> > https://twitter.com/rescienceeds
>> >
>> > And if you're familiar with Git and GitHub, you can also become a
>> reviewer:
>> > just contact us.
>> >
>> >
>> > Konrad Hinsen & Nicolas Rougier
>> >
>> >
>> > _______________________________________________
>> > Lisa_labo mailing list
>> > Lisa_labo at iro.umontreal.ca
>> > https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo
>> >
>> _______________________________________________
>> Lisa_labo mailing list
>> Lisa_labo at iro.umontreal.ca
>> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_labo
>>
>
> _______________________________________________
> Lisa_seminaires mailing list
> Lisa_seminaires at iro.umontreal.ca
> https://webmail.iro.umontreal.ca/mailman/listinfo/lisa_seminaires
>
>
-------------- section suivante --------------
Une pièce jointe HTML a été nettoyée...
URL: http://webmail.iro.umontreal.ca/pipermail/lisa_seminaires/attachments/20150906/de6865c6/attachment.html 


Plus d'informations sur la liste de diffusion Lisa_seminaires