[Lisa_seminaires] Reminder - Tomorrow 15:00 - Talk by Rion Snow

Pascal Lamblin lamblinp at iro.umontreal.ca
Dim 16 Nov 19:45:55 EST 2008


Just to remind you, because Monday is tomorrow:


Title:    Cheap and Fast  But is it Good?
Evaluating Nonexpert Annotations for Machine Learning Tasks
Speaker:  Rion Snow
Date:     Monday, November 17th, 15:00
Place:    Room 3195 (pav. André-Aisenstadt, UdeM)

Abstract:
Human annotation is crucial for many machine learning tasks but can be
expensive and time-consuming. We explore the use of Amazon's Mechanical
Turk web service, a significantly cheaper and faster method for
collecting annotations from a broad base of paid non-expert contributors
over the Web. We investigate five task in the field of natural language
processing: affect recognition, word similarity, recognizing textual
entailment, event temporal ordering, and word sense disambiguation. For
all five, we show high agreement between Mechanical Turk non-expert
annotations and existing gold standard labels provided by expert
labelers. For the task of affect recognition, we also show that using
non-expert labels for training machine learning algorithms can be as
effective as using gold standard annotations from experts. We propose a
technique for bias correction that significantly improves annotation
quality on two tasks. We conclude that many large labeling tasks can be
effectively designed and carried out in this method at a fraction of the
usual expense.

A summary of this work may be found online at:
http://blog.doloreslabs.com/2008/09/amt-fast-cheap-good-machine-learning/.

Bio:
Rion Snow is a PhD Candidate in Computer Science at Stanford University,
advised by Professors Andrew Ng and Dan Jurafsky. Rion works in the
intersection of machine learning and natural language processing, with a
focus in computational semantics. He leads the Stanford Wordnet Project,
which aims at learning large-scale semantic networks automatically from
natural text. His work on automatically inferring semantic taxonomies
received the Best Paper Award at the 2006 conference for the Association
of Computational Linguistics. His publications and contact information
may be found at his home page: http://ai.stanford.edu/~rion/.
-- 
Pascal


Plus d'informations sur la liste de diffusion Lisa_seminaires