- Lisa_teatalk - mailman.iro.umontreal.ca

Survey on New TeaTalk Schedule
by Kyung Hyun Cho 20 Jan '15

20 Jan '15

Dear all, As I have announced earlier, we are at the moment considering to move the tea talk slot from Wednesday to either the beginning of or the end of each week. Since the last email, I have received a couple of replies from the lab members. Considering those replies, we have narrowed down our options into 1. Fridays: 14h00 to 15h00 (preferred) 2. Mondays: 10h30 to 11h30 Please, reply to this email to express your preference among these options. I will take "no reply" as a sign of "either of them being fine". Best, - K

15 19

21 Jan 2015: Talk by Adriana Romero on Wednesday at 13.30 (AA3195)
by Kyung Hyun Cho 18 Jan '15

18 Jan '15

Dear all, Adriana Romero, a visiting researcher at the lab from Barcelona (!), will tell us about her latest work done here. She will teach us how to raise our neural net to become a leaner, but taller network. The talk will be at the usual place AA3195 starting from 13.30 on Wednesday (21 Jan). Hope to see many of you there! - Cho === - *Speaker*: Adriana Romero (University of Barcelona) - *Date/Time*: 13.30 - 14.30, 21 Jan 2015 - *Place*: AA3195 - *Title*: FitNets: Hints for Thin Deep Nets - *Abstract*: While depth tends to improve network performances, it also makes gradient-based training more difficult since deeper networks tend to be more non-linear. The recently proposed knowledge distillation approach is aimed at obtaining small and fast-to-execute models, and it has shown that a student network could imitate the soft output of a larger teacher network or ensemble of networks. We extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student. Because the student intermediate hidden layer will generally be smaller than the teacher's intermediate hidden layer, additional parameters are introduced to map the student hidden layer to the prediction of the teacher hidden layer. This allows one to train deeper students that can generalize better or run faster, a trade-off that is controlled by the chosen student capacity. For example, on CIFAR-10, a deep student network with almost 10.4 times less parameters outperforms a larger, state-of-the-art teacher network. - *Bio*: I am currently a PhD student at University of Barcelona, advised by Dr. Carlo Gatta, working on deep learning models and their applications to computer vision. I graduated form Universitat Autònoma de Barcelona in 2010 as Computer Engineer and from Universitat Politècnica de Catalunya in 2012 as M.Sc. in Artificial Intelligence. My previous work was focused on unsupervised sparse feature learning algorithms to train both shallow and deep networks. In August 2014, I joined LISA lab for 6 months, where I've been working with Prof. Yoshua Bengio on training thin and deep student networks from shallower and wide teacher networks.

1 0

Introduction to Tea Talk, this Wednesday @13.30, AA3195
by Kyung Hyun Cho 12 Jan '15

12 Jan '15

Dear all, This week, we will not have any tea talk. There will be a quick introductory session for new people (including those who started last autumn) on how tea talks at the lab are arranged and what one should do to schedule her/his own tea talk. So, if you're relatively new to the lab and would like to know why there are so many interesting/boring talks at the lab and how you can be yet another interesting/boring speaker, please, come to AA3195 at 13.30 this Wednesday. Best, - Cho

1 0

Tea Talks 2015
by Kyung Hyun Cho 07 Jan '15

07 Jan '15

Dear all, With the new year begins the new series of tea talks! As usual with the last year, we will host a series of (tea) talks on Wednesdays starting from 13.30 lasting one hour at the usual place of AA3195. Again as usual, the day, time and place are subject to change. You can check the schedule of the tea talks at https://docs.google.com/spreadsheets/d/1_bbHxcm4r-rs63chcKfHuBsM3roHuud4CJJ… We already have a few talks scheduled for this semester, but obviously a lot of slots are empty now. Please, feel free to contact me if you want to give a talk. I especially encourage new visitors to tell us about themselves and their research at the tea talks. Best, - K

1 0

Re: [Lisa_teatalk] Schedule of Talks before and after NIPS 2014
by Kyung Hyun Cho 16 Déc '14

16 Déc '14

Dear all, It is the final day of our Pre-/Post- NIPS seminar series tomorrow! Those two talks will also conclude the series of talks we have had this year (2014). Both of the talks will be held at the usual place at 10.30 and at 13.30. - 10.30 - 11.30 <15 million predictions per seconds: the scientific challenges at Criteo> by Nicolas le Roux (Criteo) - 13.30 - 14.30 <General Stochastic Networks for Classification> by Matthias Zohrer (TU Graz) Hope to see many of you tomorrow! - Cho *= Schedule =* Available also at: http://lisa.iro.umontreal.ca/seminars_en.html + 3 Dec 2014 (Wed) - 13.30 - 14.30 <A Multiplicative Model for Learning Distributed Text-Based Attribute Representations> by Ryan Kiros (University of Toronto) + 4 Dec 2014 (Thu) - 10.30 - 11.30 <What if computer can perceive the world like we do?> by Xinyi Chen, Peng (Eric) Ding (DeepGlint) - 13.30 - 14.30 <Learning Compact Yet Rich Models for Text> by Sarath Chandar (IBM India) + 5 Dec 2014 (Fri) - 10.30 - 11.30 <Joint Training of Deep Autoencoders> by Dr. Yingbo Zhou (University at Buffalo, The State University of New York) - 13.30 - 14.30 <How Watson Learns Superhuman Jeopardy! Strategies> by Dr. Gerald Tesauro (IBM T. J. Watson Research Center) + 15 Dec 2014 (Mon) - 10.30 - 11.30 <On the Number of Linear Regions of Deep Neural Networks> by Dr. Guido Montufar <MPI MIS> - 13.30 - 14.30 <Machine Learning with Nearest Neighbors> by Prof. Yung-Kyun Noh (KAIST) *+ 16 Dec 2014 (Tue)* * - 10.30 - 11.30 <15 million predictions per seconds: the scientific challenges at Criteo> by Nicolas le Roux (Criteo)* * - 13.30 - 14.30 <General Stochastic Networks for Classification> by Matthias Zohrer (TU Graz)*

2 3

Schedule of Talks before and after NIPS 2014
by Kyung Hyun Cho 14 Déc '14

14 Déc '14

Dear all, We will have a number of visitors at the lab before and after NIPS 2014. Many of the talks will be interesting to many of the lab members, and I would like to recommend all to be at all of the talks. Also, I'd like to warmly welcome anyone on this mailing list who's not at the LISA lab to attend the talks. Most of the talks will be held at AA3195 (Pavillon André-Aisenstadt, 2920 Chemin de la tour, Montréal). Please, check back frequently for the locations of the talks on http://lisa.iro.umontreal.ca/seminars_en.html. There may be a few more talks added in a next few days, so again, please check the schedule online frequently. Best, - Cho *= Schedule = * + *3 Dec 2014* (Wed) - 13.30 - 14.30 <A Multiplicative Model for Learning Distributed Text-Based Attribute Representations> by Ryan Kiros (University of Toronto) + *4 Dec 2014* (Thu) - 10.30 - 11.30 <What if computer can perceive the world like we do?> by Xinyi Chen, Peng (Eric) Ding (DeepGlint) - 13.30 - 14.30 <Learning Compact Yet Rich Models for Text> by Sarath Chandar (IBM India) + *5 Dec 2014* (Fri) - 10.30 - 11.30 <Joint Training of Deep Autoencoders> by Dr. Yingbo Zhou (University at Buffalo, The State University of New York) - 13.30 - 14.30 <How Watson Learns Superhuman Jeopardy! Strategies> by Dr. Gerald Tesauro (IBM T. J. Watson Research Center) + *15 Dec 2014* (Mon) - 13.30 - 14.30 <TBD> by Prof. Yung-Kyun Noh (KAIST) + *16 Dec 2014* (Tue) - 13.30 - 14.30 <General Stochastic Networks for Classification> by Matthias Zohrer (TU Graz)

2 6

[URGEN] Re: Schedule of Talks before and after NIPS 2014
by Kyung Hyun Cho 04 Déc '14

04 Déc '14

Dear all, I made a huge mistake in the last email announcing that the talk this afternoon will be at AA3195. That is wrong! The talk will start in *20 min* at *Z-209* in the main building (Roger Gaudry). Sorry about the confusion! - Cho On Wed, Dec 3, 2014 at 7:46 PM, Kyung Hyun Cho <cho.k.hyun(a)gmail.com> wrote: > Dear all, > > This is a reminder of *two* talks scheduled tomorrow. The first talk will > be given by Xinyi Chen and Peng (Eric) Ding from DeepGlint. The second talk > will be given by Sarath Chandar from IBM India. > > Both of these talks will be at AA3195. The first one starts at 10.30, and > the second one at 13.30. > > Hope to see many of you there! > - Cho > > P.S. I will probably prepare snacks only for the afternoon's talk, unless > anyone objects. Anyone who objects to this is kindly asked to bring snacks > for everyone to the morning's talk :) > > *= Schedule =* > > Available also at: http://lisa.iro.umontreal.ca/seminars_en.html > > *+ 3 Dec 2014 (Wed)* > - 13.30 - 14.30 <A Multiplicative Model for Learning Distributed > Text-Based Attribute Representations> by Ryan Kiros (University of Toronto) > > *+ 4 Dec 2014 (Thu)* > - 10.30 - 11.30 <What if computer can perceive the world like we do?> by > Xinyi Chen, Peng (Eric) Ding (DeepGlint) > - 13.30 - 14.30 <Learning Compact Yet Rich Models for Text> by Sarath > Chandar (IBM India) > > *+ 5 Dec 2014 (Fri)* > - 10.30 - 11.30 <Joint Training of Deep Autoencoders> by Dr. Yingbo Zhou > (University at Buffalo, The State University of New York) > - 13.30 - 14.30 <How Watson Learns Superhuman Jeopardy! Strategies> by > Dr. Gerald Tesauro (IBM T. J. Watson Research Center) > > *+ 15 Dec 2014 (Mon)* > - 10.30 - 11.30 <On the Number of Linear Regions of Deep Neural > Networks> by Dr. Guido Montufar <MPI MIS> > - 13.30 - 14.30 <Machine Learning with Nearest Neighbors> by Prof. > Yung-Kyun Noh (KAIST) > > *+ 16 Dec 2014 (Tue)* > - 13.30 - 14.30 <General Stochastic Networks for Classification> by > Matthias Zohrer (TU Graz) > > >

1 0

Tea Talk 26 Nov Wed @13.30 AA3195 by Antoine Chassang
by Kyung Hyun Cho 25 Nov '14

25 Nov '14

Dear all, Antoine Chassang will tell us about his work on face detection during his internship at the LISA lab. The talk will be held at the usual place AA3195 starting from 13.30 next Wednesday (26 Nov). Hope to see many of you at the talk! - Cho === - Speaker: Antoine Chassang - Date/Time: 26 Nov (Wed) 13.30 - 14.30 - Place: AA3195 - Title: Face Detection - Abstract: Whereas the state of the art is relying on viewpoint-specialized models which need additional pose-related annotations, we trained convolutional networks on a large dataset (700k faces extracted from Google Images) to learn invariance to pose. We evaluated our method on FDDB, a standard benchmark for face detection, and we show that we can achieve state of the art performance. To speed up the inference, we investigated model compression using a teacher-student formulation where the student network learns to reproduce the teacher output. The compressed model is deeper and thinner, with less parameters than the teacher.

1 1

Talk 5 Dec Fri @13.30 AA3195 by Dr. Gerald Tesauro
by Kyung Hyun Cho 21 Nov '14

21 Nov '14

Dear all, This is an early reminder of the talk by Dr. Gerald Tesauro from IBM Research Center on *5 Dec *(Friday) starting from 13.30 at AA3195. He will tell us about the exciting story behind how Watson beat human competitors on Jeopardy!. Best, - Cho P.S. From now on, you can check the schedules of the future (and past) seminars at http://lisa.iro.umontreal.ca/seminars_en.html. The same page in French will appear soon. === - Speaker: Dr. Gerald Tesauro (IBM TJ Watson Research Center) - Date/Time: 13.30 - 14.30, 5 Dec - Place: AA3195 - Title: How Watson Learns Superhuman Jeopardy! Strategies - Abstract: Major advances in Question Answering technology were needed for Watson to play "Jeopardy!" at championship level -- the show requires rapid-fire answers to challenging natural language questions, broad general knowledge, high precision, and accurate confidence estimates. In addition, Jeopardy! features four types of decision making carrying great strategic importance: (1) selecting the next clue when in control of the board; (2) deciding whether to attempt to buzz in; (3) wagering on Daily Doubles; (4) wagering in Final Jeopardy. This talk describes how Watson makes the above decisions using innovative quantitative methods that, in principle, maximize Watson's overall winning chances. We first describe our development of faithful simulation models of human contestants and the Jeopardy! game environment. We then present specific learning/optimization methods used in each strategy algorithm: these methods span a range of popular AI research topics, including Bayesian inference, game theory, Dynamic Programming, Reinforcement Learning, and real-time "rollouts." Application of these methods yielded superhuman game strategies for Watson that significantly enhanced its overall competitive record. Joint work with David Gondek, Jon Lenchner, James Fan and John Prager. - Speaker Bio: Gerald Tesauro is a Research Staff Member at IBM's TJ Watson Research Center. He is best known for creating TD-Gammon, a self-teaching neural network that learned to play backgammon at human world championship level. He has also worked on theoretical and applied machine learning in a wide variety of other settings, including multi-agent learning, dimensionality reduction, computer virus recognition, computer chess (Deep Blue), intelligent e-commerce agents and autonomic computing. Dr. Tesauro serves on the NIPS Foundation Board of Directors, and was recently elected AAAI Fellow.

1 0

Talk 21 Nov Fri @15.00 AA3195 by Dr. John Hershey
by Kyung Hyun Cho 21 Nov '14

21 Nov '14

Dear all, Dr. John Hershey (Mistubishi Electric Research Labs, US) will tell us about the connection between model-free approaches (e.g. deep neural networks) and model-based approaches (e.g. probabilistic graphical models) and how we can utilize this connection. His talk will start from 15.00 next Friday (21 Nov) at the usual place AA3195. Hope to see many of you there! - Cho === - Speaker: Dr. John Hershey (Mistubishi Electric Research Labs) - Date/Time: 15.00 - 16.00, 21 Nov - Place: AA3195 - Title: Deep Unfolding: Infusing Deep Architectures with Generative Model Inference - Abstract: Model-based methods and deep neural networks have both been tremendously successful paradigms in machine learning. In model-based methods, problem domain knowledge can be built into the constraints of the model, typically at the expense of difficulties during inference. In contrast, deterministic deep neural networks are constructed in such a way that inference is straightforward, but their architectures are rather generic and it can be unclear how to incorporate problem domain knowledge. This work aims to obtain the advantages of both approaches. To do so, we start with a model-based approach and unfold the iterations of its inference method to form a layer-wise structure. We then decouple the model parameters across layers to increase the network's learning capacity. This results in novel neural-network-like architectures that incorporate our model-based constraints, but can be trained discriminatively to perform fast and accurate inference. We show how this framework can be applied to a non-negative matrix factorization model to obtain a new kind of non-negative deep neural network, that can be trained using a multiplicative backpropagation-style update algorithm. We present speech enhancement experiments showing that our approach is competitive with conventional neural networks despite using far fewer parameters. - Bio: John Hershey is a researcher at Mistubishi Electric Research Labs (MERL), in Cambridge, MA since 2010. Prior to that, John spent 5 years as a researcher at IBM's Watson Research Center in New York, in the Speech Algorithms and Engines group, and one year as visiting researcher in the speech group at Microsoft Research, in Redmond, WA. His obtained his Ph.D. at the University of California, San Diego. He is currently working on machine learning for signal enhancement and separation, speech recognition, language processing, and adaptive user interfaces.

1 1