Statistics and Computer Science Departments
Stanford University

Jeudi 23 février, 15:30-16:30, Salle 6214 

Pavillon André-Aisenstadt, Université de Montréal, 2920 Chemin de la Tour
        

Modern machine learning systems rely on complex and distributed pipelines that require extensive hyperparameter tuning to achieve the desired performance. Careful tuning can result in significant speedups and improvements in solution quality. However, the dimensionality of the hyperparameter space often makes the use of brute-force search prohibitive. To make things worse, components can interact in unexpected ways and make joint tuning necessary. These challenges preclude non-experts from fully utilizing the potential of modern machine learning tools and call for a deeper understanding of the effect hyperparameters have on the quality and performance of a system.

In this talk, I will discuss examples of tuning large-scale learning and inference systems. I will focus on recent work that reveals a previously unknown interaction between system and algorithm dynamics when running an asynchronous learning system. Asynchronous methods are widely used for their superior throughput, but have limited theoretical justification when applied to non-convex problems. I will show that running stochastic gradient descent (SGD) in an asynchronous manner can be viewed as adding a momentum-like term to the SGD iteration. This result does not assume convexity of the objective function, so is applicable to deep learning systems. Furthermore, using a hybrid parallel architecture we can control the level of asynchrony, a new hyperparameter. Theory then implies that  jointly tuning momentum and the level of asynchrony can significantly reduce the number of iterations, necessary for an asynchronous system to achieve the same solution. This line of work provides a number of necessary components for realizing the vision of an automated machine learning pipeline.