Unsupervised Neural Network Learning via an Algorithmic Lens

 

Abstract: The tremendous success of deep learning in applications compels us to revisit its theoretical foundations. While an overarching rigorous theory for deep learning algorithms remains elusive, recent breakthroughs may pave the way towards such a theory. However, these results primarily have focused on the supervised setting, which typically relies on the availability of abundant, pristine, labeled training data. 
I will first describe several new theoretical results for neural learning algorithms for the unsupervised setting. Our approach rests on two key ideas: (1) the learned representations often themselves obey conciseness assumptions (such as compositionality, sparseness, and/or democracy), and (2) datasets often obey certain natural generative modeling assumptions. Our results can be viewed as formal evidence that (shallow) networks are indeed unsupervised feature learning mechanisms, and may shed insights on how to train larger stacked architectures. 
I will then describe an approach for unsupervised learning that succeeds in the setting of limited, coarse, unlabeled data. Our approach rests on a new generative modeling architecture, together an associated training algorithm, that is explicitly physics-aware. We demonstrate this approach in an application in computational material science, and show its benefits over the state of the art.