Interpretable Lightweight Transformer via Unrolling of Learned Graph Smoothness Priors
Speaker
Gene Cheung
Ph.D in Electrical Engineering and Computer Science, University of California, Berkeley.
Title
"Interpretable Lightweight Transformer via Unrolling of Learned Graph Smoothness Priors"
Abstract
We build interpretable and lightweight transformer-like neural networks by unrolling iterative optimization algorithms that minimize graph smoothness priors subject to an interpolation constraint. The crucial insight is that a normalized signal-dependent graph learning module amounts to a variant of the basic self-attention mechanism in conventional transformers. Unlike "black-box" transformers that require learning of large key, query and value matrices to compute scaled dot products as affinities and subsequent output embeddings, resulting in huge parameter sets, our unrolled networks employ shallow CNNs to learn low-dimensional features per node to establish pairwise Mahalanobis distances and construct sparse similarity graphs.
At each layer, given a learned graph, the target interpolated signal is simply a low-pass filtered output derived from the minimization of an assumed graph smoothness prior, leading to a dramatic reduction in parameter count. Experiments for two image interpolation applications verify the restoration performance, parameter efficiency and robustness to covariate shift of our graph-based unrolled networks compared to conventional transformers.
About Speaker
Gene Cheung received the M.S. and Ph.D. degrees in electrical engineering and computer science from the University of California, Berkeley, in 1998 and 2000, respectively. He was a senior researcher in Hewlett-Packard Laboratories Japan, Tokyo, from 2000 till 2009. He was an assistant then associate professor in National Institute of Informatics (NII) in Tokyo, Japan, from 2009 till 2018. He is now a professor and York research chair (YRC) in York University, Toronto, Canada. His research interests include 3D imaging and graph signal processing.
He has served as associate editor for multiple journals, including IEEE Transactions on Multimedia (2007--2011), IEEE Transactions on Circuits and Systems for Video Technology (2016--2017) and IEEE Transactions on Image Processing (2015--2019). He currently serves as senior associate editor for IEEE Signal Processing Letters (2021--present). He is a co-author of several paper awards and nominations, including the best student paper finalist in ICASSP 2021, best student paper award in ICIP 2013, ICIP 2017 and IVMSP 2016, best paper runner-up award in ICME 2012, and IEEE Signal Processing Society (SPS) Japan best paper award 2016. He is a recipient of the Canadian NSERC Discovery Accelerator Supplement (DAS) 2019. He is a fellow of IEEE.