Webinar: Local Competition and Stochasticity as a Nexus for Next-Generation Deep Learning

CaSToRC, the HPC National Competence Centre,
 invites you to the EuroCC and SimEA Online Seminar Series

Abstract

Deep networks have dominated breakthrough advances in machine learning in the last decade. They have enabled almost human-level accuracy in hard machine learning tasks that were previously considered intractable. The paradigm of deep networks is fundamentally different from previous neural network approaches. Deep networks are statistical machines; they exploit a huge variety, and a great fraction, of the immensely important advances of Statistical Machine Learning of the preceding decade. Among these, generative modeling and variational Bayes arguments have been extremely impactful. Despite these advances, deep networks face some major challenges: (i) they impose huge memory requirements, with compression techniques being usually relevant only in the limited context of specific network architectures; (ii) they are especially brittle to adversarial attacks; that is, carefully crafted data perturbations, easily recognizable from humans, which, nevertheless, foul the networks into incorrect decisions by exploiting vulnerabilities of the rationale underlying their objective functions; and (iii) they need immense amounts of data to learn a new task, contrary to biological systems which can generalize well given only a single, or few, examples.

In this talk, we present our work on Deep Network arguments that are founded upon the concept of Stochastic Local Competition, incarnated in the form of stochastic local winner-takes-all (LWTA) units. This type of network units results in sparse representations from each network layer, as the units are organized in blocks where only one unit generates a non-zero output. Their main operating principle lies on stochastic arguments, as the network performs posterior sampling over competing units to select the winner. Often, we can combine these LWTA arguments with tools from the field of Bayesian nonparametrics, specifically the stick-breaking construction of the Indian Buffet Process, to allow for inferring the sub-part of each layer that is essential for modeling the data at hand. Such a construction also allows for postulating stochastic network parameters (synapse weights); apart from facilitating generalization, by mitigating overfitting, this approach bears the additional benefit of enabling memory footprint reduction in a Bayesian compression fashion. Then, training is performed by means of stochastic variational Bayes, with appropriate measures for obtaining low-variance estimators. We show that these arguments give rise to: (i) dense-layer and convolutional networks that completely outperform the state-of-the-art in terms of adversarial robustness to hard benchmarks, without requiring post-hoc processes, such as adversarial training or other data manipulation techniques; (ii) (multimodal) video-to-text networks with self-attention which combine state-of-the-art accuracy with a memory footprint reduction by more than 70%; and (iii) networks that define the state-of-the-art on hard few-shot image classification benchmarks with no compromise in terms of computational efficiency.

In conclusion, we discuss initial results on how these advances can facilitate the goal of obtaining more interpretable deep networks, that transcend the black-box nature of the current paradigm. Progress to this end may catalyze wide technology adoption in products and services, as the latter requires that human experts can obtain an understanding of why, in some cases, a deep learning model generates false positives or negatives.

About the Speaker

Dr Sotirios Chatzis is an Associate Professor at the Cyprus University of Technology, and Chair of the Department of Electrical Engineering, Computer Engineering, and Informatics. Dr Chatzis is a Computer Engineering graduate and holds a PhD in Statistical Machine Learning. His research interests lie in the fields of Deep Learning and approximate Bayesian inference. Characteristic application areas include natural language understanding, video understanding, as well as unbiasedness, exploitability, and trustworthiness in the era of Machine Learning.

Since 2016, he has served each consecutive year in the program committee of the most prominent international venues in Machine Learning, namely ICML, NeurIPS, and AISTATS. This is a great honor that vouches for the recognition of his work among his peers, and the bleeding edge nature of it. He also serves as PI of several research projects funded by the European Commission and Cyprus Research & Innovation Foundation (RIF). His research laboratory currently hosts 4 full-time PhD students, 3 Postdocs, and 4 Research Assistants. It currently executes 7 research grants, 4 of them under the capacity of the project coordinator, with a total funding that exceeds Euro 1.5M for the laboratory.

Dr Chatzis has published more than 100 papers, mostly in top-tier venues. In addition, Dr Chatzis has served two terms as the elected Director of the Social Computing Research Center (SCRC) of CUT. He was elected to the Director position by the Board of SCRC.

Download the Winter-Spring 2022 Online EuroCC & SimEA Seminar Series Programme here.

The EuroCC project has received funding from the European Union’s Horizon 2020 research and innovation programme grant agreement No. 951732

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 810660

Contact This email address is being protected from spambots. You need JavaScript enabled to view it.

View all CyI events.