Publications & Preprints

On Neural Architecture Inductive Biases for Relational Tasks. G.Kerg, S. Mittal, D. Rolnick, Y. Bengio, B. Richards, G. Lajoie. Under review at the Advances in Neural Information Processing Systems (NeurIPS), 2022. arxiv code

Goal-driven optimization of single-neuron properties in artificial networks reveals regularization role of neural diversity and adaptation. V. Geadah, S. Horoi, G.Kerg, G. Wolf, G. Lajoie. biorxiv

Inductive Biases for Relational Tasks. G.Kerg, S. Mittal, D. Rolnick, Y. Bengio, B. Richards, G. Lajoie. Accepted at the ICLR 2022 workshop on Objects, Structure and Causality, 2022. openreview code

Top-down optimization recovers biological coding principles of single-neuron adaptation in RNNs. V. Geadah, G.Kerg, S. Horoi, G. Wolf, G. Lajoie. Accepted at Computational and Systems Neuroscience (COSYNE), 2022.

Continuous-Time Meta-Learning with Forward Mode Differentiation. T. Deleu, D. Kanaa, L. Feng, G.Kerg, Y. Bengio, G. Lajoie, P.-L. Bacon. Accepted at the Tenth International Conference on Learning Representations (ICLR), 2022. openreview

Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization. S. Jastrzebski, D. Arpit, O. Astrand, G.Kerg, H. Wang, C. Xiong, R. Socher, K. Cho and K. Geras. Accepted at the Thirty-eighth International Conference on Machine Learning (IMCL), 2021. arxiv

Network-level computational advantages of single-neuron adaptation. V. Geadah, G. Lajoie, G.Kerg, S. Horoi and G. Wolf. Accepted at Computational and Systems Neuroscience (COSYNE), 2021.

Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization. S. Jastrzebski, D. Arpit, O. Astrand, G.Kerg, H. Wang, C. Xiong, R. Socher, K. Cho and K. Geras. Accepted at the Proceedings of the NeurIPS 2020 Workshop on Optimization in Machine Learning (OPT), 2020. paper

Untangling trade-offs between recurrence and self-attention in artificial neural networks. G.Kerg*, B. Kanuparthi*, A. Goyal, K. Goyette, Y. Bengio and G. Lajoie. Accepted at the Advances in Neural Information Processing Systems (NeurIPS), 2020. arxiv code

Guarantees for stable signal and gradient propagation in self-attentive recurrent networks. G.Kerg, B. Kanuparthi, A. Goyal, K. Goyette, Y. Bengio and G. Lajoie. Accepted at DeepMath 2020 (Conference on the Mathematical Theory of Deep Neural Networks), 2020.

Learning Long-term Dependencies Using Cognitive Inductive Biases in Self-attention RNNs. G.Kerg*, B. Kanuparthi*, A. Goyal, K. Goyette, Y. Bengio and G. Lajoie. Accepted at the ICML 2020 Inductive biases, invariances and generalization in RL workshop, 2020. Also accepted at the Montreal AI Symposium (MAIS), 2020. paper code

Advantages of biologically-inspired adaptive neural activation in RNNs during learning. V. Geadah, G.Kerg, S. Horoi, G. Wolf and G. Lajoie. arxiv

Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics. G.Kerg*, K. Goyette*, M.P. Touzel, G. Gidel, E. Vorontsov, Y. Bengio and G. Lajoie. Accepted at the Advances in Neural Information Processing Systems (NeurIPS), 2019. arxiv code

h-detach: Modifying the LSTM gradient towards better optimization. B. Kanuparthi*, D. Arpit*, G.Kerg, R. Ke, I. Mitliagkas and Y. Bengio. Accepted at the Seventh International Conference on Learning Representations (ICLR), 2019. openreview arxiv code

Safe Screening for Support Vector Machines. J. Zimmert, C. Schröder de Witt, G.Kerg, and M. Kloft. Accepted at the Proceedings of the NIPS 2015 Workshop on Optimization in Machine Learning (OPT), 2015. paper

On Neretin's group of tree spheromorphisms. Master thesis for MSc in pure mathematics, Université Libre de Bruxelles, 2013. thesis

Expansion in groups. Essay for Part III of the Math Tripos (equivalent to Master thesis), University of Cambridge, 2012. thesis