Engineering Artificial Systems with Natural Intelligence

Citation

Raghavan, Guruprasad (2023) Engineering Artificial Systems with Natural Intelligence. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/374f-1202. https://resolver.caltech.edu/CaltechTHESIS:03172023-050019811

Abstract

Although Deep neural networks achieve human-like performance on a variety of perceptual and decision-making tasks, they perform poorly when confronted with changing tasks or goals, and broadly fail to match the flexibility and robustness of human intelligence. Additionally, artificial neural networks rely heavily on human-designed, hand-programmed architectures for their remarkable performance. In this thesis, I work towards achieving two goals: (i) development of a set of mathematical frameworks inspired by facets of natural intelligence, to endow artificial networks with flexibility and robustness, two key traits of natural intelligence; and (ii) inspired by the development of the biological vision system, I propose an algorithm that can ‘grow’ a functional, layered neural network from a single initial cell, with the aim of enabling autonomous development of artificial networks akin to living neural networks.

For the first goal of endowing networks with flexibility and robustness, I propose a mathematical framework to enable continuous training of neural networks on a range of objectives by constructing path connected sets of networks, resulting in the discovery of a series of networks with equivalent functional performance on a given machine learning task. In this framework, I view the weight space of a neural network as a curved Riemannian manifold and move a network along a functionally invariant path in weight space while searching for networks that satisfy secondary objectives. A path-sampling algorithm trains computer vision and natural language processing networks with millions of weight parameters to learn a series of classification tasks without performance loss while accommodating secondary objectives including network sparsification, incremental task learning, and increased adversarial robustness. Broadly, for achieving this goal, I conceptualize a neural network as a mathematical object that can be iteratively transformed into distinct configurations by the path- sampling algorithm to define a sub-manifold of networks that can be harnessed to achieve user goals.

For the second goal of ‘growing’ artificial neural networks in a manner similar to living neural networks, I develop an approach inspired by the mechanisms employed by the early visual system to wire the retina to the lateral geniculate nucleus (LGN), days before animals open their eyes. I find that the key ingredients for robust self- organization are (a) an emergent spontaneous spatiotemporal activity wave in the first layer and (b) a local learning rule in the second layer that ‘learns’ the underlying activity pattern in the first layer. As the bio-inspired developmental rule is adapt- able to a wide-range of input-layer geometries and robust to malfunctioning units in the first layer, it can be used to successfully grow and self-organize pooling architectures of different pool-sizes and shapes. The algorithm provides a primitive procedure for constructing layered neural networks through growth and self-organization. Finally, I also demonstrate that networks grown from a single unit perform as well as hand-crafted networks on a wide variety of static (MNIST recognition) and dynamic (gesture-recognition) tasks. Broadly, the work in the second section of this thesis shows that biologically inspired developmental algorithms can be applied to autonomously grow functional ‘brains’ in-silico.

Thesis Files