Randomized Scattering Convolutional Networks
Board Location: #88
Discipline: Computer Sciences and Information Management
Subcategory: Computer Science & Information Systems
Session: 2
Vivian White - Western Washington University
Co-Author(s): Muawiz Chaudhary, Concordia University, Montreal, Quebec; Guy Wolf, Universite de Montreal, Montreal, Quebec; Guillaume Lajoie, Universite de Montreal, Montreal, Quebec; Kameron Decker Harris, Western Washington University, Bellingham, WA
Biological neural networks are composed of neurons that respond nonlinearly and selectively to certain stimuli in their receptive fields. In artificial neural networks, the receptive fields are the weights that determine the strength of the connection between neurons across different layers. However, these weights are difficult to analyze. Our research aims to develop a better framework for understanding ANN weights by understanding how weight structure influences network performance, specifically by studying the effects of neuron-inspired weights in neural networks. We hypothesize that imbuing networks with biological structure will improve performance and provide a stronger mathematical framework for analysis than traditional unstructured weights. We replace the weights in the initialization layer of multilayer neural networks with structured weights based on V1 receptive fields. These weights are drawn from a multivariate Gaussian distribution with a covariance matrix matched to experimental data, which provides a biologically realistic and mathematically tractable model. In the supervised learning domain, we build convolutional networks with V1-inspired weights in the frozen layer and compare image classification performance against standard neural networks and wavelet scattering networks, which are popular models that perform selective filtering of stimuli by using fixed wavelets as weights. Our results show some slightly improved performance with structured weights versus unstructured weights and wavelet scattering networks. In the unsupervised learning domain, we hypothesize that networks with fixed biologically structured weights will achieve similar performance to a wavelet scattering generator. We compare our structured weights generator network against a wavelet scattering generator network in three different areas: image reconstruction, image interpolation, and image generation from white noise. Results show that these models achieve comparable performance.We are currently creating a learnable V1-like convolutional network, where we parameterize the V1-like receptive fields by their size and spatial frequency, allowing the network to learn the optimal values for these parameters. We hypothesize that letting a network learn these weights while constraining them to our V1-like model of receptive fields will improve performance and aid our understanding of the effects of biologically-inspired weights in neural networks. Additional plans include computing the random feature kernel for our networks and characterizing their function spaces to expand our theoretical understanding of weight structure.References:Harris et al. (2022).”Structured random receptive fields enable informative sensory encodings.” PLOS Computational Biology. https://doi.org/10.1101/2021.09.09.459651Angles, T., and Mallat, S. (2018). “Generative networks as inverse problems with Scattering transforms.” CoRR. https://doi.org/10.48550/arXiv.1805.06621
Funder Acknowledgement(s): Funded by the International Network for Bio-Inspired Computing (IN-BIC) through the NSF AccelNet program (2019976)
Faculty Advisor: Kameron Decker Harris, kameron.harris@wwu.edu
Role: I built all of our image classification and generation models using V1-inspired structured weights, unstructured weights, and wavelet scattering baseline models. I ran all experiments and collected all data.