The Syntax of Connectionist Networks

The Syntax of Connectionist Networks

Nicholas Shea

Abstract

Are connectionist networks more than just dumb data-fitters? Can they model any interesting aspect of human cognition? If connectionist networks genuinely represent the world, then the answer is “yes” to both questions. So it is important for enthusiasts of the parallel distributed approach to cognition to show that networks do indeed have content-bearing states. That claim has been contentious. This paper sets out a new framework for understanding connectionist networks. From this improved perspective the role of representational explanation becomes clear. In fact, content attribution is an essential part of a good explanation of how networks sometimes manage to perform correctly in response to samples from outside the training set.

The starting point is to be more careful about syntax. Of course, each particular representation is a distributed pattern of activation. But how are these points in activation space to be grouped together into vehicles of content? The written marks dog, DOG and Dog all carry the same content in public language. Similarly, to give a representational explanation of its operation, different points in the state space of a hidden layer of a network should be treated as realisations of the same vehicle of content. How should the appropriate regions of state space be individuated? Modellers do cluster analysis or principal components analysis of the hidden layer state spaces of a trained network to understand how the network achieves correct performance. Such analysis should be understood as aiming at the system’s syntax. That is, it uncovers the content-bearing items over which processing takes place.

There is nothing revolutionary in the claim that points representing the response to inputs from the training set tend to cluster together in hidden layer state space after training. However, treating these clusters as syntactic items entails two important changes to the standard theoretical framework:-

(1)Hidden layer nodes have no representational significance. Semantic dimensions correspond to clusters, and are independent of the dimensions defined by hidden layer nodes.

(2)It is wrong to think of a point in activation space as inheriting its content from a combination of the complex ‘microfeatures’ represented by each of the hidden layer nodes that contribute to its activation.

The approach has an important developmental consequence: only after training (and clustering) does the network have any contentful states. Compare the microfeatural idea. Individual hidden layer nodes have complex differential input sensitivities even under the initial assignment of a random weight matrix. The unattractive consequence of the microfeatural approach is that each point in activation space is assumed to be contentful before any representational development has taken place.

Clusters are seen to be genuinely content-bearing when they are invoked to explain how a network manages to project its correct performance on the training set to novel samples, which differ in input encoding to anything encountered during training. This allows a commonsense ascription of content to clusters. Philosophers have argued that content is fixed by an abstract resemblance between the geometrical arrangement of points in hidden layer state space and structural features of the domain being represented (Churchland 1998, O’Brien & Opie 2001), relying upon Laakso & Cottrell (2000) as a holistic test of content similarity. However, this makes connectionist content hostage to a wider philosophical issue about the metaphysical nature of representation (in virtue of what a vehicle has the content it does). The proposal about syntax advocated in this paper remains neutral.

Finally, the new framework makes clearer how networks should be interpreted as modelling some results from cognitive psychology. It even suggests some empirical predictions.

References

Churchland, P. M. (1998), ‘Conceptual Similarity Across Sensory and Neural Diversity: The Fodor/Lepore Challenge Answered’, Journal of Philosophy 95(1): 5-32.

Laakso, A. & G. Cottrell (2000), ‘Content and cluster analysis: assessing representational similarity in neural systems’, Philosophical Psychology 13(1): 47-76.

O'Brien, G. & Opie, J. (2001), ‘Connectionist Vehicles, Structural Resemblance, and the Phenomenal Mind’, section 3. In J. Veldeman (ed.), Naturalism and the Phenomenal Mind, a special issue of Communication and Cognition34: 13-38.

1