5. How we cognize space: An alternative to internalizing
5.1What does it mean to “cognize space”?
5.2Internalizing general spatial constraints: From Marr to Shepard
5.2.1Marr’s philosophy and the incorporation of Natural Constraints in the mind
5.2.2Shepard’s psychophysical complementarity
5.3Internalizing spatial properties by mapping them onto an inner space
5.3.1Brain Space
5.3.2Functional space
5.3.3Internalizing by incorporating visuomotor experience: Poincaré’s insights
5.4Sense of space
5.4.1Externalizing the spatial sense: The projection hypothesis
5.4.2Are spatial locations represented in a unitary frame of reference?
5.4.3The coordinate transformation function and as-needed translation
5.4.4What’s in a cognitive map?
5. How we cognize space: An alternative to internalizing
5.1What does it mean to “cognize space”?
One of the difficulties in understanding spatial experience is the fact that it is so extremely intuitive to us that it is unclear what we mean when we ask how we cognize space. We are like the proverbial fish trying to understand water. It seems obvious that space is the three-dimensional receptacle in which objects reside and that spatial relations are there for us to see and experience without the need for any concepts or inferences. It also seems to our modern sensibilities that space consists of a dense array of points which can be connected by straight lines. But these notions, which have been enshrined in our view of space since Euclid, may not be the right notions in terms of which we ought to describe how we perceive space and especially how we represent space in our mind when we think about it or imagine events taking place in it. But what does it mean to say that this is not the way we cognize space? How can it be that out conceptualization of space does not give a privileged place to points and lines?
I begin by trying to outline the nature of the problem that faces us. What does it mean to see the world as being laid out in space? What must the architecture of a mind be like that can do this? Given the patterns of energy that impinge on our sense organs, what must the mind do to create the particular experience of space that we have? Many of our greatest thinkers have sought to answer this question at a foundational level. Putting aside the classical Greeks, who had views about everything that matters, the problem fascinated thinkers like Kepler, who (as we saw in chapter 1) was one of the first to recognize (a) that the retinal image plays an important role in the causal chain and (b) that the gap between the retinal image and the apprehension of space would not succumb to the same style of geometrical analysis that worked so well in filling the gap between the light, the objects, and the image on the retina [Lindberg, 1976 #1599]. Rene Descartes’ arithmetization of geometry was one of the seminal accomplishments in understanding that the problem had a formal structure (not dependent on diagrams) that was amenable to rigorous study. Then in the 20th century several great French natural philosophers were stirred by the problem. Henri Poincaré [Poincaré, 1963/1913 #1201] was one of the most important of these and I will return to his views below. The problem of developing a sensory-based Euclidean geometry was raised again by Jean Nicod who, in the 1930s wrote a dissertation entitled “Geometry and the Sensory World” which laid the groundwork for a very different way of looking at this question [Nicod, 1970 #186] and which, by the way, had a profound effect on me when I began the work that lead to the FINST theory.
For Nicod the problem was that the basic building blocks of the Euclidean (and Cartesian) view are points and lines, and a way of constructing figures from them, together with the relation of congruity, none of which seemed to him like the sorts of things that perceptual systems are equipped to detect – they are complex types that collapse collections of sensory experiences into categories that make the statement of geometrical principles simple at the cost of making their connection with sensory data opaque.[1] Nicod suggested that since there are very many models of the Euclidean axioms (the Cartesian mapping of space onto ntuples of numbers being the best known) we should seek instead a way to capture Euclidean spatial properties in terms of primitives more suited for creatures with sensory systems like ours. After considering a variety of such possible primitives, he developed several “sensible geometries” based on the geometry of volumes and of volume-inclusion (or what he called “spatio-temporal interiority”) and argued that this basis is closer to our sensory capacities than one based on points and lines (one reason being that volume inclusion is detectable and is invariant with viewpoint so it can be sensed as we move through space). With the addition of a few other novel ideas (such as “succession” and “global resemblance”) Nicod set out a new direction for understanding what space might consist in for a sentient organism. While in the end he did not succeed in developing a complete formalization of geometry based on these sensory primitives he did point the way to the possibility of understanding sense-based space radically different from the Euclidean, Kantian, and Cartesian approaches that seem so natural to us. If Nicod had been able to carry out his program he might have provided a set of tools for viewing space that would have been more useful to us than the view that is thoroughly embedded in our way of thinking. But he did show us that thinking in terms of points and lines may not be the only way and indeed it may not be the most perspicuous way for cognitive science to proceed in studying the experience of space.
Another person who worked on the problem of characterizing space is Christopher Peacocke. In [Peacocke, 1992 #1668] he develops a way of characterizing the experience of space in terms of what he calls scenarios, which he defines as ways in which space can be filled. As I suggested in the previous chapter, while characterizing the phenomenal experience of space is a deep and interesting problem in its own terms, it is not clear how cognitive science can build on these ideas, since we have reason to believe that our phenomenal experience does not capture the appropriate mental structures on which one can build an explanatory theory. However, Peacocke’s work helps us to understand what might constitute the content of experience of space and what one would have to capture in order to properly express it – in what Chomsky referred to as a descriptively adequate (as opposed to an explanatorily adequate) theory. I should reiterate, however, that it is not clearthat one’s phenomenal experience of space should be taken to reveal nonconceptual content. There is reason to think that our phenomenal experience embodies both nonconceptual and conceptual content, and indeed that it is a function not just of perceptual processes, but also of inferences from general knowledge and from memory.[ZwP1] I have discussed this issue in the last chapter and will not have more to say about this approach here, except to note that the purely phenomenal content of spatial experience may be relevant to understanding certain distinctions we experience such as the qualitative difference between vision and mental imagery [Dalla Barba, 2002 #1559] or perhaps between clear perceptions that fail to be convincing as opposed to vague perceptions that seem convincingly real (a distinction that is orthogonal to perceptual content, asGestalt Psychologists recognized). In this chapter I will, instead, focus on what may be nonconceptual contents of perception and offer some comments on several approaches to understanding the nonconceptual spatial representation that postulate some form of internalizing of spatial properties.
5.2Internalizing general spatial constraints: From Marr to Shepard
Since Watson’s identification of thought with subvocal speech there has been a strong mistrust of accounts of mental phenomena that appeal to direct internalizations of external properties or phenomena. I share this mistrust and continue to believe that cognitive theories that exhibit this sort of interiorizing of externals betray our latent behaviorist instincts, our tendency to focus on observables. But not all internalizations are misled – in fact being intentional organisms entails that in some sense we have internalized (i.e., represented) aspects of the world. In prior lectures I argued that in addition to representations that are related to what they represent by the relation of semantic satisfaction we need a more direct or causally-based relation. This, in turn, suggests that other sorts of internalizations besides conceptual or intentional ones that constitute our knowledge of the world, play a role and therefore that we could learn by taking a second look at the general issue of internalization. In what follows I will consider several theories that are shaped by some aspect of the internalizing approach – several of which represent, in my view, useful ways to look at the problems of nonconceptual representation and at least one that has led us hopelessly astray.
5.2.1Marr’s philosophy and the incorporation of Natural Constraints in the mind
Computational vision perhaps more than any other approach has faced squarely the problem of the indeterminacy of visual information in relation to what we perceive. As is well known the inversion of the mapping from a distal scene to the retinal image(s) is indeterminate – it is a many-one mapping. But the visual system computes a univocal inversion – we almost always see a unique spatial layout despite the ambiguity of the incoming information. How we can do this has been the subject of speculation for many years, with the prevailing view in the mid 20th century being that visual interpretation is influence by expectancy based on knowledge of the world, an in particular knowledge of the scene in question. James Gibson questioned this assumption, insisting that the information was there in the ambient light if we only looked for it in the right way. But it was David Marr (and others working on computer vision at the time) who made the case convincingly that vision does not need (and indeed, is unable to use) information from our general store of knowledge in interpreting a scene [Pylyshyn, 1999 #965]. Rather, the reason it comes to a univocal interpretation of spatial layouts is that it is unable to entertain other alternative hypotheses compatible with the sensory evidence. And the reason for that is the existence of what Marr called “Natural Constraints” which consist in very general constraints that were compiled into the visual system through evolution. It’s not that the visual system knows that the scene before it consists of rigid objects, but rather that it is so constituted that only interpretations consistent with the rigidity of most objects are available to it.[ZwP2] If you knew that the objects were not rigid it would make no difference to the interpretation that vision would provide.
This idea, though not entirely unprecedented, was revolutionary when combined with a program of researchin computational vision. The task then became to uncover the various natural constraints that are built into vision and to show how a system that respected these constraints could see spatial layouts. This led to a series of projects usually entitled “structure from X” where the X’s such cues as motion, shading, stereo, contour, and so on. This is a sense of internalizing of “constraints” that is both theoretically plausible and empirically validated – at least in many cases. The approach is closely related to a similar goal in linguistics where both language learning and sentence comprehension are underdetermined process: The data on the basis of which languages are learned and on the basis of which sentences are parsed are similarly impoverished. What is assumed to enable the learning of a native language in the face of this indeterminacy is the innate brain structures described by Universal Grammar which prevent the infinite number of humanly inaccessible languages from being learned or the similar infinite range of sentence parsings from being considered. Similarly the interpretation of visual signals is constrained by internalized natural constraints.[2]
The question of whether these constraints allow for processing of spatial information without conceptualization is an interesting one. If space were represented without first carrying out inferences (which requires conceptualization), it would be a good candidate for a nonconceptual form of representation. This is exactly what the natural constraints idea proposes. It says that the representation of space enabled by the internalized natural constraints can be achieved without inferences and therefore without conceptualizing the sensory information. A major question that this story raises is whether the constraints apply in thought as well as in perception. Does the fact that we cannot imagine a 4-dimensional space related to the fact that we cannot perceive one? Perhaps. Or perhaps it is because we don’t know what such a thing would look like! In general it is not clear how our ability to conceptualize and imagine spatial layouts can be explained by general constraints of the sort that are postulated in vision. While thinking is, by definition, carried out with conceptualized representations, yet there is no principled reason why a nonconceptual representation could not play a role if the cognitive architecture made this possible. We can, after all, make use of the nonconceptual representations in diagrams and other drawings, so why not in thought? I will return to this question later, but just to anticipate; one of the problems with such a proposal is that imagination is creative in ways whose boundaries are unknown. While we may not be able to imagine a 4 dimensional space, we can imagine objects moving in very nearly any way at all, with or without maintaining rigidity, with or without obeying the laws of physics or the axioms of geometry. And this plasticity of thought is a major problem for any internalization theory, as we will see.[ZwP3]
5.2.2Shepard’s psychophysical complementarity
Roger Shepard [Shepard, 2001 #1657] has take the idea of internalization even further. Citing the example of circadian rhythms which internalize the earth’s daily light-dark cycle, he argues that many reliable generalizations of human behavior can be traced to our internalizing universal properties of the world. His general argument is based on the evolutionary advantage of being wired in that way – a sort of Leibnizian “preestablished harmony” between world and mind. But such an argument should lead us to expect that universal physical properties would be internalized, for what is more important that correctly anticipating where and when a rock will fall on you? Yet this is not the case; dynamics does not seem to have a foothold either in vision or in thought. What appears to be internalized, according to Shepard, are principles of kinematic geometry. Because of these internalized principles we tend to perceive (and imagine) objects as traveling through the simplest geodesics in a 6 dimensional space (3 dimensions of translation and 3 of rotation). Shepard presents some interesting examples of this principle, involving apparent motion of asymmetrical 2 dimensional forms that tend to be seen as traveling in 3D according to a screw transformation, as predicted by the “simple geodesic” story.
These are interesting and suggestive ideas and if there is anything to the internalization principle these are certainly good candidates. But neither Marr’s Natural Constraints nor Shepard’s geodesics represent an internalizations of space so much as principles of abduction that determines what hypotheses the organism is able to entertain and which explain the choices that the visual system makes when faced with ambiguous inputs. In that respect the are like Universal Grammar. The alternative to such constraints is the conceptual story: rather than internalizing properties, we learn about them (by induction) and then we draw inferences from them. In the end this cannot be the case for all cases of induction, for reasons that I have discussed throughout these lectures, namely that our beliefs must eventually make contact with the world. Internalizing is a way of incorporating principles in a nonconceptual way, which gives it much of its appeal. It is not surprising, therefore, that the only candidates for internalization are ones that apply to modular parts of the perceptual systems where general reasoning is not permitted because of encapsulation.
But some people have taken internalization proposals such as Shepard’s as evidence that the properties of Euclidean space are somehow internalized [as, for example, a space defined by states of assemblies of neurons – as proposed by some of the commentators on Shepard’s paper, such as \Edelman, 2001 #1658]. If they are, then there is little evidence of their operating in thought – particularly in mental imagery. We can easily imagine objects breaking kinematic principles, traveling along nongeodesic paths and violating just about any principle or constraint you can think of. And this, as I remarked earlier, is what leads where most paths to cognitive architecture eventually end up – with the view that the mind is much more like a Turing Machine, than like any multidimensional space.
5.3Internalizing spatial properties by mapping them onto an inner space
Before proceeding with this survey of approaches to representation of space I need to introduce two topics that are much more of an ongoing concern in the mental imagery debate than they are to the issue of spatial representation. But they constitute proposals for how space is represented, so it may be fitting to say a word or two about them. They are both strong forms of an internalization proposal – it is the proposal that space is internalized by a space inside the brain. The strongest form of this proposal actually locates the space of mental representation in the literal space of the visual cortex. The weaker form, that most people cite when they are pushed into a corner, is the idea of a “functional space”. Because that proposal is a muddle, I will take this opportunity to clear it up before proceeding with discussion of a more serious possibility for what I call the sense of space based on connections with the motor system.