Cross-cultural dynamics of picture and text

Wolfgang Wildgen (University of Bremen, Faculty of Languages and Literatures, Germany)

Abstract: The comparison of text and picture across cultures has to ask for fundamental schemata, their stability/instability and the levels of content filling for these schemata. On this basis a comparison of cultural variants of the media: text (spoken/written) and picture/sculpture becomes possible. The complexity of sign structures is related to their dimensionality, the force-fields of a void sign space (in one or two dimensions), the reduction of complexity and the coding of ‘lost’ information. The basic dynamics are assessed by a short analysis of Leonardo’s “Last Supper”; the cross-cultural aspects in time are described with reference to abstract paintings (Kandinsky, Klee) and to satirical deformations of Leonardo’s topic (Buñuel, Smudja).

Key words: dimensionality, Leonardo, Klee, abstraction, satirical deformation

  1. Introduction

If language is basically a linear structure (de Saussure’s assumption), then it is fundamentally different from the symbolic organization of pictures (e.g. paintings, photos) and even more different from sculptures and architecture. The dimensionality of any organization of signs is a basic determinant of its structure; it imposes other degrees of freedom and asks for other restrictions (this is a general insight also valid in the material world of physics and chemistry and could be called a fundamental law of Morphodynamics; cf. Thom, 1972). Nevertheless, one has the intuition that sentences/texts and pictures have many features in common, they can cooperate, interact as in comics, illustrations, emblems, etc., they may be translated into one another and, what is more crucial, they respond to the same cognitive system with perception, motor-control, memory and imagination which is independent from the specific semiotic modality chosen.

Similar problems of integration starting from sensorial inputs with different spatial, temporal and dynamic characteristics arise with all symbolic forms, e.g. language, art, myth (religion); techniques, ethical rules (laws), economic rules of exchange (monetary systems); cf. Cassirer 1923-29, Wildgen (2003c), and Wildgen (2004a: Chapter 9). In the case of text/picture the questions, therefore, are:

What (spatial/temporal/dynamic) organization do text and picture have in common (we may expand the comparison to sculpture and architecture)? Can we conceive a semiotic framework in which both the similarities and the differences between text (sentence) and picture are mapped? What is the syntax allowed or furthered by dimensional features and as a consequence what kind of symbolic creativity is possible or dominant in one mode or another? Finally different cultural contexts select, enable or restrict these possibilities, a fact which may be responsible for many cross-cultural differences and may be overwhelmed in a process of “mondialisation” (cultural globalization).

  1. How linear is language? A first confrontation with the dimensionality of pictures

The linearity of language is much more controversial than Saussure’s course (edited by his students) makes believe. There exists a phenomenological multi-dimensionality in intonation and paralinguistic information which points to parallel information channels and interaction (structural binding) between these channels and the proper information channel of language. One could argue that in the case of parallel lines of coding the basic organization is still linear and the interaction may be restricted to points of coordination (cf. parallel computation which coordinates the activities of several linear computers). Another complication that has to be considered is the direction of the process of speaking/listening (writing/reading). The simplest model was that of a linear automaton which from left to right reads one element, replaces it or not and goes on. The discussion in Chomsky (1957) on the format of generative (production) grammars already showed the restrictions of a linear (unidirectional) automaton. Not only do discontinuous constituents occur, the action of the automaton must consider information given in specific places in the sequence already passed; thus, the linearity of language must accept moves back and forth and the range of these moves depends on the information given in single places (constituents). As a consequence the linearity of language has a dominant direction (on the time axis) but it also has a memory of relevant places in the past and reacts to structural places not yet reached (but asked for, necessary to come). In figure1 I try to give a schematic (not a precise) picture of the kind of linearity involved in language processing (mainly in language production).

a) Unidirectional process

b) Dependence on past or future steps of the process

c) Garden path and reanalysis

garden path

reanalysis

Figure 1:Major deviations of the unidirectional linearity of language.

Another counterargument to linearity in language processing could be the duality of syntagmatic and paradigmatic relations (cf. Jakobson, 1990: 59). The syntagmatic relations correspond roughly to the contents of figure1. The paradigmatic relations open a field of choices or variations. At any moment, and before any move on the line is made, the speaker may consider a set of possibilities allowed by the prior choices. The set of alternatives opens a second dimension (of freedom) governed by semantic/ pragmatic/sociolinguistic choice criteria (cf. Wildgen, 1977). The choice made is responsible for the style, the literary quality, and the rhetorical effect of the uttered sequence. The stylistic choices open a multi-linear field of associations and form-meaning correspondences, which is definitely not subjected to the strict linearity of linguistic production (consider the difficult choices made by a poet). I consider therefore stylistic (and esthetic) variation to go beyond the linearity of language.

In the case of pictures (e.g., drawings or paintings) it is immediately obvious that the eye which reads and the hand which draws/paints the picture operate basically in two dimensions. Even in the case of drawing where the hand performs linear moves, these have many different directions, i.e., they have many orientations in a two-dimensional space. In paintings, linear (directed) strokes may be performed (and even be visible as in van Gogh’s paintings), but what is dominant is the composition of a surface out of sub-surfaces; therefore the composition is not linear and the esthetically important neighborhood of colors and shapes is defined in two dimensions.[1] If a perspective is constructed, a third dimension in space is simulated and hierarchical orders between figures or topics in the painting may exist. This shows that in the case of pictures many phenomena mentioned in the discussion of language reappear again, they concern now the transition between a two-dimensional base space and three-dimensional interpretations. If words, sentences, texts have characteristic linear boundaries, pictures have two-dimensional boundaries. Thus, the shape of the frame: be it rectangular, quadratic, circular, oval, etc., and all the dynamics inherent in a picture are influenced by the fact that they end at this border-line or start from it (cf. Plümacher, 2003).

  1. From one-dimensional language to two-dimensional pictures

We shall first consider the one-dimensional space (of language) that is devoid of linguistic signs, i.e., silence, and the two-dimensional space (of pictures) that is blank. What kind of dynamics may we find in this “virgin”-situation? In the case of language one may infer that silence just began (communication stopped) or that silence will finish (communication is just about to start). The void linear space has implicitly a vector pattern of begin/end, as illustrated in the central window of Figure 2.

Figure 2:Virtual dynamics of “silence” in communication.

In the case of a frame without picture, a blank canvas, the situation is more complex. The “linear” space of silence of language has as its correlate a (denumerable infinite) set of regular surfaces (I neglect non- or semi-regular surfaces): the equilateral triangle, the square, the regular polygons with 5, 6, 7, …n corners. For simplicity sake, I shall just consider the square. What are the dynamics of a square frame (without picture). It is obvious that the corners, the diagonal which links them, the regular grid of squares which may compose it, or may be inscribed or circumscribed define a whole family of implicit paths and thus dynamical potentialities. Figure 3 shows this basic observation and adds typical paths (cf. for further comments on the “pictorial base space”, Saint-Martin, 1987: chapter4).

Figure 3:Virtual dynamics of a square (diagonal, horizontal/perpendicular and spiral force-lines).

The comparison of language and picture shows that the transition from d(dimension) =1 to d=2 leads to a dramatic increase in the potential dynamics: A frame without picture (pictorial “silence”) has a complexity which goes far beyond that of a one-dimensional linguistics “silence”. Before we begin to fill the void spaces, we should ask, if this increase of latent structure continues steadily with d=3, d=4, … The answer which already impressed Plato (or the dialogue partner of Socrates Timaeus) is that the story does not go on as one would guess. The major reason is that, although we find infinite regular polygons, we only find five regular polyhedrons (the Platonic solids), in 4-space we find six regular hypersolids, in 5-, 6-, 7-space only three (cf. Stewart, 1989: 91). This means that the dimensionality does not induce a monotonic increase in the number of basic forms, on the contrary it involves restrictions which reduce this number. In order to complete somewhat the argument (which cannot be followed in detail here) one has to consider, that there is still a steady increase in the number of corners (and therefore of implicit dynamic fields, cf. above):

  • two end points in a line segment
  • four corners in a square
  • eight corners in a cube
  • sixteen corners in a four-dimensional cube

32, 64, 128 … corners if we increase further the dimensionality of the cube (cf. Stewart, 1989: 90f.)

To the non-monotonic increase (even dramatic decrease) in the number of regular entities (dimension d= 1, 2, 3, 4 …n) corresponds a dramatic clash in the stability of unfoldings (process-types). This is the heart of Thom’s classification theorem (cf. Wildgen, 1982: 7-18 for an introduction).

This short summary of basic regularities discovered in geometry and differential topology helps us to understand that the transition from one dimension to two, three, four does have dramatic structural consequences and it would be a silly mistake to believe that one has just to add some more features to the body of results obtained in the case of one-dimensional structures (e.g., language) in order to describe pictures which are basically two-dimensional. Another silly argument would be that if pictorial structures are very different (qualitatively different) from linguistic ones, one should just forget the results of linguistic analysis and begin the analysis of pictures ex ovo, as if they had nothing to do with language. In both cases do we have symbolic forms (cf. Cassirer, 1923-1929, and Wildgen, 2003c) and basically these symbolic forms use the same perceptual, mnemonic and imagistic resources. The dimensionality is therefore the key to the difference between language and pictures. The common (cognitive) base of both modalities allows for the blending of linguistic and pictorial signs and their contribution to one universal type of human understanding.

  1. Implicit force-fields and the organization of content

The space of silence in language may be filled by a sentence (we simplify the real processes). It inherits the borders of this space such as: beginning/end and is governed by relative probabilities in a linear sequence, i.e., the set of possible first constituents and dependent on it of second constituents, etc. The production grammars put forward since Markov’s first proposals (by Harris, Chomsky, and others) elaborated this basic idea (and added the special cases of context sensitivity, transformation, reanalysis, etc.). I will just take this tradition as given and ask how a similar process may look like in the case of picture-production/analysis.

First, we have seen that even the ideal paths in a square (let alone non-ideal or chaotic paths) are multiple. I have mentioned the diagonals, the square grid (vertical and horizontal symmetry lines) and a spiral moving from the outside to the center or vice-versa.[2]In producing a picture (on a void surface) these force fields are relevant and they depend naturally on the shape of the picture (be it rectangular, square, circular, elliptical, etc.). A strong preference is given to rectangular frames which are near to the ideal (the square) but introduce a basic asymmetry.[3]

If we take the painting the “Last Supper” of Leonardo da Vinci (cf. Wildgen, 2004b, 2004a: chapter 6) the prominent table of the supper fills the basic horizontal line and the head (ear) of Christ marks the intersection with a vertical line of symmetry. The diagonals correspond to the slightly deformed lines of perspective (see the ceiling and the tapestry at left and right) that produce the illusion of three-dimensionality. Figure 4 reconstructs the basic force fields.

Figure 4:The force fields in Leonardo’s “Last Supper”.

As this example shows, all three force fields we analyzed in the case of a void frame are used to organize specific contents (surfaces, figures, persons in space) in Leonardo’s mural painting. The head (ear) of Jesus is at the center of all force fields. The sub-centers of the groups of apostles lie in the intersections between the horizontal axis and the symmetric spiral which end at Jesus’ head (ear). The rectangle of the whole fresco breaks the symmetry of the (ideal) square.[4] The perspective generates a subdivision of the background space into three equal zones. In the central zone are situated: Jesus, John (at the right of Jesus), and Thomas, James Major (at the left of Jesus); Judas is already outside of this field although he has the second position at the right of Jesus. Peter and Philip are at the intersections of these fields. Geometrically we have a blending of two orders: the symmetrical subdivision of the group of apostles into 6+6 and (3+3)+(3+3) and the three background fields with Jesus and three apostles in the middle and four apostles at the right and the left (Judas has his arms on the table and thus sits in a plane nearer to the spectator); this order is basically 4+4+4 (+ Judas in a frontal position). The table organizes the spatial distribution of the persons, which are all in the lower part of the frame (which is therefore in a vertical asymmetry); the same is true for the trunks, heads, hands of the persons above the table and the feet below; there is a clear dominance of the body parts above the table. Thus, the geometrical rigor of the force lines is broken by a set of asymmetries. The information of the picture is at the first level of analysis a breaking and deformation of symmetries and corresponding force fields. Our analysis only considered the fundamental restructuring of a space void of content but structured by force-fields dependent on a frame. As soon as specific contents: a person or a configuration of persons, objects (e.g., flowers, fruit, dead animals in a “nature morte”) or abstract configurations are introduced, these contents “graft” local spaces and dynamics upon the dynamically organized pictorial frame. Thus Jesus and his twelve apostles implant their own configuration into the painting. The new dynamical relations may be gravitational (the apostles sit or stand at the table), map events and actions (giving, holding something), or symbolic acts (gestures and/or glances). This means that the two-dimensional space contains several sub-spaces that introduce their own structure and dynamics into the picture. They may be coordinated by the overall structure but they still create conflicts, oppositions, deformations in the already deformed base space. The basic content complexes organized in Leonardo’s painting are:

  1. The table in the foreground.
  2. The perspective of the dining room, the windows, the landscape visible through the window, the subdivision of the background into three equal sub-fields.
  3. The arrangement of 12 apostles (grouped by 4x3) on both sides of Jesus.
  4. The gestures (body poses) and glances of Jesus and his apostles superimpose a further dynamical structure (cf. Wildgen, 2004b).

The blending of these different content complexes constitutes the central message of the painting. At the same time it creates a pattern of structural layers which is not basically different from what we know about linguistic structures:

  • The basic (linear) dynamics of the sentence define a starting field where we often find a subject, a middle field where verbal constituents and valence governed noun phrases/pronouns are found and (sometimes) a closing field. The specific filling depends on the type of language and on the pragmatics of the utterance in question.
  • The verb in the center of a valence pattern introduces a local space that partially controls the linear dynamics. The dynamical patterns of verb valences have been described in dynamic semantics (cf. Wildgen, 1982, 1994).
  • In the periphery of valence patterns the hierarchically nested nominal, verbal and adjectival phrases complete the picture and adverbial modifiers or inflectional markers further specify the time/mode/aspect (TMA) of the central event/action reported.

In the case of classical paintings, which transport a narrative content and may be “translated” into a text or illustrate a given text, the basic organization of the painting adapts the patterns found in language to the conditions of a two-dimensional representation and its inherent dynamics (which are different from a linear pattern, although they can embed such patterns).

If we continue this line of thought to sculpture and architecture, new types of restrictions are added, which may overwhelm the patterns found in sentences and pictures. Thus the sculpture as a freestanding physical object is submitted to the gravitational force field. (The objects represented in the painting should not contradict our gravitational imagination but gravitation does not affect them directly.) Thus we may wonder, if Mary in Leonardo’s painting of St. Anne may fall from St. Anne’s lap, but in a corresponding sculpture gravitational forces may really destroy this unstable configuration. Therefore sculptors like Henry Moore formulated as the central aim of their art that sculptures must “stand” or “lie” naturally. In an architectural design the physical, technical restrictions become dominant, because a building must be statically and functionally “consistent”. The domain of artistic freedom left for a semiotic message is therefore heavily restricted by static and functional considerations.