Language & Communication 32 (2012) 205215

Contents lists available at SciVerse ScienceDirect

Language & Communication

journal homepage: www.elsevier.com/locate/langcom

Aristotle, Saussure, Kress on speech and writing: Language as paradigm for the semiotic?
Edward McDonald
School of International Studies, University of New South Wales, Sydney 2052, NSW, Australia

a r t i c l e

i n f o

a b s t r a c t
Aristotle famously dened writing in a way that made it dependent on speech; and Saussure, in mapping out a place for language as part of a new eld of semiology, has been seen as continuing this so-called logocentric bias in Western thinking about signs. Kress, by contrast, in dening a new eld of social semiotics, emphasizes the differing materiality of speech and writing as modes with related yet importantly distinct affordances. This paper will use Saussures many-sided questioning of language, to show that Kresss theoretical and descriptive project within Social Semiotics still needs something like Saussures model of linguistic meaning, and to suggest that a clearer theorization of language has positive implications for our understanding of other semiotic modes. 2012 Elsevier Ltd. All rights reserved.

Article history: Available online 21 June 2012 Keywords: Speech Writing Language Social Semiotics Multimodality Affordances

1. Introduction: Moving away from language Spoken words are the symbols of mental experience and written words are the symbols of spoken words. Just as all men have not the same writing, so all men have not the same speech sounds, but the mental experiences, which these directly symbolize, are the same for all, as also are those things of which our experiences are the images. (Aristotle: On Interpretation, trans. Cook 1938: 16a 3) Aristotle famously dened writing in a way that made it dependent on speech; and Saussure, in mapping out a place for language as part of what he envisaged as the new eld of semiology (1916), has been seen as continuing this so-called logocentric bias (Derrida, 1967) in Western thinking about signs. Kress, from the very beginning of his work moving out from language (Kress and Hodge, 1979), used Saussure and the traditions deriving from him as the structuralist counter-example to his new social semiotics. As Kress has come to focus increasingly on multimodality, to the point where he now regards Social Semiotics as one tradition within the broader area of multimodal analysis (Jewitt, 2009), Kress has claimed that this focus on materiality marks a clear break from Saussurean semiotics (2009, p. 57): The focus on materiality marks two decisive moves: one is away from abstractions such as language, the linguistic system or grammar, towards specicity, the materiality of modes, developed in social uses. The other is linking modes as means for representation with the bodyliness of humans: through the physiology of sound and hearing, of sight and seeing, of touch and feeling, of taste and tasting, a recognition that humans make meaning through all these means and a realization that these means of engaging with the world are linked and make meaning jointly. That brings the possibility of recognizing meaning as embodied and provides a means of getting beyond separations in abstractions such as mind and body, affect and cognition. This as a general program sounds all very well. But a supposed need to move away from abstractions surely does not mean that the theorist is also released from the necessity of dening the actual mechanisms whereby meanings are made in different modalities; nor can an appeal to bodilyness or embodiment, a popular theme in recent scholarship across a
E-mail address: e.j.mcdonald@unsw.edu.au 0271-5309/$ - see front matter 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.langcom.2012.04.007

206

E. McDonald / Language & Communication 32 (2012) 205215

range of disciplines (though perhaps more as a general invocation than a detailed research program see critique in Ruthrof (2000)), make up for specic accounts of how bodily affordances are employed for semiotic purposes. Furthermore, as I will show below, scholars such as Kress working within Social Semiotics exhibit an unthinking reliance on the meanings provided by language in order to ground their interpretations of other modalities, without acknowledging that the symbolic mode of language, whether operating by itself or in concert with other modalities in multimodal texts, itself needs to be theorised, including its key feature of arbitrary or unmotivated meaning. Kress, however, seems to assume an unproblematically natural relationship between the meanings we want to express and the means through which we express them, thus marking out his own Social Semiotics as clearly distinct from a Saussurian semiotics which placed great importance on the conventional nature of semiotic resources (Kress, 2010, pp. 6465, original emphasis): [I]n Saussurian semiotics, if I want to be understood, I do so by learning the social rules of use of the semiotic resources which those around me know and use. If I dont know them, Im in trouble. In Social Semiotics, if I want to be understood, by preference I use the resources that those around me know and use to make the signs which I need to make. If I am not familiar with those resources, I make signs in which the form strongly suggests the meaning I want to communicate. Many of us have found ourselves in the latter situation and survived, using signs of gesture, of drawing, of pointing. Those signs however have to be as transparent, as iconic as I can possibly make them. Such a formulation, as well as confusing the employment of semiotic systems with their description by the analyst, seems to be recommending an almost improvisational approach to the use of signs which ignores the highly conventionalised nature of much semiosis, particularly that involving language. Kress, however, is highly critical of Saussures notion of arbitrariness as taking away agency from the sign-maker (2010, p. 64, original emphasis): The notion of arbitrariness goes directly against the notion of the sign-makers interest in the making of signs and meaning. Arbitariness (in Saussures conception of it) and motivation each point to social principles: arbitrariness points to the strength of social power as convention and motivation points to plausibility and transparency of the relations of form and meaning in the sign. The apt relation of material form and cultural meaning is an expression of the sign-makers interest in two ways: matching form and meaning satises the sign-makers wish for an apt realization of their meaning and that, in turn, is needed crucially in communication as a guide for the recipient in their interpretation. Convention points to social agreement and power in sign-use. Motivation points to the need for transparency as a means towards shared recognition in the relation of form and meaning in communication. In an ideal semiotic world, individual sign-makers might well be able to attain this degree of transparency in satisfying their wish for an apt realization of their meaning, but a more plausible analysis would surely have to recognise that individuals learn to operate within a large degree of conventionality. Furthermore, Kress sees Saussure as claiming a special status for language among semiotic systems, a claim which went along with the high era of abstraction in the social sciences in which concepts like Saussures langue and parole were commonplace, as opposed to the specic, material turn of a multimodal Social Semiotics (Kress, 2010, p. 13, original emphasis): Ferdinand de Saussure suggested, at the very beginning of the twentieth century, that Linguistics in its theory and terminology as the then most advanced study of any semiotic system, might sufce to furnish forth the wedding tables for the semiotic feasts to come in the new century. This seemed a reasonable hope, given that that expectation was expressed in the high era of abstraction in the Social Sciences: Linguistics and Semiotics included. In certain areas of the study of language in particular, concepts such as language, langue, parole, stood in for the tendency towards achieving a grasp of the particular via very high degrees of abstraction. By entire contrast, the study of modes in multimodal social semiotics focuses on the material, the specic, the making of signs now, in this environment for this occasion. In its focus on the material it also focuses on the bodilyness of those who make and remake signs in constant semiotic (inter)action. It represents a move away from high abstraction to the specic, the material; from the mentalistic to the bodily. This sense of reacting against the supposed imperialistic sway of language and linguistics also comes out in a recent collection on multimodality edited by Jewitt (2009, p. 1) where Jewitt sees this eld as moving out and away from language, dening its starting point as being to extend the social representation of language and its meanings to the whole range of representational and communicational modes or semiotic resources for making meaning that are employed in a culture such as image, writing, gesture, gaze, speech, posture (p. 1, emphasis added). While this stance from one point of view simply reects the historical background of much Social Semiotic work on multimodality in literacy studies (e.g. New London Group, 1996; Kress, 1997; Cope and Kalantzis, 2000), it does create the danger of other modalities being theorised as, in effect, different kinds of languages, rather than as modalities in their own right. 2. Language vis--vis other sign systems So where did Saussure place language and the study of language within the human sciences? Not indeed in lofty isolation, but as part of a new science of signs as part of social life whose scope he set out in a seminal passage (1916/1957, p. 16, original emphasis):

E. McDonald / Language & Communication 32 (2012) 205215

207

A science which studies the role of signs as part of social life is conceivable; it would be part of social psychology, and consequently of general psychology; I shall call it semiology (from the Greek semeion, sign). Semiology would show what constitutes signs, what laws govern them. Since the science does not yet exist, no-one can say for certain what it would be; but it has a right to exist, a place staked out in advance. Linguistics is only a part of the general science of semiology; the laws discovered by semiology will be laws applicable to linguistics, the latter will circumscribe a well-dened area within the mass of anthropological facts. When Saussure went on to explain why he saw language as having a special status within that eld, he saw this status as depending crucially on the arbitrary nature of linguistic signs (1916/1957, p. 68): [W]hen semiology becomes organized as a science, the question will arise whether or not it properly includes modes of expression based on completely natural signs, such as pantomime. Supposing that the new science welcomes them, its main concern will still be the whole group of systems grounded on the arbitrariness of the sign. . . .Signs that are wholly arbitrary realize better than the others the ideal of the semiological process; that is why language, the most complex and universal of all systems of expression, is also the most characteristic; in this sense linguistics can become the master pattern for all branches of semiology although language is only one particular semiological system. As I read this, what Saussure is stressing here is the difculty of accounting for language as a semiotic system: precisely because of its high degree of arbitrariness, it presents the greatest challenge to a semiological account. Kress, however, is highly critical of the whole notion of arbitrariness (2010, pp. 6566, original emphasis): Saussures mistaken assumption that the relation of signier and signied is an arbitary one was, as is all theory, a product and realization of the social conditions of his time. Here are three objections. First, arbitrariness takes no account either of the patent facts of the histories (of change) of semiotic resources (see in this respect Raymond Williamss (1985) Keywords) nor of the facts of contemporary sign-making practice in every instance. Second it rests on a confusion on Saussures part about the characteristics and the levels at which signier and signied operate. Third, it denies agency to those who make meaning in making signs: in wishing to buttress one pillar of his social theory the force of collective power; or the power of the collective he ignored the source of that power, namely the agency of individuals in their action collectively. For much the same reason the wish for a plausible social theory which does not negate the energy and signicance of individual action I stress the agency of socially formed individuals acting as sign-makers out of socially shaped interest with socially made resources in social interactions in communities. To accuse Saussure, the supreme historical linguist, of being unaware of histories (of change), let alone taking the inventor of synchronic linguistics to task for ignoring the facts of contemporary sign-making, seem more than a little paradoxical, to say the least. Moreover, Kress seems to assume as unproblematic a transparent and highly motivated relationship between the forms of signs and their meanings, a relationship that ascribes the highest degree of agency to the individual sign-makers, while seemingly ignoring the obvious fact, one which Saussure continuously grappled with, of the largely unmotivated nature of linguistic signs. Furthermore, even in the case of the seemingly most motivated kinds of signs such as icons, exactly how the icon signies is far from well understood, as pointed out by Machin (Machin, 2009, p. 183): One of the major problems that has haunted traditional semiotic approaches from the start is the icon. At the root of this problem is how much we can think of images as being like language or composed of abstract symbols that are like words. . . .The problem is to do with whether the icon is a sign at all. And if they are not signs this raises the problem of whether they work like language. Can we think of a photograph as a sign? Can we think of light and dark in a photograph or movie still, a saturated colour on a design, thick or thin borders on a page layout in the same way we think about words? All these things reference the real world rather than symbolising them in the manner of words. For multimodality this has all sorts of implications for how we break down images into elements, think about them in terms of communicative functions, and to know at which level to place them into systems of choices. What is problematic with icons, which have a more or less motivated relationship to their referents, becomes even more problematic with symbols, whose relationship to their referents is more or less wholly conventional; unless, of course, we continue to hold, with Aristotle, that they are meaningful in relation to identical images of experience inside everyones head. Saussures innovation was to replace this commonsense view of linguistic meaning with a far less intuitive characterization of language as not reecting but rather constituting meaning through a mutual delimitation of concept and sound (2002/2006, p. 60, original emphasis) [E]xisting signs MECHANICALLY produce, by their simple presence and the accidental state of their DIFFERENCES at every moment of the language, an equal number not of concepts, but of values understood to be mutually differentiated . . .This contrasting of values, which is a PURELY NEGATIVE fact, becomes a positive fact, because as each sign forms an antithesis with the sum of other comparable signs at a given time, progressing from general to specic categories, it quite independently of us ends up delimited in its own value.

208

E. McDonald / Language & Communication 32 (2012) 205215

At the same time as apparently dismissing the notion of arbitrariness altogether, in his actual analyses Kress assumes an understanding of the symbolic reference provided by language to underpin his interpretations, as shown in the following discussion of the use of pitch in languages (2009, p. 56): In tone-languages pitch is used among other things for lexical purposes: with the same syllabic (or multisyllabic) form, difference in pitch tone produces different words. In English, pitch-movement is used predominantly for grammatical purposes, for forming questions or statements, as well as for the expression of affect. In English, pitch is used for lexical purposes too, but in a lesser way: try saying yes to mean are you really sure? or maybe or to express sceptical half-agreement. These variants use the same syllable yes; in fact they are different words. This analysis takes for granted an understanding of the linguistic meaning of the vocable yes, but does not explain how this arbitrary combination of sounds gains this meaning: in other words, while depending on the symbolic reference provided by language i.e. the meaning of yes to ground his interpretation, Kresss analysis renders this process of symbolic reference itself entirely invisible. David Machin identies a similar paradox in many Social Semiotic analyses of the visual as involving post hoc analysis. . .[i]n other words we have an understanding of something and then construct our concepts round this (Machin, 2009, p. 188). Such a reliance on sources outside the semiotic text to interpret what is going on within the text stems, it seems to me, from a failure to theorise explicitly how meaning is created in a particular modality. If there is no explicit link theorised between contextual meanings and textual patterning, then the only thing analysts have to fall back on are their own interpretative skills. On the one hand, this leads to the phenomenon dubbed by feminist scholar Donna Haraway the God trick, referring to that analytical sleight of hand whereby the analyst poses as if in possession of an omniscient vantage point from which to know the social world (quoted in DeNora (2000, p. 3)). On the other, it leads to language willy-nilly being taken as the paradigm for other sign systems, since concepts originally developed for the analysis of language are simply transferred to the description of other modalities. 3. A science of signs as part of social life In trying to resolve these contradictions, Saussures many-sided questioning of this thing we call language is still, it seems to me, our best guide. This is particularly the case if we acknowledge that Saussure as a theorist was far more concerned to raise the right questions than to come up with the right answers, something already pointed out by Bhler many years ago (1934/1990, pp. 78): Ferdinand de Saussures Course in General Linguistics is anything but an account of results. Instead, the Course reects throughout, and excitingly, the methodological scepticism of a researcher who knows his craft and its results just as well as any other does, but cannot refrain from carrying out his own version of the purifying test of Descartess Meditations on the linguists ndings. His lectures, which were posthumously rounded out to form a book, must have been guided tours through the working drafts of a creative mind of great stature, one still struggling with the problems. I am convinced that we are only at the beginning of the historical inuence of Saussures work, of his sketches on the topic of language theory. Kress himself, in an earlier work, acknowledges this very dialogic and unnished nature of Saussures ideas as they were interpreted for most of the twentieth century (1985, p. 86): Saussures text is constructed precisely out of and in the difference between contradictory discourses: the romantic nineteenth-century discourses of freedom, change and of the social (in this case language) as a species of the natural, and the discourses (embodied in this case in the writings, for instance, of Durkheim) of the rule-governed system, of the social as subject to its own laws, and of the individual as subject to the social. The fact that one reading proved predominant is explained in my account by the predominance of the latter set of discourses at the time when Saussures lectures appeared as a written objectied text. What seems to have been very much a live dialogue for Saussure himself became for his readers a settled undimensional text. In his more recent work, however, Kress seems to have reacted against what he sees as the rigidity of such interpretations of Saussure by going to the other extreme, failing to accord much if any role to the collective, and privileging individual agency to an extent that ies in the face of accepted understandings of human semiosis. So what challenges does Saussure still present for a multi-modal Social Semiotics for, in his own words, a science which studies the role of signs as part of social life? The rst challenge derives from a problem Saussure never ceased to grapple with, that of accounting for the lack of a natural relationship between sound and meaning in language (2002/2006, p. 147, original emphasis): [O]ther institutions are to various degrees all based on NATURAL relationships. . .. For instance, a nations laws, or political system, or even fashion, even its whimsical sartorial fashions, which can never ignore the given [proportions] of the human body. It follows that all changes, all innovations. . .are always dependent on the basic principle in force in this very area, which is to be found in the very depths of the human soul. But language and writing are NOT BASED on a natural relationship between things. There is never in any way a link between a certain sibilant sound and the shape of the letter S, and similarly it is no harder for the word vache than the word vacca to refer to a cow.

E. McDonald / Language & Communication 32 (2012) 205215

209

From this it follows that the basic reality of language consisting not of elements but of relationships (2002/2006, p. 188, original emphasis): In linguistic phenomena, element and character are always the same thing. A characteristic of language, and of any semiological system, is that it does not allow any difference between what distinguishes something and what constitutes it (because the things in question are signs, which have no other aim, essence, than to be distinct). Any linguistic fact consists of a relationship, and nothing but a relationship. One of the earliest major scholars to pick up on this emphasis on relationships rather than elements or components was Hjelmslev, who saw this new conception as subsuming the traditional one (Hjelmslev, 1953/1961, p. 22): [T]he important thing is not the division of an object into parts, but the conduct of the analysis so that it conforms to the mutual dependences between those parts, and permits us to give an adequate account of them. . .. [B]oth the object under examination and its parts have existence only by virtue of these dependences; the whole of the object under examination can be dened only by their sum total; and each of its parts can be dened only by the dependences joining it to other coordinated parts, to the whole, and to its parts of the next degree, and by the sum of the dependences that these parts of the next degree contract with each other. After we have recognized this, the objects of nave realism are, from our point of view, nothing but intersections of bundles of such dependences. That is to say, objects can be described only with their help and can be dened and grasped scientically only in this way. The dependences, which nave realism regards as secondary, presupposing the objects, become from this point of view primary, presupposed by their intersections. A relational point of view poses a much greater challenge to the analyst to identify the networks of internal contrasts through which a semiotic system makes meaning without seeking direct correlates for these contrasts in the world beyond the text. This does not mean, a supposed deciency for which Saussure has been roundly criticised (e.g. Ogden and Richards, 1923), that semiotic systems operate in isolation from their material and social contexts, but rather that, as Saussure points out (2002/2006, p. 154), the fact that the sign can only begin to be truly known when it is understood that it is something not only transmissible, but intrinsically designed to be transmitted means that one of the prime tasks of the analyst is to reveal the internal logic of the system, a logic which in the case of language depends crucially for its power on the fact that there is no motivated relationship between its expressions and their interpretations (see further discussion on this point below). The second challenge derives from the fact that Saussure was concerned in the rst instance to lay the theoretical foundations within semiology of a new kind of linguistics: in other words, he set himself the task of mapping out a new framework for a general linguistics, not for a general semiology. Saussure does not, therefore, provide much explicit guidance for those working on other semiotic systems; though one suspects he would have been less eager than many theorists working within Social Semiotics to simply transfer the insights of linguistics across semiotic boundaries. As Machin notes in relation to the explosion in multimodal studies in recent years (2009, pp. 181182): In linguistics one justication for the need for become multimodal in our analysis is that communication itself has become more multimodal (Kress and Van Leeuwen, 2001) leading to the demise of the authority of the written text. A purely linguistic, or monomodal, analysis will, therefore, miss much of how texts create meaning. But if we are indeed staring at the end of the domination of monomodal linguistic communication should we then be using models that were designed to study language to think about everything else? Would this new state of affairs not suggest rather that linguists should be looking outside of their own theoretical models rather than simply exporting their own? For several thousand years there has been debate about the nature of images, how they represent, and what they have in common with language, through Plato, Freud, Foucault, Goodman, and Habermas, not to mention a century of work in semiotics since Charles Sanders Pierce. Should linguists not rst consider this work and the problems that it has encountered, especially if these ideas have also frequently dealt with visual communication in terms of its similarity to, or difference from, language? Machin provides a roll call of thinkers whose contribution is crucial for understanding how to theorise non-linguistic semiotic modes such as images. But even if, moving on from Saussures work, we concentrate on placing language within a broader typology of semiotic systems, there are further issues which he only touched upon, or did not treat at all, that are crucial for theorising the materiality and sociality of language in a way that puts it on a par with other semiotic systems. 4. The materiality of language The rst has to do with whether the material expression plane of language has any necessary relation to its interpretation plane. In this regard, Saussure approvingly quotes an earlier contemporary who answered this question in the negative (2002/2006, p. 150, original emphasis): In one of the last chapters of Life and Growth of Language Whitney says that people have used voice to give signs to their ideas, just as they might have used gesture or anything else, because they found it easier to use the voice. In our opinion, these two lines, which seem a huge paradox, [represent] the most accurate philosophical idea about language yet, but furthermore, our day-to-day experience of objects that we analyse would benet greatly from this notion. This is because it establishes the fact that language is merely one case of the sign among others and may not be judged independently.

210

E. McDonald / Language & Communication 32 (2012) 205215

To some extent, this may seem simply to be a restatement of the claim put forward by Bacon, citing Aristotle, over four centuries ago (1605, Book ii, original emphasis): For the organ of tradition, it is either speech or writing: for Aristotle saith well, WORDS ARE THE IMAGES OF COGITATIONS, AND LETTERS ARE THE IMAGES OF WORDS; but yet it is not of necessity that cogitations be expressed by the medium of words. FOR WHATSOEVER IS CAPABLE OF SUFFICIENT DIFFERENCES, AND THOSE PERCEPTIBLE BY THE SENSE, IS IN NATURE COMPETENT TO EXPRESS COGITATIONS. And therefore we see in the commerce of barbarous people, that understand not one anothers language, and in the practice of divers that are dumb and deaf, that mens minds are expressed in gestures, though not exactly, yet to serve the turn. And we understand further, that it is the use of China, and the kingdoms of the high Levant, to write in characters real, which express neither letters nor words in gross, but things or notions; insomuch as countries and provinces, which understand not one anothers language, can nevertheless read one anothers writings, because the characters are accepted more generally than the languages do extend; and therefore they have a vast multitude of characters, as many, I suppose, as radical words. The obvious difference here is that Saussures model expressly denies the existence of cogitations separate from their specic means of expression, which would seem to imply that there is some necessary relationship between the sounds of which words are made up and the concepts which they express. We may begin to broach this complex question by taking up the two examples cited by Bacon of means of expression which do not depend on sound: what would now be called sign languages, and Chinese characters. In both cases, we need to distinguish (a) the origin of signs in (more or less motivated) iconic or indexical expressions from (b) their functioning as (more or less unmotivated or arbitrary) symbols in a semiotic system. The role of gesture in sign languages may seem to provide an iconic, natural mode of expression similar to what has been claimed for Chinese characters. In fact, gesture in signed languages has been shown to play exactly the same role as sound in spoken languages: although many signs are clearly iconic or indexical in origin, as elements of a sign language they have exactly the same symbolic function as sounds in spoken languages (Johnston and Schembri, 2007, pp. 2226). The case of Chinese characters may seem to present an iconic system of representation that is not directly linked to, or dependent on, speech as hailed by one sinologist: something unique among the literary languages of the world, a system by which ideas are presented directly to the mind by visual images, with little assistance from the phonetic principle (Creel, 1936, p. 105). More than one sinologist since (e.g. Ames and Rosemont, 1999, pp. 289290) have joined Creel in arguing that Chinese characters do not in fact represent speech, in the sense of spoken language, but some more abstract form of language, one mediated through images rather than sound. Such arguments depend on, as a critic of Creels has observed, the basic assumption that a Chinese graph is a more or less eternal symbol for an idea, from which it follows that the most useful information about it is given through an analysis of its technique of ideal representation (Kennedy, 1953, p. 490). Unfortunately for such claims, any sustained examination of Chinese characters as part of a writing system, as opposed to Just So Stories about the composition of individual characters (McDonald, 2009), shows that the character script represents units of the (spoken) language. Generally speaking in the Chinese writing system, each Chinese character represents a syllable, which in most cases is at the same time a morpheme. This syllabic-morphemic nature of the script shows up very clearly when Chinese characters have been borrowed for other languages: such as in Japanese, where simplied forms of certain characters developed into the individual symbols of the kana syllabaries; or for the traditional Vietnamese system known as Chu Nom where specially devised indigenous characters were used to represent native Vietnamese syllabic-morphemes. Nevertheless such examples do raise the issue of what is the inuence of the expression plane of languages, whether spoken or signed, on their interpretation plane; and what exactly it is that remains constant when the expression plane is transposed from a sonic/aural one to a graphic/visual one, as in the transfer between spoken and written language. Kress, for one, would draw the boundary between the two very sharply, claiming that (2009, p. 56) the differences between speech and writing may be as or more signicant than the similarities. This makes it surprising that speech and writing are subsumed under one label, language. From a social semiotic perspective, the shared label obscures their distinctness as modes with related yet importantly distinct affordances. . .. However, when it comes to characterizing their similarities, his description is more gnomic than enlightening (2006, p. 56): In alphabetic cultures, speech and writing share even though in signicantly different ways features of syntax and lexis; with that go the distinctly different material potentials for meaning of sound and of graphic stuff, both also socially shaped. Kresss vague formulation that spoken and written language share. . . features of syntax and lexis begs the question of how such features are shared. By rendering invisible the symbolic reference of language, at the same time as depending on it to ground his interpretations, Kress effectively takes the nave philosophical stance criticized by Saussure as nomenclaturist (1916/1957, p. 50) by assuming the existence of pre-existing ideas to which spoken words or written graphs may be attached. The radical implication of Saussures alternative position, something he characterised as the Dual Nature of Language to use the title of his recently discovered manuscript (included in Saussure (2002/2006)), is that there are in fact no pre-existing ideas to which words, or graphs, may attach themselves: there are only sounds and concepts mutually delimited into meanings. In Saussures terms, then, what is shared by both written and spoken forms of a language is precisely the

E. McDonald / Language & Communication 32 (2012) 205215

211

same langue, to repeat part of an earlier quotation an equal number not of concepts, but of values understood to be mutually differentiated (2002/2006, p. 60, original emphasis). The written form of a language corresponds to a spoken language, not in the sense that it is a transcription of spoken discourse, but in that it shares the same linguistic system, that is, the same network of mutually differentiated sounds and meanings. The fact that in written language, these sounds are, as it were, translated into graphic form does not negate their essential differentiating function with respect to the meanings of a language. 5. Sound as a semiotic medium Such a formulation, however, still leaves unanswered the question of the nature of sound as a semiotic medium. Linguists, from Saussure onwards, have shown little interest in the question of why sound, and if they are only concerned with language, there is perhaps no obvious reason why they should. We have to look to musicology for guidance here: specically to those scholars seeking to characterize the use of sound in music, who tend to do so in explicit contrast with language, since for both systems the paradigmatic means of expression is the human voice. One of the most penetrating thinkers in this area, Burrows, in his Sound, Speech, Music sees the contribution of sound as essential and determining (1990, p. 9): Chief vehicle of thought is speech, and of speech, sound, and I will argue that sound is far more to speech than a passive conveyance. I suggest rather that human thought has evolved its expansiveness and freedom in large part through exploiting the unique capacity of vocal sound for rapidity of articulation in detachment from the world of enduring spatial objects. Burrows explains how the physiological characteristics of vocal sound become semioticized in a way that provides the individual with a strategically advantageous sonic mining of its situation in the world (1990, p. 30): Vocalization may be thought of as a freely manipulable representation and advertisement of the life that underlies it. . .. In the most blandly literal sense, the voice is expression, a pressing outward past the partially yielding obstacle of the vocal folds. The dialogue between diaphragm and larynx can be thought of as a symbolic displacement to the bodys interior of the interface between self and world, a displacement that has a strategic advantage over its original because, in this wholly embodied form, outside and inside are both inside where the self gets to play both sides of the game. . .. The resultant signal, the sound of the voice, can be read as the selfs sonic miming of its situation in the world. Burrows then goes on to identify the specically semiotic power of vocal sound in economically representing objects, in contrast to the iconic visual mode (1990, pp. 5152): The loss of one kind of power, the kind inherent in the resemblance of a token to its referent, is more than made up for by gains that are also attributable to detachment from appearances: nonlikenesses are both more generalizable and more freely combinable in novel congurations than are likenesses, and speech exploits both possibilities in reaching for a new kind of power. Precisely because the sounds produced by the voice have no spatial characteristics of their own, they can, if we choose, stand for anything in space, or for any place, or for everywhere or nowhere. . . .Breaking with the look and the heft of things is a necessity in vocal communication, and on a massive scale we have made a virtue of it by capitalizing on a whole new range of manipulative possibilities that it opens up. It is at this point that Burrows thesis intersects with Saussures description of the limits of langue or the language system, limits which are at the same time the source of its power (2002/2006, pp. 5051): [I]t hardly needs pointing out that the differences between terms making up a language system in no way correspond, even in the most perfect language, to the relationship between things. . . .[T]he impression left by a material object will never be sufcient to create a linguistic category. There are thus only ever negative terms, in each of which an object in imperfectly housed; by the same token, the object will be parcelled out into several terms. Complaining about the inaccuracy of language, however, suggests ignorance of the source of its power. It is quite impossible to prevent a single, specic thing being variously referred to as a house, a construction, a building, a structure (a monument), a block, a dwelling, a residence. . . The existence of material facts, then, like the existence of facts of another order, has no bearing on langue. Langue is ever on the move, pressed forward by its imposing machinery of negative categorization, wholly free of materiality, and thus perfectly prepared to assimilate any idea that may join those that have preceded it. Burrows arguments could, it seems to be, easily be transposed mutatis mutandis to the case of sign languages, since the manipulatory and combinatory potential of expressions produced by the hands and the face in signed languages approaches if not equals those produced by the tongue and lips in spoken languages, an analogy that is especially compelling in view of the known evolutionary link between the development of manual dexterity in the human species and the development of language (Arbib, 2002). Both media allow for a large repertoire of distinct expressions that at the same time are easily combined to compose complex messages. Both media, when utilised semiotically, also exhibit the characteristic feature which has variously been named structural order (Tesniere, 1959, p. 16), or structure in depth (Hockett, 1987: 16), in other words,

212

E. McDonald / Language & Communication 32 (2012) 205215

that we can recognize (layers of) relationships between meaningful elements that are not simply a function of their relative sequence. Jakobson saw such features as characteristic of both language and music as sonic semiotic systems (1967/1971, p. 341): [B]oth music and language present a consistently hierarchized structure, and. . .musical as well as verbal signs are resolvable into ultimate, discrete, rigorously patterned components which, as such, have no existence in nature but are built ad hoc. These hierarchical relationships between linguistic units reected in folk linguistic notions of syllables forming words which form phrases which in turn form sentences are commonly understood in terms of Saussures distinction of associative now more commonly called paradigmatic relationships of substitution from syntagmatic relationships of combination (1916/1959, p. 123), and Firths distinction of colligation between grammatical categories from collocation between lexical items (1956/1968, pp. 179181). 6. The sociality of language It is when we acknowledge this high order of complexity that we read into the acoustic signal which is the physical manifestation of language that Saussures insistence on both sides of the linguistic sign the sound image or signier and the concept or signied being conceptualized as psychological entities becomes comprehensible. At the same time, as suggested by his denition of the broader eld of semiology in which he envisages linguistics taking its place as the study of signs as part of social life, this fundamentally psychological nature of the linguistic sign is part of a linguistic system whose functioning is inextricably social (2002/2006, p. 120, original emphasis): Language is always being considered within the human individual, a false viewpoint. Nature gives us man ready made for articulated language, but not actually in possession of it. The language system is a social fact. The individual, organized with a view to speaking, may only use the vocal apparatus in the context of his community moreover, the individual only feels the need to use it when interacting with that community. Nor is it justied to accuse Saussure of being unaware of discourse, or of unduly privileging the abstract system over the concrete manifestations of that system (2002/2006, p. 197, original emphasis): Langue is created only with a view to discourse; but what separates discourse from the language system, and what allows us to say that a language system enters into action as discourse at any given moment[?]. . . What is needed in order for us to get the idea that someone wants to signify something by using the terms that are available in a language system? This is the same question as asking what is discourse, and at rst sight the answer is simple: discourse consists, even if only in a rudimentary way, and in ways of which we are unaware, of conrming a link between two of the concepts that appear cloaked in a linguistic form, while the language system at rst consists only of isolated concepts that are waiting to be put into relation with each other so that meaningful thought may be expressed. He elaborates further on this point, emphasizing both the temporal priority of discourse and the semiotic primacy of the internal store that is langue (2002/2006, p. 81): The educational fact that we may learn sentences before knowing words is of no real consequence. It is like observing that langue rst enters into our internal store via discourse; this is something we have said, and which is indisputable. But just as the sound of a word, something which also entered our mind this way, becomes an impression which is wholly independent of discourse, so our mind continually abstracts from discourse what is necessary so that the word alone remains. The way the word came to be xed is unimportant, once the operation is completed, as long as we accept that it is the dominant unit. It is true that Saussures insistence on langue, the language system, as the fundamental principle of classication in linguistics may be understood as taking away agency from the speaking subject (2002/2006, p. 64): All modications, be they phonetic or grammatical (analogical) occur exclusively within discourse. At no time do individuals revise their inner mental store of langue, and detachedly create new forms. . .with the intention. . .of inserting them into future discourse. All innovation comes about through improvisation, when someone speaks, and from there enters either into the inner store of the listener or the speaker, being thus produced in relation to discursive language. However, it is clear from this account that Saussure does recognize individual agency, at least in the improvisation of new forms, but that these must be socially accepted among speakers in order to function as part of the linguistic system. From one point of view, this is simply acknowledging the fact that language use depends on a more or less agreed set of conventions which cannot be wilfully modied by individual speakers without risking a breakdown in communication. But from another point of view, it stresses the ineradicably social nature of language (2002/2006, p. 64): [L]angue ows between people, it is social. If I ignore this condition, if for instance I sit down at my desk to make up a language, nothing that I shall say about language will be true, or necessarily true.

E. McDonald / Language & Communication 32 (2012) 205215

213

7. Towards an inclusive semiology If we now move forward from the picture provided by Saussure for the study of language, what might an inclusive but not linguistically-biased semiology look like? Saussure provides some useful pointers here (2002/2006, p. 54, original emphasis): 1. Language is merely a specic case of the theory of signs. But this very fact alone totally prevents it from being something simple (or something whose nature the mind can grasp directly), even though in fact, within the general theory of signs the specic case of vocal signs might not be incalculably more complex than all the specic known cases, such as writing, numerals etc. 2. The crucial response of the study of language concerning the theory of signs, and the eternally new horizon it will open up. . ., will be to impart to that theory a whole new aspect of the sign, the fact that the sign can only begin to be truly known then it is understood that it is something not only transmissible, but intrinsically designed to be transmitted. One interesting trajectory in this regard is that sketched out by Ruthrof in his Semantics and the Body, putting forward an intersemiotic interpretation of the key concept of meaning, building particularly on the work of Peirce, which attempts to put all semiotic behaviour, including the linguistic, on a equal footing (Ruthrof, 1997, p. 24): From this perspective, meaning is regarded as a virtual event in which at least two, and normally several, sign systems are associated with one another, such that visual, tactile, haptic, and other non-verbal signs are engaged in the activation of linguistic expressions. Non-linguistic meaning is likewise conceived in this way. This requires a redescription of perception in signicatory terms. . . .Reference. . .is regarded as a specic link between sign systems, rather than as a relation between two incompatible domains: the signs of language, and a non-semiotic external world. In this way, the world as we know it is signicatory, rather than a physicalistically given. Ruthrof goes onto argue for a conception of meaning based on acts, rather than the objects or states of affairs in the world of traditional semantics, acts which moreover are meaningful in a certain community, a model he indeed later dubs a kind of social semiotics (p. 36), whose axiomatic assumptions he sketches out as follows (Ruthrof, 1997, p. 33, original emphasis): Suppose meaning is not in any way a feature of language but a broader feature of social doing, of which language is a part. Let us say that social doing of any kind is regarded as either meaningful or meaningless by a community. . . .A community knows its world because its members have imposed and continue to impose a signicatory matrix on whatever there is. . . .[M]eaning is the realization by a community of the relation between different sign systems. Members of the community are dened primarily by their ability to negotiate such relations according to the communitys recipes for interpretation. . . .[R]eality is the result of the corroboration of one sign system by at least one other sign system. Ruthrofs insistence here on meaning-making as taking place in communities, as well as what we might call his semiotic democratism in contrast to what he dubs the linguistic imperialism (1997, p. 186, 2000, p. 22) characteristic not only of most theories of semantics, but I would argue, of many multimodal approaches are both useful emphases to maintain. A complementary approach is taken by Norris in the context of multimodal interactional analysis with her identication of the fundamental unit of analysis as being the mediated action (2005, p. 13, original emphasis): In multimodal interactional analysis, the mediated action is the unit of analysis, and since every action is mediated, I will simply speak of action as the unit of analysis. . .. This use of action as a unit of analysis may seem confusing at rst sight. However, let us think about the specic example of a meeting among three friends to illustrate the usefulness of this unit of analysis. The meeting is taken to be a higher-level action. This higher-level action is bracketed by an opening and a closing of the meeting and is made up of a multiplicity of chained lower-level actions. All intonation units that an individual strings together become a chain of lower-level actions. All gesture units that an individual performs become a chain of lower-level actions. All postural shifts that an individual completes become a chain of lower-level actions. All gaze shifts that an individual performs become a chain of lower-level actions, and so on. Consequently, all higher-level actions are made up of multiple chains of lower-level actions. In this approach, language is simply one communicative mode among many (2005, p. 2): Previously, language has been viewed as constituting the central channel in interaction, and nonverbal channels have been viewed as being subordinate to it. While much valuable work on the interplay between the verbal and the nonverbal has been established, I believe that the view which unquestionably positions language at the center limits our understanding of the complexities of interaction. Therefore I will step away from the notion that language always plays the central role in interaction, without denying that it often does. Language. . .is only one mode among many, which may or may not take a central role at any given moment in an interaction. In this view, gesture, gaze, and head movement may be subordinated to the verbal exchanges going on as has been shown in much research. However, gesture, gaze, and head movement may also take the superior position in a given interaction, while language may be subordinated or absent altogether. Or more pithily (2007), Sometimes we use language, but always we communicate.

214

E. McDonald / Language & Communication 32 (2012) 205215

Nevertheless, an emphasis on the material affordances of the expression plane cannot replace a detailed understanding of the means by which meanings are distinguished on the interpretation plane, a challenge Kress seems to gloss over (2009, p. 55, original emphasis): Alphabetic writing is a border category in this respect: it is spatially displayed, yet it leans on speech in its logic of sequence in time, which is mimicked in writing by the spatial sequence of elements on the line on which writing is displayed. . .. However, while the relations of elements of an image can usually be read in an order shaped by the interest of the viewer, the reading of writing is governed by linearity and directionality left to right or right to left and by the ordering of syntax. Unlike image, writing is not, dominantly and nally, organized by the logic of space; as readers we are bound by the orderings of syntax and the directionality of the line. In alphabetic cultures, speech and writing share even though in signicantly different ways features of syntax and lexis; with that go the distinctly different material potentials for meaning of sound and of graphic stuff, both also socially shaped. A full semiotic account of written language needs to take account not only of the directionality of the line but also of the ordering of syntax, understanding the latter as a multi-dimensional kind of patterning that goes beyond, though it never escapes from, the linearity of the linguistic sequence (Tesnire, 1959, pp. 1618). Again, the implied contrast with Chinese writing here is misplaced: whether in the traditional convention of vertical lines of characters read from top to bottom, right to left, or the modern Western-inuenced convention of reading horizontally from left to right, Chinese writing obeys the same directionality as the written form of any other language. So to nish up by attempting to answer the question posed by the title of this paper: in what sense, if any, may we meaningfully regard language as being paradigm for the semiotic? To repeat the careful formulation of Saussure already quoted above (2006, p. 54): Language is merely a specic case of the theory of signs. But this very fact alone totally prevents it from being something simple (or something whose nature the mind can grasp directly), even though in fact, within the general theory of signs the specic case of vocal signs might not be incalculably more complex than all the specic known cases, such as writing, numerals etc. Saussure, unlike many linguists and semioticians since (e.g. Barthes, 1964/1968), shows himself agnostic on the question of whether language actually is incalculably more complex than other systems. However, to repeat a point made earlier, although we cannot fault Saussure, the linguist, for being primarily concerned with language, from the point of view of a general semiology, it is the very difculty of accounting for the largely symbolic, unmotivated nature of linguistic signs that makes a clear understanding of linguistic meaning very hard to attain. At the same time, whether or not we regard language as functioning as the (potential) interpreter of all other semiotic systems, it is undeniable that it tends to be used that way by many semioticians, especially Social Semioticians, very often in the same breath as denying that language has any special status. A refocusing on communication as multimodal may well be being forced on us by developments in communicative technologies, as Machin suggests, but it also represents a long overdue recognition of the importance of embodied semiotics in much human interaction, as Ruthrof shows. However, the bodily cannot be taken as an unquestioned positive whose meanings are natural and therefore open to easy explanation: its semiotic affordances as employed in systems as diverse as gaze, movement, gesture not to mention speech and singing still need to be theorised so that the particular network of negative distinctions created in each case may be understood (Thibault (2004) represents an interesting step in this direction). Much multimodal work in the Social Semiotic tradition seems curiously visually-biased, and at the same time largely unproblematically analogising concepts from linguistics for the analysis of other semiotic systems, without seemingly feel much need, as Machin notes, to engage with existing scholarship in those areas. If multimodal studies is to be anything more than simply a new eld for linguists or ex-linguists to play in, it will need to face up to the challenges of theorising all semiotic systems on their own terms, as well as in interaction with each other, within a overall eld for which Saussures recommendation for a study of the role of signs as part of social life still provides the best pointer. References
Ames, Roger T., Rosemont Jr., Henry., 1999. The Analects of Confucius: A Philosophical Translation. Ballantine Books, New York. Arbib, M.A., 2002. The mirror system, imitation, and the evolution of language. In: Dautenhahn, Kerstin, Nehaniv, Chrystopher (Eds.), Imitation in Animals and Artifacts. The MIT Press, Cambridge MA, pp. 229280. Bacon, Francis, 1605/1998. The Advancement of Learning. Renascence Editions, An Online Repository of Works Printed in English Between the Years 1477 and 1799. <http://www.luminarium.org/renascence-editions/adv1.htm>. Barthes, Roland, 1964. lments de smiologie Communications 4, Paris: Seuil. (1968). In: Lavers, Annette, Smith, Colin (trs.) Elements of Semiology. Hill and Wang, New York. Bhler, Karl, 1934/1990. In: Goodwin, D.G. (tr.), Theory of Language: The Representational Function of Language. Benjamins, Amsterdam. Burrows, David, 1990. Sound, Speech, and Music. The University of Massachusetts Press, Amherst, MA. Cook, H.P., 1938. Aristotle: De Interpretatione. Loeb Classical Library, London. Cope, B., Kalantzis, M. (Eds.), 2000. Multiliteracies. RoutledgeFalmer, London. Creel, Herrlee Glessner, 1936. On the Nature of Chinese Ideography. Toung Pao xxxii, 85161. DeNora, Tia., 2000. Music in Everyday Life. Cambridge University Press, Cambridge. Derrida, Jacques, 1967. In: Spivak, Gayatri Chakravorty (tr.), De la grammatologie (1974). Of Grammatology. Johns Hopkins University Press, Baltimore. Firth, J.R., 1956/1968. Linguistics and translation. In: Palmer, F.R. (Ed.), Selected Papers of J.R. Firth 195259. Longmans, London, pp. 8495.

E. McDonald / Language & Communication 32 (2012) 205215

215

Hjelmslev, Louis, 1943/1953/1961. Prolegomena to a Theory of Language. English Translation by Francis J. Whiteld. Indiana University Publications in Anthropology and Linguistics, Baltimore (IJAL Memoir 7), second ed. University of Wisconsin Press, Madison (slightly revised (1961)). Hockett, Charles, 1987. Refurbishing our Foundations: Elementary Linguistics from An Advanced Point of View. Current Issues in Linguistic Theory, vol. 56. Benjamins, Amsterdam. Jakobson, Roman, 1967/1971. Visual and auditory signs. In: Proc. AFCRL Symposium on Models for the Perception of Speech and Visual Form reprinted in Selected Writings, vol. 2. Word and Language, The Hague, Mouton, pp. 334344. Jewitt, Carey. (Ed.), 2009. The Routledge Handbook of Multimodal Analysis. Routledge, London. Johnston, Trevor, Schembri, Adam, 2007. Australian Sign Language: An Introduction to Sign Language Linguistics. Cambridge University Press, Cambridge. Kennedy, George A., 1953. Reviews of Books: Literary Chinese by the Inductive Method, vol. III. The Mencius, Books IIII; Talks on Chinese History (Junggwo Lishr Jyanghwa); Ching Documents. An Introductory Syllabus. Journal of the American Oriental Society, vol. 73, no.1, pp. 2730. Kress, Gunther, 1985. Linguistic Processes in Sociocultural Practice. Deakin University Press, Geelong. Kress., Gunther., 1997. Before Writing: Rethinking Paths to Literacy. Routledge, London. Kress, Gunther, 2009. In: Jewitt, Carey (Ed.), What is Mode? pp. 5467. Kress, Gunther., 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. Routledge, London. Kress, Gunther, Hodge, Robert, 1979. Language as Ideology. Routledge & Kegan Paul, London. Kress, Gunther, van Leeuwen, Theo, 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. Arnold, London. Machin, David, 2009. In: Jewitt, Carey (Ed.), Multimodality and Theories of the Visual, pp. 181190. McDonald, Edward, 2009. Getting over the walls of discourse: Character Fetishization in Chinese Studies. Journal of Asian Studies 68.4, 11891213. New London Group, 1996. A pedagogy of multiliteracies: designing social futures. Harvard Educational Review 66, 60-92. Norris, Sigrid, 2005. Analyzing Multimodal Interaction: A Methodological Framework. Routledge, New York. Norris, Sigrid, 2007. Can We Really Speak of Semiotic Margins?: A Theoretical Un-focussing of Language Semiotic Margins: Reclaiming Meaning, University of Sydney (1012 December). Ogden, C.K., Richards, I.A., 1923. The Meaning of Meaning. Kegan Paul, Trench, Trubner, London. Ruthrof, Horst, 1997. Semantics and the Body: Meaning from Frege to the Postmodern. University of Toronto Press, Toronto. Ruthrof, Horst, 2000. The Body in Language. Cassell, London. Saussure, Ferdinand de, 1916. Cours de linguistique gnrale. Baskin, Wade (tr.) (1957). Course in General Linguistics. Philosophical Library, New York. Saussure, Ferdinand de, 2002. crits de linguistique gnrale. Sanders, Carol, Pires, Matthew (trs.) (2006). Writings in General Linguistics. Oxford University Press, Oxford. Tesnire, Lucien, 1959. lments de syntaxe structurale [Elements of structural syntax]. C. Klincksieck, Paris (second edition revised and corrected 1965). Thibault, Paul J., 2004. Brain, Mind and the Signifying Body: An Ecosocial Semiotic Theory. Continuum, London.

Edward McDonald is senior lecturer in the School of International Studies at the University of New South Wales, and has taught Chinese language, music, linguistics, and semiotics at universities in Singapore, China, and New Zealand. His research interests lie in the areas of embodied performative semiotic systems in particular language and music, multimodality and systemic functional theory, translation and cross-cultural communication, Chinese grammar and discourse, ideologies of language, and the history of linguistics. His most recent publication is Learning Chinese, Turning Chinese: challenges to becoming sinophone in a globalized world (Routledge, 2011).