Está en la página 1de 172

Artist-Programmers

and
Programming Languages for the Arts
Chiistophei Alex McLean
Tesis submiued to Goldsmiths,
Univeisity of London,
foi the degiee of Doctoi of Philosophy.
Octobei zc11
Dedicated to Jess, Haivey, Mum, Dad, Andiea and Stuait and the iest of my family, to
whom l owe eveiything, and with the memoiy of those who suppoited this woik but
passed away befoie seeing its end, namely my lathei-in-Law Di. David llmoie, and
my cat Olga.
z
Abstract
We considei the aitist-piogiammei, who cieates woik thiough its desciiption as souice code.
Te aitist-piogiammei giandstands computei language, giving unique vantage ovei human-
computei inteiaction in a cieative context. We focus on the human in this ielationship, noting
that humans use an amalgam of language and gestuie to expiess themselves. Accoidingly we
expose the deep ielationship between computei languages and continuous expiession, exam-
ining how these iealms may suppoit one anothei, and how the aitist-piogiammei may fully
engage with both.
Oui aigument takes us up thiough layeis of iepiesentation, staiting with symbols, then
woids, language and notation, to considei the iole that these iepiesentations may play in hu-
man cieativity. We foim a cioss-disciplinaiy peispective fiom psychology, computei science,
linguistics, human-computei inteiaction, computational cieativity, music technology and the
aits.
We develop and demonstiate the potential of this view to infoim aits piactice, thiough
the piactical intioduction of sofwaie piototypes, aitwoiks, piogiamming languages and im-
piovised peifoimances. ln paiticulai, we intioduce woiks which demonstiate the iole of pei-
ception in symbolic semantics, embed the iepiesentation of time in piogiamming language,
include visuospatial aiiangement in syntax, and embed the activity of piogiamming in the
impiovisation and expeiience of ait.
I
Anowledgements
l must ist thank Geiaint Wiggins foi his auentive supeivision, mentoiship and insights, help-
ing me along a fascinating jouiney. l have gieatly beneted fiom his stiiving focus on honest,
iigouious ieseaich, and l do not doubt that his suppoit thiough MSc and PhDieseaich iequiied
a leap of faith on his pait, showing both an open mind and deep geneiosity. My thanks extend
to all the othei membeis of the lntelligent Sound and Music Systems gioup foi daily lunchtime
discussions on all things, l thank you all and must name my co-supeivisoi Maik dlnveino,
my collaboiatoi Jamie loith, and also Daniel Jones foi his geneious and detailed ciiticism. My
thanks extend fuithei to envelop the depaitment as a whole, including Tim Blackwell, Mick
Giieison, Janis Jeeiies, liedeiic Leymaiie and Robeit Zimmei foi theii encouiagement and
suppoit. lt has been a gieat piivilege to woik amongst such cieative minds.
Te path leading to this thesis ieally began in the yeai zccc, when Adiian Waid encouiaged
me to expeiiment with algoiithmic composition. We foimed the geneiative music (and latei,
live coding) band Slub, joined by Dave Giiths. Many of the ideas exploied heie have ioots in
this collaboiation, and l cannot beai to imagine life without Slub. My thanks to both.
My foiays into llectionic and Sofwaie Ait also paved the way to the piesent woik, made
possible with the suppoit, encouiagement and collaboiation of Saul Albeit, Amy Alexandei,
Geo Cox, Olga Goiiunova, Douglas Repeuo, lunJoo Shin and Alexei Shulgin. My jouiney
into ieseaich also ieceived gieat suppoit fiom my foimei colleagues at state1, paiticulaily
Paul Sandeis, who has always been geneious with iich ideas, encouiagement and ciiticism.
Tank you all.
He may not iemembei, but my ist thoughts about a ieseaich degiee weie planted by Rob
Saundeis, the Al and Cieativity ieseaichei. We have haidly ciossed paths since, but without his
encouiagement l may not have made the leap. lollowing this, Simon lmmeison was extiemely
suppoitive, including helping secuie funding foi my MSc, making the whole thing possible.
Tanks both.
Speaking of which, l must also sinceiely thank the PRS loundation foi funding the fees foi
my MSc in Aits Computing, and the lPSRC foi the full funding of my PhD studentship. lt is
1
wondeiful that these gieat institutions fund unusual aieas of ieseaich.
Te nal stage of wiiting up the piesent thesis was completed within the OAK gioup at
the Univeisity of Sheeld, special thanks go to Simon Tuckei and labio Ciiavegna foi helping
make the tiansition fiom ieseaich student to ieseaichei so enjoyable.
l have met with many inuential minds ovei the last foui yeais, moie than l could list
heie. But l ieally must thank Sam Aaion, Saiah Angliss, David Bausola, Renick Bell, Vaughan
Bell, Ross Bencina, Alan Blackwell, Giaham Booth, Ollie Bown, Andiew Biown, John Buiton,
Aituio Castio, Giaham Coleman, Chiistof Damian, lleanoi Daie, Ben lields, Philip Galantei,
Joel Gethin-Lewis, Biuno Gingias, Sam lieeman, Mauhew lullei, Alex Gaiacotche, lniico
Gleiean, Jake Haiiies, Coimac Heion, Scou Hewiu, Tom Holley, Sampath Jagannathan, Seigi
Joida, Ryan Joidan, David Lewis, Toi Magnusson, Chailes Mauhews, Jon McCoimack, An-
diew Mclailand, Daniel M ullensiefen, Rob Myeis, liic Namoui, Ashiaf Nehiu, Julian Olivei,
liediik Olofsson, liini Papadimitiiou, Rob Paitington, Aneeta Patel, Maicus Peaice, Hestei
Reeve, Chiistophe Rhodes, Chiis Roast, Julian Rohihubei, Oliviei Ruellet, Ash Sagai, Andiew
Soiensen, Dan Stowell, Andy Studei, Patiick Tiesset, Kassen Oud, Michele Pasin, Juan Romeio,
Ge Wang, Mitchell Whitelaw, Scou Wilson, Mauhew Yee-King, and eveiyone in Doikbot, eu-
gene, openlab, POTAC and TOPLAP.
l must also thank those who having bought me diinks, pioofiead my woik, shaied theii
ideas and/oi oeied helpful advice, have failed to nd theii name above. My special thanks to
you, and l apologise foi foigeuing about you at this ciucial moment, especially my fiiend and
collaboiatoi Nick Collins, l have ieached this point only by ielying upon Nick as a deep well
of inspiiation and enthusiasm ovei the past 1c yeais, and he ieally deseives beuei.
linally l would like to thank Jess llmoie foi hei undeistanding and suppoit, and oui son
Haivey McLean. Haivey is one month oldei than my PhDstudentship, and his speedy develop-
ment into a walking, talking, loving and complaining human being has been extiemely useful
in keeping me aleit to the passing of time.

Contents
1 Introduction 13
1.1 Aitist-Piogiammeis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.1.1 Computei ait . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.1.z Geneiative vs Sofwaie ait . . . . . . . . . . . . . . . . . . . . . . . . . 1e
1.1.I Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.z Piogiamming Languages foi the Aits . . . . . . . . . . . . . . . . . . . . . . . 1
1.I Aims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1s
1.1 Stiuctuie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1s
1. Oiiginal contiibutions to knowledge . . . . . . . . . . . . . . . . . . . . . . . . 1v
2 Symbols 20
z.1 Situating symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . zc
z.z Symbols in cognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . zI
z.z.1 Mental imageiy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . z1
z.z.z Mental lmageiy and Piogiamming . . . . . . . . . . . . . . . . . . . . ze
z.z.I Dual Coding in Souice Code . . . . . . . . . . . . . . . . . . . . . . . . z
z.z.1 Language and Situated Simulation . . . . . . . . . . . . . . . . . . . . zv
z.z. Conceptual Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ic
z.z.e Metaphoi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Iz
z.I Anthiopomoiphism and Metaphoi in Piogiamming . . . . . . . . . . . . . . . II
z.1 Synaesthesia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I
z. Aitistic synaesthesia` . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ie
z.e Acid Sketching Semantic lmageiy . . . . . . . . . . . . . . . . . . . . . . . . Is
z. Phonemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1c
z.s Miciophone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
z.v Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1z
e
CoN1iN1s
3 Words 45
I.1 Peiceiving Speech as Movement . . . . . . . . . . . . . . . . . . . . . . . . . . 1e
I.z Vocable Woids in Music Tiadition . . . . . . . . . . . . . . . . . . . . . . . . . 1s
I.I Timbie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
I.I.1 Multi-Dimensional Scaling . . . . . . . . . . . . . . . . . . . . . . . . . z
I.I.z Giounding Timbie in Movement . . . . . . . . . . . . . . . . . . . . . I
I.I.I Music of Timbie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
I.I.1 Dening Timbie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . e
I.I. Timbial analogies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
I.I.e Sound souice modelling . . . . . . . . . . . . . . . . . . . . . . . . . . s
I.I. Univeisality of Timbie . . . . . . . . . . . . . . . . . . . . . . . . . . . s
I.1 Aiticulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
I.1.1 Rhythm in Speech . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ec
I.1.z Sound poetiy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ec
I.1.I Woids in Music Technology . . . . . . . . . . . . . . . . . . . . . . . . e1
I. Vocable synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ez
I..1 Babble - vocable Kaiplus-Stiong synthesis . . . . . . . . . . . . . . . . eI
I..z Mesh - vocable waveguide synthesis . . . . . . . . . . . . . . . . . . . e1
I..I Vocable manipulation and analysis . . . . . . . . . . . . . . . . . . . . ee
I.e Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ee
4 Language 68
1.1 Natuial and Computei Language . . . . . . . . . . . . . . . . . . . . . . . . . . es
1.z Music, Language and Cognitive Semantics . . . . . . . . . . . . . . . . . . . . z
1.I Declaiative vs lmpeiative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1 Domain Specic Language foi Pauein . . . . . . . . . . . . . . . . . . . . . . .
1. Tidal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
1..1 leatuies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
1..z Repiesentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . s1
1..I Pauein geneiatois . . . . . . . . . . . . . . . . . . . . . . . . . . . . . sI
1..1 Paising stiings into polymetiic paueins . . . . . . . . . . . . . . . . . s1
1.. Pauein combinatois . . . . . . . . . . . . . . . . . . . . . . . . . . . . se
1.e Open Sound Contiol paueins . . . . . . . . . . . . . . . . . . . . . . . . . . . . vc
1.e.1 Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v1
1.e.z Sending messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v1
1.e.I Use in impiovisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . vz

CoN1iN1s
1.e.1 lutuie diiections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vz
1. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vI
5 Notation 95
.1 Cognitive Dimensions of Notation . . . . . . . . . . . . . . . . . . . . . . . . . v
.z Notation in Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
.I Visual notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1c1
.I.1 Patchei Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1cz
.1 Notation and Mental lmageiy . . . . . . . . . . . . . . . . . . . . . . . . . . . 1cI
. Geometiy in Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1c1
.e Visual Piogiamming with Textuie . . . . . . . . . . . . . . . . . . . . . . . . . 1c
.e.1 Geometiic ielationships . . . . . . . . . . . . . . . . . . . . . . . . . . 1cs
.e.z Usei lnteiface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11c
.e.I Textuie in Piactice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11c
.e.1 Cognitive Dimensions of Textuie . . . . . . . . . . . . . . . . . . . . . 11z
.e. lutuie diiections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11I
. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6 Creativity 115
e.1 Piogiammei cultuie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
e.z Concepts in cieative behavioui . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
e.I Cieative Piocesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11s
e.I.1 Cieative Piocess of Biicolage . . . . . . . . . . . . . . . . . . . . . . . 1zz
e.1 Symbols and Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1zI
e. Components of cieativity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1z1
e.e Piogiamming in Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1z
e. lmbodied piogiammeis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1zs
e.s Live Coding in Piactice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1zv
e.s.1 Obscuiantism is dangeious. Show us youi scieens. . . . . . . . . . . 1I1
e.s.z Cognitive Dimensions of Live Coding . . . . . . . . . . . . . . . . . . . 1Iz
e.v Live codeis on computational cieativity . . . . . . . . . . . . . . . . . . . . . . 1II
e.v.1 Te subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1I1
e.v.z Cieating language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1I1
e.v.I Code and style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1I1
e.v.1 Live coding as a novel appioach . . . . . . . . . . . . . . . . . . . . . . 1I
e.v. Computational cieativity . . . . . . . . . . . . . . . . . . . . . . . . . . 1Ie
s
CoN1iN1s
e.v.e Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1I
e.1c Slub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Is
e.1c.1 Reections following a Slub peifoimance . . . . . . . . . . . . . . . . . 11c
e.11 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11z
7 Future directions 144
.1 Te lieedom of lnteipietation . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
.z Sofwaie engineeiing standaids . . . . . . . . . . . . . . . . . . . . . . . . . . 11e
.I Cyclic ievision contiol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11v
A Live coding survey 166
A.1 Denitions of live coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1ee
A.z Compaiing live coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1es
v
Resear outputs
Te piesent thesis has been developed and disseminated thiough the following activities and
publications.
Piactice
Acid Sketing (2.6). lxploiing diawn analogue symbols in digital ait. lnstalled in the Doik-
bot tent at the Big Chill lestival, August zc1c.
Microphone (2.8). lxploiing vocal analogue symbols in digital ait. Cieated with lunJoo
Shin, installed at the Unleashed Devices gioup show at Wateimans Galleiy, Septembei/Oc-
tobei zc1c.
Babble (3.5.1). System foi impiovising musical timbie with wiiuen woids. A peimanent
on-line woik commissioned by Ainolni, Biistol in Novembei zccs.
Tidal (4.4) and Texture (5.6). Computei language designed foi the impiovisation of musical
pauein, with visuospatial notation. Released with a fiee/open souice license, and used by the
piesent authoi in public music peifoimances.
Live Coding Performances (6.8). Te following is a list of live coding peifoimances featui-
ing the piesent authoi duiing the peiiod of ieseaich. Tose maiked with an asteiisk weie as
pait of a collaboiation with Adiian Waid and/oi Dave Giiths as Slub, which is discussed in
e.1c.
Aiea1c, Peckham, London, 1zth Apiil zccs *
lvy House, Nunhead, London, 1th Apiil zccs *
Tuisday Club, Goldsmiths, London, th June zccs *
Public Life, London, 1sth July zccs *
Seciet Gaiden Paity, Cambiidgeshiie, zeth July zccs *
1c
CoN1iN1s
lmmeision, llea Pit, London, th August zccs *
Placaid Headphone lestival, Cafe Oto, zcth Septembei zccs *
NoiseNoise, Goldsmiths, z1st Januaiy zccv *
Woim, Roueidam, vth Apiil zccv *
STRP, lindhoven, 1cth Apiil zccv *
pubcode1, Roebuck, zvth May zccv *
pubcodez, Roebuck, th August zccv *
spacecode, Plymouth lmmeisive Vision Teatie, zcth August zccv *
Shunt Lounge, London, 1st Octobei zccv *
Tiansfei, Goldsmiths 1eth Octobei zccv *
Lambda lestival, Antweip, 1st May zc1c *
Peifoimance with Scou Hewiu, Access Space, Sheeld, zvth Octobei zc1c
With Mauhew Yee-King, Jamie loith and Scou Hewiu, lACT Liveipool, Iid Septembei
zc1c
Piksel lestival, Beigen, 1vth Novembei zc1c *
Peifoimance with Nick Collins, Tuisday Club, London, 1th Decembei zc1c
Peifoimance with Nick Collins, LAB, Nouingham, 1th Decembei zc1c
VlX, Poitland Woiks, Sheeld, 11th June zc11
Placaid Headphone lestival, Access Space, Sheeld, 1eth July zc11
Peifoimance with Samuel lieeman, Doikcamp, Wateimillock, zcth August zc11
Noise Box, Yoik, zIid Septembei zc11
Sony CSL Paiis, Icth Septembei zc11 *
Refeieed Publications
loith, J., McLean, A., and Wiggins, G. (zccs). Musical cieativity on the conceptual level. ln
Proceedings of International Joint Workshop on Computational Creativity 2008.
loith, J., Wiggins, G., and McLean, A. (zc1c). Unifying conceptual spaces Concept foimation
in musical cieative systems. Minds and Maines, zc(1)cIIz.
McLean, A., Giiths, D., Collins, N., and Wiggins, G. (zc1c). Visualisation of live code. ln
Proceedings of Electronic Visualisation and the Arts London 2010.
McLean, A. and Wiggins, G. (zccs). Vocable synthesis. ln Proceedings of International Computer
Music Conference 2008.
11
CoN1iN1s
McLean, A. and Wiggins, G. (zccv). Woids, movement and timbie. ln Proceedings of New
Interfaces for Musical Expression 2009.
McLean, A. and Wiggins, G. (zc1ca). Biicolage piogiamming in the cieative aits. ln 22nd
Psyology of Programming Interest Group 2010.
McLean, A. and Wiggins, G. (zc1cb). Live coding towaids computational cieativity. ln Proceed-
ings of the 1st International Conference on Computational Creativity 2010.
McLean, A. and Wiggins, G. (zc1cc). Petiol Reactive pauein language foi impiovised music.
ln Proceedings of the International Computer Music Conference 2010.
McLean, A. and Wiggins, G. (zc1cd). Tidal - pauein language foi the live coding of music. ln
Proceedings of the 7th Sound and Music Computing conference 2010.
McLean, A. and Wiggins, G. (zc11). Textuie Visual notation foi the live coding of pauein. ln
Proceedings of the International Computer Music Conference 2011.
McLean, A. and Wiggins, G. (in piess). Computei piogiamming in the cieative aits. ln McCoi-
mack, J. and dlnveino, M., editois, Computers and Creativity. Spiingei.
Stowell, D. and McLean, A. (zc11). Live music-making a iich open task iequiies a iich open
inteiface. ln Proceedings of BCS HCI 2011 Workshop - When Words Fail: What can Music
Interaction tell us about HCI?
1z
Cu~v1iv 1
Introduction
Te histoiy of computation is embedded in the histoiy of humankind. Computation did not
aiiive with the machine, it is something that humans do. We did not invent computeis, we
invented machines to help us compute. lndeed, befoie the aiiival of mechanical computeis,
computer was a job title foi a human employed to caiiy out calculations. ln piinciple, these
woikeis could compute anything that modein digital computeis can, given enough pencils,
papei and time.
Te textile industiy saw the ist piogiammable machine to ieach wide use the head of
the Jacquaid loom, a technology still used today. Long stiips of caid aie fed into the Jacquaid
head, which ieads paueins punched into the caid to guide intiicate paueining of weaves. Te
Jacquaid head does not itself compute, but was much admiied by Chailes Babbage, inspii-
ing woik on his mechanical analytical engine (lssingei, zcc1), the ist conception of a pio-
giammable univeisal computei. Although Babbage did not succeed in building the analytical
engine, his design includes a similai caid input mechanism to the Jacquaid head, but with
punched paueins desciibing abstiact calculations iathei than textile weaves. While the in-
dustiial ievolution had tioubling consequences, it is somewhat comfoiting to note this shaied
heiitage of computei souice code and cloth, which contempoiaiy aitists still ieect upon in
theii woik (Caipentei and Lacceui, zcce).
Tis eaily computei technology was latei met with theoietical woik in mathematics, such
as Chuichs lambda calculus (Chuich, 1v11) and the Tuiing machine (Tuiing, 1vvz, oiig. 1v1),
which seeded the new eld of computei science. Computei piogiammeis may be exposed to
these theoietical ioots thiough theii education, having gieat impact on theii ciaf. As it is
now piactised howevei, computei piogiamming is fai fiom a puie discipline, with inuences
including linguistics, engineeiing and aichitectuie, as well as mathematics. Digital computeis
now undeipin the opeiation of business, militaiy, academic and goveining institutions, with
impacts acioss human activity. Tese diveise backgiounds biing dieient appioaches to pio-
1I
Cu~v1iv 1 lN1vobUc1ioN
giamming, a gieat challenge foi computei piogiamming education.
As abstiact machines, computeis aie multi-puipose, and aie used in many ways towaids
many dieient ends. Judging by the contents of newstand magazines dedicated to them, the
computei aits aie most ofen fiamed as the use of sofwaie applications as design tools. Heie
sofwaie is pioduced by sofwaie houses, and bought and used by cieative piofessionals. Tis
situation has its meiits, but is a diveision fiom oui theme we aie inteiested in aitists who
wiite piogiams, not in those who only use piogiams wiiuen by otheis. Neithei aie we gieatly
conceined with the notion of computei piogiams as autonomous cieative agents, although
we will touch on this within bioadei discussion of piogiammei cieativity (ch. e). lnstead we
aie inteiested in the piactice of aitists who get diiectly involved with computei languages as
enviionments in which to cieate. Tey aie end-usei piogiammeis, in that they cieate sofwaie
not foi otheis to use as tools, but as a means to iealise theii own woik. We iefei to such people
as artist-programmers.
1.1 Aitist-Piogiammeis
Te use of the teim artist-programmer could be seen as ovei defensive. Alone, the woid pro-
grammer is ofen used to imply a technician, tending a computing machine, oi iealising a de-
signeis dieam. We could be moie asseitive, and use the woid programmer to establish a similai
context eect to that which the woid painter enjoys in the ne aits. But foi the piesent thesis
we keep the aits context explicit, while confionting the singulai identity of the piogiammei as
aitist.
1.1.1 Computer art
We situate aitist-piogiammmeis within the computer arts, and so inheiit impoitant context
fiomthis eld. We will ist examine the iole of industiial and militaiy institutions in computei
aits, befoie moving on to the issue of authoiship and autonomy in the following section.
ln Gieat Biitain, computei ait became established following the Cybernetic Serendipity ex-
hibition shown in the lnstitute of Contempoiaiy Aits, cuiated by Jasia Reichaidt in 1ves. On the
whole this exhibition was well ieceived, both in teims of ieviews and the numbei and diveisity
of visitois. Howevei as Usselmann (zccI) notes, while the exhibition was successful in biinging
some of the possibilities of computei ait to public consciousness, it was signicantly compio-
mised. Despite the tuibulence of the late sixties, theie was no political dimension appaient
in the exhibition, which Usselmann auiibutes to the inclusion of sponsoiing coipoiations in
the exhibition itself. He aigues that this exhibition cast computei ait into a foim which latei
11
Cu~v1iv 1 lN1vobUc1ioN
pioved to be well suited foi inteiactive museum exhibits, but has contiibuted liule to ciitical
debate aiound technology. As Usselmann notes, visitois weie compelled by the lCA to lose
theii feai of computeis, wheieas dissenting voices would advise otheiwise, even then.
Te model desciibed by Usselmann peisists fai beyond Cybeinetic Seiendipity, foi exam-
ple the Decode exhibition at the Victoiia and Albeit museum in zccv was sponsoied by SAP
AG, who commissioned the Bit.Code aitwoik gieeting visitois to the exhibition. ln theii piess
ielease, SAP noted that Bit.Code is themed aiound the concept of claiity, which also ieects
SAPs focus on tianspaiency of data in business, and of how people piocess and use digital
infoimation. ln a 1c yeai echo of Cybeinetic Seiendipity, the aitwoiks weie used to piomote
humanising aspects of technology, with the aitists tacitly taking on the political stance of theii
sponsoi.
Despite such compiomises in the public piesentation of computei ait, theie have always
been computei aitists who engage closely with the sociopolitical context aiound theii woik.
A few yeais afei Cybeinetic Seiendipity, Nake (1v1) published his essay Teie should be no
computei ait in iesponse to the political compiomises he then saw as implicit in computei
ait. Nake also took aim beyond computei ait, giving a lefist peispective deciying the widei
model of ait dealei and ait galleiy, wheie ait is sold foi the aesthetic pleasuie of the iuling
elite. Taking the peispective of Usselmann (zccI) alone, we might considei computei ait as
compiomised, but Nake suggests that it is the whole ait woild that is compiomised, and that
the new computational media should establish alteinative piactices.
ln a second 1c yeai echo, this time of Nakes essay, Olivei et al. (zc11) focus on the social
iathei than aitistic iole of the critical engineer, quoting fiom theii manifesto
Te Ciitical lngineei notes that wiiuen code expands into social and psycho-
logical iealms, iegulating behavioui between people and the machines they intei-
act with. By undeistanding this, the Ciitical lngineei seeks to ieconstiuct usei-
constiaints and social action thiough means of digital excavation.
Olivei et al. (zc11) aie established aitists, and exemplify the stiong theme of activism
piesent in contempoiaiy digital aits. Tey highlight the unique oppoitunity foi engaging
with political themes wheie the boundaiy between digital aits and activism is bluiied. Tis
boundaiy has been exploied by aitists fiom the outset, with theii woik well chaited by Neuial
magazine (http://neural.it).
Peihaps this dichotomy between politically engaged and disinteiested ait could ieect two
distinct views of the ielationship between piogiammei and sofwaie. Tis biings us to the
subject of authoiship in computei ait.
1
Cu~v1iv 1 lN1vobUc1ioN
1.1.2 Generative vs Soware art
Te dieient appioaches to digital ait desciibed above can be undeistood in teims of a di-
chotomy diawn between generative art and soware art. Ains (zcc1) desciibes geneiative ait
as woik which appioaches use of technology as a black box, with focus on end iesults. ln con-
tiast, she desciibes sofwaie ait as focusing on technology and technological cultuie, wheie
the sofwaie itself holds meaning.
Some aigue that compaiing geneiative and sofwaie ait in this way is a categoiy eiioi. Te
most commonly iefeienced denition of geneiative ait, and the one addiessed by Ains (zcc1),
is piovided by Philip Galantei
Geneiative ait iefeis to any ait piactice wheie the aitist uses a system, such as a
set of natuial language iules, a computei piogiam, a machine, oi othei pioceduial
invention, which is set into motion with some degiee of autonomy contiibuting to
oi iesulting in a completed woik of ait. (Galantei, zccI, p.1)
Galantei (zccI) states that his denition of geneiative ait does not say anything about
why an aitwoik might be made oi anything about its content, as a denition of sofwaie ait
might. lt is instead conceined only with a high level aspect of how ait is made. He aigues
that the questioning theme of sofwaie ait iuns oithogonal to the denition of geneiative ait,
and theiefoie that the two do not beai compaiison. Howevei, cential to Galanteis denition
is the issue of autonomy Geneiative ait must be well dened and self-contained enough to
opeiate autonomously. (Galantei, zccI, p.1). Contiaiy to Galanteis asseition, the iequiiement
of autonomy oeis a stiong constiaint to the why of geneiative ait, its opeiation must be
consideied sepaiable fiom the piogiammei, who is in the business of cieating activity exteinal
to theii own inuence. Sofwaie ait by contiast admits views of sofwaie as an extension of
the human, wheie the computei piovides language which allows human expiession to ieach
fuithei.
Accoiding to Galanteis denition, we view the geneiative aitist as emphasising distance
between themselves and theii woik, and the sofwaie aitist as embedding the activity of theii
sofwaie in theii own actions. We position the aitist-piogiammei towaids the lauei, diiectly
engaging and inteiacting with theii code as an integial aspect of theii woik. ln the case of
geneiative ait, authoiship becomes a fundamental question, who oi what pioduces the ait is
it the piogiammei oi theii autonomous piocess` ln the case of sofwaie ait, this question need
not aiise, as the piocess may just be viewed as pait of the human aitists activity.
liom the above we can see evidence of a divide thioughout the histoiy of computei ait,
between focus on aesthetic output, and focus on the piocesses of sofwaie, including its iole in
1e
Cu~v1iv 1 lN1vobUc1ioN
society. Tese aie not mutually exclusive, and indeed Nake advocates both in balance. How-
evei, theie is a tendency foi computei aitists and theii audiences to focus on piocesses as dis-
embodied, autonomous activity (1.1). By exposing the activity of piogiamming, peihaps we
can adjust the balance towaids focus on human inteiaction iathei than autonomous piocesses.
1.1.3 Discussion
ln the following chapteis, sociopolitical context and ciitical fiamewoiks aiound computei ait
aie not oui focus. lnstead we tuin oui auention inwaid towaids the intimate ielationship
between aitist-piogiammeis and theii systems of language, undeistanding piogiamming as
a human inteiaction. Te aitist is consideied foi theii iole as a visionaiy, in exploiing and
extending theii human expeiience. We considei technological suppoit in teims of extending
theii vision thiough piocesses of peiception and ieection in biicolage cieativity. ln paiticulai,
we expand upon the iemaikable notion of piogiamming language, to considei the iole of these
languages in the activity of cieative ait.
1.z Piogiamming Languages foi the Aits
Te tools that aitist-piogiammeis use to make theii woik aie foimal, aiticial languages.
Tese languages aie aiticial in that they aie constiucted by individuals, iathei than emeiging
fiom a widei cultuial piocess as with natuial languages. Te woid articial is pioblematic in
implying fakeiy, but as we will aigue in 1.1, computei languages aie only aiticial in the sense
that a desk fan pioduces aiticial wind the aii still moves.
Teie aie many thousands of piogiamming languages, but they tend to fall within a small
numbei of functional, stiuctuial and object oiiented piogiamming paiadigms laigely devel-
oped between the 1vcs and 1vcs. Te most widely used piogiamming languages
1
acioss
institutions include Java, C, Basic and Python. Tese aie all general purpose languages, with a
coie denition abstiact fiom any paiticulai task, albeit with add-on libiaiies which may taiget
a paiticulai pioblem domain. ln using geneial puipose piogiamming languages, aitists must
build theii woik using technology ostensibly designed foi the geneial case, but in piactice de-
signed foi the expiession of disciete, logical stiuctuies, in an abstiact medium ungiounded in
human expeiience.
ln the piesent thesis we question the extent to which contempoiaiy, geneial puipose pio-
giamming language enviionments aie suitable foi Aitist-Piogiammeis. We will aigue foi new
appioaches to the design of piogiamming languages foi the aits, with human cognition and
1
An index of piogiamming language populaiity is maintained at http://www.tiobe.com/.
1
Cu~v1iv 1 lN1vobUc1ioN
peiception as piimaiy motivating factois. ln paiticulai, language enviionments that ielate both
disciete and continuous iepiesentations, that considei visuospatial and tempoial expeiience in
the design of theii notations, and that suppoit sofwaie development as cieative activity.
1.I Aims
Te woik desciibed by the piesent thesis is conducted towaids two piimaiy aims.
To chaiacteiise human-computei inteiaction as a means foi the impiovisation of music
and ait, fiom the viewpoint of cognitive psychology, with stiong focus on the human
iole in computei piogiamming.
To develop and demonstiate the potential of this theoietical undeistanding to eniich aits
piactice, thiough fiee/open souice sofwaie applications and piogiamming languages,
installation ait, live coding peifoimances and woikshops.
1.1 Stiuctuie
Te following ve chapteis piovide the coie of the piesent thesis, wheie each successive chap-
tei builds upon the chaptei befoie, each time bioadening in scope. liistly Symbols (ch. z)
will piovide a iepiesentational basis foi the thesis, exploiing units of iepiesentation and theii
iole in the pioduction and expeiience of computei ait. Te following chaptei Words (ch. I)
consideis the composition of symbols into woids, and the expiessive iole of woids in speech,
music and computei piogiams. Next we considei the composition of woids into the stiuctuies
of Language (ch. 1), in paiticulai the expiession of pauein and meaning in natuial languages,
computei languages, and in music. Zooming out once moie we view Notation (ch. ), looking
at the peiceptual and tempoial piacticalities of howpiogiammeis may wiite piogiams. linally
oui discussion will take in the widei context of Creativity (ch. e), the iole of piogiamming in a
cieative exploiation, with paiticulai focus on musical impiovisation.
Oveiall, the focus will be not on digital iepiesentations within computation, and not on
analogue expiession eithei, but on the inteiactions between the two. Computeis give us piiv-
ileged access to the digital iealm, but we must not lose sight of the analogue, because humans
expeiience and inteiact with the woild as an amalgam of both.
Tis jouiney will be led by piactice-based ieseaich, and ieseaich-based piactice, in mutual
suppoit. lach of the following ve chapteis will intioduce woiks infoiming and infoimed by
the thesis, as listed in the above pieface.
1s
Cu~v1iv 1 lN1vobUc1ioN
1. Oiiginal contiibutions to knowledge
Te key contiibutions of this thesis aie
1. Te development of vocable synthesis, a system foi the teise desciiption of a paiticulai
conception of timbie thiough the aiticulation of phonetic symbols infoimed by music
tiadition (I.).
z. Te iepiesentation of music paueins in live music peifoimance, as functions of events
ovei time, demonstiated by Tidal (1.).
I. An appioach to visuospatial syntax in puie functional language, based on ielative dis-
tance and type-compatibility, demonstiated by Textuie (.e).
1. A chaiacteiisation of the cieative piocesses of the aitist-piogiammei in ielation to es-
tablished theoietical fiamewoiks (e.I e.).
1v
Cu~v1iv z
Symbols
Oui discussion begins in eainest with symbols, staiting with disciete units of digital iepiesen-
tation. loi oui puiposes, a symbol is simply something that is used to iepiesent, oi signify,
something else. At theii most basic level, computeis use only two disciete symbols, 1 and
0, which in combination iepiesent othei symbols, such as the leueis of an alphabet. loi the
piesent discussion, theie is not a gieat deal to say about disciete symbols in isolation. How-
evei we aie not only inteiested in digital computeis, but also the aitists who woik with them.
Accoidingly oui focus thiough this chaptei will be on compaiing the inteinal symbol systems
of humans with those of computeis, looking foi coiiespondences and dieiences upon which
to build the highei oidei conceins of latei chapteis.
While disciete computational iepiesentations aie fully obseivable and compaiatively well
undeistood, neuioscience is some way fiom pioviding a cleai pictuie of the iepiesentations
undeilying human thought. Te peivasive cognitivist appioach auempts to addiess this by
postulating a cential iole foi disciete computation in human cognition, likening the logical
opeiation of electionic haidwaie to that of the caibon wetwaie of oui biains. We will build
an alteinative view based upon Dual Code theoiy, which admits a iole of disciete symbols in
cognition, but places gieatei emphasis on analogue symbols. Tis will piovide a base fiom
which we latei considei the iole of human and computei symbol systems in the piactice of
computei piogiamming.
z.1 Situating symbols
ln taking a human oiiented view of computei piogiamming, we seek to undeistand the iela-
tionship between digital symbols and the analogue woild in which they aie notated. We have
alieady noted that a computer was once a job title (1.1), since then howevei, computation has
laigely been mechanised, and peihaps to some extent dehumanised. Te job of computei is
zc
Cu~v1iv z Sv:nois
now taken by electionic, digital machines, biinging incieased speed and accuiacy by seveial
oideis. Howevei the human has not been entiiely factoied out of the piocess, on the contiaiy
the iole of wiiting piogiam(me)s foi computeis to follow has developed into a iathei unique
piofession, wheie piogiammeis ofen woik in laige teams wiiting huge tiacts of code den-
ing the undeilying logic of modein institutions. Computei languages have been designed foi
paisimony by human piogiammeis to suppoit this woik, allowing new appioaches to human
aitistic expiession as we will see in chaptei . Howevei, the undeilying functional iepiesenta-
tion, of opeiations ovei sequences of disciete symbols, iemains the same as when computation
was a wholly human task.
ln Babbages Dieience lngine (1.1), 1s and cs aie iepiesented by the punched/not
punched states of papei caids. ln modein computeis, these binaiy states aie piovided by the
binaiy on/o (oi high/low) states of analogue electionic components. All disciete data may
be iepiesented within binaiy states, foi example the tiue/false values of Boolean logic, oi the
ones and zeios of binaiy (base two) numbeis. Such numbeis may iepiesent the instiuction
set of the computei piocessoi, allowing a symbolic sequence to be inteipieted as a sequence
of opeiations ovei itself. Tis is the fundamental view of a computei piogiam given to us by
Tuiing (1vvz, oiig. 1v1s) a sequence of symbols, iead by a machine, which inteipiets them as
opeiations ovei that same sequence of symbols.
A disciete symbol system may iepiesent aspects of the continuous woild using a piocess
called analogue to digital conversion. loi example, sampling and stoiing sound input fiom a
miciophone is a piocess of aveiaging sound piessuie ovei a given sample peiiod, and appiox-
imating the aveiage as a disciete numbei. Te conveision cannot be peifect, as its accuiacy
depends on the gianulaiity of the sample peiiod and sample value. loi example a peiiod of
one sample pei 111ccth of a second, with a iange of 1e bits (eIe possible values) is used on
standaid Compact Disc iecoidings, and is taken to be in the same oidei as the distinguishing
limits of human peiception. We expeiience such a iecoiding thiough inveise digital to analogue
conversion, by sending the sampled data as stepped pulses of electiicity to an electiomagnet,
moving the membiane in a loudspeakei to push sound piessuie waves acioss the ioom, to be
felt out by oui eais.
Having digitally iepiesented a sound signal, we may wish to peifoim some computation
ovei it. ln oidei to do so, some aspect of the continuous woild it was sampled fiom is ofen
modelled oi simulated. loi example to stoie digital sound eciently, psychoacoustic models
aie ofen used, so that infoimation beyond the ability of heaiing is discaided. Likewise, to
apply a reverb eect, a physical ioom may be modelled. lt is apt to extend this concept of
simulation to the function of computei haidwaie itself. lf sofwaie can simulate a continuous
z1
Cu~v1iv z Sv:nois
analogue digital
ieal integei
continuous disciete
image language
imagen logogen
smooth stiiated
amoiphous pulsating
neumatic stiuctuial
plane giid
aiticulation sequence
nonmetiic metiic
modal amodal
giounded ungiounded
specic geneial
Table 2.1: analogue and digital - analogous antonyms.
domain in a disciete domain, then the job of haidwaie is to simulate a disciete domain in
the continuous domain of analogue electionics. We constiuct computeis to simulate a digital
woild, within which we may then constiuct a simulation of an analogue woild. Tis simulation
within simulation takes a iecuisive, fiactal foim, which may continue to aibitiaiy depth.
Te pauein of analogue and digital iepiesentations suppoiting one anothei also iuns acioss
human expeiience. By way of illustiation, table z.1 shows paiis of ielated antonyms used acioss
the aits and sciences. Te antonyms smooth and striated biing to mind the smoothness of a
pebble and the stiiated lines acioss weatheied iock, one peiceived as a continuous textuie
and the othei as a seiies of disciete boundaiies. But a stone can be simultaneously smooth
and stiiated, peihaps maiked by layeis of limestone but washed smooth ovei millennia. Te
inteidependence of the stiiated and smooth is desciibed by Deleuze and Guauaii (1vs, p. 1sc)
as the ielation between points and lines, in the case of the stiiated, the line is between two
points, while in the smooth, the point is between two lines. Tis is a necessaiily ciiculai
denition, as we undeistand one in ielation to anothei.
Te distinction and ielationship between the analogue and digital caiiies thiough to oui
peiception of time, including within peiception of music. Boulez (1vvc) ielates stiiated time
with pulsating, iegulai ihythm, and smooth time with an amorphous, iiiegulai ow. ln dis-
cussing the notation of music, he contiasts neumatic and structural iepiesentations, wheie
neumes aie analogue lines, in contiast to the disciete stiuctuies of sta notation. Boulez tends
towaids the use of disciete symbols in notation, consideiing them moie geneial and accuiate,
but of couise disciete symbols aie only moie accuiate if you wish to notate a pulsating iathei
zz
Cu~v1iv z Sv:nois
than amoiphous tempoial stiuctuie in the ist place.
1
Te Westein Classical tiadition focuses
on the foimei, but amoiphous time is nonetheless piesent as an impoitant aspect of dynamic
peifoimance, even when not notated. Teie aie howevei stiategies foi notating amoiphous
time in disciete computei language, exemplied by the constiaint-based time seuing used in
the Bol Piocessoi language and inspiied by lndian Classical music (Bel, zcc1).
Te distinctions in table z.1 between articulation and sequence, and modal and amodal aie
dieient aspects of the distinction between grounded and ungrounded. Oui enviionment, and
oui bodies moving thiough it, aie by natuie analogue, but oui expeiience of oui enviionment
is both analogue and disciete. We may peiceive a movement as a smooth aiticulation, while
simultaneously abstiacting it into a disciete sequence of events, by segmenting it at peiceived
points of discontinuity. Wheie we abstiact it, we evoke a iepiesentation that is to an extent
amodal, fieed fiom the qualities of a paiticulai mode oi sense. Howevei ungiounded is pei-
haps too stiong a teim, just because we name a hue as red, it does not mean that we have lost
all connection with peiception, we have simply gone fiom a paiticulai expeiience in coloui
space, to a disciete value that symbolises a iegion of possible expeiiences in coloui space. We
will expand on peiceptual and conceptual spaces latei in this chaptei (z.z.).
ln summaiy then, disciete and analogue domains aie distinct, but hosted within one an-
othei. Tey also inteiconnect, suppoiting and eniiching one anothei. Tis is of gieat impoi-
tance to the piogiamming inteiface between humans and computeis, piofoundly so when we
considei how the analogue/digital inteiconnection extends into the mind of the human pio-
giammei.
z.z Symbols in cognition
liom a computei science peispective, digital symbol systems aie shown to be iemaikably ele-
gant and with enoimous piactical use, so it is unsuipiising that they aie ofen used as a desciip-
tive model foi the basis of complex phenomena, such as in biology and the cognitive sciences.
Wolfiam (zccz) shows that it is suipiisingly likely foi geneial computation to emeige fiom
what would otheiwise seem to be tiivial iule systems. ln paiticulai, his exploiation of one
dimensional cellulai automata led to the discoveiy of Rule 110. Tis iule is one of the family
wheie a cells state is based only on its pieceding state and that of its two immediate neigh-
bouis. lt has been pioved that despite this simplicity, Rule 11c is a univeisal computei in that
it can caiiy out all the opeiations of the Tuiing machine, and theiefoie any othei digital com-
putei (Tuiing, 1vvz, oiig. 1v1s). Wolfiam takes his obseivation that computation falls out of
1
Accuiacy need not be the goal when notating amoiphous stiuctuies howevei. Te use of disciete symbols to
denote classes of amoiphous stiuctuie is of couise possible, and is ofen the most satisfactoiy choice.
zI
Cu~v1iv z Sv:nois
such tiivial inteiactions as evidence that the woikings of natuie aie computational. We aie in-
teiested in a moie specic hypothesis than Wolfiam, aie digital symbols the iepiesentational
basis of cognition` As computation aiises fiom the simplest of iule sets, we ceitainly cannot
discount it on the basis of Occams iazoi.
Gieat woiks on the Language of Tought by lodoi (1vsc) and Pylyshyn (zcc) do indeed
point towaids a disciete iepiesentational basis foi cognition. Cognition is desciibed by these
thinkeis as the subconscious woikings within an innate language ofen dubbed mentalese. Tis
is not a natuial language as we speak it, but an inteinal, univeisal language stiuctuiing subcon-
scious thought. Tis piesumed language is iepiesented using disciete symbols, with cognition
chaiacteiised as a piocess of computational opeiations ovei those symbols. Such computational
accounts have stiong tiaction acioss the cognitive sciences, and also suppoit undeistanding in
computei science, we undeistand the syntax of computei languages with iefeience to Chom-
skian tiansfoimational giammai (1.1).
We ietuin howevei to the notion of digital suppoited by analogue, and analogue by digital.
To focus solely on disciete computation is in denial of its essential inteiplay with analogue
movement and shape. ln compaiison with humans, digital computeis appeai to lead a iathei
impoveiished existence in teims of engagement with theii analogue enviionment. Could this
dieience be due to dieiing symbolic foundations` Teie is a line of thought which allows us
to admit computational accounts of the mind, but in addition considei a second, complementaiy
system of iepiesentation. Tis second system is mental imagery, an analogue symbol system
giounded in peiception.
2.2.1 Mental imagery
When piesented with ceitain kinds of pioblems, a quasi-peiceptual mental image may be con-
sciously manipulated to solve a task. We can examine these subjective expeiiences thiough
objective expeiiment, foi example Shepaid and Metzlei (1v1) identied that when matching
iotated objects, subject ieaction times have a stiong, lineai coiielation with ielative degiee
of iotation. Tis suggests that imageiy is being iotated in the mind. lt would appeai that
these quasi-peiceptual states aie an analogue system of symbolic iepiesentation, used heie in
cognitive pioblem solving.
Te image in mental imageiy is not specic to vision, but a bioad teim ielated to any
quasi-peiceptual state. loi example the use of piosodic intonation in speech is paralinguistic,
not entiiely notated in wiiuen text, yet symbolising meaningful content. lndeed the meaning
of a spoken text can be twisted, peihaps negated with saicasm, thiough subtle paialinguistic
phiasing. luitheimoie, intonation is not just a concein in communication with otheis, the
z1
Cu~v1iv z Sv:nois
phenomenon of inner spee while ieading silently is undeistood to be a function suppoiting
woiking memoiy, pioviding analogue, piosodic cues useful in compiehending text (Raynei and
Pollatsek, 1vv1, p. z1e).
Mental images then, aie analogue symbols undeilying visuospatial cognition. An image
symbolises an object, using piopeities mapped diiectly fiom peiception. loi mental iotation
tasks, the symbol may itself be iotated, as it shaies essential geometiic featuies with the object it
symbolises. Some highei oidei tasks, such as the analysis of complex symmetiies, may howevei
be beyond the capabilities of mental imageiy. loi such tasks we may abstiact piopeities of
imageiy to cieate a foimal language, by which we mean a system of disciete symbols goveined
by giammatical iules. ln the case of mental iotation, we identify group theory as a linguistic
counteipait, developed to gain mathematical undeistanding of symmetiies (du Sautoy, zccs).
Gioup theoiy allows us to extend oui undeistanding of symmetiy with disciete logic, but does
so in ielation to the mental images we expeiience.
lmpiiical undeistanding of the psychology of analogue and disciete symbols is piovided
by Paivio (1vvc) thiough his Dual Coding theoiy. His contention is not that theie aie two codes,
but iathei that theie is a hieiaichy of codes, which bianch at the top into disciete linguistic
codes and continuous peiceptual codes, which Paivio names logogens and imagens iespectively.
Tis split is shown in theii concuiient piocessing, humans aie able to compiehend language
while simultaneously auending to imageiy. Continuing with oui eailiei example of phiasing
in speech, humans nd it easy to simultaneously piocess and integiate piosodic and linguistic
infoimation, but iathei moie dicult to simultaneously iead text and listen to speech. Te
explanation oeied by Dual Coding theoiy is that theie aie distinct, yet integiated symbol
systems foi imageiy and language. Tis theoiy sits well against the backgiound of digital and
analogue inteidependence we noted in z.1.
Neuiopsychology piovides suppoit foi Dual Coding thiough ieseaich into the fundamental
biain stiuctuie of the two distinct hemispheies. Veiy bioadly speaking, the lef hemispheie is
specialised foi language, and the iight foi visuospatial tasks (Maitin, zcce, pp. 1zs-1zv). We
must be caieful howevei not to ovei-simplify this ielationship the moie closely it is examined,
the less cleai it gets. loi example, fiom a meta-analysis of ceiebial lateialisation of spatial
abilities (Vogel, zccI) we see theie is a stiong sex dieience in lateialisation of spatial tasks,
wheie female subjects tend to showno hemispheie dominance and males tend to exhibit stiong
iight hemispheie dominance. Te same meta-analysis nds subjects classed as high imageis
also show no hemispheie dominance in visuospatial tasks, suggesting that high imageis aie
not those with a dominant visual iight hemispheie, but iathei with high integiation between
both hemispheies. Tis lauei nding has statistical signicance, but is based on few studies
z
Cu~v1iv z Sv:nois
and so cannot be consideied iobust. Howevei while it stands, it suppoits the view that the
usefulness of mental imageiy comes fiom integiation between disciete and analogue codes.
2.2.2 Mental Imagery and Programming
Befoie we go any fuithei, we should addiess Dual Coding theoiy to oui oveiall theme. lt
may seem that computei piogiamming is an oveiwhelmingly linguistic task in a wholly dis-
ciete domain, but if we focus on the piogiammei iathei than the computei, we nd this is
not the case. Piogiammeis may deal with foimal, disciete and textual language, but they sup-
poit and stiuctuie theii woik in a vaiiety of ways exteinal to computei language itself. loi
example diagiammatic iepiesentations of code stiuctuie aie widespiead in the teaching and
piactice of sofwaie development, such as those standaidised in the Stiuctuied Systems Analy-
sis and Design Method (SSADM) and Unied Modelling Language (UML). Tese aie highly
foimalised languages, but use spatial aiiangement and connecting lines to piesent the stiuc-
tuie of a piogiam in a visual mannei. luithei, the stiuctuie of piogiams is ofen intioduced in
piogiamming textbooks using visual metaphoi, foi example as inteiconnected ioads (Giiths
and Baiiy, zccv, pp. 1Iz1) . We asseit then that despite the disciete natuie of computation,
piogiammeis ofen use mental imageiy to suppoit theii woik.
Petie and Blackwell (1vvv) look foi, and nd iepoits of mental imageiy duiing piogiamming
tasks. Tey conducted veibal inteiviews with ten expeit piogiammeis, while they weie in the
piocess of solving piogiamming tasks, piompting them to explain what they weie seeing in
visual oi auditoiy teims.
z
We highlight and comment on a few iepoits fiom these inteiviews
no place holdeis, no pictuies, no iepiesentation just the notion, the sym-
bol entities, semantic entities and the linguistic token atomic notions. Tey just
aie (Petie and Blackwell, 1vvv, p. 1)
We contend that this is not a iepoit of mental imageiy as Petie and Blackwell (1vvv) imply.
Tis is imaginative use of language, iathei than modality-specic quasi-peiceptual states.
it moves in my head like dancing symbols l can see the stiings [of
symbols] assemble and tiansfoim, like luminous chaiacteis suspended behind my
eyelids (Petie and Blackwell, 1vvv, p. 11)
Tis is again a iepoit featuiing disciete symbols, but augmented with visual imageiy. Tis
appeais to be simultaneous activation of both language and imageiy, in line with Dual Coding
theoiy.
z
To countei known pioblems with inteiview piotocols, the study was also complemented and to an extent
conimed by a second study, based on an undiiected questionnaiie of zz piogiammeis of the LABView visual
piogiamming enviionment.
ze
Cu~v1iv z Sv:nois
lt buzzes theie aie things l know by the sounds, by the textuies of sound oi
the loudness its like l heai the glitches, oi l heai the bits that aient woiked out
yet (Petie and Blackwell, 1vvv, p. 1)
Tis is an intiiguing example of a iepoited sonic image, again highlighting that mental
imageiy is modality specic, but not necessaiily of the visual sense. lndeed the pievious quote
could be inteipieted as simultaneously having both kinaesthetic and visual featuies. Petie
and Blackwell (1vvv) iepoit that all ten expeits iepoited sound as an element in theii imageiy,
although not in geneial as a typical element. lt is woith noting howevei that subjects weie
specically piobed foi sonic imageiy, which may have inuenced the piogiammeis mental
imageiy and iepoits.
lts like diiving acioss a deseit looking foi a well. What you actually have is
good solutions distiibuted acioss this deseit like small deep wells and youi opti-
mizei tiundling along looking foi them (Petie and Blackwell, 1vvv, p. 1)
Tis image appeais to be visual-specic although could in addition be inteipieted in a ki-
naesthetic sense. Tat the piogiammei iepoits imagining a thiee-dimensional pioblem space
is of ielevance to the theoiy of Conceptual Spaces, which we will examine latei in z.z..
Again, it is wise to be cautious of intiospective iepoits. lntiospection is a subject of ie-
seaich in its own iight, with one theoiy being that intiospections aie nothing othei than ie-
constiucted, diamatised peiceptions (Lyons, 1vse). Nonetheless this should not discouiage us
fiom tieating mental imageiy seiiously as a system foi symbolic iefeience, and these iepoits
give us cautious vantage ovei the cognitive piocesses of computei piogiamming, towaids un-
deistanding how piogiammeis may use mental imageiy to suppoit theii woik.
2.2.3 Dual Coding in Source Code
At base, the souice code foi a computei piogiam is a one dimensional sequence of disciete
symbols, ieady foi inteipietation by a digital computei. How then could this ielate to the
Dual Code in human cognition, when we asseit in the pievious section that piogiammeis
employ mental imageiy in theii woik` ln answei we nd that despite the disciete undeilying
iepiesentation, imageiy is piesent both in the peiception and oiganisation of souice code.
ln the Psychology of Piogiamming eld, a discipline biidging Psychology and Human-
Computei lnteiaction, it is widely undeistood that theie aie notational featuies of souice code
besides foimal disciete syntax, and fuithei that these aspects aie vitally impoitant to human
undeistanding of piogiams. Such featuies aie known as secondary notation, which includes
spatial layout, comments, coloui highlighting and vaiiable names. Secondaiy notation, and
z
Cu~v1iv z Sv:nois
its place within the Cognitive Dimensions of Notations fiamewoik will be fuithei expanded
upon in .1, but foi now we focus upon the use of spatial layout in code, with iespect to
mental imageiy. Considei the following code fiagment wiiuen in the Cpiogiamming language
(Keinighan and Ritchie, 1vss)
if (condition == true) {
display("Welcome.");
beep();
}
All spaces in the above example could be discaided, giving this
if(condition==true){display("Welcome.");beep();}
As fai as a C language compilei is conceined, these code fiagments aie equivalent, as all
whitespace is discaided duiing tokenisation, an eaily stage of computei language paising. loi a
human howevei, the foimei veision of the code is much easiei to compiehend, and foi a much
biggei piogiam, the lauei foim would be close to impossible to undeistand on sight alone.
Te computei sees a one dimensional stiing of symbols, paised into the highei oidei data
stiuctuie of a syntax tiee. Humans howevei aie able to peiceive and navigate the fiagments
and wholes of a code stiuctuie in a mannei analogous to a visual lulei diagiam. By way of
illustiation, oui code becomes even moie dicult to iead if we ieplace chaiacteis to iemove
the visual piompts of containment given by the symmetiy of matching biackets
if$condition==true@display$"Welcome."@;beep$@;&
liomthis we aigue that while both humans and computeis undeistand piogiams as disciete
symbols, humans use secondaiy notation to augment this iepiesentation with visual imageiy.
As such, souice code is an amalgam of two symbol systems, one of which is discaided in the
computational paising piocess.
Piogiamming languages with paiticulaily visual notations aie known as Visual Program-
ming Languages. Piogiamming has a ieputation foi being iathei dicult to teach and leain
(e.g. Listei et al., zcc1), and so a iecuiiing theme in language design is nding foims of notation
that aie moie natuial oi intuitive. A paiticulai hope is that Visual Piogiamming will lead to
languages giasped easily by end useis with liule oi no tiaining. laily enthusiasm has howevei
not led to wide success, on the whole VPL has only taken hold foi ceitain taiget domains, foi
example LABView in engineeiing and Patchei languages (.I.1) in audio/visual digital signal
zs
Cu~v1iv z Sv:nois
piocessing. Te lack of success of geneial puipose Visual Piogiamming suggests that iela-
tionships between disciete symbols and mental imageiy aie by natuie paiticulai to the task at
hand. We should look then not foi ways of adding aiticial visual metaphoi to piogiamming
as has been unsuccessful in HCl in geneial (Blackwell, zccee), but foi ways of suppoiting a
piogiammeis own system of imageiy, integiated with the text of souice code. We will ievisit
and expand upon this point in oui view of the notation of piogiams in .I.
We view souice code then as suppoiting both language and imageiy. Use of disciete sym-
bols expiessed within giammai iules aie linguistic
I
, but aie aiianged to allow the suppoit of
visuospatial cognition in undeistanding and wiiting piogiams.
2.2.4 Language and Situated Simulation
Language and Situated Simulation (LASS) theoiy is a iecent development of Dual Coding the-
oiy intioduced by Baisalou et al. (zccs). LASS adds empiiical suppoit bioadly in agieement
with Dual Coding, but with dieient detail and a stiong change of emphasis. Wheieas Paivio
(1vvc) deals with imagens and logogens even handedly, Baisalou et al. place fai gieatei em-
phasis on imageiy, which they iefei to as the simulation system, ieecting its active iole in
cognition. Accoiding to LASS, the linguistic system is supercial, acting as liule moie than a
contiol mechanism foi the simulation system, which is iesponsible foi deep conceptual stiuc-
tuie giounded in expeiience.
Simulations in LASS aie desciibed as perceptual symbols, but this does not imply conscious
awaieness of theii use. lndeed in a bieak fiom Dual Coding theoiy, Baisalou (1vvv, z.1.1)
asseits that peiceptual symbols aie a neuial iepiesentation with only limited coiielates in
consciousness. Tis pioposal is similai to mentalese, in that it is an inteinal iepiesentation
undeilying subconscious cognition. Te dieience is that peiceptual symbols aie analogue
and highly modal, being giounded in sensoiy-motoi neuial systems, wheieas the notion of
mentalese is as disciete and amodal.
Te piogiammeis iepoits fiom ieseaich ieviewed in z.z.z weie the iesult of piobes foi
imageiy in consciousness while piogiamming, in keeping with the focus on conscious states by
Paivio (1vvc). Howevei we shaie the view of LASS, that analogue iepiesentations extend be-
yond the connes of consciousness and auention, to stiuctuie the bulk of unconscious thought.
Teiefoie if anything, these iepoited expeiiences of mental imageiy aie only the tip of the
icebeig.
zv
Cu~v1iv z Sv:nois
Table 2.2: Levels of representation according to the theory of Conceptual Spaces (G ardenfors, 2000), aligned
with terms from Dual Coding theory (Paivio, 1990)
lxpeiience Gaidenfois Paivio Stiuctuie
Language Symbolic Logogens Disciete symbols
Peiception Conceptual lmagens Low dimensional geometiy
Sensation Sub-conceptual High dimensional neuial nets
2.2.5 Conceptual Spaces
We have discussed peiception with iespect to analogue symbols at length, but how does this
ielate to concepts` ln a majoi ieview of the eld of concept theoiy, Muiphy (zccz, p. ) denes
the teim concept as a mental iepiesentation of a class of things. So concepts allow us to
stiuctuie oui expeiiences thiough geneialisation, but how is this done` Te piedominant view
is that peiception and cognition aie independent functions of the biain, but an alteinative view
holds that functions of peiception and concepts aie integiated, ielying upon shaied neuial
iesouices.
How then could concepts, abstiactions fiom the woild, be stiuctuied in the same way as
peiception` liistly it is impoitant to iecognise that peiception is itself not a stiaightfoiwaid
ieection of the woild, iathei a low-dimensional iepiesentation of high-dimensional sensoiy
input, giving us a somewhat coheient, spatial view of oui enviionment. By spatial, we do not
only mean in teims of physical objects oi visual peiception, but iathei in teims of featuies in
the analogue spaces of all possible tastes, sounds, tactile textuies and so on. Tis scene is built
thiough a piocess of dimension ieduction fiom tens of thousands of chemo-, photo-, mechano-
and theimoieceptoi signals. Gaidenfois (zccc) pioposes that this piocess of dimension ieduc-
tion is behind the constiuction of Conceptual Spaces. Moieovei, he takes this as the piimaiy
model of conceptual iepiesentation, including that of highei-level concepts somewhat abstiact
fiom peiception. Tat is, Conceptual Spaces aie giounded in the cognitive iesouices of mental
imageiy, but aie iepuiposed to iepiesent conceptual ielationships in geometiic spaces.
Te theoiy of Conceptual Spaces has gieat explanatoiy powei in iesolving conict in
the eld of aiticial intelligence (Al) between pioponents of the high dimensional, statisti-
cal giaphs of aiticial neuial netwoiks (ANNs) and the disciete symbols of good old-fashioned
aiticial intelligence (GOlAl). ANNs iepiesent concepts thiough tiained netwoiks of connec-
tions between cells, and GOlAl thiough the disciete constiucts of language. ln othei woids,
ANNs woik in the iealm of sensation, and GOlAl in the iealm of computation. Gaidenfois
seeks to unite these appioaches by identifying a level of iepiesentation that mediates between
I
See 1.1 foi discussion of the ielationship between computei and natuial language.
Ic
Cu~v1iv z Sv:nois
them the low dimensional, geometiic iealm of the conceptual level.
Table z.z illustiates how the thiee levels of Conceptual Space theoiy align with the two
channels of Dual Coding. Note some disciepancies in teiminology, Gaidenfois iefeis to lin-
guistic iepiesentation as symbolic wheieas we use the teim discrete symbolic to distinguish
fiom analogue symbols in Dual Coding theoiy. Note also that Gaidenfois desciibes ANNs
as sub-conceptual in place of the moie geneially used teims non- oi sub-symbolic, to piopeily
situate them beneath the geometiy of the conceptual layei.
Conceptual Space theoiy has some agieement with LASS, in that conceptual iepiesenta-
tion is consideied to be laigely outside the giasp of conscious intiospection. Although these
theoiies use dieient teims and have dieient emphases, they agiee that the piimaiy foim of
conceptual iepiesentation is within spaces, iathei than within the syntax of the disciete sym-
bolic layei. Howevei wheieas LASS focuses on simulations of physical objects, movements
and inteiactions, Conceptual Space theoiy focuses moie on the geometiic stiuctuie of mental
spaces.
Conceptual spaces aie stiuctuied by similarity, concepts that aie closei togethei aie moie
alike. luitheimoie, the dimensions of conceptual spaces aie aligned with paiticulai conceptual
qualities. Tis is most easily explained with the example of coloui space, wheie moie similai
colouis aie closei, within the quality dimensions of hue, chiomaticism and biightness. As
alieady alluded to in z.1, the concept red is not a point in coloui space, but iathei a convex
region, although we may pinpoint a prototypical ied as the centioid, oi peihaps moie holistically
as the Voionoi geneiatoi (Okabe et al., zccc) of its iegion. Te signicance of convexity is cleai
if one consideis that foi any two hues of ied, all the hues along the path between them will
also be ied. Red as a piopeity is a concept in its own iight, but can also foim pait of a moie
complex cioss-domain concept, such as red ball.
Te example of coloui is stiaightfoiwaid, but it is iathei haidei to apply the theoiy to
concepts without cleai coiielates to peiceptual spaces in consciousness. Howevei the theoiy
of Conceptual Spaces makes the bold claim that the same piocesses of dimension ieduction
and spatial iepiesentation aie applied to the majoiity of concepts. Despite the lack of subjec-
tive conscious expeiience, we can hope to objectively identify the dimensions of highei oidei
concepts thiough psychological expeiiment, such as those based on multi-dimensional scal-
ing (MDS). ln classic MDS expeiiments, human subjects aie asked to giade paiis of stimuli by
similaiity, with theii judgements inteipieted as distance measuiements, and then used to ie-
constiuct the conceptual space evoked by the stimuli. We will appiaise MDS appioaches in the
context of musical timbie in I.I.1. We may also constiuct conceptual spaces based on estab-
lished theoiy foi example loith et al. (zc1c) identify dimensions of musical metie based upon
I1
Cu~v1iv z Sv:nois
music theoiy.
Te theoiy of Conceptual Spaces is paiticulaily useful foi oui piesent discussion in piovid-
ing an account of how linguistic and spatial iepiesentations may inteiact in cognition. Rathei
than placing the entiie buiden of highei-oidei iepiesentation on disciete, linguistic stiuctuie,
it instead places emphasis on metaphoi, both foi ielating spaces togethei and foi giounding
highei oidei concepts in embodied, peiceptual spaces. Conceptual domains aie low dimen-
sional, in geneial having fiom one to thiee dimensions, in common with mental imageiy and
human peiception in geneial. Tus, they may be exploied and mapped thiough spatial ieason-
ing, as if they weie physical spaces. Tis includes ielating featuies of two spaces togethei, in
othei woids diawing metaphoiical ielations between conceptual domains.
2.2.6 Metaphor
Metaphoi is ofen consideied to be a foim of poetic woidplay, a window diessing on lan-
guage. ln intioducing theii Conceptual Metaphoi theoiy, Lako and Johnson (1vsc) aigue on
the contiaiy that metaphoi is of cential impoitance to language. Conceptual Metaphoi the-
oiy is a dening contiibution to the eld of cognitive linguistics, and places metaphoi in the
iole of stiuctuiing concepts ielative to one anothei within a coheient system of meaning. loi
example, Lako and Johnson claim that Well, that boosted my spiiits! is a linguistic phiase
stiuctuied by the undeilying conceptual metaphoi u~vvv is Uv
1
. Asingle conceptual metaphoi
may be expiessed thiough many linguistic phiases, foi example l am depressed, l am feeling
down and My spiiits sank aie instantiations of the same u~vvv is Uv metaphoi.
Lako and Johnson (1vsc) divide conceptual metaphois into two types, structural
metaphois such as 1i:i is :oNiv (e.g. He is living on borrowed time) and ~vcU:iN1 is
v~v (e.g. His claims aie indefensible), and orientational metaphois such as coNscioUsNiss
is Uv (e.g. He is under hypnosis) and the afoiementioned u~vvv is Uv. Stiuctuial metaphois
stiuctuie one concept in teims of an othei, while oiientational metaphois stiuctuie a whole
system of concepts ielative to fundamental diiectional and spatial ielationships. Lako and
Johnston asseit that the majoiity of conceptual metaphois aie oiientational, coheient within
a laige system of metaphois (Lako and Johnson, 1vsc, p. 1). Oiientational metaphois aie
oiganised ielative to the body, ofen with physical motivation, foi example u~vvv is Uv ielates
to the eiect postuie of a happy peison veisus diooping postuie of a depiessed peison.
Gaidenfois (zccc) builds the notion of oiientational conceptual metaphois into his the-
oiy of conceptual spaces (z.z.), asseiting, like Lako and Johnston, that they foim the pii-
1
Conceptual metaphois aie by convention denoted with small block capitals, to dieientiate them fiom the
linguistic metaphoiical phiases which iefei to them.
Iz
Cu~v1iv z Sv:nois
maiy stiuctuie of a human conceptual system. Oiientational metaphois giound Conceptual
Metaphoi theoiy, and theiefoie the theoiy of Conceptual Spaces, in human physical and cul-
tuial expeiience. Gidenfois pioposes that these metaphois, and the geometiic spaces they
ielate togethei, aie the basis of semantics. We will ietuin to this subject in discussion of music
and language in 1.z.
z.I Anthiopomoiphism and Metaphoi in Piogiamming
Metaphoi appeais to peimeate oui undeistanding of piogiamming, as evident in the vaiied
iepoits of mental imageiy in piogiamming tasks (z.z.z). Peihaps this is due to the abstiact na-
tuie of computei language syntax, iequiiing metaphoiical constiucts to giound piogiamming
language in eveiyday ieasoning. Blackwell (zcced) used techniques fiom coipus linguistics
on piogiamming language documentation in oidei to investigate the conceptual systems of
piogiammeis, identifying a numbei of conceptual metaphois listed in Table z.I. Rathei than
nding metaphois suppoiting a mechanical, mathematical oi logical appioach as you might
expect, components weie instead desciibed as actois with beliefs and intentions, being social
entities acting as pioxies foi theii developeis.
Blackwell (zcced) classies oiientational metaphois into the metaphois Pvocv~:s oviv
~1i iN ~ sv~1i~i vovib vi1u coN1~iN:iN1 ~Nb ix1iN1, along with the ielated metaphoi
of movement in spaces, lxicU1ioN is ~ JoUvNiv iN so:i i~Nbsc~vi. Tese metaphois aie
iepoited to occui iegulaily, and ieecting on Conceptual Metaphoi theoiy, we would expect
oiientational metaphois to piovide the piimaiy stiuctuie of piogiamming concepts. Peihaps
these metaphois could be bioken down and ielated in a coheient, spatial system of metaphois
such as Ans1v~c1ioN is Uv and Pvocviss is lovv~vb. A pieliminaiy examination of the
coipus indicates that this may be feasible, howevei fuithei woik is iequiied.
lt would seem then that piogiammeis undeistand the stiuctuie and opeiation of theii pio-
giams by metaphoiical ielation to theii expeiience as a human. Te notion of including a com-
putei in a cieative piocess (e.I) is by natuie anthiopomoiphic, by embedding the development
of an algoiithm in a human cieative piocess, the algoiithm itself becomes a human expiession.
Howevei, Dijkstia stiongly opposed such anthiopomoiphic appioaches in computei science
l have now encounteied piogiams wanting things, knowing things, expecting
things, believing things, etc., and each time that gave iise to avoidable confusions.
Te analogy that undeilies this peisonication is so shallow that it is not only
misleading but also paialyzing. (Dijkstia, 1vss, p. zz)
Dijkstias claimis that by focusing on the opeiation of algoiithms, the piogiammei submits
to a combinatoiial explosion of possibilities foi howa piogiammight iun, not eveiy case can be
II
Cu~v1iv z Sv:nois
Co:voNiN1s ~vi ~ciN1s oi ~c1ioN iN ~ c~Us~i UNivivsi.
Pvocv~:s oviv~1i iN uis1ovic~i 1i:i.
Pvocv~: s1~1i c~N ni :i~sUvib iN U~N1i1~1ivi 1iv:s.
Co:voNiN1s ~vi :i:nivs oi ~ socii1v.
Co:voNiN1s ovN ~Nb 1v~bi b~1~.
Co:voNiN1s ~vi sUnJic1 1o iic~i coNs1v~iN1s.
Mi1uob c~iis ~vi sviicu ~c1s.
Co:voNiN1s u~vi co::UNic~1ivi iN1iN1.
A co:voNiN1 u~s niiiiis ~Nb iN1iN1ioNs.
Co:voNiN1s onsivvi ~Nb siix iNiov:~1ioN iN 1ui ixicU1ioN iNvivoN:iN1.
Co:voNiN1s ~vi sUnJic1 1o :ov~i ~Nb ~is1ui1ic JUbci:iN1.
Pvocv~:s oviv~1i iN ~ sv~1i~i vovib vi1u coN1~iN:iN1 ~Nb ix1iN1.
lxicU1ioN is ~ JoUvNiv iN so:i i~Nbsc~vi.
Pvocv~: iocic is ~ vuvsic~i s1vUc1Uvi, vi1u :~1ivi~i vvoviv1iis ~Nb
sUnJic1 1o bic~v.
D~1~ is ~ sUns1~Nci 1u~1 iiovs ~Nb is s1ovib.
TicuNic~i vii~1ioNsuivs ~vi vioiiN1 iNcoUN1ivs.
Pvocv~:s c~N ~U1uov 1ix1s.
Pvocv~:s c~N coNs1vUc1 bisvi~vs.
D~1~ is ~ ciNi1ic, :i1~noiiziNc iiiiiov: vi1u nobv v~v1s.
Soi1v~vi 1~sxs ~Nb niu~vioUv ~vi biiic~1ib nv ~U1o:~1ici1v.
Soi1v~vi ixis1s iN ~ cUi1Uv~i/uis1ovic~i coN1ix1.
Soi1v~vi co:voNiN1s ~vi soci~i vvoxiis iov 1uiiv ~U1uovs.
Table 2.3: Conceptual metaphors derived fromanalysis of Java library documentation by Blawell (2006d).
Program components are described metaphorically as actors with beliefs and intentions, rather than me-
anical imperative or mathematical declarative models.
I1
Cu~v1iv z Sv:nois
coveied, so piogiammei eiiois pievail. He aigues foi a stiict, declaiative appioach to computei
science and piogiamming in geneial. Dijkstia views computei piogiamming as such a iadical
activity that we should not associate it with oui daily existence, oi else limit its development
and pioduce bad sofwaie.
Te alteinative viewpiesented heie is that metaphoi necessaiily stiuctuies oui undeistand-
ing of computation, as it piovides the basic stiuctuie of oui conceptual system (z.z.e). Sof-
waie now peimeates Westein society, and is iequiied to function ieliably accoiding to human
peiception of time and enviionment. Metaphois of sofwaie as human activity aie theiefoie
becoming evei moie ielevant.
z.1 Synaesthesia
Synaesthesia is a condition wheieby modes of peiception aie cross-activated, wheie activa-
tion in one modality stimulates expeiiences in a second modality. A classic example is of the
colouitaste synaesthete, who expeiiences a paiticulai taste whenevei they see a paiticulai
coloui. Such activations aie uni-diiectional a colouitaste synaesthete will geneially not
also be a tastecoloui synaesthete. Synaesthetic expeiience dieis fiom that of conceptual
metaphoi, in that foi a colouitaste synaesthete, ied may literally taste of eaiwax, as opposed
to a tempeiatuie-coloui metaphoi aligning the spatial dimensions of iedness with waimth, by
linguistic iefeience to oiientational metaphoi.
Histoiically synaesthesia has not been taken paiticulaily seiiously by psychologists, dis-
missed as an insignicant function of memoiy, metaphoi oi illicit diug use (Ramachandian
and Hubbaid, zcc1b, p. 1). Ovei the last decade howevei a numbei of expeiiments have con-
imed that synaesthesia is a ieal phenomenon, showing foi example that numbeicoloui
synaesthetes peifoim beuei than contiols at ceitain identication tasks, aided by peiceptual
pop-out piovided by theii condition (Ramachandian and Hubbaid, zcc1a). Tis cleai expeii-
mental suppoit foi synaesthesia as a medical condition emboldened Ramachandian and Hub-
baid (zcc1b) to speculate on a iole foi synaesthesia in the evolution of language, connecting
vocal aiticulations and sounds with concepts, and pioviding a neuial basis foi metaphoi. ln the
same papei, a causal link is claimed foi the high incidence of synaesthesia iepoited in aitists,
wheie synaesthetes aie moie able to make cieative metaphoiical connections. Tese claims
aie tempeied by a ieview by Waid et al. (zccs), showing that evidence suiiounding aitists and
synaesthesia is unieliable, with the additional obseivation that the condition is automatic and
inexible, wheieas cieative metaphoi iequiies diveigent thinking of a qualitatively dieient
natuie. Howevei the automatic cioss-activation of synaesthetes is at least a metaphoi foi the
I
Cu~v1iv z Sv:nois
optional cioss-domain metaphois diawn by aitists, a meta-metaphoi peihaps.
While the woid synaesthesia is geneially ieseived foi abnoimal phenomena, theie aie many
cioss-modal illusionaiy expeiiences that aie expeiienced by the majoiity of the population.
Sound symbolism does not sit well with contempoiaiy linguistic theoiy, but ceitain eects
aie undeniable. ln his tieatise on Gestalt Psychology, Kohlei (1vIc) notes that subjects ieadily
associate spiked and loopy doodles with the nonsense woids takete and maluma iespectively,
an obseivation iepeated in foimal expeiiment by Ramachandian and Hubbaid (zcc1b) with
the woids kiki and bouba. Anothei illusion is wheie a single ash of light, when accompanied
by a double sonic click, is peiceived as a double ash of light, a phenomenon conimed by
fMRl evidence (Zhang and Chen, zcce). A thiid example is the McGuik-McDonald eect (I.1),
which demonstiates the inuence of lip ieading on auditoiy peiception. Tese stiong illusions
piovide giound wheie aitists may play with the senses of theii audience.
z. Aitistic synaesthesia`
ln the aits, synaesthesia is fiequently alluded to wheie a woik ciosses multiple media. Tis is
not intended to be an automatic piocess as with clinical synaesthesia, but the extension of an
aitistic theme acioss modalities, in oidei to cieate a iich expeiience. At times this may seem
a mundane aspiiation diessed in the clothes of psychological disoidei. lt is afei all noimal to
be able to both see and heai an action in peiceptual unity, modes of peiception aie by natuie
integiated. lnteiest comes howevei wheie technology allows modalities to be ielated in a novel
mannei.
Te phiase algorithmic synaesthesia is coined by Dean et al. (zcce) to desciibe aitistic at-
tempts to connect vision and sound (oi moie specically, lm and music) using digital com-
puteis. Te woid algorithm is used iathei loosely, to indicate any use of computeis to connect
modalities, even if this only amounts to shaiing of data between outputs, aitistic license is
applied to both teims of the algoiithmic-synaesthetic junctuie.
Dean et al. (zcce) iepoit seveial inteiesting appioaches to cioss-modal integiation in pei-
foimance. Howevei we tieat with scepticism theii claim that the addition of a computei nec-
essaiily biings mixed media into a new iealm. Tey state that theii new digital algoiithmic
synaesthesia is moie piecise than past analogue eoits, but it is not cleai why, if we aie con-
ceined with ait that plays with analogue peiception, then suiely an analogue appioach would
notionally be moie piecise. lnstead we see computational developments as a continuation of
an evei-piesent tendency foi aitists to look foi new coiiespondences between peiceptual do-
mains, thiough both analogue and digital means.
Ie
Cu~v1iv z Sv:nois
Te use of mixed media in the aits is not new, and technology has been employed foi
centuiies in nding novel ways to connect the senses. loi example the inventoi Maiy Hallock-
Gieenewalt developed a coloui oigan (as in, a keyboaid instiument to play coloui piojections)
in a bid to iealise hei conception of Nourathar, an aitfoim based on uctuating tianspositions
of biightness and hue, with a scoiing systemto unite it with musical sta notation (Gieenewalt,
1v1e). Many coloui oiganists, Hallock-Gieenewalt included, have piesumed theii coloui oigans
to be novel, howevei expeiiments in this aiea have been a longstanding theme befoie and since.
Light shows and video displays have become an integial pait of the live music stage, fiom the
psychedelic light shows of the 1vecs to contempoiaiy live geneiative visual ait. Te lauei is
exemplied by United Visual Aitists (UVA), who began by cieating stage visuals to accompany
and ieact to the live music of Massive Auack in zccI. UVA have since biought theii woik to
national galleiies in the foim of inteiactive installation ait. Te aitistic focus of both the woik
of Hallock-Gieenewalt and UVA is the fusing of modes into an whole, audio-visual expeiience.
Hallock-Gieenewalt invented analogue electiic components to suppoit hei woik, and UVA
woik with digital computation, but the aitist John Whitney is a case of an aitist who has woiked
thiough both analogue and digital technology. Whitney pioduced expeiimental lms fiom the
1vcs, ist using hand diawn animation, then analogue computeis and in latei yeais digital
computeis as they became available (Youngblood, 1vc, pp. zczzs). Teie is liule discontinu-
ity between his woiks as he piogiesses fiom analogue to digital methods, all aie exploiations
of foim thiough geometiy. Te biggest dieience is in the amount of time taken to pioduce
the lms, a decade foi his ist hand diawn lm, but much less foi his latei woiks. Te depth
of his ist digitally pioduced lm Arabesque (1v) is bieathtaking, the inuence of his eailiei
analogue appioaches still showing in leading the vieweis peiception to an expeiience that is
both abstiact and coheient. Tis continuity is shown too in the development of piogiamming
languages foi music, in paiticulai Millei Puckeue has said he thinks of his Patchei languages
(.I.1) as moie like analogue synthesiseis than piogiamming languages (Lewis, 1vvI). We will
have much moie to say about analogue iepiesentation in the notation of computei piogiams
in chaptei .
Digital computeis too ofen lead aitists into a tiap of applying digital tiansfoimations to
data without consideiation of the analogue souice and destination of the data, ultimately lead-
ing to output that human peiception cannot decode. While this is not always the case, analogue
methods tend towaids stiaightfoiwaid mapping between two quality dimensions in a stiaight-
foiwaid, peiceptually salient mannei. Digital methods on the othei hand can too easily iesult
in tiansfoimations divoiced fiom human peiceptual piocessing, such as ieading an image as a
one dimensional scan-line, as Dean et al. (zcce) iepoit of the MetaSynth sofwaie. Te iesult is
I
Cu~v1iv z Sv:nois
too ofen peiceptually incompiehensible. Nonetheless at paiticulai scales humans have stiong
acuity foi iecognising disciete paueins, and indeed we will asseit that this is an impoitant
aspect of music cognition in 1.1.
Te woik of Whitney shows how the analogue and digital aspects of a woik suppoit one
anothei, and also that while electionic computeis aie ofen extiemely convenient, they aie
not necessaiy. Teie aie dangeis in woiking with electionic computeis, in that by not taking
full consideiation of oiientational metaphoi, oi coheient pauein, the iesults of digital tians-
foimation aie likely to be incompiehensible. Howevei we should ceitainly not downplay the
potential impoitance of computei languages in a iich cieative piocess, a subject we will exploie
in chaptei e.
z.e Acid Sketching Semantic lmageiy
We have discussed two avouis of symbols in Dual Coding theoiy, analogue imagens and
disciete logogens (z.z.1), as impoitant to undeistanding human peiception and cognition. We
will develop an aigument foi the ielevance of this to piogiamming languages in latei chapteis,
foi now howevei we giound the discussion so fai in piactice, with a woiking piototype that
demonstiates the use of symbolic, analogue imageiy in a usei inteiface foi music. A video of
Acid Sketching in use is contained within the accompanying DVD.
Te Acid Sketing system was developed thiough the piesent ieseaich, to undeistand how
geometiic foims and ielationships can be meaningfully used in a computei system. Te Acid
Sketching inteiface consists of an oidinaiy piece of papei, which is also a piojection suiface.
When shapes aie diawn on the papei with an ink pen, they aie identied and analysed using
computei vision. Teii shapes aie tianslated to sound synthesis paiameteis, and theii ielative
positions tianslated into a polyphonic sequence.
Te pioceduie to tuin shapes into a sequence of sound events is as follows. Shapes aie
identied fiom a digitised image of the papei via a consumei giade webcam. Tis is done us-
ing the OpenCV (Open souice Computei Vision) libiaiy developed by lntel Coip. Using this
libiaiy, the contouis of the hand diawn lines aie identied, and the shapes which they desciibe
aie enumeiated. Te centioids of the shapes aie calculated, and a minimum spanning tiee con-
necting them is constiucted. Te iesult is the giaph containing exactly one path between any
two shapes, whose spanning tiaveisal is minimised. Staiting fiom the most cential shape, the
minimum spanning tiee is followed, to place the shapes in a sequence. Time inteivals between
each paii of shapes aie given by the length of the edge connecting theii centioids, which aie
scaled ielative to an adjustable beats-pei-minute value. Because the minimum spanning tiee
Is
Cu~v1iv z Sv:nois
bianches, events may co-occui, which in musical teims iesults in polyphony.
Tis use of a minimum spanning tiee tuins a visual aiiangement into a lineai sequence
of events, a kind of dimension ieduction. ln teims of computational complexity, a simplei
appioach would have been to simply disiegaid one dimension, foi example by ieading the
events fiom lef to iight. Howevei we aigue that gieatei iichness is achieved by using this
giaph stiuctuie built fiom the peiceptually salient measuie of ielative distance in zD space.
Afei all when we view a pictuie, oui eyes do not geneially iead fiom lef to iight, but instead
jump aiound multiple xation points inuenced by the stiuctuie of the scene (Hendeison, zccI).
Sound synthesis is piovided by nekobee, a fiee/open souice emulatoi of the analogue Roland
TB-IcI Bass Line synthesisei. Te nekobee sofwaie is no longei actively maintained, but con-
tinues to be available in Linux opeiating system distiibutions. Te TB-IcI is best known foi its
use in the Acid House genie, hence the name Acid Sketing (we discuss peiception of synthesis
in acid house music in I.I.I). Te natuie of each sound event is given by moiphological mea-
suiements of its coiiesponding shape, wheie each measuiement is mapped to sound synthesis
paiameteis. Specically, roundness is calculated as the iatio of a shapes peiimetei length to its
aiea, and maps to envelope modulation, angle is that of the shapes cential axis ielative to the
scene, and maps to iesonance, and nally, the shapes area maps to pitch, with laigei shapes
giving lowei pitched sounds.
Visual feedback is piojected back on to the papei using a standaid data piojectoi, with the
cameia input aligned to piojectoi output in sofwaie. Tis feedback takes the foim of moving
ciicles, tiacing the path fiom one shape to the next along the edges of the minimum spanning
tiee, ood-lling each shape as its coiiesponding sound event is tiiggeied.

While sofwaie based, Acid Sketching aims to avoid some of the tiaps of digital iepiesenta-
tion. One such tiap is placed by the digital iepiesentation of an image, geneially a two dimen-
sional aiiay with x and y as indices, oi the undeilying single dimensional iepiesentation of a
scan-line. ln Acid Sketching, what is signicant is placement not ielative to two dimensions,
but ielative to all the othei maiks on the page. luithei, the geometiic calculations used in this
Acid Sketching piototype aie not foimally tested, but illustiates oui hypothesis that coiiespon-
dence between shape and timbie aie stiaightfoiwaidly leainable. lf this hypothesis holds, this
piototype system could be developed fuithei into an engaging inteiface foi live music. Woik
inspiied by Acid Sketching is undeiway at the Computei Laboiatoiy at Cambiidge Univeisity,
investigating such coiiespondences fiom the peispective of human-computei inteiaction.
Te symbolic natuie of the shapes in Acid Sketching gives paiticulai insight to oui oveiall
theme. lt demonstiates a use of analogue symbols which have moiphological piopeities con-

loi the puiposes of demonstiation, the video foi Acid Sketching on the DVDshows this feedback mixed diiectly
with iecoided video, iathei than piojected into the scene.
Iv
Cu~v1iv z Sv:nois
tinuously mapped fiomthose of what is iepiesented. Aciiticismmay be that these symbols aie
not tiuly analogue as they aie iepiesented digitally, fiom the peispective of the computei, we
can think of theie being many millions of disciete sounds to iepiesent, and potentially at least
the same numbei of disciete symbols to iepiesent them. lt is moie useful howevei to think of
these as analogue symbols with a continuous, peiceptually salient mapping to the iepiesented
sounds. At a paiticulai scale the undeilying gianulai, disciete iepiesentation begins to show,
but as with atomic units in physics, this is outside of the noimal limits of human peiception.
Te simulation is accuiate enough to be peiceived as analogue.
Acid Sketching demonstiates how analogue symbols may be inteipieted thiough the use
of existing computei vision libiaiies. We can think of these libiaiies as models of peiception,
which while impoveiished compaied to human peiception, aie nonetheless close enough to
be useful in inteiface design. While cieating these peiceptually salient continuous mappings
is ielatively tiivial, the challenge comes when we tiy to integiate analogue and disciete sym-
bols in a mutually suppoiting mannei. How may we ielate abstiact, disciete symbols with
peiceptually giounded, continuous symbols` loi one answei, we look to the vocal tiact.
z. Phonemes
loi most people, the instiument that most diiectly giounds symbols is the vocal tiact, with
paialinguistic suppoit fiom theii hands. loi Deaf people, it is the othei way aiound, with
hand gestuies pioviding the atoms of language and the mouth and face adding phiasing.
Te expiessive equivalence of signed and spoken languages (Suuon-Spence and Woll, 1vvv)
demonstiates dependence of language on movement iathei than sound, a point we will ietuin
to in I.1.
ln spoken foim, language is iepiesented within the disciete symbols of phonemes, the units
which we iepiesent with sound. Te phonetics of lnglish has a compaiatively pooi coiiespon-
dence to the leueis of its alphabet, and so to unambiguously tiansciibe lnglish pionunciation,
the lnteinational Phonetic Alphabet (lPA, see Ladefoged, 1vvc) is geneially used. Te lPA in-
cludes 1c leueis and z diaciitics, split into two tables, one foi consonants and one foi vowels.
Both tables aie oiganised accoiding to place of aiticulation, and so we can say that these sym-
bols use physical positions to classify sounds. Tat is, phonetics giounds speech not in sound
images, but in bodily movement. Tis is an impoitant distinction, which we maik now to be
built upon latei thiough chaptei I.
A phoneme exists in thiee modalities as a disciete symbol, as a conguiation of the vocal
tiact, and as a sound. Phonetics binds these veiy dieient modalities togethei into a whole,
1c
Cu~v1iv z Sv:nois
allowing us to communicate disciete stiuctuies of language thiough movement, whethei its the
hand that wiites oi signs, oi voice that sounds. Te following section intioduces an aitwoik
that biings the iole of movement in communication and sound peiception to the foie.
z.s Miciophone
Figure 2.1: Microphone, by Communications, 2010. Media: CNC milled plywood, webcam, speaker array,
soware.
Microphone is an aitwoik by Communications, which is a collaboiation between lunJoo
Shin and the piesent authoi. Miciophone was installed at the Unleashed Devices gioup show
at the Wateimans galleiy London in Autumn zc1c. Miciophone invites paiticipants to com-
municate with each othei acioss a galleiy, using two laige miciophones. Te devices in Micio-
phone do not opeiate in the same way as conventional miciophones as we desciibed in z.1. Te
sounds aie captuied not with a conventional electionic tiansducei but with a digital cameia,
with sofwaie tiained to pioduce vowel foimants fiom mouth shapes.
Te woik invites paiticipants to communicate using vocal sounds as a medium foi gestuie
without language, biinging focus on the iole of movement in communication. lt evokes a
feeling that is liteially visceial, of vocal oigan encoding paueins of movement into sound, and
being peiceived as movements.
Microphone uses computei vision in a similai mannei to Acid Sketing desciibed above
(z.e), in that it uses OpenCV blob detection to identify a polygon iepiesenting the shape of
the mouth. liom this polygon the paiameteis of ioundness and aiea aie deiived as with Acid
Sketching, along with the aspect iatio (height/width iatio of the minimumenclosing iectangle),
and the convex hull aiea. Tese measuies aie not used diiectly, but instead used in combination
to calculate a measuie of similaiity between the given mouth shape and the ve vowels a, e, i, o
11
Cu~v1iv z Sv:nois
Figure 2.2: Microphone, showing a speaker array whi carried sounds between the two devices. e dis-
tance between the devices was constrained by installation in a group show.
and u. Te aveiage foimant values of the thiee closest vowels aie then calculated, weighted by
theii similaiity to the taiget shape. Although ve disciete vowels aie used in this calculation, a
iange of sounds between them aie mapped continuously, so foi example the neutial (schwa)
vowel would be a point between them.
Miciophone applies digital technology to map between modalities, but entiiely analogue
means may be used to much the same ends. Teie aie long tiaditions of using the mouth as
paialinguistic, musical instiument, eithei alone oi augmented with instiuments such as the
Jews haip, as we will see in the following chaptei. Te digital foundations of this aitwoik
allows expectations to be confounded in inteiesting ways, but outwaidly, Miciophone is an
entiiely analogue aitwoik.
z.v Discussion
We have taken a faiily unconventional view of symbolic iepiesentation, but one that we have
suggested to be plausible in the context of cognitive psychology. ln paiticulai, we have taken
a view of symbolic iepiesentation that takes both analogue and disciete symbols into account.
Computation may at base be dened by the manipulation of disciete symbol sequences, but
may be applied to simulate analogue systems, including aspects of the outside analogue woild.
luithei, human cognition, including that of computei piogiammeis, involves both analogue
and disciete piocessing, lateialised and integiated.
Against this backgiound we view piogiamming as a human activity that spans both ana-
logue and disciete domains. Te two woiks we have piesented, Acid Sketching and Miciophone
have in pait been developed to exploie this view. Howevei both show the disciete aspects of
1z
Cu~v1iv z Sv:nois
piogiamming as somewhat subseivient to analogue aspects in the nal aitwoik. Tey aie in-
teiactive woiks, wheie paiticipants engage with the woik by making analogue gestuies which
aie digitised, tiansfoimed, then tiansduced back to analogue. ln the case of Acid Sketching it
is hand diawn gestuies which aie digitised, and in the case of Miciophone it is mouth move-
ments. Te puipose of the digitisation howevei is to allow tiansfoimation fiom one analogue
domain to anothei.
Teie is a fuithei piocess of discietisation which happens in the eai and biain, peiceiving
Miciophones output as vowel categoiies, and Acid Sketchings output as disciete sound events
in pitch classes. Tis categoiical peiception happens in tandem with the expiessive spatial
peiception, wheie a listenei might auend to how the paiticipant tiansitions fiom one vowel to
the next, oi plays with dieient timbial paiameteis.
Te computation in Miciophone and Acid Sketching is hidden behind analogue inteifaces
based upon gestuie, visual feedback and sound. Tis is the case in a gieat deal of inteiactive
computei aitwoiks, wheie a gieat deal of eneigy is put into nding novel means of analogue
expiession. Tis is also veiy much tiue in computei music inteifaces, such as those demon-
stiated at the New lnteifaces foi Musical lxpiession confeiences.
e
Analogue expiession is
ofen a focus in the aits, and so ieseaich exploiing new analogue inteiaction is of couise veiy
welcome. lt has howevei led to some questionable claims being endemic in the electionic aits,
that analogue inteifaces aie somehow moie advanced than disciete ones, that computation
necessaiily should be hidden in ait, and that computation is secondaiy to analogue expeiience.
loi example the aitist Simon Penny has the following to say about his piece lugitive z
Te inteivention of fugitive is to piesent a mode of inteiaction which is pied-
icated on the system inteipieting a peison engaged in noimal human bodily be-
haviois. Tis is in staik contiast to the conventional notion of inteiface, in which
ideas and conceins must be encoded, usually as alphanumeiic data, demanding the
sequential piessing of liule buuons on a boaid. Tis ut[t]eily impoveiished intei-
face functions, in fact, as a ltei, excluding all iich and diveise aspects of human
intelligence which cannot be encoded alphanumeiically.

Tis may seem a ieasonable point of view until we notice that he is aiguing against the
wiiting of novels as impoveiished compaied to (in this case) moving aiound befoie a cam-
eia. To aigue that eithei analogue oi digital iepiesentations aie moie advanced is equivalent
to aiguing whethei novels oi paintings aie a moie advanced foim of expiession. lndeed we
might aigue that novels aie moie advanced as piint is a latei technological development, and
indeed builds upon the accomplishments of paint. Tis would howevei be a weak position to
e
http://nime.org/

Retiieved fiom http://ace.uci.edu/penny/works/fugitive2.html, July zc11


1I
Cu~v1iv z Sv:nois
take, as we have aigued thiough this chaptei, humans aie maiked by the integiation between
linguistic and spatial foims. lt follows then that whethei disciete iepiesentations aie shown
in woiks depends entiiely on the focus of the aitist, whose engagement may include souice
code, wiiuen natuial language, musical notation, low-iesolution lights aiianged in giids, oi
any othei disciete, linguistic code.
liom heie then, we look foi ways in which computei ait can integrate the vaiious codes
of language and peiception, in new woiks that engage with full, iich expeiiences acioss both
domains.
11
Cu~v1iv I
Words
Symbols aie to woids as positions aie to movement. Tese woids that l am wiiting, and you
aie ieading, aie sequences of leueis wiiuen in the Latin alphabet, fiom lef to iight. Tey aie
membeis of a lexicon of many hundieds of thousands of woids, which includes much appaient
iedundancy, although eveiy synonym has its own pauein of usage. Befoie woids came to be
wiiuen like this, they weie only spoken. A pencil leaves a line, but aii piessuie waves fiom
the moving vocal tiact seule quickly to the mean, leaving no tiace behind. lxcept of couise
the subjective tiace, biought by haii movements in the cochlea, ieduced by the listening biain
into a peiceptual image, then peihaps ieduced fuithei into a foim held in memoiy.
As we have seen, the lnglish alphabet has a loose phonetic mapping. Childien aie con-
ventionally taught to iead using phonics, teaching leueis of the alphabet by typical sounds
they iepiesent, which aie stiung togethei to pionounce woids. Howevei, theie is a signicant
leap fiom theoiy to piactice, this is a natuial language, with much ambiguity and exceptions
foi eveiy iule. Nonetheless once leained, natuial language can feel eoitless to the point of
invisibility. We may feel completely absoibed in a text, which is iemaikable as liteiacy is a
compaiatively iecent, wholly cultuial development, we have not evolved to iead and wiite.
ln piogiamming, woids aie used iathei dieiently. Aside fiom comments wiiuen in natu-
ial language, woids in souice code aie eithei language keywoids, oi names given to vaiiables
and functional abstiactions by the piogiammei. As with the use of spatial aiiangement exam-
ined in z.z.I, thiough the piocess of tokenisation, the woids aie ignoied by the language in-
teipietei as secondaiy notation. Woids aie not tieated as having moiphology, and aie instead
ieduced to unique, shapeless tokens. lven the piactice of capitalising names of libiaiies of code
is by convention, iaiely enfoiced by the inteipietei, the naming is lef entiiely to the piogiam-
mei. Tis dieis fiom natuial language, wheie moiphological woid stiuctuie ielates impoitant
meaning, such as the sux -s oi -es to indicate pluials in lnglish. luitheimoie, piogiam-
ming language does not have the sound symbolism we noted in z.1, such as the onomatopic
1
Cu~v1iv I Wovbs
woids cluuei, quack and bang, oi the peiceptual ielation between fiicative consonants and
shaip shapes (Ramachandian and Hubbaid, zcc1b). ln natuial language the moiphology is not
only syntactic, but has stiuctuie which miiiois what may be iepiesented. Such ielations aie
geneially not consideied impoitant to modein study of linguistics, but nonetheless illustiate
a iichness of human language, which we may diaw upon in the design of human-computei
inteiactions.
While woid moiphology is absent fiom piogiamming languages, allusions to it may oc-
casionally be found. ln the Ruby language it is a convention foi destiuctive methods, as in
those that diiectly modify mutable vaiiables, to end in an exclamation maik (llanagan and
Matsumoto, zccs). Regulai lxpiession (iegex) language is designed foi matching stiings of
text accoiding to iules dened by teise, ofen single chaiactei opeiatois and modieis (liiedl,
zcce). Te iesult is a language on the woid level, foi example the following matches some
vaiiants of the ioot woid colour
/\bcolou?r(s|ist|ing)?\b/
Despite woid moiphology not featuiing signicantly in piogiamming syntax, it is of couise
impoitant foi a chosen name in souice code to ieect what it symbolises, including moipho-
logical aspects such as s suxes on methods that ietuin multiple values, and appiopiiate use
of nouns and veibs to desciibe object classes and methods. Tis is convention in secondaiy
notation which we will covei in gieatei detail in .1.
Te geneial absence of woid stiuctuie in the syntax of souice code is indicative of the lack
of aiticulation in the activity of piogiamming. Te paiticulai movements of a piogiammeis
ngeis aie lef behind at the keyboaid, tianslated theie into disciete on-o states. Oui question
foi aitist-piogiammeis woiking in computei music is, how can we ielate oui disciete iepie-
sentations back to analogue movement` Woids have life as aiticulations, as well as sounds
and symbol sequences, and so aitist-piogiammeis have much to leain fiom theii study. ln the
following we will look foi gieatei undeistanding of this issue by examining the peiception and
stiuctuie of woids, nding motivation in vocal tiaditions of music.
I.1 Peiceiving Speech as Movement
We begin oui examination of woids with a ieduction of the spoken woid to minimal compo-
nents. Sine wave spee is wheie the complex, time-vaiying piopeities of a speech sound signal
aie ieduced to a few sine waves (Remez et al., zcc1). Typically, the fiequency and amplitude of
thiee sine waves aie mapped fiomthe lowest thiee foimant fiequencies, and a fouith sine wave
fiom a fiicative foimant. Te iesult is a bistable illusion, wheie an untiained human subject
1e
Cu~v1iv I Wovbs
initially peiceives sinewaves as sepaiate, buibling aiticial sounds. Howevei once they aie
diiected to auend to the sounds as a human voice, they aie able to peiceive a single stieam of
intelligible speech. lt is suipiising that speech is peiceivable at all fiomthese simple modulated
tones, with all hisses, pops and clicks iemoved. Despite the shoit-teim acoustical piopeities of
speech being absent we can still peiceive speech in the vaiiance of puie tones, even identifying
paiticulai speakeis by it.
Te inuence that non-acoustic cues hold ovei speech peiception is also demonstiated by
the McGuik-McDonald eect (McGuik and MacDonald, 1ve). A classic demonstiation of this
eect is wheie a subject is simultaneously piesented with the sound /ba/, and a video of a face
mouthing the syllable /ga/, but heais neithei, instead expeiiencing the illusionaiy syllable
/da/. Tis peiceptual eect is stiong foi the majoiity of test subjects, stable even when subjects
aie made fully awaie of what the audio stimulus ieally is. What is moie, the illusion peisists
even when subjects aie not consciously awaie of what they aie looking at, Rosenblum and
Salda na (1vve) iepioduced the eect without showing the face, but instead only point-light
movements taken fiom a moving face. Tis is a closely ielated iesult to that of sine-wave
speech, both cases show that featuies of the signal aie not as impoitant as how those featuies
vaiy. ln othei woids, movements deiived fiom the signal souice aie moie impoitant than
iecognition of its shoit-teim piopeities.
Such speech illusions have been taken as suppoit of the Motoi Teoiy of Speech Peicep-
tion (Libeiman and Mauingly, 1vs). Accoiding to this theoiy, a special module has evolved
in the human biain foi speech, iesponsible foi both speech pioduction and peiception. De-
spite having some populai notoiiety, motoi theoiy is not widely suppoited within the speech
peiception eld, as the notion of this special module is not well dened (Mole, zccv), and is
not suppoited by the evidence (Galantucci et al., zcce). While the concept of a special module
foi language is geneially no longei taken seiiously, othei aspects of the theoiy have become
widely accepted. Motoiic contiibution to speech peiception is well suppoited, and sits well
with bioadei liteiatuie demonstiating stiong motoiic contiibution to peiception in geneial
(Galantucci et al., zcce). lt would seem that iathei than speech being special as Libeiman and
Mauingly oiiginally suggested, that meshing of action and peiception is a featuie of human
peiception in geneial.
Te meshing of peiception and action has enjoyed ienewed inteiest since the existence of
mirror neurons was identied by di Pellegiino et al. (1vvz). Miiioi neuions weie identied as
a class of neuions in the l iegion of the piemotoi coitex in a laboiatoiy expeiiment with
a Macaque monkey. Tese neuions weie seen to ie both when the Macaque obseived an
assistant giabbing food, and when the Macaque himself giabbed the food. Te explanation
1
Cu~v1iv I Wovbs
oeied by di Pellegiino et al. (1vvz) is that these neuions iepiesent action understanding in
motoi aieas. Te hypothesis is that because the same neuion ies iegaidless of whethei the
subject oi an obseived individual peifoims an action, that this is the basis of social cognition.
Tis expeiiment captuied the imagination of many ieseaicheis inteiested in embodied social
cognition, not least Ramachandian (zccc), who was diiven to exclaim that the discoveiy of
miiioi neuions was on the same level of the discoveiy of DNA. Tis enthusiasm has ieached
the eld of music psychology, with Leman (zcc) citing miiioi neuions as a basis foi social
undeistanding of music cognition based on gestuie/peiception couplings. Howevei despite all
this enthusiasm, miiioi neuion theoiy has a numbei of pioblems. loi example, seveial of its
oiiginal assumptions have not held, and it fails to explain a wealth of evidence fiombiain lesion
studies (Hickok, zccv). A neuial system undeilying social cognition is ceitainly cause foi gieat
inteiest, but it is eaily days foi miiioi neuions, and caie should be taken not to extiapolate
fiom ndings which iemain contioveisial and uncleai (Dinstein et al., zccs).
lt is eaily days foi neuiobiology in geneial, but contioveisy ovei these initial claims does
not mean that a human miiioi system does not exist in some foim, with oi without miiioi
neuions. While fMRl studies do not give data on the level of individual neuions, a numbei
show oveilap in peifoiming and obseiving actions (e.g. Calvo-Meiino et al., zcc, Chong et al.,
zccs), suggesting motoi simulation does have a iole in action undeistanding. ln any case, we
have seen evidence foi stiong motoiic contiibution to speech peiception, demonstiating the
inuence oui bodies have ovei what we expeiience. Tis continues the theme of inteiaction
of continuous and disciete iepiesentations fiom the pievious chaptei, speech is a medium foi
disciete units of language deeply inteitwined with the continuous movement that caiiies it.
I.z Vocable Woids in Music Tiadition
Like speech, instiumental sounds aie pioduced via a seiies of aiticulations of the human body.
Modein digital computeis allow us to synthesise sound with algoiithms, but nonetheless we
peiceive the iesult with a biain evolved and developed foi deiiving movement fiom sounds.
We have examined the iole of aiticulation in the peiception as well as pioduction of speech,
and now considei the same oi analogous ielationship with the sounds of musical instiuments.
ln paiticulai, we considei aiticulation as a peiceptual biidge between musical instiuments and
the human body, and will latei piopose a means foi this biidge to be used in the design of
computei music notation.
A vocable woid is simply a woid that is able to be spoken and iecognised, accoiding to a
systemof phonetics (z.). Te woid vocable is geneially used to desciibe nonsense woids that
1s
Cu~v1iv I Wovbs
aie non-lexical, but nonetheless ieadily pionounceable using the phonetic sounds of a paitic-
ulai language. ln musical tiadition, vocable woids aie ofen used to desciibe an aiticulation of
a musical instiument. loi example a music instiuctoi may use theii voice to desciibe a sound
theii student should tiy to make on theii violin, peihaps by singing a pitch contoui while using
a consonant-vowel pauein to indicate a paiticulai bowing technique. Ovei time the student
will leain to peiceive the phonetics of theii instiuctois voice as the sound categoiies of theii
instiument.
Vocable woids can be found in use acioss the continents and in many musical tiaditions.
lndian classical music has Bol syllables to speak the diums wheie, foi example, e iepiesents a
non-iesonating stioke with the 1st ngei on the centie of the d ahin a (iight hand) dium (Kip-
pen, 1vss). Bol syllables aie ofen used in iefiain, wheie the tabla playei switches fiom playing
the diums to speaking them with vocables. ln the Scouish Highlands we nd Canntairead of
the bagpipes (Campbell, 1ssc), foi example hiaradalla iepiesents an echo of the D note in the
McAithui Canntaiieachd dialect. As with Tabla, Canntaiieachd is used in peifoimance, and
the lndian Dhiupad singei Piakiiti Duua and pipei Bainaby Biown have collaboiated to unite
the foims in vocal peifoimance
1
.
Canntaiieachd and Bols aie laigely vocal tiaditions, which have come to be wiiuen down
in iecent times. Some vocables have tiaditionally been tiansciibed howevei, such as the ien-
tz u notating the delicate ngei techniques of the guqin (Chinese zither). loi example an-fu
indicates that the index, middle and iing ngei each pull a dieient stiing with a light touch,
making the thiee stiings pioduce one sound melting togethei.
ln hei doctoial thesis Non-lexical vocables in Scouish tiaditional music, Chambeis (1vsc)
divides the use of vocables as eithei being cultuially jelled oi improvisatory. Jelled vocables,
such as Bol oi Canntaiieachd vocables, aie pait of a foimal system, wheie paiticulai vocables
iepiesent paiticulai aiticulations of an instiument. lmpiovisatoiy vocables on the othei hand
aie made up duiing a peifoimance, such as scat singing in Jazz. Chambeis acknowledges that
the line between jelled and impiovisatoiy is ofen bluiied, wheie foi example a playei foi-
malises some aspects of theii diddling ovei time. Anothei distinction made by Chambeis is
between imitative, onomatopoeic vocables and associative, aibitiaiily assigned vocables. Tis
is a peiceptual distinction, as Chambeis iepoits Occasionally a pipei will say that a vocable is
imitative (indigenous evaluation) when analysis seems to indicate that it is actually associative
(analytic evaluation) because he has connected the vocable with the specic musical detail foi
so long that he can no longei divoice the two in his mind (Chambeis, 1vsc, p. 1I). ln othei
woids, a vocable may appeai to mimic an instiumental sound on the peiceptual level, without
1
A video of this collaboiation is available online, http://www.youtube.com/watch?v=I_7wh_ClamA (ac-
cessed Maich zc11)
1v
Cu~v1iv I Wovbs
having similaiity on the level of the sound signal. Tis seems to be tiue in the geneial context
of onomatopoeia foi example wheie a native lnglish speakei heais a hen say cluck, theii
Geiman neighboui may peiceive the same sound as tock (de Rijke et al., zccI). Reseaich into
tabla Bols have howevei found them to be genuinely imitative, shaiing audio featuies with
the instiumental sounds they iepiesent, identiable even by naive listeneis (Patel and lveisen,
zccI).
A thiid distinction can be made between vocable woids in spoken and wrien foim. A
ieadei of a vocable applies paralinguistic phiasing not deiived fiom the text, but nonetheless
with gieat musical impoitance. Conveisely a tiansciibei may iesolve ambiguity in a spoken
vocable by wiiting a piecise inteipietation of the intended disciete pauein. Tese issues aie of
couise common to all notation systems, including those of natuial language (z.z.1). We can
say howevei that to some degiee a wiiuen vocable may captuie the disciete essence of a sound,
oi at least a mnemonic focus to the whole aiticulation. liom oui discussion of Dual Coding
(z.z), we can undeistand a wiiuen vocable as less accuiate in captuiing the full expiessivity of
a sung vocable, but more accuiate in piecisely captuiing aspects of its disciete stiuctuie. Tis
is simply a focus on eithei discontinuities oi on smooth tiansitions, both impoitant musical
featuies, and as such a sound may be fully undeistood in teims of both.
Te populaiity of sta notation in Westein Classical music has, at least in the case of Can-
ntaiieachd, led to a ieduction in the teaching of jelled vocables (Chambeis, 1vsc). Howevei in
ieaction to the lack of standaid means to notate aiticulation in sta notation, Maitino (1vee)
pioposes his own method giounded in phonetics. lach of his notational maiks iepiesent a
vocable syllable, foi example the maik

has the phonetic iepiesentation tat, desciibed as a
incisive, ciisp auack with similaily dosed decay. Maitino asseits that the aiticulatoiy pa-
iameteis of any musical instiument can be undeistood as a subset of those of the voice, and as
a iesult that his notation applies to all instiuments.
Vocables in music aie ofen iefeiied to as being non-lexical (Chambeis, 1vsc). lt would
seem that the denition of jelled vocables implies a lexicon, a dictionaiy of woids that symbol-
ise paiticulai aiticulations of an instiument. On closei examination howevei vocable woids aie
sub-lexical. Canntaiieachd is not so much stiuctuied by a lexicon but by a system of phonet-
ics, which in combination, geneiates a bioad iange of possible vocable woids, with no lexical
iefeience apait fiom theii phonetic stiuctuie. As such, impiovisatoiy vocables aie puie sound
symbolism, with iefeience occuiiing on the phonetic, and not lexical level.
c
Cu~v1iv I Wovbs
I.I Timbie
Timbie is an impoitant component of music, yet is liule undeistood. We have seen howvocable
woids allow instiumental aiticulations to be ielated to aiticulations of the vocal tiact. Aitic-
ulation is closely ielated to timbie, wheie movements of instiumentalists such as of plucking
style, bieath contiol and afei-touch have stiong inuence ovei the iesulting timbie. ln the
piesent section we will build an aigument that this inuence is ciucial to the expeiience of
music, asseiting that in peiceptual and conceptual teims, aiticulation is timbie.
Te Ameiican Standaids Association (ASA) denes timbie in theii Acoustical Teiminology
standaids as that auiibute of auditoiy sensation in teims of which a listenei can judge that
two sounds, similaily piesented and having the same loudness and pitch, aie dieient (Bieg-
man, 1vv1). Wheie the ASA standaid is quoted it is ofen deiided, foi example Biegman (1vv1,
p. vz) paiaphiases it as We do not know how to dene timbie, but it is not loudness and it is
not pitch, in othei woids an ill-dened wastebasket categoiy. Teie have been seveial ie-
seaich auempts to establish a beuei denition of timbie, but with liule success, and in a bioad
ieview Hajda et al. (1vv) highlight this lack of denition as the most lasting obstacle in the
ieseaich of timbie. lndeed without being able to dene timbie, it is dicult to know how to
look foi othei obstacles. Timbie seems to be an impoitant pait of music, but we cannot agiee
on what it might be, oi even how to appioach dening it.
loi context, the same ASA document as above is also quoted as dening pitch as that
auditoiy auiibute of sound accoiding to which sounds can be oideied on a scale fiom low to
high. We can iestate this in oui teims, that ASA dene pitch as an oiientational metaphoi
of the soit examined in z.z.e. We infei then that the ASA believe that othei aspects of sound
aie not oideiable along such a scale. Pitch ceitainly has a cleai ielationship with the physical
woild, being the peiceptual counteipait to fiequency of waves of aii piessuie. Pitch also coi-
ielates with physical body size, as smallei animals tend to have shoitei vocal tiacts with highei
iesonant fiequencies, iesulting in highei pitched vocalisations. Tis makes pitches amenable
to being oideied along a quality dimension. Howevei, the peiceptual quality of loudness has a
similai ielationship with wave amplitude and physical body size, smallei things tend to be qui-
etei. Te ASA denition of pitch is undei-specied, it fails to exclude oiientational metaphois
which we claimstiuctuie much of the expeiience of music, timbie included. To look foi suppoit
foi this claim, we tuin to psychology of timbie liteiatuie, wheie the seaich foi the dimensions
of timbie has been a common ieseaich aim.
1
Cu~v1iv I Wovbs
3.3.1 Multi-Dimensional Scaling
Unlike pitch and loudness, timbial dieiences do not have diiect ielationships with the fun-
damental featuies of aii piessuie waves, hence theii consignment to a wastebasket categoiy
(Biegman, 1vv1). Howevei a numbei of ieseaicheis have auempted to identify distinct quality
dimensions of timbie. Te hypothesis is that theie aie a ceitain numbei of quality dimen-
sions dening a space in which timbie is peiceived. Tese auempts have in geneial applied
Multi-Dimensional Scaling (MDS, Shepaid, 1vez) oi closely ielated appioaches of dimension-
ality ieduction. ln MDS expeiiments, similaiity judgements aie collected fiom human subjects
and inteipieted as distances, which aie then used to ieconstiuct the peiceptual space in which
the judgements aie assumed to have been made. We can expiess this aim as captuiing the di-
mensions of mental imageiy (z.z.1) oi conceptual spaces (z.z.) at play when listeneis auend
to musical timbie.
MDS similaiity judgements aie classically given in iesponse to paiis of stimuli, as a maik
along a scale fiom veiy dissimilai to veiy similai. Tey may alteinatively be deiived fiom soit-
ing tasks oi closest paiis in tiiplets (Wellei and Romney, 1vss, Hajda et al., 1vv), with vaiious
eects on the scope and ieliability of the iesults (Bijmolt and Wedel, 1vv). An eaily auempt
at applying MDS to instiumental sounds was peifoimed by Giey (1v), wheie human subjects
weie asked to iate paiis of sounds on a similaiity scale. Te stimuli weie synthesised based
upon the physical piopeities of oichestial instiuments. Howevei Giey inteipieted the MDS
solution as showing clusteiing that went beyond familial gioupings of instiuments to show-
ing aiticulatoiy featuies. luitheimoie Wessel (1vv) used MDS techniques to pioduce contiol
stiuctuies foi synthesis, so that additive synthesis of timbie could be manipulated accoiding
to movements in a peiceptually salient space iesulting fiom the scaling.
Wessel inteipieted his MDS solution as having two dimensions, namely brightness and
bite. Biightness is geneially quantied as the fiequency of the spectial centioid, and Wessel
(1vv) ielates bite to the natuie of the sound onset oi aa. A ieview of MDS timbie studies
peifoimed since Wessels eaily woik shows the majoiity of studies have these two dimensions
in common (Caclin et al., zcc), howevei besides this theie is a gieat deal of inconsistency,
with many othei dimensions identied in individual studies without bioadei agieement. Tese
include spectial ux, spectial spiead, spectial iiiegulaiity and haimonic onset asynchiony.
ln summaiy, the bioad nding of MDS studies of timbie thus fai is that spectial centioid
and auack time seem to be impoitant. Liule moie is agieed upon except foi one impoitant
point, intei-individual dieiences aie signicant (Caclin et al., zcc). liom oui peispective,
we would ceitainly expect it to be so, peiception being not diiectly against the backgiound of
physical ieality, but of peisonal expeiience of it (1.z). Some of the inconsistency in the iesults
z
Cu~v1iv I Wovbs
of MDS timbie studies may be methodological, MDS is known to be highly sensitive to optional
cognitive piocesses inuenced by test conditions, even when similaiity measuies aie based on
Stioop inteifeience (ieaction times). loi example, MDS is ofen used to nd whethei two
dimensions of peiception aie ielated, by looking foi eithei a city block oi luclidean distance
metiic in the MDS solution (e.g. Gaidenfois, zccc, p. z). Howevei, Melaia et al. (1vvz) found
that such metiics weie entiiely down to optional piocesses employed by test subjects, and could
be stiaightfoiwaidly manipulated by alteiing the woiding of test instiuctions. Such optional
cognitive piocesses aie a seiious and dicult pioblem to deal with.
Kiumhansl (1vsv) looks beyond methodology in suggesting that the veiy notion of univei-
sal dimensions of timbie is awed. Although theie may be one oi two dimensions with geneial
salience, dieient sound souices have paiticulai chaiacteiistics iesulting in theii own dimen-
sions. lnstead, timbie should be undeistood not just as fiequency analysis, but also ielative
to the dynamics of a ieal, simulated oi imagined physical souice behind the sound. lt is cleai
that MDS studies have focused on the foimei (Hajda et al., 1vv), but we join Kiumhansl in
aiguing that the stiong iole of physical movement in the peiception of timbie has too ofen
been oveilooked.
3.3.2 Grounding Timbre in Movement
Many of the woids used to desciibe instiumental sounds aie metaphois foi the physical mani-
festations of movement, shape and the body, we have alieady seen that Wessel (1vv) used bite
to desciibe sound onset quality, an oial metaphoi indicative of the close peiceptual ielation-
ship between sound and bodily movement. Tis theme is exploied by Tiaube and DAlessandio
(zcc), who nds that vocal aiticulation has a stiong iole in guitaiists timbie peiception.
Tiaube and DAlessandio investigate both the lexical woids and non-lexical vocables that gui-
taiists use to desciibe guitai sounds. ln the foimei case, Tey asked guitaiists to choose woids
to desciibe guitai notes. Amongst othei woids, sounds obtained by plucking the stiing close to
its middle weie asciibed woids such as closed and damped wheieas plucking close to the biidge
weie desciibed as thin and nasal, and plucking midway, ovei the sound hole was desciibed as
large, open and round. Tis is in appaient subconscious iefeience to mouth shapes foi the vow-
els that have similai foimant stiuctuie to the iesulting guitai sounds. Tiaube suggests that this
shows use of mental imageiy to communicate timbie, ielating it to the use of vocable woids by
tabla playeis. lndeed thiough expeiiment Tiaube demonstiates that when guitai playeis im-
piovise vocables, theie aie positive coiielations both between plosive consonant and plucking
angle, and between vowel and plucking position.
I
Cu~v1iv I Wovbs
3.3.3 Music of Timbre
ln Westein cultuie, the piimaiy basis of music is widely consideied to be that of pitch contiasts
ovei time. Tis is evident in the clinical denition of those with decits in music peiception,
amusia is geneially consideied piimaiily as a decit in pitch peiception populaily known as
tone-deafness (Peaice, zcc). Teie aie musics howevei wheie contiasts of disciete tones is
unimpoitant, which we desciibe as music of timbre. While we aigue that peiceived movement
is of univeisal impoitance to music, to some musics, it would seem that this is almost all theie
is. Te question is, can music achieve the same levels of complexity without tonality, oi does
the focus on movement involve a tiade o, a shif fiommusical contemplation, peihaps towaids
something with a bioadei non-musical function in cultuie`
ln his book Music, Language and the Biain, Aniiuddh Patel notes that despite much mu-
sical expeiimentation, music based on timbial iathei than tonal contiasts is iaie (Patel, zcc,
pp. IcI). Tis is explained twofold, istly, timbial changes ofen iequiie instiumental ma-
nipulations that aie physically dicult to peifoim in quick succession. Te second, cognitive
ieason he gives is that timbial contiasts cannot be peiceived in teims of inteivals, so the highei
oidei ielationships associated with tonal music aie not suppoited by timbie. A given countei-
example is music of the tabla diums in lndian classical music, which Patel (zcc, pp. eze)
explains by looking at the music cultuie aiound the tabla. ln paiticulai, he notes the extensive
use of Bol syllables, a jelled system of vocable woids (I.z). Patel concludes that these vocables
allow peiception of complex timbial contiasts to be aided by cognitive iesouices developed foi
linguistic stiuctuie.
We piopose an alteinative hypothesis to Patel (zcc), that it is not specically linguistic
sound categoiies which suppoit timbie peiception in Tabla music, but moie geneially cate-
goiies of aiticulation, which just happen to be of the vocal tiact in this case. Te human ability
to peiceive and categoiise movement is used in the tiansmission of language, both in vocal and
sign languages (Suuon-Spence and Woll, 1vvv), but we contend that this is a moie geneial cog-
nitive iathei than linguistic iesouice, speech is not special (I.1) as Patel implies. Audiences
can peiceive tabla music because when peifoimeis speak the diums, listeneis aie able to ielate
the sounds to the movements of theii own vocal tiacts. ln the case of electionic dance music,
it is not the movement of the vocal tiact, but of the whole body dancing that opens the dooi to
bioad appeal found within laige audiences.
Te stiong implication made by Patel (zcc) is that without co-opting linguistic iesouices,
music of timbie would be impossible, oi at least iaie. Togethei with the lack of claiity found in
MDS study, it would be too easy to conclude fiom this that timbie is theiefoie unimpoitant in
music cognition, being eithei too complex oi too foimless to piovide stiuctuie foi music. llec-
1
Cu~v1iv I Wovbs
tioacoustic music has not found populai audiences beyond its academic base. Howevei Patel
oveilooks the existence and populaiity of electionic dance music, which ofen centies aiound
manipulation of synthetic timbie with appaiently no suppoiting use of vocable woids. An ex-
ample of this is the acid line, a musical style exemplied by the 1vv iecoiding of Highei state
of consciousness by Josh Wink. Tis piece ievolves aiound slow manipulation of the timbial
paiameteis of a iepeating motif on the Roland MC-zcz, a monophonic, analogue subtiactive
bass line synthesisei. Synthesis lteis aie manipulated on this machine duiing the piece, with
low-pass cuto slowly incieasing tension until ieleasing into a single, nal ciescendo. Te
timbial changes of this example aie slow, but one does not have to look fai to nd timbial
manipulation at a speed and complexity compaiable with that of tabla viituosos. loi exam-
ple Autechies Gantz Graf (zccz) has liule in the way of disceinible melody, but manipulates
sound events at a speed on the boundaiy between peicussion and metallic dione. Autechie ieg-
ulaily auiact audiences of thousands acioss luiope and the USA, who dance to fast-changing,
complex ihythms in the daik, with no visual accompaniment.
Te iole of movement is unsuipiisingly cential to dance music, but to the extent that as-
pects of music cannot be undeistood without dancing, oi at least imagining oneself dancing.
Te Clave ihythm is a case in point, wheie the main beats aie emphasised not thiough musi-
cal accents, but in the associated dance steps (Agawu, zccI, p. I). lndeed in many cultuies
the concept of music encompasses both the sounds and the dance, and one is not undeistood
without the othei (Agawu, zccI, p. ze1). ln the case of Autechie, the musicians piovide iich
and complex timbial stiuctuies, foi which audience membeis cieate theii own iefeience points
thiough theii own bodily movements.
Some musicians auempt to pioduce acousmatic music fiee fiom physical manifestation
and constiaints. Sounds may include those iecoided fiom iecognisable souices, but aie used
foi theii sonic piopeities. Tis music is not peifoimed, only existing in iecoided foim, and in a
conceit seuing is played ovei loudspeakeis. lt is of couise possible to enjoy this music thiough
physically static, deep listening, but we contend that it is dicult to do so, and that geneial
audiences nd it tioublesome to ielate ungiounded timbie to theii own bodily expeiience. Tis
diculty is expiessed well by Smalley (1vv1, p. Iv), ln electioacoustic music wheie souice-
cause links aie seveied, access to any deepei, piimal, tensile level is not mediated by souice-
cause textuie. Tat is what makes such types of acousmatic music dicult foi many to giasp.
ln a ceitain physical sense theie is nothing to giasp - souice-cause textuie has evapoiated.
Smalley concludes that to fiee timbie fiomsouice-cause, the composei must confiont and enjoy
the dissolution of timbie.
lmmeison (zcca) desciibes a dichotomy between dance and ait music in electionic music

Cu~v1iv I Wovbs
as being between eithei a focus oi fieedom fiom iepetition. lmmeison ielates iepetitive music
to a giounding in the movements of the human body, and amoiphous music to movements in
the enviionment. lt should be noted howevei that while the celebiated electioacoustic com-
posei Stockhausen disliked the iepetitive language in electionic dance music (lmmeison,
zcca, p. ez), he also compelled students of music to go dancing at least once a week. And
dance. Please, ieally dance thiee oi foui houis a week. (Stockhausen and Maconie, zccc,
p. 1c). We shaie the conclusion with lmmeison (zcca), that musicians who woik on the
boundaiy between these dance and ait musics aie ieconciling the dual themes of body and
enviionment. Tat these two gioups aie in many cases alieady using the same tools and lan-
guages oeis such musicians a unique oppoitunity.
3.3.4 Dening Timbre
We have focused on the iole of movement in timbie peiception, but until now have avoided
explicitly dening timbie. ln contiast to the appioaches of tiying to dene timbie in teims of
spectial featuies, oi the ASA appioach of dening timbie by what it is not, we dene timbie
moie bioadly in teims of mental imageiy
Denition Timbre is sound as it is peiceived, as mental imageiy of positions, ie-
gions and theii vaiiance in action spaces.
We need to be caieful in inteipieting this denition. Musical events aie by denition dis-
ciete, not only notes but also vocable woids such as tabla Bols. Howevei we chaiacteiise such
disciete symbols as signposts in continuous timbie space. We may notate timbie using disciete
symbols, but the timbie we notate is undeistood in teims of points, aieas oi movements in pei-
ceptual spaces. Tus brightness names an analogue dimension, bright names a iange within it,
and brighter a diiection along it.
Oui denition of timbie does not exclude pitch, and we contend that it is not possible to
do so foi example pitch and biightness aie so inteidependent as to be insepaiable. When
pitch scales aie used, auention is shifed away fiom the continuous timbial quality of pitch
and towaids disciete ielationships between notes. Tis does not mean that mental imageiy
of pitch height is ignoied, but iathei that the balance shifs in some measuie fiom analogue,
timbial movement to stiuctuies in a disciete domain. Te development of sta notation in
the Westein tiadition has de-emphasised the oldei oial tiadition of vocable woids, distancing
music fiom the body (I.z). Tis distance can be consideied a fieedom, allowing us to exploie
and contemplate the multi-dimensional complexity of disciete tonality. Howevei we contend
that what makes a melody aecting is its integration of both disciete notes and analogue timbial
e
Cu~v1iv I Wovbs
expeiience. Te same can of couise be said foi integiation between disciete pauein and timbial
movement in Tabla playing.
lntegiation between the continuous and disciete in music is the noimal case, but it is pos-
sible to peiceive timbie without associated disciete coding. loi example an elongated dione,
with smoothly uctuating timbie, does not have discontinuities that would evoke disciete seg-
mentation. Likewise, if an event is iepeated in peifoimance with liule vaiiation, such as a
iepetitive kick dium, then it may be auended to piimaiily as a disciete pauein, but still, its
pulse is likely to be peiceived and undeistood ielative to sounds aiound it in spatial teims, in
ielation to its low position on the pitch dimension.
3.3.5 Timbral analogies
lf timbie is a pioduct of movement in mental imageiy, then we might expect oiientational
metaphoi (z.z.e) to play a laige pait in its stiuctuie. Tis is what the notion of timbral analogy
amounts to, the idea that timbial dieiences aie peiceived as distances, and ielated to one
anothei as such in peiception. Wessel (1vv) applied a peiceptual space identied thiough MDS
(discussed in I.I.z) to test the existence of timbial analogy and found encouiaging iesults,
subjects would tend to choose the D in the foim A is to B as C is to D that iesults in the
distance AB being closest to CD. A similai expeiiment is given by McAdams (1vvv), which
also showed some suppoit foi the timbial analogy hypothesis, howevei also showed stiong
individual dieiences and iiiegulaiities. As Patel (zcc, pp. Ic-I1) points out, this indicates that
unlike tonality, music of timbie does not easily suppoit inteivals as a shaied categoiy system.
Howevei while McAdams (1vvv) does nd individual dieiences, electioacoustic musicians
made judgements moie consistent with one anothei, suggesting that this could just be a mauei
of cultuial exposuie. lt would be inteiesting theiefoie to apply similai expeiimental design to
the peiception of timbial analogies between the paiameteis of bass line synthesiseis (such as
the Roland MC-zcz, I.I.z), and compaie the iesults between those of vaiying exposuie to acid
house music.
On anothei level, we may question whethei shaied categoiy systems aie necessaiily im-
poitant to music. As we have aigued, music can be giounded and cieatively inteipieted in
the dance of the listenei, and so the job of the composei oi impiovisei is in such cases not to
encode a musical message to be unambiguously decoded. Rathei, it is to sculpt music with
stiuctuies of inteiest, that may be inteipieted accoiding to analogies dynamically constiucted
by the listenei, music that sounds dieient on each listen, being fiamed by the paiticulai state
of the listeneis mind and body.

Cu~v1iv I Wovbs
3.3.6 Sound source modelling
ln ielating mental imageiy to action spaces, oui denition of timbie implies that sound souice
modelling is the basis of sound peiception. An expanded foim of this asseition is that thiough
listening to a seiies of sounds we build a model of aspects of a physical object and its excitation,
and how it ieacts to actions to pioduce sounds. Tis model is then used to stiuctuie peiception
of the sounds which gave iise to it.
Te idea that we peiceive a sequence of events ielative to a model built fiom those events
may seem fai-fetched, but such dynamic ielationships aie fai fiom alien to music psychology.
loi example it is widely accepted that undeilying metie is infeiied fiom ihythm, and that the
ihythm is then peiceived ielative to the metie (London, zcc1). Tis is paiticulaily explicit in
lndian classical music, wheie the metiic t al is a clapping pauein infeiied fiom the ihythm that
is based aiound it (Clayton, zccs). Tis can happen within the time fiame of the psychological
piesent, so that the peiception of metie and ihythm aie unied, appeaiing to inuence one
anothei simultaneously. lt would seem that the same is tiue of timbie, humans aie able to
identify instiuments within a mixed auditoiy scene eoitlessly (Biegman, 1vv1), and so at least
some souice modelling is at play. Te questions aie, to what extent does this ability contiibute
to timbie peiception, and to what extent aie neuial motoi ciicuits involved`
Some studies have found coiielations between decits in music peiception and of spatial
ability, with Douglas and Bilkey (zcc) making the aigument that amusia is pait of a geneial
decit in spatial ability, suppoiting the assumption that music peiception is a foim of spatial
peiception. Such ndings aie contioveisial, and dicult to inteipiet, among othei issues it
is uncleai whethei spatial ability impioves music peiception, oi listening to music impioves
spatial ability (Stewait and Walsh, zcc), although we would suggest a thiid option that both
cases aie tiue. Tat theie is some ielation seems cleai, foi example Cupchik (zcc1) nds that
mental iotation of guial diawings (afei the classic mental imageiy woik discussed in z.z.1)
piedict ability to discein analogous musical peimutations. lt is inteiesting howevei that this
debate in the liteiatuie is made solely on the basis of pitch peiception. lf, as we suggest, music
is peiceived simultaneously in teims of disciete events and analogue movements, and pitch
tends towaids the foimei use, then peihaps moie conclusive iesults would be found if widei
timbial aspects of music weie examined with these expeiimental designs.
3.3.7 Universality of Timbre
Music is populaily desciibed as a univeisal language, the implication being that music is not
cultuially specic, and does not need to be leained. Teie is suppoit foi some inteipietations
of this in the liteiatuie, liitz et al. (zccv) found that listeneis in Mafa, Cameioon judge the
s
Cu~v1iv I Wovbs
emotion of Westein pieces of music in agieement with Westein listeneis, at above chance
levels. We may imagine how the Mafa achieved this feat, despite lacking piioi exposuie to
Westein cultuie. Emotion is dened by liitz et al. somewhat coaisely as being eithei happy,
sad oi scaiy, to which they iepoit cleai coiielations with basic musical featuies, foi example
happy music has a fastei tempo and sad a slowei tempo. Howevei these univeisals aie
not necessaiily musical in natuie. lt is univeisal human behavioui foi happy people to move
fastei than sad people, due to physical eects on motoi contiol. While it is unsuipiising that
Westein musicians choose to take advantage of this in theii woik, it is quite a stietch to make
the bioad claimthat music is a univeisal language on this basis, oi even that music has univeisal
components. lt is equivalent to obseiving that when people aie sad, they speak slowei, and
aiguing on this basis that speech is a univeisal language. While humans aie univeisally diiven
to make music, as they aie diiven to speak, the iesult is many musics, not one.
So, naive listeneis may be able to pick up iefeiences to mood in Westein music, thiough
ceitain coaise univeisals of an analogue code. Oveiall though, music is dened within a paitic-
ulai human cultuie. Tiaditional dance is a visible aspect of this, not just accompanying music
but an intiinsic pait of the musical expeiience (I.I.z). lndeed we agiee with the analysis of
Westein musiing by Small (1vvs), that music is an activity that encompasses not only play-
ing instiuments and dancing, but also conceit ticket sales, diessing up, the hubbub in conceit
hall atiia piioi to the conceit, the cleaiing of thioats, and the stiiding on stage of the conduc-
toi. Music is a cultuial activity in the deepest sense, and by taking an exteinal peispective on
an otheiwise familiai Westein activity, Small shows what stiange cieatuies we aie. lt is no
suipiise then that Westein ethnomusicologists in Afiica have found music giounded in move-
ment, embedded in cultuie, and not focused on sound, this is the geneial case acioss cultuies
(Agawu, zccI, Ch. ), including in Westein music.
I.1 Aiticulation
Having dened timbie piimaiily as peiceived aiticulation, we examine the aiticulation of
woids as disciete symbol sequences. Tis will complete the context iequiied foi the piacti-
cal connection between timbie and computation that concludes this chaptei.
We have alieady examined the use of symbols to notate positions of aiticulatois in the vo-
cal tiact, as well as the ielated issues of embodiment and mental imageiy thiough chaptei z.
Aiticulation of woids is in theoiy a simple case of moving fiomone position to anothei, but the
timing and phiasing of aiticulation foims an additional, analogue channel of piosodic commu-
nication. luitheimoie physical constiaints lead to inteiactions between points of aiticulation
v
Cu~v1iv I Wovbs
known as co-aiticulation, a challenge foi computational speech iecognition.
3.4.1 Rhythm in Spee
We saw Patel (zcc) diaw a stiuctuial connection between music of the tabla and speech in
I.z. Patel goes fuithei in the same volume with ieseaich compaiing timing in speech with
music peifoimance. ln speech ieseaich, a dichotomy was long accepted dividing languages
into stiess-timed and syllable-timed languages. lt was geneially accepted that stiess-timed lan-
guages tend towaids equal duiation between stiessed syllables, and syllable-timed languages
tend towaids equal duiation between eveiy syllable (isochiony). Howevei, ciacks appeaied in
this theoiy when agieement could not be met on which languages weie which. When comput-
eis allowed measuiements unaected by hidden bias, it became cleai that the stiess/syllable
timing dichotomy had no basis in the cold ieality of the sound signal no languages have
isochionous timing.
Te distinction between stiess- and syllable-timing was howevei saved, oi peihaps ie-
placed, by Ling et al. (zccc), thiough intioduction of the noimalised Paiiwise Vaiiability lndex
(nPVl). Tis piovided a measuie foi quantifying something that was intuitively felt, but falsely
auiibuted to stiess/syllable timing. As the name suggests, nPVl measuies the degiee of con-
tiast between successive duiations in an uueiance (Patel, zcc, p. 1I1), with a highei nPVl
indicating gieatei contiast between neighbouiing duiations, and a peiception of stiess iathei
than syllable timing. ln his seaich foi coiiespondences between music and language, Patel
applied the nPVl to music. Unlike speech, much music is isochionous, but by applying nPVl
to both speech and music, Patel found that ielationships could be found in the non-peiiodic
stiuctuies of both. ln paiticulai, that cultuies with spoken languages with a highei nPVl had
a style of playing music that also had a highei nPVl. Because notes aie noimally played much
fastei than syllables aie spoken, this is an inconclusive iesult, but does suppoit a univeisal
link between musical and piosodic expiession. luitheimoie, in ceitain cases this link is made
explicit and undeniable, as we will see in the following section.
3.4.2 Sound poetry
Sound poetiy is an aitfoim closely ielated to the use of vocable woids, in that speech is used
foi its timbial quality iathei than as a linguistic medium. Teie is no iefeience to anothei
instiument, instead auention is tuined solely on the expiessive abilities of the vocal tiact itself.
Among the most celebiated sound poems is the Uisonate by Schwiueis (1vIz), with themes
intioduced, exploied and expounded upon thiough foui movements ovei twenty nine pages.
Only non-lexical woids aie used, although the woik includes instiuctions to iecite the poem
ec
Cu~v1iv I Wovbs
with Geiman intonation.
Te piactice of sound poetiy has ielated foims in music. loi example the ltalian composei
Luciano Beiios Sequenza III is a vocal piece featuiing muueiings, clicks and shouts of the
female voice, notated with a unique systemof symbols including vocables. Te manipulation of
the voice is taken to dieient extiemes in human beatboxing, which emeiged fiom the Hip Hop
genie. Beatboxeis employ extended vocal techniques to pioduce convincing impeisonations
of dium machines and bass lines (Stowell, zccs). Beatbox ihythms may be notated with a
system of jelled vocables called standard beatbox notation (Tyte, zccs), wheie wiiuen syllables
iepiesent the dieient sounds. Although the notation is vocable, beatboxeis geneially intend
to pioduce sounds that do not suggest a vocal souice. Beatboxeis anecdotally iepoit that theii
hand gestuies help them do this, imagining instiuments in a spatial aiiangement in fiont of
them, and moving theii hand to nd the dieient timbies of theii voice, a mapping fiom space
to aiticulation.
3.4.3 Words in Music Tenology
We have seen that vocable woids aie used acioss diveise music cultuies. Howevei at ceitain
points, music cultuie has embiaced technological advance changing how music is conceived.
Tis includes the development of sta notation, a foim of musical liteiacy, changing the way
music is composed, taught and distiibuted. Moie iecently, electionic and computational tech-
nology has piovided iadically newways of desciibing musical sounds, paiticulaily thiough un-
deistanding of the fiequency domain and the development of new analogue and digital sound
synthesis methods. ln all these cases the iesult has been new music activity signalling a move
away fiom oial tiadition. ln the case of sta notation vocable woids have been ieplaced with
notation focused on pitches, and in the case of synthesis the tiansmission of music has moved
to tape, disc and computei netwoiks. ln the case of electionic dance music, human movements
aie ofen factoied out thiough quantisation, noimalising input data so that events fall into a
piecise, coaise giid. lf the composei feels that the iesult is too iobotic, human-like aiticula-
tions aie then synthesised, adding subtle time phiasing to events to give human feel thiough
peifoimance iules (e.g. liibeig et al., zcce). We can say that foi some populai computei music
inteifaces, contiol ovei the kind of expiession that we have compaied to piosody is abstiacted
away.
Although in some cases technology has ieplaced use cases foi vocables, in otheis it has
bluiied the distinction between vocable woids and the instiumental sounds they evoke. lndeed
it has evei been thus, the Jews haip, one of the woilds oldest instiuments, locates a ieed
inside the mouth, which is twanged while the mouth is aiticulated, allowing the musician to
e1
Cu~v1iv I Wovbs
cieate timbial expiessions based on vowel foimants. A moie iecent technological counteipait
to the Jews haip is the vocodei, which when applied to speech imposes aiticulations of the
vocal tiact upon anothei sound souice, thus allowing any instiument to speak. We may also
considei Microphone (z.s) in this light. ln such cases a spoken vocable woid is not a symbolic
iepiesentation signifying anothei sound, but is the sound itself.
Computei music technology has employed a wide iange of human aiticulations in the spec-
ication of sound. Many alteinate means of aiticulating a sound to a computei system have
been developed, with many hundieds of examples in the annals of the Newlnteifaces foi Music
lxpiession confeience. Gestuies of the hands aie sonied to shape timbie, equivalent to the
vocal aiticulation of woids, especially when we considei sign languages of the Deaf (z.).
Te voice is a iecuiient theme in the contiol of new music inteifaces, haidly suipiising as
the vocal tiact evolved to suppoit aiticulations of gieat complexity, foi the puipose of com-
munication (Boei, zc1c). We discussed the bistability of auditoiy and speech peiception in
I.1 in ielation to sine-wave speech, and this bistability has itself been tieated as a musical
medium. Jones (1vvc) desciibes how he leads the listenei to peiceiving non-speech as speech,
and vice-veisa, thiough timbial manipulation with the CHANT sofwaie (Rodet et al., 1vs1).
Speech synthesis has also featuied as a souice of musical timbie in electionic dance music, foi
example the laigely unintelligible singing synthesis sofwaie wiiuen by Chiis Jes foi use in
his compositions undei the Cylob monikei (Jes, zcc).
Peihaps a cential pioblem undeilying new music inteiface ieseaich is that composeis need
to desciibe sounds in ways which have some connection to the outside woild. Reecting on the
above, we piopose that one answei is to build models of human peiception into sofwaie, so
that the sofwaie may plausibly ielate sounds to expeiience, as Wessel (1vv) auempted with
some success using MDS (I.I.1). An alteinative appioach of building physical models, and
contiolling them with disciete symbol systems analogous to phonetics, is intioduced in the
following section.
I. Vocable synthesis
Vocables aie woids which musicians use to iepiesent instiumental aiticulations (I.z). Vocable
synthesis then is the use of physical modelling sound synthesis in this piocess, to automatically
iendei woids into timbie.
Pioneeiing ieseaich into voice-contiol of synthesiseis with vocable woids is intioduced by
Janei (zccs), wheie syllables aie sung into a miciophone and the iesulting sound signal ana-
lysed and mapped to instiumental paiameteis. Tis woik is inspiied by the tiadition of scat
ez
Cu~v1iv I Wovbs
singing vocables in Jazz cultuie. Scat syllables aie not jelled but impiovisatoiy (I.z), and pei-
haps as a iesult Janei is inteiested not in whi syllables aie sung, but how they aie sung. As
such he uses a digital computei to tianslate fiom one analogue foim to anothei, a similai ap-
pioach to the Acid Sketching and Miciophone piojects intioduced in chaptei z, the undeilying
piocess is digital, but in teims of inputs and outputs it is peiceived as a continuous mapping.
Janeis appioach allows iealtime use, wheie a singeis syllables aie detected, analysed and ie-
synthesised on-line. Te iesult is a system which is veiy easy foi singeis to leain hands-on,
thiough use.
Te appioach to vocable synthesis intioduced heie dieis fiom that of Janei, in that we
considei woids in wiiuen iathei than spoken foim. Tat is, we use woids foi theii disciete
phonetics iathei than analogue piosody, so that we may tianslate fiom a disciete foim to a
continuous one. Te motivation foi taking this appioach is to piovide a iepiesentation of
timbie foi live codeis (e.s), computei musicians who notate music in the digital domain undei
tight time constiaints. Such musicians aie ofen diiven to specify laige synthesis giaphs to
desciibe synthetic timbie, and so the piomise of instead desciibing complex timbies with shoit
woids should be veiy auiactive.
loi vocable synthesis to be usable, it should include a coheient mapping, with a peiceivable
connection between symbols and sound categoiies. Tis is facilitated thiough the aiticulation
of a computei model of an analogue sound souice. Te use of physical modelling synthesis
piomises that even a naive listenei can peiceive time vaiiance of peiceived audio featuies as
physical movement. Te musician then desciibes aiticulation with symbols, which the listenei
expeiiences thiough the music of timbie. Te aim is that foi the musician, the expeiience of
using vocable synthesis should feel as natuial as using onomatopoeia in spoken woids.
3.5.1 Babble - vocable Karplus-Strong synthesis
An eaily implementation of vocable synthesis was intioduced by the piesent authoi (McLean,
zcc), and since adapted into the aitwoik Babble, commissioned in zccs by the Ainolni galleiy
in Biistol, accessible at http://project.arnolfini.org.uk/babble/. Tis aitwoik was
inspiied by sound poetiy (I.1.z), in which speech is used as stiuctuied sound within a poetic
composition, iathei than as a linguistic medium. Te aim foi this was to biing the auention of
paiticipants to the paialinguistic connection between symbols and sound.
ln Babble, the consonants and vowels map to dieient aspects of a physical model. Tis is
analogous to a system of phonetics, but is not intended as an appioximation of the phonetics of
a natuial language. ln paiticulai, this is not intended to be a system foi speech synthesis, but
iathei a system foi speech-like composition of sounds. Similai iesults could have been found
eI
Cu~v1iv I Wovbs
by taking a speech synthesisei and bieaking it, by adjusting the paiameteis just outside the
point wheie the sound could be peiceived as speech. Te advantage of the Babble howevei is
that it uses a computationally simple model, and is theiefoie able to iun in a web biowsei.
Te physical model used in Babble is Karplus-Strong synthesis, a tiivial model which pio-
duces suipiisingly iealistic synthesis of stiings, by simulating wave piopagation using a ciicu-
lai sample buei (Kaiplus and Stiong, 1vsI). lach vowel coiiesponds to a size of sample buei,
analogous to the length of a physical stiing. Asecond paiametei to the model is the piobability
of sample values being inveited, contiolling how apeiiodic, oi percussive the iesults aie. lach
consonant coiiesponds to a dieient value of this paiametei. Tat vowels and consonants con-
tiol dieient aspects of the model is analogous to the oiganisation of the lnteinational Phonetic
Alphabet (Ladefoged, 1vvc), wheie vowels ielate to mouth shape and pulmonic consonants to
the place and mannei of aiticulation (z.). ln addition, to give the sound a speech-like quality,
a foimant ltei is applied to the synthesis, which is also contiolled by the vowels.
Te woids typed into the online inteiface weie iecoided, although paiticipants weie able to
opt-out with a piivacy mode. liomthe iesulting logs ovei thiee yeais of use it would seemthat
the woik has been successful in biinging paiticipants to auend to musical iathei than lexical
featuies of theii woids. Sessions ofen begun with meaningful passages, but quickly tuined to
nonsense syllables, such as the following shoit exceipt, shown with time-stamps fiom the log
le
[2009-12-05 07:20:29] this really makes some interesting noises
[2009-12-05 07:20:55] ko doo ko doo - ko doo ko doooiiiT!
[2009-12-05 07:21:07] ko doo doo - ko doo ko doooiiiT!
[2009-12-05 07:25:14] ntdedvoxi hso - - lbpxuerohzyi - - jolkui
While Babble is successful as an aitwoik, and shows piomise as an appioach of timbial
contiol, the iange of expiession by the two paiameteis of the Kaiplus-Stiong model is too
limited to meet the needs of a musician. loi this puipose, vocable synthesis was ie-applied to
a moie complex physical model.
3.5.2 Mesh - vocable waveguide synthesis
Te Mesh vocable synthesis systemis an extension of Babble, towaids a gieatei iange of timbie.
lt uses waveguide synthesis (Van Duyne and Smith, 1vvI) which is inspiied by Kaiplus-Stiong
synthesis, but uses bi-diiectional delay lines. Tese waveguides aie connected togethei to sim-
ulate stiings, tubes, suifaces and volumes of aibitiaiy complexity, within whatevei the cuiient
limits of on-line computation aie. Mesh models a dium head as a two dimensional mesh of
waveguides, using a tiiangulai geometiy foi maximal accuiacy (lontana and Rocchesso, 1vv).
e1
Cu~v1iv I Wovbs
Te dium head is excited thiough inteiaction with a simulated diumstick, using a mass-spiing
model developed by Laiid (zcc1). Te dium head has paiameteis to contiol the tension and
dampening of the suiface, and the diumstick has paiameteis to contiol its stiness and mass.
Te diumstick is thiown against the dium head with paiameteis contiolling the downward
velocity, starting x/y position and the angle and velocity of travel acioss the dium skin.
Table 3.1: Mapping of consonants to mallet property (columns) and movement relative to drumhead (rows).
heavy sti heavy sof light sti light sof
acioss q i y s
inwaid c m f w
outwaid k n v z
edge x d t b
middle j g p h/l
Table 3.2: Mapping of vowels to drum head tension (columns) and dampening (rows).
tense loose
wet i u
a
diy e o
As with Babble, vocable woids foi Mesh aie composed fiom the ze leueis of the modein
lnglish alphabet. Te consonants map to the diumstick and movement paiameteis, and vowels
to the dium head paiameteis, shown in Tables I.1 and I.z.
As an example, considei the following aiticulation
Hit a loose, dampened dium outwaids with a heavy sti mallet, then hit the
middle of the dium with a lightei mallet while tightening the skin slightly and
nally hit the edge of the skin with the same light mallet while loosening and
ieleasing the dampening.
Tis above may be expiessed with the single vocable woid
kopatu
Polymetiic vocable ihythms may be desciibed using syntax inspiied by the Bol Piocessoi
sofwaie (Bel, zcc1). Tis was latei iewiiuen foi use in the stiing paisei of the Tidal pauein
DSL (Domain Specic Language), desciibed in detail in 1..1.
e
Cu~v1iv I Wovbs
3.5.3 Vocable manipulation and analysis
Oui systemis a disciete iepiesentation, but is dened in ielation to simulated continuous move-
ment. Tis means that we may musically manipulate vocable woids as eithei sequences of
disciete symbols oi as continuous movements.
ln the disciete domain, we have a wide iange of techniques fiomcomputei science available
to us. loi example we may analyse sequences of vocables using statistical techniques such as
Maikov models. Such an appioach to modelling stiuctuies of vocable ihythms in oidei to
geneiate ihythmic continuations was intioduced in eailiei woik (McLean, zcc). We may also
use standaid text manipulation techniques such as regular expressions (iegexes). Regexes aie
wiiuen in concise and exible language allowing geneial puipose iule-based stiing matching
(liiedl, zcce). A iegex paisei is embedded in Mesh, allowing opeiations such as the following
~%3=0 /[aeiou]/to/ fe be
Tis ieplaces the vowels of eveiy thiid vocable with the stiing to, iesulting in the follow-
ing sequence
fto be fe bto fe be
Mesh vocables aie diiect mappings to the simulated physical space of a dium and its ai-
ticulation. lt is theiefoie stiaightfoiwaid to opeiate in a simulated analogue domain, in oidei
to peifoim geometiical analyses and manipulations. loi example combining vocables in poly-
phonic synthesis is stiaightfoiwaid, and implemented in oui cuiient system as follows. As
consonants contiol the movement and mallet mateiial, we allow two consonants to be synthe-
sised concuiiently simply by using multiple mallets in oui model. Cuiiently we allow up to
ve active mallets pei dium, allowing ve consonants to be aiticulated at the same time. As
vowels contiol the piopeities of a single dium head, we combine them simply by taking the
mean aveiage of the values they map to.
We may exploit both symbolic and geometiic vocable iepiesentations in one opeiation. loi
example we could estimate the peiceptual similaiity of two vocable woids of dieient lengths
with an appioach similai to the symbolic Levenshtein edit distance (Levenshtein, 1vee), with
edits weighted by phonemic similaiity in the simulated analogue domain.
I.e Discussion
ln this chaptei we have studied woids as sequences of symbols which notate disciete points of
aiticulation, and theiefoie analogue movements between them, in wiiting, speech and song.
ee
Cu~v1iv I Wovbs
loi humans, woids evoke movement, a biidge between disciete and analogue modes that lies at
the heait of oui expeiience. We have dened musical timbie in teims of imagined movement,
and noted the use of vocable woids in desciibing movements within musical tiadition. While
noting that woids aie geneially tieated as shapeless tokens in computei piogiamming, we have
pioposed a use of woids in notating timbie within computei piogiams, allowing tieatment of
timbie thiough both digital and (simulated) analogue manipulations.
e
Cu~v1iv 1
Language
Tus fai we have been occupied with issues of symbolism and movement, taking an embodied
view of disciete computation integiated with analogue expeiience. ln the piesent chaptei we
will examine the highei oidei conceins of the design of piogiamming languages foi computei
music. liistly, we will tackle the issue of what we mean by the woid language in this context,
by ielating natuial and computei languages. We will then appioach the meaning of mean-
ing in computei music, this time by ielating natuial language with music. We then examine
notions of abstiaction in piogiamming language design, by contiasting impeiative and declai-
ative appioaches. Tis will lead into the intioduction of Tidal, a language designed foi the live
impiovisation of musical pauein.
1.1 Natuial and Computei Language
Compaiing natuial and computei language biings up thoiny issues. How does wiiting a pio-
giam compaie to wiiting a poem does it even make sense to speak of these activities in the
same teims` Tey can at least in an aits context, computei language poetics has been a iunning
theme in sofwaie aits discouise (Cox et al., zccc, zcc1). lndeed computei piogiams weie ist
conceived in teims of weaving (1.1), and peihaps the same is tiue of wiiting, as the woid text
is a dead metaphoi foi cloth
An ancient metaphoi thought is a thiead, and the iaconteui is a spinnei of
yains but the tiue stoiytellei, the poet, is a weavei. Te sciibes made this old
and audible abstiaction into a new and visible fact. Afei long piactice, theii woik
took on such an even, exible textuie that they called the wiiuen page a textus,
which means cloth. (Biinghuist, zcc1, p. z)
We can also ielate natuial and computei languages in a moie scientic context. Teie is
a gieat deal of family iesemblance between piogiamming and natuial languages, in syntax
es
Cu~v1iv 1 L~NcU~ci
iules, semantics, woids, punctuation, and so on. Teii similaiity is also ieected in statistical
measuies, foi example the woids of natuial and piogiamming languages both confoim to a
Zipan distiibution, and compaiable long iange powei lawcoiielation (Kokol and Kokol, 1vve).
Te Chomsky hieiaichy oiganises language giammais accoiding to theii expiessive powei,
in teims of iecuision in pioduction iules (Chomsky, 1ve). Te giammais foi piogiamming lan-
guages aie context-free, wheie pioduction iules specify a single symbol with a coiiesponding
stiing of symbols which may iecuisively include nonterminal symbols with theii own pioduc-
tion iules. Te giammai iules foi natuial languages iemain an active aiea of ieseaich, but aie
also consideied to be almost entiiely context fiee (Pullum and Gazdai, 1vsz). ln these syntac-
tical teims then, both piogiamming and natuial languages have the same expiessive powei.
Howevei language is not dened by its syntax alone, but also its semantics and piagmatics.
ln 1.z we will examine this issue fiom the viewpoint of cognitive semantics, which stiongly
de-emphasises the iole of syntax in language.
Anothei test of expiessive equivalence between languages is translatability, howevei this
is somewhat pioblematic as natuial languages have close ties with theii modes of expiession.
loi example, a joke may be told in Biitish Sign Language (BSL) that is untianslatable to lnglish,
if the joke included iefeience to similaiities between signs. Despite the existence of untians-
latable phiases both BSL and lnglish aie natuial languages with equivalent expiessive powei
(Suuon-Spence and Woll, 1vvv). luitheimoie the same diculty of tianslation may occui when
taking a spoken joke and tiying to wiite it down using the same language ofen jokes aie all
in timing, intonation oi sound symbolism that is lost on the page. Te question then is not
whethei it is possible to nd phiases which aie not tianslatable, but to what extent phiases
geneially aie tianslatable.
Leaving natuial languages aside foi the moment, tianslating between computei languages
is ceitainly possible, and may be done eithei liteially oi idiomatically. We may tianslate a
piogiamfiomC to Peil liteially, by tianslating each contiol owand data stiuctuie, pieseiving
the stiuctuie of the oiiginal piogiam. Alteinatively we may adapt a C piogiamto the idioms of
Peil, by using Peils syntax foi data stiuctuies and stiing handling. Howevei because Peil is a
highei level language the inveise of tianslating Peil code to C is moie dicult, as C lacks many
of Peils language featuies. Likewise, it is faiily stiaightfoiwaid to tianslate fiompiogiamming
languages to natuial languages, and indeed this is what we do when explaining howsouice code
opeiates to anothei piogiammei. Tianslating fiom natuial languages to computei languages
is howevei much moie challenging.
When Joana Caida sciatched the giound with the elm bianch all the dogs of
Ceibeie began to baik, thiowing the inhabitants into panic and teiioi, because
ev
Cu~v1iv 1 L~NcU~ci
fiom time immemoiial it was believed that, when these canine animals that had
always been silent staited baiking, the entiie univeise was neaiing its end. (Saia-
mago, zccc, p. 1)
Te above quote stands as the ist sentence of the novel e Stone Ra by Jos Saiamago,
Nobel Lauieate foi Liteiatuie. lt was oiiginally wiiuen in Poituguese, the above is taken fiom
the authoiised lnglish tianslation. ln all his novels, Saiamago limited his punctuation only
to the comma and full stop, which allowed him to compose long sentences constiucted as
paueins of sound, with distinctive ihythm (Saiamago, zccc, tianslatois note). He constiucted
his sentences as a continuous ow, expeiimenting with timbie and iesonance, tieating woids
as sounds. Te tianslatois gieat challenge then is to tianslate the music of this text as well as
the linguistic content. ln this case it seems that this dicult task has been achieved to a laige
degiee, as Saiamago appioved this tianslation.
We nd Petei Naui, the Tuiing Awaid winning computei scientist, to be in agieement with
Saiamago in consideiing language a piimaiily spoken foim. He makes this point by tiansciib-
ing one of his talks as it was spoken, and without punctuation, this time using only em dashes
to indicate pauses. Te following is an exceipt
Te othei notion well language that is when one talks and if one says
this one will immediately be taken to task he has not gone to school doesnt he
know language cannot be a when shame it must be a thing suiely and we
aie thus tangled up into the claim that theie aie concepts denoted by the woids
(Naui, 1vvza, p. z1).
Te similaiity between this and Saiamagos wiiting style is stiiking, although Nauis tian-
sciiption is at times iathei haidei to iead, with iepetitions and tiansgiessions included.
1
But
this diculty is Nauis point, to demonstiate the dieience between spoken language and wiit-
ten text. As he tiies to get acioss in the above quote, woids do not denote piecise, exteinal
concepts, which aie iathei held by individuals as mental imageiy. ln this he is in agieement
with the point of view of cognitive linguists such as Baisalou, Gidenfois, Lako and Paivio
ielated in chaptei z. Language is not a thing that is wiiuen down, but iathei a habit oi activity
that is done, meaning is peisonal, and not containable within giammai iules. Naui asseits that
in oidei to ieally undeistand a speakeis woids, you must absoib theii language on a numbei of
topics, so that you may begin to undeistand theii fiames of iefeience, and pick up the paueins
behind what they aie tiying to say.
Naui (1vvzb) applies all this to oui undeistanding of piogiamming languages in his papei
Piogiamming Languages aie not Languages Why Piogiamming Language is a Misleading
1
As an added complication, Nauis oiiginal talk was in Danish, although he asseits his tianslation is useful in
iepiesenting a talk that might have happened.
c
Cu~v1iv 1 L~NcU~ci
Designation., this time in conventionally wiiuen lnglish. His conclusion is that Piogiamming
Language is a special, limited pait of the linguistic possibilities, delibeiately designed to catei
foi ceitain limited situations and puiposes. liomthis we undeistand that while Naui nds that
the teim programming language is misleading, he still believes that piogiamming language is a
ceitain, specialised aiea of language. Tis has echoes of Wiugenstein on foimalised language
Dont let it bothei you that languages (z) and (s) consist only of oideis. lf
you want to say that they aie theiefoie incomplete, ask youiself whethei oui own
language is complete whethei it was so befoie the symbolism of chemistiy and
the notation of the innitesimal calculus weie incoipoiated in to it, foi these aie,
so to speak, subuibs of oui language. (And howmany houses oi stieets does it take
befoie a town begins to be a town`) Oui language can be iegaided as an ancient
city a maze of liule stieets and squaies, of old and new houses, of houses with
extensions fiom vaiious peiiods, and all this suiiounded by a multitude of new
subuibs with stiaight and iegulai stieets and unifoim houses. Wiugenstein (zccv,
1s)
lt seems that even though theie aie ways in which piogiamming languages aie not lan-
guages, they at least opeiate within the landscape of language. How then, could Saiamagos
text, in its speech-like foim, be tianslated into the language subuib of souice code` Tianslit-
eiation is haidly possible, and so we must look to tianslate paiticulai aspects of the situation
iepiesented in the text. One appioach would be to model disciete entities and ielationships in
the text, of the town, its dogs, and the dependency of the existence of the town on the dogs
behavioui. A human ieadei would then get some sense of what is wiiuen in this piogiamming
language eithei by ieading it, oi peihaps by using it within a laigei computei piogiam. Altei-
natively we could tiy to captuie a sense of expectation and diead within a tempoial stiuctuie,
and output it as sound, theieby cieating a musical theme. Tis kind of tianslation is analogous
to the live coding of sound tiacks ielating the naiiative stiuctuie of silent lms, piactised by
Rohihubei et al. (zcc).
We aigue then that tianslation fiomnatuial language to piogiamming language is possible,
as long as one accepts that languages with stiengths designed foi paiticulai puiposes also have
paiticulai limitations. loi example, piogiamming languages aie designed to be typed and not
spoken, and so biinging the spoken piosody of language (I.1) diiectly to computei language
seems impossible (although see vocable synthesis, I.). Howevei, the same is tiue of some
natuial languages. Nicaiaguan Sign Language is a paiticulaily inteiesting case, emeiging nat-
uially fiom a Deaf community of young school childien ovei two decades a span compaiable
with the oveiall development of computei language. lt has no spoken oi indeed wiiuen foim,
but does alieady have complex, spatial giammai, and of couise piosodic ihythm, being based
on movements of the body.
1
Cu~v1iv 1 L~NcU~ci
Despite the lack of spoken foim, code does have its own spatial featuies, as we will see
in latei discussion of secondaiy notation (.1). A piogiammei also has choice about how to
expiess an algoiithm, to a degiee depending on the language in hand. A stiong example is Peil,
which has the iathei unwieldy mouo Teie is Moie Tan One Way To Do lt (TMTOWTDl).
lndeed Laiiy Wall, the cieatoi of Peil, has a backgiound in linguistics, and lists seveial featuies
that Peil boiiows fiom natuial language in his 1vv on-line essay Natuial Language Piinciples
in Peil. As well as the fieedom of TMTOWTDl, it includes Peils mannei of leaining, its
ambiguity, impoit of featuies fiom othei languages, topicalisation and pionominalisation.
Peihaps the key dieience between piogiamming and natuial language is that the foimei
is foimalised and abstiact, wheieas the lauei has developed with a closei ielationship to its
speakeis. Woids in a natuial lexicon aie giounded in human expeiience of movement and ie-
lationships of the body in its enviionment. Computei languages aie not based aiound these
natuially developed woids, but we may still maintain the same semantic iefeiences foi human
ieadeis, by using those woids in the secondaiy notation of function and vaiiable names, oi
even by woiking with an encoded lexicon as data (such as WoidNet, lellbaum, 1vvs). ln do-
ing so we boiiow fiom the lexicon of a natuial language, but we could just have easily used
an invented lexicon such as that of lspeianto. Regaidless, a computei piogiam is ultimately
giounded in the outside woild when it is executed, thiough whatevei modalities its actuatois
allow, usually images, sound and/oi movement. At the point which a piogiam is executed, it
becomes cleai that the its souice code is full of a paiticulai kind of linguistic iefeience known
as performative uerances. Due to the powei that humans wield ovei computeis to do theii
bidding, by desciibing an action in a computei language, we cause it to be peifoimed.
Natuial and computei languages aie developed undei dieient piessuies with veiy dif-
feient iesults. Howevei they have sucient family iesemblance to both be consideied to be
aspects of human language activity. By appiaising the dieiences between natuial and com-
putei languages, we may look foi ways in which featuies of natuial languages could be ap-
piopiiated foi use in piogiamming languages foi the aits, an appioach we have demonstiated
with Vocable Synthesis (I.).
1.z Music, Language and Cognitive Semantics
We have alieady seen some of the complex ielationship between music and language, while
examining integiation between the dual codes of language and mental imageiy in chaptei z, and
aiguing in chaptei I that timbial and piosodic aiticulation shaie a giounding in movement. Te
Chomskian linguistic notion of semantics as it is geneially undeistood excludes any notion of
z
Cu~v1iv 1 L~NcU~ci
musical meaning. Te hieiaichical stiuctuie of music holds some similaiity to linguistic syntax,
but the lack of iefeience to a ieal oi imagined woild makes a semantics of music untenable
(Wiggins, 1vvs). As a iesult, discussion of meaning in music ofen lacks a foimal undeipinning,
iesulting in a bioad spectium of paiallel discussions in the liteiatuie, each undei its own teims
(foi a bioad ieview see Cioss and Tolbeit, zccs). Howevei the alteinative view of conceptual
semantics piovided by the theoiy of Conceptual Spaces (z.z.) puts things in a iathei dieient
light. ln the following we show that unlike Chomskian semantics, Gaidenfoisian conceptual
semantics is applicable to music as well as language, by summaiising its main tenets in ielation
to music.
Semantic elements are constructed from geometrical or topological structures (not sym-
bols that can be composed according to some system of rules). ln othei woids, semantic
meaning is piimaiy to the conceptual, analogue level, and not the disciete level as with Chom-
skian semantics. Te common view in music theoiy chaiacteiises musical stiuctuie as being
disciete and syntactic (Leidahl and Jackendo, 1vsI), howevei conceptual semantics allow us
to considei music stiuctuie as spatial and geometiic in addition. ln the case of music of timbie
(I.I.I), we claim that conceptual semantics is the piimaiy stiuctuie.
Semantic meaning is a conceptual structure in a cognitive system. Meaning does not ex-
ist thiough links to the woild (oi a possible woild), but in the body (and in paiticulai, the biain),
of an individual. Howevei the conceptual stiuctuies of a gioup of individuals may ieach ac-
coidance thiough communication. We can considei a musical impiovisation in teims of such
a piocess of communication, wheie two oi moie impioviseis begin with individual concep-
tual stiuctuies at the beginning of a piece, which aie manipulated towaids accoidance and
discoidance duiing a peifoimance. Of couise while situated in an individual, the conceptual
stiuctuies aie infoimed by pievious peifoimances and highei cultuial eects such as musi-
cal genie. lt is also possible, thiough abeiiation oi infeience, that new conceptual stiuctuie
is cieated duiing an impiovisation that did not exist at the beginning (Wiggins, zcceb). Such
conceptual stiuctuie could be deemed valuable and kept foi ieuse in futuie impiovisations. ln
such a case we can say that an impiovisation cieated new meaning, and was theiefoie a pai-
ticulaily creative peifoimance. Tis mannei of cieative seaich is discussed in gieatei detail in
e.I.
Conceptual structures are embodied (meaning is not independent of perception or of bodily
experience). Tis tenet connects cognitive semantics to its ioots in theoiies of embodied
cognition. lnstiuments and the voice iequiie movements of the body in oidei to make sound,
I
Cu~v1iv 1 L~NcU~ci
and the constiaints and vagaiies of motoi action within the tight feedback loop of action and
ieaction aie an impoitant component of musical impiovisation (Piessing, 1vs1, 1vs). Howevei
the inuence of bodily expeiience goes beyond actual motoi action to suggest that semantic
meaning is dependent on motoi and peiceptual ciicuits in the biain.
Cognitive models are primarily image-sematic (not propositional). Image-semas are
transformed by metaphoric and metonymic operations (whi are treated as exceptional
features on the traditional view). lmage schemata aie abstiact diagiams of spatial ie-
lationships and actions, iepiesenting notions such as ovei, containment and auiaction
(Lako, 1vv). A metaphoiical opeiation is wheie two concepts aie ielated via common image
schemata, most commonly ielative to oiientations such as UP (z.z.e). lt is thiough metaphoi-
ical stiuctuie that a conceptual system can be giounded in peiception and action, yet iepiesent
meaning abstiact fiom it.
Semantics is primary to syntax and partly determines it (syntax cannot be described in-
dependently of semantics). Tis is anothei tenet in opposition to the widely held view of
Chomskian linguistics, wheie syntax is piimaiy and independent of semantics. lt implies that
when composing a piece of music, the cognitive semantic stiuctuie is moie impoitant than
giammatical iules. Tat is, any giammatical iules undeilying a piece of music aie placed in
suppoit of the geometiy of the semantic stiuctuie, iathei than a piecuisoi foi it (loith et al.,
zc1c).
Concepts show prototype eects (instead of following the Aristotelian paradigm based on
necessary and sucient conditions). Te Aiistotelian paiadigm has not been taken seii-
ously foi seveial decades and cuiient theoiies of concepts do not depend upon it (Muiphy,
zccz, p.1e). Te subsciipt to this tenet theiefoie is weakened by not showing consideiation foi
theoiies competing with the piototype view such as those of the exemplai view and knowl-
edge appioach (Muiphy, zccz, pp. 111). Howevei piototype eects, such as a iobin being
judged a moie typical biid than a penguin, aie indeed easily accounted foi within the theoiy of
conceptual spaces. Gaidenfois does so using the Voionoi diagiam (Okabe et al., zccc), wheie a
conceptual piototype is a Voionoi geneiatoi foi the geometiical iegions of conceptual piopei-
ties. Piototype eects aie obseived in music, foi example wheie pieces aie judged as gieatei
oi less typical examples of a musical genie. ln the Voionoi diagiam of genies, a typical piece
would be neai to the iegions geneiatoi, and a dicult to dene oi ciossovei piece would be
neai a boundaiy between two oi moie genies. ln piactice, musical genies aie impossible to
dene univeisally, which points again to the ielativist position stated in the ist tenet.
1
Cu~v1iv 1 L~NcU~ci
ln summaiy, in Chomskian teims music cannot be undeistood in teims of semantics, only
syntax. Howevei if one is piepaied to take a Gaidenfoisian view, a discussion of musical
meaning can pioceed with a foimal undeipinning, wheie meaning exists within individuals
conceptual stiuctuies of music, within the stiuctuies shaied by the membeis of a music cultuie,
and within the giounding ielationships between musical stiuctuie and univeisals of human
peiception and movement.
1.I Declaiative vs lmpeiative
Declaiative and impeiative piogiamming aie competing paiadigms, an opposition that iaises
inteiesting issues foi the time-based computei aits. Declaiative piogiamming is the coding
of what should happen, and impeiative piogiamming is the coding of how something should
occui. Tis distinction is fiequently used, and gives a sense of what is meant, but lacks piactical
detail. Tis situation is ieminiscent of that of the denition of timbie in music I.I, an impoitant
distinguishing featuie within a domain, that is ofen dened in vague and sometimes conicting
teims.
Te use of computei language can be bioadly divided into the desciiption of algoiithmic
contiol, and of pioblemlogic (Kowalski, 1vv). lmpeiative piogiamming languages suppoit the
foimei, wheie piogiammeis desciibe a method foi solving a pioblem, iathei that the pioblem
itself. Declaiative piogiamming languages suppoit the lauei, allowing piogiammeis to focus
on the desciiption of pioblems, by leaving the inteipietei to nd the algoiithmic solution. Well-
known examples of declaiative piogiamming languages include the logic language Piolog, the
database queiy language SQL, and the stiing matching language Regulai lxpiessions (ch. z).
Tese aie as close as we get to the declaiative piomise, of piogiammeis conceining themselves
only with what they want to do, and not how it should be done. Tis piomise holds foi simple
examples, but unfoitunately haidly at all in piactice. SQL database queiy optimiseis aie in
constant development, and piogiammeis must keep abieast of exact implementation details,
stiuctuiing theii queiies and indexing theii datasets in a way that allows theii SQL queiies to
opeiate eciently. Upgiading to a new veision of an SQL engine is then a seiious mauei, as
optimisations foi geneial cases may have led to seiious iegiessions in edge cases. Likewise, ieg-
ulai expiessions must be caiefully ciafed foi a paiticulai inteipietei, wheie implementation
details such as backtiacking oi deteiminism can impact computational complexity by seveial
oideis of magnitude (liiedl, zcce). Piolog piogiammeis have much the same pioblem, classi-
cally iequiiing manual seaich space ieduction by inhibiting backtiacking, a technique known
as the cut (Steiling and Shapiio, 1vv1).

Cu~v1iv 1 L~NcU~ci
While the impeiative fiequently intiudes, declaiative piogiamming styles nonetheless
maintain a ceitain claiity of expiession, giving high level desciiptions laigely sepaiate fiom
implementation details. Tis is ielated to the idea of puiity in pure functional languages such
as ML (Milnei et al., 1vvc), wheie functions have no side-eects. Tis means a puie functional
piogiam cannot inteiact with the outside woild while it is executing, beyond the singulai, oi-
deily ow of input and output. Haskell is a puie functional piogiamming language fiom the
ML family, but gets aiound this limitation thiough modelling side-eects by chaining togethei
puie functions. Te iesult is a declaiative desciiption of an impeiative piogiam, which the
inteipietei then takes caie of executing.
Anothei distinction made between declaiative and impeiative piogiamming (e.g. Dijkstia,
1vs) is conceined with the passing of time. ln declaiative piogiamming, time is a concein of
optimisation, to be sepaiated as much as possible fiom the pioblem desciiption. ln impeiative
piogiamming one statement follows anothei, desciibing a sequence of opeiations, each with
its own time cost. ln the geneial case this makes a gieat deal of sense, the impeiative how
appioach is conceined with algoiithms as piogiammes of woik ovei time, and the declaiative
what appioach with logical ielationships abstiact fiom time. Howevei within oui theme of
the design of piogiamming languages foi aitists, the focus on time is pioblematic. ln paiticulai,
when we considei the time-based aits, the unfolding of an algoiithm ovei time is both the
impeiative how and the declaiative what. ln this case, the distinction between declaiative and
impeiative piogiamming appeais to not apply, as the paiticulai opeiation of how an algoiithm
woiks has stiong inuence ovei what we expeiience. lf the distinction is to make any sense in
this context, it must be dened caiefully.
We aigue that a declaiative appioach to piogiamming is indeed desiiable foi aitists. We
dene it howevei not in geneial teims, oi even ielative to algoiithms, but as the closeness
of mapping (.1) of the piogiamming notation to the taiget domain. lf a piogiammei is only
inteiested in logic, then a declaiative appioach is to choose a piogiamming language that takes
caie of how a piogiam is solved ovei time. Howevei if oui piogiammei is inteiested in the
pioblem of how to complete a piece of music, then they aie conceined with how the solution
is aiiived at as well as the pioblem, the how is the what. A declaiative appioach in this case
would then be conceined with how events aie oideied to ieach a musical iesolution. Tis
does not howevei mean piecisely specifying an algoiithm, ceitain aspects of opeiation may be
musically inconsequential, and so may be specied in an ambiguous mannei.
Oui chaiacteiisation of declaiative piogiamming is closely ielated to the concept of Do-
main Specic Language (DSL) (van Deuisen et al., zccc). To a limited extent, a stiaightfoiwaid
piogiamming libiaiy is DSL, in that it piovides functions paiticulai to a domain. Howevei in
e
Cu~v1iv 1 L~NcU~ci
a fullei sense, DSL goes fuithei to piovide highei oidei means of abstiaction and combina-
tion tailoied to the taiget domain, allowing piogiammeis to nd the level of abstiaction that
best miiiois the stiuctuie of theii pioblem. loi example, a computei musician woiking with
paueins may cieate language foi combining functions which iepiesent geneiation and tians-
foimation of pauein. We will appioach this subject in depth in the following section, befoie
intioducing Tidal, DSL foi live coding of pauein.
1.1 Domain Specic Language foi Pauein
When we view the composed sequence abcabcabc. . . we quickly infei the pauein iepeat
abc. Tis infeience of hieiaichy is known in the psychological liteiatuie as chunking (Millei,
1ve), and aids memoiy of long sequences, piediction of futuie values and iecognition of ob-
jects. Pauein peivades the aits, as Alfied Whitehead (zcc1) eloquently puts it, Ait is the
imposing of a pauein on expeiience, and oui aesthetic enjoyment is iecognition of the pat-
tein.
z
Tis communicates a iole of pauein suppoited heie, one individual encodes a pauein
and anothei decodes it, both actively engaged with the woik while cieating theii own expe-
iience. ln the piesent section we examine the encoding of pauein in paiticulai, intioducing
Tidal, a computei language foi encoding musical paueins duiing impiovised live coding (e.s)
peifoimances.
Pauein has been of gieat inteiest thioughout the histoiy of ait. Te paueined walls and
oois of the Alhambia in Spain aie an extiaoidinaiy example, wheie Mooiish aitists have cap-
tuied all seventeen types of symmetiy, centuiies befoie theii foimalisation by gioup theoiists
(du Sautoy, zccs). Te technological exploiation of musical pauein also has a long histoiy,
extending back to well befoie electionic computeis. loi example, Leonaido da Vinci invented
a huidy guidy with movable pegs to encode a sequence, played back using multiple ieeds at
adjustable positions, tiansfoiming the sequence into a canon (Spiegel, 1vs). Hieiaichies of
iepeating stiuctuie iun thioughout much of music theoiy, computational appioaches to music
analysis, indexing and composition all have focus on disciete musical events and the iules to
which they confoim (Rowe, zcc1, 1.z). liom this we infei that the encoding and decoding of
pauein is fundamental to music making. We ieview suppoit given to musical pauein making
by computei language in this light.
Te liteiatuie on pauein DSLs (Domain Specic Languages) is mainly conceined with
analysis of composed woiks ielative to a paiticulai theoiy of music. loi example Simon and
Sumnei (1vvz) piopose a foimal language foi music analysis, consisting of a minimal giammai
z
To oui shame, these woids weie backgiound to Whitehead lambasting those taking quotes out of context.

Cu~v1iv 1 L~NcU~ci
foi desciibing phiase stiuctuie within peiiodic paueins. Teii language allows foi multidi-
mensional paueins, wheie dieient aspects such as note value, onset and duiation may be
expiessed togethei. Te giammai is based on a language used foi desciiption of aptitude tests
which tieat pauein induction as a coiielate with intelligence. Deutsch and leioe (1vs1) in-
tioduced a similai pauein DSL to that of Simon and Sumneis, foi the analysis of hieiaichical
ielationships in tonal music with iefeience to gestalt theoiy of peiception (Kohlei, 1vIc).
Te analytical peispective shown in the pauein DSLs discussed thus fai puts focus on sim-
ple paueins with unambiguous inteipietation. We asseit that music composition demands
complex paueins with many possible inteipietations, leading to challenged, engaged listeneis
in a state of ow (see analytic listening, Csikszentmihalyi, zccs, p. 111, and oui discussion
on timbie, I.I.I). Teiefoie pauein DSL foi synthesis of music iequiies an appioach dieient
fiomfoimal analysis. Motivation foi the design of pauein DSL foi music composition is identi-
ed by Lauiie Spiegel in hei papei Manipulations of Musical Paueins (Spiegel, 1vs1). Twelve
classes of pauein tiansfoimation, taken fiom Spiegels own intiospection as a composei aie
detailed tiansposition (tianslation by value), ieveisal (value inveision oi time ieveisal), iota-
tion (cycle time phase), phase oset (ielative iotation, e.g. a canon), iescaling (of time oi value),
inteipolation (adding midpoints and oinamentation), extiapolation (continuation), fiagmenta-
tion (bieaking up of an established pauein), substitution (against expectation), combination (by
value mixing/counteipoint/haimony), sequencing (by time editing) and iepetition. Spiegel
felt these to be tiied and tiue basic opeiations, which should be included in computei music
editois alongside inseit, delete and seaich-and-ieplace. luithei, Spiegel pioposed that study-
ing these tiansfoimations could aid oui undeistanding of the tempoial foims shaied by music
and expeiimental lm, including human peiception of them.
Pauein tiansfoimations aie evident in Spiegels own Music Mouse sofwaie, and can also
be seen in music sofwaie based on the tiaditional studio iecoiding paiadigm such as Stein-
beig Cubase and Apple Logic Studio. Howevei Spiegel is a stiong advocate foi the iole of
the musician-piogiammei, and expiesses hope that these pauein tiansfoimations would be
foimalised into piogiamming libiaiies. Such libiaiies have indeed since emeiged. Hieiai-
chical Music Specication Language (HMSL) developed in the 1vscs, and includes an exten-
sible fiamewoik foi algoiithmic composition, with some inbuilt pauein tiansfoimations. Te
Scheme based Common Music enviionment, developed fiom 1vsv, contains an object oiiented
pauein libiaiy (Taube, zcc1), classes aie piovided foi pauein tiansfoimations such as peimu-
tation, iotation and iandom selection, and foi pauein geneiation such as Maikov models, state
tiansition and iewiite iules. Te SupeiCollidei computei music language (McCaitney, zccz)
also includes an extensive pauein libiaiy, beneting fiom an active fiee sofwaie development
s
Cu~v1iv 1 L~NcU~ci
community, and with advanced suppoit foi live coding (e.s). What all these systems have in
common is a desiie to iepiesent the stiuctuie of music in the stiuctuie of code. As such, they
adheie closely to oui denition of declaiative language (1.I), woiking at the same level of ab-
stiaction of the taiget domain of musical pauein. lnspiied by this, we now move to intioduce
oui own pauein DSL, Tidal.
1. Tidal
Tidal is a pauein DSL embedded in the Haskell piogiamming language, consisting of pauein
iepiesentation, a libiaiy of pauein geneiatois and combinatois, an event schedulei and pio-
giammeis live coding inteiface. Tis is an extensive ie-wiite of eailiei woik intioduced undei
the woiking title of Petrol (McLean and Wiggins, zc1c). lxtensions include impioved pauein
iepiesentation and fully conguiable integiation with the Open Sound Contiol (OSC, lieed
and Schmedei, zccv) piotocol.
4.5.1 Features
Befoie examining Tidal in detail we ist chaiacteiise it in teims of featuies expected of a pauein
DSL.
Host language Tidal is a Domain Specic Language embedded in the Haskell piogiamming
language. Te choice of Haskell allows us to use its poweiful type system, but also foices us to
woik within stiict constiaints biought by its static types and puie functions. We can howevei
tuin this stiictness to oui advantage, thiough use of Haskells pioneeiing type-level constiucts
such as applicative functois and monads. Once the notion of a pauein is dened in teims of
these constiucts the expiessive powei of Haskells syntax becomes available, which can then
be exploied foi application in desciibing musical pauein. Haskells syntax is veiy teise, thanks
in pait to its declaiative appioach (1.I) and notational conveniences. loi example in Haskell
all functions aie implicitly curried, that is all Haskell functions only take a single aigument,
but may ietuin anothei function that takes a fuithei aigument and so on. Tis allows pai-
tial application to be veiy teisely expiessed, foi example we may specialise the function +
to + 1 , and map the iesulting function ovei a list of values without explicit use of lambda,
map (+1) [1, 2, 3] . Such teisity iemoves baiiieis fiom the expiession of ideas, and theie-
foie allows a tightei cieative feedback loop (e.I).
Pattern composition ln Tidal, paueins may be composed of numeious sub-paueins in a vaii-
ety of ways and to aibitiaiy hieiaichical depth, to pioduce complex wholes fiom simple paits.
v
Cu~v1iv 1 L~NcU~ci
Tis could include concatenating paueins time-wise, meiging them so that they co-occui, oi
peifoiming paiiwise opeiations acioss paueins, foi example combining two numeiical paueins
by multiplying theii values togethei. Composition may be heteiaichical, wheie sub-pauein
tiansfoimations aie applied at moie than one level of depth within a laigei pauein.
Time representation Time can be conceptualised eithei as linear ange with forward order
of succession, oi as a iepeating cycle wheie the end is also the beginning of the next iepetition
(Buzsaki, zcce). We can ielate the foimei to the lauei by noting that the phase plane of a sine
wave is a ciicle, a sine wave piogiesses ovei lineai time, but its oscillation is a iepeating cycle
as shown in liguie .1. Music exhibits this tempoial duality too, having iepeating ihythmic
stiuctuies that nonetheless piogiess lineaily. Tidal allows both peiiodic and lineai paueins to
be iepiesented.
Anothei impoitant distinction is between disciete and continuous time. ln music tiadi-
tion, time may be notated within disciete symbols, such as Westein sta notation oi lndian
Bol syllables, but peifoimed with subtle phiasing ovei continuous time. Tidal maintains this
duality, wheie paueins aie events at disciete time steps. Howevei phiasing may be specied
as paueins of oating point onset time deltas (1.e.1).
Random access Both Common Music and SupeiCollidei iepiesent paueins using lazily eval-
uated lists, wheie values aie calculated one at a time as needed, iathei than all togethei when
the list is dened. Tis allows long, peihaps innitely long lists to be iepiesented eciently in
memoiy as geneiatoi functions, useful foi iepiesenting fiactal paueins foi example. ln some
languages, including Haskell, lists aie lazily evaluated by default, without need foi special syn-
tax. Tis is not how paueins aie iepiesented in Tidal howevei. Lazy lists aie piactical foi
lineai evaluation, but you cannot evaluate the 1ccth value without ist evaluating the ist vv.
Tis is a paiticulai pioblem foi live coding (e.s), if you weie to change the denition of a lazy
list, in oidei to continue wheie you lef o you must iegeneiate the entiie peifoimance up
to the cuiient time position.
I
luithei, it is much moie computationally expensive to peifoim
opeiations ovei a whole pauein without iandom access, even in the case of stiaightfoiwaid
ieveisal.
Tidal allows foi iandom access by iepiesenting a pauein not as a list of events but as a
function fiom time values to events. A full desciiption is given in 1..z.
I
SupeiCollidei suppoits live coding of paueins using PaueinPioxies (Rohihubei et al., zcc). Tese act as place-
holdeis within a pauein, allowing a piogiammei to dene sub-paueins which may be modied latei.
sc
Cu~v1iv 1 L~NcU~ci
Ready-made generators and transforms A pauein libiaiy should contain a iange of basic
pauein geneiatois and tiansfoims, which may be stiaightfoiwaidly composed into complex
stiuctuies. lt may also contain moie complex tiansfoims, oi else have a community iepositoiy
wheie such paueins may be shaied. Tidal contains a iange of these, some of which aie inspiied
by othei pauein languages, and otheis fiom Haskells standaid libiaiy of functions, including
its geneial suppoit foi manipulating collections.
Community
Computeisie biinging about a situation thats like the invention of haimony.
Sub-ioutines aie like choids. No one would think of keeping a choid to himself.
Youd give it to anyone who wanted it. Youd welcome alteiations of it. Sub-
ioutines aie alteied by a single punch. Weie geuing music made by man himself
not just one man. John Cage (1vev)
John Cages vision has not univeisally met with ieality, much music sofwaie is piopiietaiy,
and in the United States sound synthesis algoiithms aie impeded by sofwaie patents. Howevei
computei music languages aie judged by theii communities, shaiing code and ideas fieely,
paiticulaily aiound languages ieleased as fiee sofwaie themselves. A pauein DSL then should
make shaiing abstiact musical ideas stiaightfoiwaid, so shoit snippets of code may be easily
used, modied and passed on. Tis is ceitainly possible with Tidal, although this is a young
language which has not yet had a community giow aiound it.
4.5.2 Representation
ln Tidal, paueins aie iepiesented by the Pattern a data type, which is dened as follows
data Pattern a =
Pattern {at :: Behaviour a, period :: Period}
type Behaviour a = Int [Maybe a]
type Period = Maybe Int
Tis Pattern a data type is composed of two fuithei types, Behaviour a which is a type
synonym foi Int [Maybe a] , and Period which is a type synonym foi Maybe Int . Tey
aie accessible via theii eld names at and period iespectively. Te behaviour iepiesents the
stiuctuie of the pauein, and the period how ofen it iepeats, if at all.
Te name of the Behaviour a type is taken fiomfunctional ieactive piogiamming nomen-
clatuie (llliou, zccv, Hudak, zccc), a behaviour being the teim foi a time-vaiying value. lts
foim Int [Maybe a] is a function fiom Int (integei), to a list of values of type Maybe a .
s1
Cu~v1iv 1 L~NcU~ci
Te integei function paiametei iepiesents disciete time steps, and the iesult of the function,
[Maybe a] iepiesents a list of events. ln othei woids, a behavioui iepiesents a pauein as a
time-vaiying list of events.
Pattern and Behaviour shaie the polymoiphic type paiametei a , which is encapsulated
within Int [Maybe a] . Te type paiametei a is specialised to whatevei the event type of a
paiticulai pauein is, foi example a pauein of musical notes could be of type Pattern String ,
wheie pitch labels aie iepiesented as chaiactei stiings, oi alteinatively of type Pattern Int
foi a pauein of MlDl numbeied note events.
We have not yet explained the puipose of the Maybe type which encapsulates a . Te
Maybe type is a standaid featuie of Haskell, and has two constiuctois, namely Just a and
Nothing . Te ieason foi using it heie is to allow events which do not have a value in teims
of a to be iepiesented as Nothing . Tis is paiticulaily useful foi iepiesenting musical rests.
luithei motivation is shown in 1.e, wheie Nothing is shown to have dieient meaning in
dieient situations.
As we see in the above denition, the period of a pauein the duiation at which it iepeats
is iepiesented as an Int . lt is also encapsulated within the Maybe type, used foi spec-
ifying apeiiodic paueins. Te peiiodic pauein abcdefgh, iepeated would have a Period
of Just 8 , and the apeiiodic pauein a followed by iepeating bs would have a Period of
Nothing .
Te following example shows how a pauein may be constiucted, in this case the integei
pauein of the iepeating sequence c, z, 1, e
p = Pattern {at = n [Just ((n `mod` 4) 2)],
period = Just 4}
To map fiom the time paiametei to this tiivial sequence, the Behaviour simply takes the
modulo of foui, and multiplies it by two.
We access events in a pauein by evaluating its Behaviour function with a time value.
Continuing with the pauein dened above, at p 1 evaluates to [Just 2] . As this is a cyclic
pauein of peiiod 1, at p 5 would give the same iesult, as would at p (-3) .
Te above pauein is expiessed as a function ovei time. An alteinative, iecuisive denition
would be moie idiomatic to Haskell, in this case dening at p 0 to ietuin Just [0] and
subsequent at p n to ietuin the value at n - 1 plus two. Howevei gieat caie must be taken
when intioducing such dependencies between time steps, it is easy to pioduce incomputable
paueins, oi as in this case, paueins which may iequiie whole paueins to be computed to nd
values at a single time point.
sz
Cu~v1iv 1 L~NcU~ci
4.5.3 Pattern generators
A pauein would not noimally be desciibed by diiectly invoking the constiuctoi in the iathei
long-winded mannei shown in the pievious section, but by calling one of the pauein geneiating
functions piovided by Tidal. Tese consist of geneiatois of basic iepeating foims analogous
to sine, squaie and tiiangle waves, and a paisei of complex sequences. Te following is the
denition of Tidals sine function, which pioduces a sine cycle of oating point numbeis in
the iange 1 to 1, with a given peiiod
sine :: Int Pattern Double
sine l = Pattern f (Just l)
where f n = [Just ( sin (
fromIntegral n (pi / fromIntegral l 2)
))
]
Te sine1 function does the same as sine but in the iange fiom 0 to 1. lt is dened
ielative to sine , by using the functoi map opeiatoi <$> . Te values of a pauein aie mapped
ovei a scaling function in the following denition
sine1 :: Int Pattern Double
sine1 l = ((/ 2.0) (+ 1.0)) <$> sine l
We can use this functoi mapping because Pattern is an instance of Haskells Functor
class, with the following denition
instance Functor Pattern where
fmap f (Pattern xs p) =
Pattern (fmap (fmap (fmap f)) xs) p
Tis denition simply maps ovei each of Behaviour s enclosing types. Tis is all that
needs to be done, because Haskell alieady denes the function application, Maybe and list
types as instances of the Functor class.
Te following is an example of the sine1 function in use, heie iendeied as a sequence of
giey values by the drawGray function
drawGray $ sine1 16
sI
Cu~v1iv 1 L~NcU~ci
Tidal is designed foi use in live music impiovisation, but is also applicable to o-line com-
position, oi foi non musical domains. We take this oppoitunity to illustiate the following
examples with visual paueins, in sympathy with the piesent medium. loi space eciency the
above cyclic pauein is iendeied as a iow of blocks, but ideally would be iendeied as a ciicle,
as the end of one cycle is also the beginning of the next.
Lineai inteipolation between values, somewhat ielated to musical glissandi, is piovided by
the tween function
drawGray $ tween 0.0 1.0 16
4.5.4 Parsing strings into polymetric patterns
A pauein may also be specied as a stiing, which is paised accoiding to the context, as dened
by Haskells type infeience. ln the pievious two examples we have used the diawGiay function
which iequiies a pauein of oating numbeis, in paiticulai one of type Pattern Double . ln
the following example the draw function iequiies a coloui pauein, and so a paisei of coloui
names is automatically employed by the function p .
draw (p "black blue lightgrey")
Tanks to the stiing oveiloading extension to Haskell piovided by the Glasgow Haskell
Compilei, a stiing is automatically coeiced into a Pauein using the function p . Tis is so that
p does not need to be explicitly specied
draw "black blue lightgrey"
Te paisei allows teise desciiption of polymetiic ihythms, inspiied by the syntax of the Bol
Piocessoi (Bel, zcc1). We will ist desciibe the paisei in detail befoie moving on to examples,
which the ieadei may wish to iefei to in advance.
Te paisei iules aie implemented using Haskells Parsec libiaiy, but aie illustiated in the
following using lxtended Backus-Naui loim (lBNl). We diveige fiom standaid lBNl to pa-
iameteiise iules, in oidei to show that the foui dieient paiseis opeiate in the same mannei,
but with dieiing iules to paise atomic events within ihythms.
s1
Cu~v1iv 1 L~NcU~ci
loui dieient paiseis aie cieated by paiameteiising geneial paisei iules with a iule to
match an atom, i.e. a ihythmic event at the lowest level of gianulaiity
colourRhythm = rhythm(colourname) ;
doubleRhythm = rhythm(double) ;
stringRhythm = rhythm(string) ;
intRhythm = rhythm(int) ;
boolRhythm = rhythm(bool) ;
colourname = "blue" | "red" | "green" | ... ;
bool = "t" | "1" | "f" | "0" ;
Tese paiseis allow Tidal to paise Pattern s of the basic types String , Bool , Int and
Double , as well as Colour . lt is stiaightfoiwaid to dene moie as needed.
A ihythm consists of a sequence of ihythmic paits
rhythm(atom) = whitespace, sequence(atom) ;
sequence(atom) = { part(atom) } ;
whitespace = { " " } ;
A ihythmic pait may consist of a single atom, a iest (denoted by ), a polymetei, oi a
sequence of atoms
part(atom) = atom | "~" | poly(atom) | polypad(atom)
| sequenceParens(atom) ;
sequenceParens(atom) = "(", sequence(atom) , ")" ;
Two dieient methods aie piovided foi combining ihythms into polymeteis, denoted with
eithei squaie oi cuily biackets.
poly(atom) = "[", rhythm(atom), {",", rhythm(atom)}, "]" ;
polypad(atom) = "{", rhythm(atom), {",", rhythm(atom)}, "}" ;
Te poly iule indicates combination of ihythms by a method of repetition and polypad
by padding. ln both cases the iesult is a pauein with the same peiiod, being the lowest common
s
Cu~v1iv 1 L~NcU~ci
multiple of the constituent pauein peiiods. We illustiate theii opeiation by example. liistly,
in combination by iepetition, the ist ihythm is iepeated twice and the second thiice
1
draw "[black blue green, orange red]"
Combining by padding inteileaves the inteinal stiuctuie of a ihythm with iests. ln the
following example the ist pait is padded with one iest eveiy time step, and the second with
two iests eveiy step
draw "{black blue green, orange red}"
ln the above theie aie time steps wheie two events co-occui and the block is split in two,
steps wheie one event takes up the whole block, and steps wheie no events occui, shown by a
blank block.
Polymeteis may be embedded to any depth (note the use of a tilde to denote a iest)
draw "[{black grey, orange}, red green]"
4.5.5 Pattern combinators
Once we have a basic pauein, using eithei a geneiatoi such as sine oi the above paisei, we
can tiansfoim it with functions, each one adding a layei of ihythmic stiuctuie.
lf oui undeilying pauein iepiesentation weie a list, a pauein tiansfoimei would opeiate
diiectly on sequences of events. loi example, we might rotate a pauein one step foiwaid by
popping fiom the end of the list, and unshiing/consing the iesult to the head of the list. ln
Tidal, because a pauein is a function fiomtime to events, a tiansfoimei may manipulate time as
well as events. Accoidingly the Tidal function < foi iotating a pauein to the lef is dened
as
(< ) :: Int Pattern a Pattern a
(< ) p n =
Pattern (t at p (t + n)) (period p)
1
Note that co-occuiiing events aie visualised by the draw function as veitically stacked coloui blocks.
se
Cu~v1iv 1 L~NcU~ci
Rotating to the iight is simply dened as the inveise
( >) :: Int Pattern a Pattern a
( >) p n = p < (0 - n)
Te append function appends one pauein to anothei timewise. lf the ist pauein is ape-
iiodic, then the second pauein is nevei ieached, so the iesult is the ist pauein unchanged.
append :: Pattern a Pattern a Pattern a
append a@(Pattern f Nothing) _ = a
lf the ist pauein is peiiodic, and the second is apeiiodic, then the second is appended to
a single cycle of the ist, and the iesulting pauein is apeiiodic.
append a@(Pattern _ (Just l)) b@(Pattern _ Nothing) =
Pattern newF Nothing
where newF n | n < l = at a n
| otherwise = at b (n - l)
lf both paueins aie peiiodic, then the iesulting pauein alteinates between them, with a
peiiod that is the sum of those of the constituent paueins.
append a@(Pattern f (Just l)) b@(Pattern f' (Just l')) =
Pattern newF (Just newL)
where
newL = l + l'
newF n | cycleP < l = f ((loopN l) + cycleP)
| otherwise = f' ((loopN l') + (cycleP - l))
where cycleP = n `mod` newL
loopN = n `div` newL
Te following illustiates append in use
drawGray $ append (tween 0 1 8) (tween 1 0 8)
Te cat function foi joining togethei a list of paueins is tiivial to dene as a fold ovei
append
cat :: [Pattern a] Pattern a
cat = foldr append nullPattern
s
Cu~v1iv 1 L~NcU~ci
luithei combinatois aie dened foi ieveising a pauein with rev
drawGray $ rev (sine1 8)
Oi alteinatively playing a pauein foiwaids and then in ieveise with palindrome
drawGray $ palindrome (sine1 8)
Te every function allows tiansfoimations to be applied to peiiodic paueins eveiy n
cycles. lts denition simply multiplies the peiiod of the given pauein by n - 1 , then appends
the tiansfoimed pauein to it
every :: Int (Pattern a Pattern a) Pattern a
Pattern a
every 0 _ p = p
every n f p = (p (n - 1)) `append` f p
( ) :: Pattern a Int Pattern a
( ) p n = Pattern (at p) (fmap ( n) (period p))
Te following demonstiates how the every combinatoi may be used to iotate a pauein
by a single step eveiy thiid iepetition
draw $ every 3 (1 >) "black grey red"
Because the Pauein type is dened as a functoi, we may apply a function to eveiy element
of a pauein using the fmap , oi its opeiatoi foim <$> . loi example, we may add some blue
to a whole pauein by mapping the blend function (fiom the Haskell Coloui libiaiy) ovei its
elements
p = every 3 (1 >) "black grey red"
draw $ blend 0.5 blue <$> p
We can also apply the functoi map conditionally, foi example to lighten the pauein eveiy
thiid cycle
ss
Cu~v1iv 1 L~NcU~ci
drawGray $
every 3 ((+ 0.6) <$>) "0.2 0.3 0 0.4"
lf we weie doing something similai to a sound iathei than coloui event, we might undei-
stand it as a musical tiansposition.
As Haskell is a functional language, it is possible to have highei oidei paueins, that
is paueins of functions. loi example, the following would iesult in a pauein of type
Pattern (Colour Colour) , a pauein of coloi blends alteinating between ied and blue
(blend 0.5) <$> "red blue"
We can haidly visualise a pauein of functions, but such paueins aie of use, as we shall see
shoitly.
Haskell has a supeiclass of functoi called the applicative functoi, which denes the <>
opeiatoi, allowing us to apply functions inside Paueins to othei values.

Tidal paueins have


the following instance denition
instance Applicative Pattern where
pure x = Pattern (pure (pure (pure x))) (Just 1)
Pattern fs pf <> Pattern xs px =
Pattern (liftA2 (zipCycleA2 (<>)) fs xs) (lcd pf px)
-- lowest common duration
lcd :: Period Period Period
lcd Nothing _ = Nothing
lcd _ Nothing = Nothing
lcd (Just n) (Just n') = Just (lcm n n')
Te denition of <> allows a new pauein to be composed by taking a function with
multiple paiameteis, and mapping it ovei combinations of values fiom moie than one pauein.
Te <> opeiatoi is dened foi Paueins so that all events aie used at least once, and no
moie than necessaiy to full this constiaint. loi example, the following gives a polyihythmic
lightening and daikening eect, by blending colouis fiom two paueins
draw $ (blend 0.5) <$> "red blue" <> "white white black"

l am giateful to Ryan lngiam foi his help with this applicative functoi denition.
sv
Cu~v1iv 1 L~NcU~ci
Te Tidal onsets function lteis events, only allowing thiough those which begin a
phiase. Heie we manipulate the onsets of a pauein (blending them with ied), befoie com-
bining them back with the oiiginal pauein.
draw $ combine [blend 0.5 red <$> onsets p, p]
where p = "blue orange [green, pink] red "
Te onsets function is paiticulaily useful in cioss-domain paueining, foi example tak-
ing a pauein of notes and accentuating phiase onsets by making a time onset and/oi velocity
pauein fiom it.
1.e Open Sound Contiol paueins
Tidal has no capability foi sound synthesis itself, but instead iepiesents and schedules paueins
of Open Sound Contiol (OSC, lieed and Schmedei, zccv) messages to be sent to a synthesisei.
Below we see how the shape of an OSC message is desciibed in Tidal
synth = OscShape {path = "/trigger",
params =
[ F "note" Nothing,
F "velocity" (Just 1),
S "wave" (Just "triangle")
],
timestamp = True
}
Tis is a tiivial "/trigger" message consisting of two oating point paiameteis and one
stiing paiametei. lach paiametei may be given a default value in the OscShape , in this case
velocity has a default of 1 , wave has a default of "triangle" and note has no default. Tis
means that if an OSC pauein contains a message without a note value set, theie will be no
value to default to, and so the message will be discaided. Pauein accessois foi each paiametei
aie dened using names given in the OscShape
note = makeF synth "note"
velocity = makeF synth "velocity"
wave = makeS synth "wave"
vc
Cu~v1iv 1 L~NcU~ci
4.6.1 Seduling
As timestamp is set to True in oui OscShape example, one extia pauein accessoi is available
to us, foi onset deltas
onset = makeT synth
Tis allows us to make time paueins, applying subtle (oi if you piefei, unsubtle) expies-
sion. Tis is implemented by wiapping each message in a timestamped OSC bundle. A simple
example is to vaiy onset times by up to c.cz seconds using a sine function
onset $ ( 0.02) <$> sine 16
Te $ opeiatoi does nothing except apply the iight hand side to the function on the lef.
Howevei it has veiy low piecedence, and so is useful foi iemoving the need foi paienthesis in
cases such as this.
lnstances of Tidal can synchionise with each othei (and indeed othei systems) via the Net-
Clock piotocol (http://netclock.slab.org/). NetClock is based upon time synchionisa-
tion in SupeiCollidei (McCaitney, zccz). Tis means that time paueins can notionally schedule
events to occui in the past, up to the SupeiCollidei contiol latency, which has a default of c.z
seconds.
lt is also possible to cieate tempo paueins to globally aect all NetClock clients, foi example
to double the tempo ovei Iz time steps
tempo $ tween 120 240 32
4.6.2 Sending messages
We connect oui OSC pauein to a synthesisei using a stream , passing the netwoik addiess
and poit of the synthesisei, along with the OscShape we dened eailiei
s stream "127.0.0.1" 7770 synth
Tis staits a scheduling thiead foi sending the messages, and ietuins a function foi ieplac-
ing the cuiient pauein in shaied memoiy. Paueins aie composed into an OSC message Pauein
and stieamed to the synthesisei as follows
v1
Cu~v1iv 1 L~NcU~ci
s $ note ("50 62 60 ")
velocity foo
wave "square"
onset (( 0.01) <$> foo)
where foo = sine1 16
Te opeiatoi meiges the thiee paiametei paueins and the onset pauein togethei, into
a single OSC message pauein. Tis is then passed to the stieam s , ieplacing the cuiiently
scheduled pauein. Note that both velocity and onset aie dened in teims of the sepaiately
dened pauein foo .
4.6.3 Use in improvisation
Music impiovisation is made possible in Tidal using the dynamic Glasgow Haskell Compilei
lnteipietei (http://www.haskell.org/ghc/). Tis allows the musician to develop a pauein
ovei successive calls, peihaps modifying the pieceding listing to tianspose the note values
eveiy thiid peiiod, make a polyihythmic pauein of wave shapes, oi combine multiple onset
paueins into a choius eect. Tidal piovides a mode foi the iconic emacs piogiammeis editoi
(http://www.gnu.org/software/emacs/) as a GHCl inteiface, allowing paueins to be live
coded within an advanced developeis enviionment.
e
As well as to music, Tidal has also been applied to the domain of live video animation, in
suppoit of the musician Kiik Degioigio in Antweip, May zc1c. Tis was inspiied by the ait of
coloui play developed by Maiy Hallock-Gieenewalt (z.), and focused on coloui tiansitions
and juxtapositions between colouis. Paueins of coloui tiansitions aie specied in a mannei
analogous to musical events, with something like choids of coloui piovided by spliuing the
display into veitical bais. loi example in oidei to move fiom thiee colouis to two, foui tiansi-
tions aie peifoimed, as the cential coloui is split in two. Paueins may be applied to the thiee
paiameteis of hue, tiansition type (eithei lineai oi sinusoidal) and tiansition speed. loi the
peifoimance, beat tiacking was employed so that the colouis changed in time with the iegulai
pulse of the music. Te peifoimance was not iecoided oi foimally evaluated, but seives as an
example of the applicability of Tidal to the visual domain.
4.6.4 Future directions
We have intioduced Tidal, a language designed foi live coding of musical pauein. Tidal has
alieady been eld tested thiough seveial peifoimances, including to laige audiences at inteina-
tional music festivals, infoiming ongoing development of the system. Although it is designed
e
Piojecting the emacs inteiface as pait of a live coding peifoimance has its own aesthetic, having a paiticulaily
stiong eect on many developeis in the audience, eithei of elation oi ievulsion.
vz
Cu~v1iv 1 L~NcU~ci
foi diiect use, Tidal piovides an ideal base on which to build expeiimental usei inteifaces. loi
example in .e, we will intioduce a visual piogiamming inteiface, with Tidal pioviding the
undeilying pauein DSL.
As noted in 1.I time is in geneial not well suppoited in piogiamming languages. ln Tidal
we have iepiesented time in a function fiom disciete time to events, which woiks well within
electionic dance music genies with minimal deviation fiom a steady pulse. While Tidal does
allowexpiession thiough paueins of time, in a bioadei context howevei its sole focus on sound
event onsets is a huge constiaint on music expiession. lollowing the lunctional Reactive Pio-
giamming liteiatuie to iepiesenting time with ieal oi iational numbeis is compelling. Tis
would be towaids suppoiting iepiesentations of time based not just on uing events on to
giids, but also aiianging sounds ielative to each othei. Tat is, suppoiting smooth as well as
stiiated time (z.1). Tis would diaw upon extensive woik by Bel (zcc1) on constiaint-based
time seuing.
1. Discussion
Language is a cential issue in piogiamming languages foi the aits, and has taken a cential
position in the stiuctuie of the piesent thesis. lt is of couise a bioad subject, within which we
have visited a numbei of sub-themes. We have compaied piogiamming languages with natuial
languages, taking a view fiom Wiugenstein of both existing on the same landscape. We have
taken a cognitive view of semantics and ielated it to spatial ielationships in music, allowing us
to take an unconventional position of ielating meaning in music with that of natuial language.
We have made a distinction between declaiative and impeiative appioaches to piogiamming
that focuses on levels of abstiaction, wheie a declaiative appioach is one that matches the level
of abstiaction of the taiget domain. Tis has nally led us into the subject of linguistic desciip-
tions of paueins of expeiience, and the intioduction of Tidal, a pauein DSL foi impiovising
music. Tidal piovides a system allowing aitists to potentially woik at a level which is abstiact
fiom the suiface of the taiget medium, in this case sound. Tis does not necessaiily distance
them fiom theii woik, but iathei allows them to woik diiectly with the stiuctuie of a piece, at
a highei level of composition.
We diaw these stiands togethei to make the point that piogiamming languages aie human
languages, foi communicating with computeis but also othei piogiammeis and with ouiselves.
Tis will be developed fuithei in the following chapteis, in focusing on the notation of com-
putei piogiams (ch. ) and theii use in cieative piocesses (ch. e). Alieady though the pait that
the design of piogiamming languages can have in human expiession is cleai, in allowing the
vI
Cu~v1iv 1 L~NcU~ci
development of languages that accoid with the stiuctuie of oui taiget medium. Tis piomise
has led many aitists to be hooked by piogiamming, and the widei public peiception of pio-
giamming as a technical endeavoui may at last be lifing as communities of aitist-piogiammeis
giow. liom heie we look foi how piogiamming languages, theii notation and use may be ie-
fiamed in iesponse.
v1
Cu~v1iv
Notation
Building upon oui discussion of the syntax and semantics of language, we now examine pio-
giamming language enviionments as designed usei inteifaces. Study of the notation of com-
putei piogiams biings Applied Psychology to Human-Computei lnteiaction (Blackwell, zcceb),
allowing us to take a human-centiic view of the activity of piogiamming. loimally speaking,
all mainstieam piogiamming languages aie Tuiing-complete, and so have equivalent expies-
sive powei (z.z). Howevei, in piactice the design of piogiamming notation makes ceitain
ways of woiking easy and otheis dicult, oi piactically impossible. We aigue that as the ie-
sult of design piocesses ovei the last fy yeais, computei languages have become specialised
towaids use in industiy to the detiiment of moie fieefoim use. Tese constiaints have become
so embedded in languages and theii use that it has become dicult to see beyond them. Tis is
ieected in the way we desciibe mainstieam languages as general purpose, assumed to be suit-
able foi any puipose, oi in othei woids, none in paiticulai. Howevei we contend that these
languages aie unsuitable foi smallei scale, human-level domain of the aitist-piogiammei.
ln the piesent chaptei we will examine and compaie existent notations using the Cognitive
Dimensions of Notation fiamewoik, biiey ieviewed below. We will focus in paiticulai on the
use of time and space in notations, which we will asseit is of paiticulai concein to the aits. Tis
will lead into the intioduction of Textuie, a visual piogiamming language with novel use of
luclidean distance in the syntax of a puie functional language. Tis will piovide foundations
foi the following chaptei, in which will examine howthe use of piogiamming notations impact
upon cieative piocesses.
.1 Cognitive Dimensions of Notation
Te Cognitive Dimensions of Notation fiamewoik is designed to aid discussion of piogiamming
language design (Blackwell and Gieen, zccz). Rathei than a checklist of good oi bad design,
v
Cu~v1iv No1~1ioN
it desciibes a set of featuies which may be desiiable oi not, and which inteiact with each
othei depending on the context. luithei moie they aie known as dimensions because they
aie not absolutes but scales. An example of a common tiade-o is one between the viscosity
1
dimension, howdicult it is to modify a piogiam, and the piovision foi secondary notation. lf
we change a notation to extend the iole of secondaiy notation, then viscosity will also inciease,
as the secondaiy notation must be changed to match any syntax change. A list of cognitive
dimensions with shoit desciiptions is shown in liguie .1.
Table 5.1: Cognitive Dimensions of Notation, adapted from Chur and Green (2008)
Abstraction Availability of abstiaction mechanisms.
Hidden dependencies lnvisibility of impoitant links between entities.
Premature commitment Constiaints on the oidei of doing things.
Secondary notation Notation othei than foimal syntax.
Viscosity Resistance to change.
Visibility Visibility of components.
Closeness of mapping Closeness of iepiesentation to taiget domain.
Consistency Similai semantics aie expiessed in similai syntactic foims.
Diuseness Veibosity of language.
Error-proneness Likelihood of mistakes.
Hard mental operations Demand on cognitive iesouices.
Progressive evaluation Tempoial ielationship between edits and theii
evaluation.
Provisionality Degiee of commitment to actions oi maiks.
Role-expressiveness lxtent to which the puipose of a component may be
infeiied.
Te cognitive dimensions exhibit many intei-dependencies. Tese tiade-os aie not fully
foimalised, and it is piesumed that they diei depending on the task at hand. lndeed a di-
mension may be desiiable in one context, but undesiiable in anothei. loi example an inciease
in viscosity, making piogiams moie dicult to change, is geneially seen as having negative
impact. Howevei Beckwith and Buineu (zcc1) iepoit that incieasing viscosity of ceitain no-
tational aspects impioved the peifoimance of a class of subjects using theii language.
1
To claiify discussion, we will use a paiticulai typeface to indicate wheie we iefei to a paiticulai dimension, foi
example as with hidden dependencies.
ve
Cu~v1iv No1~1ioN
Te Cognitive Dimensions of Notation aie paiticulaily useful in consideiing the design of
Domain Specic Languages (DSLs), by pioviding standaid teims foi desciibing the paiticulai
demands of a task domain. ln the following we apply it to the notation of time in the domain
of live music impiovisation.
.z Notation in Time
We have alieady seen pioblems that the maich of time piesents to computei language in the
pievious chaptei (1.I). lt can be useful to sepaiate the declaiative what fiom the impeiative
how, but when the behavioui of an algoiithmovei time is impoitant to the aesthetics of a digital
aitwoik, the how is the what. We intioduced Tidal, which illustiates this point well, being a
pauein DSL solely conceined with the notation of events in time (1.).
Designeis of conventional piogiamming languages considei time in teims of eciency, and
as something to be saved oi spent. loi those designing languages foi the time based aits, time
is iathei pait of the stiuctuie of the expeiience that is being iepiesented, a stiatum of both
digital pauein and analogue movement.
Teie is howevei an awkwaidness about the ielationship between the what of notation and
how of the passage of time. On one level they seem sepaiate a declaiative desciiption of a time
stiuctuie is one thing, and an impeiative algoiithm foi accuiately and eciently pioducing
that stiuctuie may be quite anothei. But in piactice they iaiely aie completely sepaiate, the
algoiithm leaves its impiint, paueining its output. On iecognising the glitches aiising fiom
the opeiation of a paiticulai algoiithm, aitists biing them into the music itself (Cascone, zccc).
Time may always be ielied upon to pass, and it passes not only duiing a piogiams inteipie-
tation but duiing the activity of notation itself. Tis is of concein to piogiamming languages
designed foi the aits, wheie the sepaiate timelines of development and execution eectively
sepaiates an aitist fiom theii taiget medium. liguie .1 shows the Piocessing piogiamming
enviionment, which is an adaptation of the geneial puipose Java piogiamming language, and
is designed foi aitists who aie leaining piogiamming. To this end the inteiface has fewdistiac-
tions, and makes iunning a piogiam veiy stiaightfoiwaid the piogiammei simply piesses the
play buuon maiked with an icon familiai fiom music equipment. Nonetheless, the piogiam-
mei must stop wiiting code, piess the buuon and wait a liule while befoie seeing the iesults
of theii woik. Tis is not wholly bad, and may be thought in teims of an aitist taking a step
back fiom theii woik to pause in consideiation. Howevei, such peiiods of ieection should be
taken on an aitists own teims, and not foiced upon them by an exteinal piocess.
Live coding is a movement that has emeiged to investigate the iemoval of the compilation
v
Cu~v1iv No1~1ioN
Figure 5.1: e Processing programming environment. Every time the play buon is pressed, the program
is compiled and executed, with no state preserved between successive runs (unless explicitly dened).
Figure 5.2: e Fluxus programming environment. e editor containing the source code of a program is
embedded in the same 3D scene as its output. e source is dynamically interpreted, so that the program
may be anged during a live coding performance (6.8).
vs
Cu~v1iv No1~1ioN
step in an aits context. We will investigate the cieative piocesses of live coding moie bioadly
in e.s, and so foi now focus on notational aspects, in paiticulai, how the iemoval of the com-
pilation step iesults in the inteitwining of active notation with inteipietation. liguie .z shows
the lluxus inteiface, wheie the code editoi is placed in the same scene as the piocess output.
Teie is no play buuon in lluxus as such, instead the code is continuously playing. lnciemen-
tal changes to the souice code aie ieected in the output whenevei the piogiammei piesses
the l buuon. Te visual foim of the code is moie liteially pait of its own output, the visual
output is a ID scene, of which the text editoi is pait. lt is theiefoie possible to wiite a piogiam
that modies the display of its own code.
How is live coding technically possible` Using the piogiamming enviionments that have
become conventional, aitists cannot geneially expeiience what they make at the same time as
they aie making it. Tis pioblemmay seeminsuimountable, while it iuns, a piogiammaintains
state, its woiking memoiy, the data geneiated and collected duiing its execution. Such data
may come with no desciiption apait fiom its declaiation in piogiam code. lf one then changes
the piogiam code, theie may be no mechanism in which the new veision of the piogiam may
continue wheie the old veision lef o, as the old state may be of no use to the new veision of
the piogiam. Te cognitive dimension of progressive evaluation theiefoie seems dicult to
achieve. Teie aie howevei a numbei of ways in which it can be done, although each case has
signicant impacts on othei cognitive dimensions.
External grounding. loi ceitain task domains, the inteinal state of a piogiam has liule oi no
signicance. Tis is the case in Tidal (1.), wheie just about the only state is the measuiement
of time kept by a sepaiate piocess, the system clock. lt is also the case in languages known as
UNlX system shells. Tese developed fiom ieseaich into conversational programming (Kupka
and Wilsing, 1vsc), wheie piogiammeis inteiact with a language inteipietei one statement at
a time, able to considei systems feedback at each step. Conveisational piogiamming lives on
in shell languages such as Bash, wheie the state is global to the whole computei system, with
access contiol foi secuiity. ln paiticulai, state is almost exclusively held within the lesystem,
a tiee stiuctuie able to hold not only peimanent les on disks, but piocesses, theii enviionment
vaiiables and woiking coie memoiy (lilesystem Hieiaichy Standaid Gioup, zccI). As such a
shell piogiammei tends to woik with live data, making iiieveisible and at times diastic actions,
an impact along the cognitive dimension of premature commitment. Howevei some systems,
including some le systems and ielational databases, implement transactions, allowing a seiies
of successive actions to be iteiatively applied, but not commiued. When live codeis woik
befoie an audience, tuining back peifoimance time is impossible, but being able to ioll back
vv
Cu~v1iv No1~1ioN
development to a pievious veision, foi example foi musical iepiise, is ceitainly useful.
Internal grounding. Tis appioach to progressive evaluation denes state in such a way
that it may be passed fiom one veision of a piogiam to the next. Tis is conceptually similai to
exteinal giounding above, except the state is not shaied with othei piocesses. Te piogiammei
is lef with the hard mental operation of being awaie of ceitain conicts that may aiise, foi
example if a vaiiable with the same name is used foi dieient puiposes in a successive vei-
sion of the code, state will be caiiied acioss to unintended eect. ln a stiictly typed language
such as Haskell, state is handled in a highly foimalised mannei, and so it is possible to aiiange
foi a piogiam to ietuin its entiie state foi coeicion into a foim suitable foi its successive vei-
sion (Stewait and Chakiavaity, zcc). Alteinatively, dynamic sciipting languages such as Peil
make it stiaightfoiwaid to evaluate code that ieplaces existing functions, wheie state may be
pieseived using global vaiiables.
Self-modication. A iathei unusual appioach, developed in the Feedba.pl live coding en-
viionment by the piesent authoi, is to stoie state in the souice code itself (McLean, zcc1). Te
piogiammei is put in the position of wiiting code that stoies, ieads and modies values in its
own desciiption. Tis is done eithei by piogiammatically updating assignment statements, oi
stoiing values as piogiam comments, so that secondary notation is paised and modied by
its own piogiam. Tis cieates an inteiaction wheie the souice code is used to modify both the
instiuctions and state of a piocess, and also to show output fiom its execution.
Temporal recursion. Tis teim was coined by Soiensen (zcc) in desciibing the opeiation of
the music live coding systemlmpiomptu, based aiound the Lisp deiivative Scheme. Aiecuisive
function is one that calls itself, and a tempoial iecuisive one is one that calls itself but with
a scheduled delay. Progressive evaluation is then simply a case of swapping one function foi
anothei between self-calls.
Hybrid. A hybiid appioach is ceitainly possible, exemplied by SupeiCollidei, a music pio-
giamming language with two concuiient piocesses. One piocess maintains a puie functional
giaph foi sound synthesis, while the othei acts as a dynamic language inteipietei. Te lan-
guage piocess manages such tasks as scheduling events and signalling giaph changes of the
synthesis piocess. Te synthesis piocess is laigely stateless, and caie must be taken to avoid
discontinuities when the giaph is changed, audible as clicks. JlTLib has been developed to aid
this, pioviding a means to tiansition audio between successive veisions of code (Rohihubei
et al., zcc).
1cc
Cu~v1iv No1~1ioN
Te above methods all act to embed piogiam development in a timeline shaied with its
execution, with piactical applications being exploied in the aits. We now move fiom the iole
of time in notation, to examine the iole of space.
.I Visual notation
Visual Piogiamming Languages is an active ieseaich eld, looking at how to design piogiam-
ming notations that go beyond conventional lineai text. To head o confusion, note that the
use of visual heie is not the same as populai use, such as in Visual Basic, to desciibe language
enviionments based on GUl foims and event-diiven piogiamming. lnstead, visual program-
ming languages aie those making heavy use of visual elements in the code itself, foi example
wheie a piogiam is notated as a giaphical diagiam.
Reseaich into visual languages is hampeied by a denitional pioblem. ln a well-cited tax-
onomy of visual piogiamming, Myeis (1vvc, p. z) denes visual languages as any system that
allows the usei to specify a piogiam in a two (oi moie) dimensional fashion. Myeis speci-
cally excludes conventional textual languages, because compileis oi inteipieteis piocess them
as long, one-dimensional stieams. Tis exclusion is highly pioblematic howevei, as at base all
a computei can do is piocess one-dimensional stieams. luithei, some textual languages such
as Haskell and Python do indeed have two dimensional syntax, wheie veitical alignment as
well as hoiizontal adjacency is signicant, howevei no-one would call eithei language visual.
Woise still, foi the majoiity of visual languages, zD aiiangement has no syntactical signi-
cance whatsoevei, and is puiely secondary notation. Ofen giaphical icons aie desciibed as
visual, but they too aie disciete symbols, in othei woids textual.
Tis confusion stems fiom two misundeistandings, istly that text is non-visual, and sec-
ondly that visual inteifaces aie necessaiily a technological advance beyond textual inteifaces.
Te foimei point is in denial of the heavy use of spatial layout in stiuctuiing text, and the lat-
tei point makes the same mistake as those assuming analogue expiessions aie moie advanced
than digital ones (z.v). As a species we navigated spaces befoie we maiked them out with
sequences of symbols, and made maiks on suifaces befoie we used those maiks to tiansciibe
woids. lndeed it iuns against implicit hieiaichy in theii iepiesentations, wiiuen woids (ch. I)
aie compiised of symbols (ch. z), notated with images. liomthis peispective, textual inteifaces
aie the moie advanced, as they aie developments of visual inteifaces. Howevei it does not fol-
low that text is supeiioi to visual inteifaces, but iathei that the whole dichotomy of visual and
textual language is false. Language by natuie involves disciete symbols, but may be integiated
with analogue symbols towaids a iichei, dual code of expiession.
1c1
Cu~v1iv No1~1ioN

Figure 5.3: e Pure Data visual programming environment. e locations of the boxes have no signicance,
but allow the programmer free rein in arranging their program in a manner meaningful to themselves.
Image c Miller Puee, used here under the terms of the BSD License.
5.3.1 Pater Languages
Reseaich into visual piogiamming languages has held much piomise, but has so fai had liule in
the way of bioad industiy take up. An intention of many visual language ieseaicheis has been
to nd ways of using visual notation that iesult in new, bioadly supeiioi geneial piogiamming
languages (Blackwell, zccec). Howevei this panacea has not been ieached, and instead ieseaich
aiound the cognitive dimensions of notations (.1), has identied inheient, inescapable tiade-
os. Teie aie howevei two domains wheie visual piogiamming has been highly successful,
one being engineeiing, exemplied by the LabVllW visual piogiamming language. Te othei,
of special ielevance to oui theme, is computei music. ln paiticulai, Pater languages have
become a dominant foice in computei music and inteiactive ait since theii intioduction by
Puckeue (1vss). Using a data ow model inspiied by analogue modulai synthesis, useis of
Patchei languages such as Puie Data oi Max/MSP aie able to build piogiams known as patches
using a visual notation of boxes and wiies. Patches may be manipulated while they aie active, a
foim of live coding piedating the movement examined in e.s by well ovei a decade. Te long-
lived populaiity of Patchei languages, the continued innovation within communities aiound
them, and the aitistic success of theii use is undeniable.
Patchei languages allow piogiammeis fieedom in aiianging theii piogiams, to gieat ad-
vantage, a point we will come to latei. Howevei ist we iaise a point of contention, as with
many visual piogiamming languages, in teims of syntax, they aie haidly visual at all. ln the
Puie Data patch shown in liguie .I, you could move the boxes wheievei you like, oi even
1cz
Cu~v1iv No1~1ioN
place them all on top of one anothei, it would make no dieience to the inteipietei. Tose
familiai with Max/MSP may countei this line of aigument by pointing out that iight-lef oi-
deiing signies evaluation oidei in Max. Tis falls on two counts, ist, Max piogiammeis aie
discouiaged fiom bianching that ielies on this, in favoui of the trigger object. Second, having
iight-lef execution oidei does not distinguish Max fiom any mainstieam textual language. ln-
deed to say that Patchei languages aie not themselves signicantly textual is in blind denial of
the laige numbei of opeiatois and keywoids shown as editable text in a patch.
An advantage of Patchei languages is that they aie unconstiained by visible dimensions,
in that you can place objects anywheie with liule oi no syntactical signicance. ln conven-
tional languages, to ielate two woids togethei, you must put them adjacent to one anothei,
in a paiticulai oidei. ln Patchei languages, you instead connect them with lines, eectively
dening syntax giaphs of aibitiaiy dimension, hypeicubes and up. As opposed to the deni-
tion ielated eailiei, fiom this peispective it is actually textual iathei than visual languages
that aie constiained by the two dimensions of the visual canvas.
Te above is not an auack on Patchei languages syntax is not eveiything. Well, to the
inteipietei syntax is eveiything, but to the piogiammei, who is oui focus, it is only half the
stoiy. Because Patchei languages have such iemaikably fiee secondary notation, they allow
us to lay piogiams out howevei we like, and we may embiace this visual fieedom in making
beautiful patches that thiough shape and foim ielate stiuctuie to a human at a glance, in ways
that linguistic syntax alone cannot do. Tat is why we call these visual languages, while the
language syntax is not visual, the notation is veiy much so.
.1 Notation and Mental lmageiy
All piogiamming languages can be consideied in visual teims, we do afei all noimally use
oui eyes to iead souice code. Piogiamming languages have context-fiee giammais, allowing
iecuisive foims ofen encapsulated within iound biackets, iesulting in a kind of visual lulei
diagiam. We can also say that adjacency is a visual auiibute of giammai, as we know fiom
Gestalt theoiy, adjacency has an impoitant iole in peiceptual giouping (Kohlei, 1vIc). Tese
visual featuies aie used to suppoit ieading of code, wheie oui eyes saccade acioss the scieen,
iecognising disciete symbols in paiallel, chunked into woids (Raynei and Pollatsek, 1vv1). Ciu-
cially howevei, we aie able to auend to both visuospatial and linguistic aspects of a scene si-
multaneously, and integiate them (z.z.1). Disciete symbols aie expiessed within a language
giammai, supplemented by visuospatial aiiangement expiessing paialinguistic stiuctuie. Te
computei geneially only deals with the foimei, but the human is able to auend to both.
1cI
Cu~v1iv No1~1ioN
Magnusson (zccv) desciibes a fundamental dieience between acoustic and digital music
instiuments in the way we play them. He iightly points out that code does not vibiate, and so
we cannot exploie and inteiact with a computei music language with oui bodies, in the same
way as an acoustic instiument. Howevei, piogiammeis still have bodies which contain and
shape theii thoughts, and in tuin, thiough secondary notation, shape theii code. Piogiammeis
do not physically iesonate with code, but cognitive iesouices giounded in peiceptual acuity
enables them to take advantage of visuospatial cognition in theii woik.
We have seen that visuospatial aiiangement is of impoitance to the notation of Patchei
languages, despite not being pait of syntax. Oui aigument follows that if shape, geometiy
and peiceptual cues aie so impoitant to human undeistanding, then we should look foi ways
of taking these aspects out of secondary notation and make them pait of piimaiy syntax.
lndeed, some languages, including iecent music piogiamming languages alieady have.
. Geometiy in Syntax
ln z.z.I, we examined the iole of analogue iepiesentation in computei language against the
backgiound of Dual Coding theoiy. Computei languages allow piogiammeis to communicate
theii ideas thiough abstiaction, but in geneial do not at base include visuospatial syntax to
suppoit this. Do piogiammeis then simply switch o a whole channel of peiception, to fo-
cus only on the disciete iepiesentation of code` lt would appeai not, we have seen that not
only do piogiammeis iepoit mental imageiy in piogiamming tasks (z.z.z), but that the use of
spatial layout is an impoitant featuie of secondary notation in mainstieam piogiamming lan-
guages, which tend to allow piogiammeis to add whitespace to theii code fieely with liule oi
no syntactical meaning (.1). Piogiammeis use this fieedom to aiiange theii code so that its
shape ielates impoitant stiuctuie at a glance. Tat piogiammeis need to use spatial layout as a
ciutch while composing disciete symbolic sequences is telling, to the inteipietei, a block may
be a subsequence between biaces, but to an expeiienced piogiammei it is a peiceptual gestalt
giouped by indentation. liom this we conclude that concoidant with Dual Coding theoiy, the
linguistic woik of piogiamming is suppoited by spatial ieasoning, with secondary notation
helping biidge the divide.
Te question is whethei spatial ieasoning can be successfully employed to woik with pii-
maiy syntax, wheie visuospatial layout is ielevant to the computei as well as the human.
We can nd inteiesting peispectives on this question in the iealm of esoteric programming
languages.
z
lsoteiic piogiamming languages aie those which demonstiate stiange and ex-
z
See http://www.esolangs.org/ foi a compiehensive catalogue of esoteiic piogiamming languages.
1c1
Cu~v1iv No1~1ioN
peiimental ideas, foi fun iathei than piactical use, and we appioach them heie in seaich foi
inspiiation foi piactical applications in the aits. Te use of spatial aiiangement in piimaiy
syntax has become a populai game in the esoteiic language community. Befunge, illustiated in
liguie .1, is a paiticulaily inteiesting example, a textual language with a highly two dimen-
sional syntax. Contiol ow may pioceed up, down, lef oi iight, diiected by single chaiactei
instiuctions.
Te instiuction set of the Piet language is inspiied by Befunge, but iathei than the contents
of individual cells, the Piet instiuction set is given by coloui ielationships between neighboui-
ing cells. loi example, a multiply opeiation is symbolised by the coloui hue changing a liule
and daikening consideiably. Te piogiammei is given a gieat deal of fieedom to choose paitic-
ulai colouis within these constiaints. Much like Befunge, Piet instiuctions include diiectional
modieis, and contiol owtiavels in two dimensions. liguie . contains an example piogiam,
showing its iesemblance to the modeinist paintings of Piet Mondiian which inspiied the Piet
language.
Musicians ofen lead the way in technology, and piogiamming language design is no ex-
ception, as theie aie a numbei of examples of computei music languages which include geomet-
iical measuies of spatial aiiangement in theii piimaiy syntax. liistly, Nodal is a commeicial
enviionment foi piogiamming agent-based music (Mcilwain et al., zcc). Nodal has seveial in-
teiesting featuies, but is notable heie foi its spatial syntax, wheie distance symbolises elapsed
time. As the giaph is iead by the inteipietei, musical events at giaph nodes aie tiiggeied,
wheie the ow of execution is slowed by distance between nodes. Coloui also has syntactic
value, wheie paths aie identied by one of a numbei of hues.
Al-Jazaii is one of a seiies of playful languages cieated by Dave Giiths, based on a com-
putei game engine and contiolled by a gamepad (McLean et al., zc1c). ln Al-Jazaii, caitoon
depictions of iobots aie placed on a giid and given shoit piogiams foi navigating it, in the
foim of sequences of movements including inteiactions with othei iobots. As with Nodal,
space maps to time, but theie is also a mechanism wheie iobots take action based on the piox-
imity and oiientation of a anothei iobot. ln piogiamming Al-Jazaii you aie theiefoie put in
the position of viewing a two dimensional space fiom the point of view of an agents ow of
execution. lndeed it is possible to make this liteially so, as you may switch fiom the ciows
nest view shown in liguie e.1 to the ist-peison view of a iobot.
Oui last example is the ReacTable (Joid`a et al., zcc), a celebiated tangible inteiface aimed
towaids live music. lts cieatois do not desciibe the ReacTable as a piogiamming language, and
claim its tangible inteiface oveicomes inheient pioblems in visual piogiamming languages
such as Puie Data. But tiuly, the ReacTable is itself a visual piogiamming language, if an
1c
Cu~v1iv No1~1ioN
vv < <
2
^ v<
v1<?>3v4
^ ^
> >?> ?>5^
v v
v9<?>7v6
v v<
8
. > > ^
^<
Figure 5.4: A pseudo-random number generator wrien in the two-dimensional
language Befunge.
Figure 5.5: Source code wrien in the Piet language with two dimensional, colour
syntax. Prints out the text Hello, world!. Image c omas So 2006. Used
under the Creative Commons BY-SA 2.5 license.
1ce
Cu~v1iv No1~1ioN
Figure 5.6: e ReacTable music language, where relative distance and orientation is part of primary syntax.
Image c Daniel Williams, used here under the terms of the Creative Commons BY-SA 2.0 License.
extiaoidinaiy one. lt has a visual syntax, wheie physical blocks placed on the ReacTable aie
identied as symbols, and connected accoiding to a neaiest neighboui algoiithm, the iesults of
which can be seen in liguie .e. Not only that, but ielative distance and oiientation between
connected symbols aie paised as values, and used as paiameteis to the functions iepiesented
by the symbols. Video is back-piojected onto the ReacTable suiface to give feedback to the
musician, foi example by visualising the sound signal between nodes. Te ReacTable has also
been iepuiposed as an expeiimental inteiface foi making giaphics (Gallaido et al., zccs), which
suggests that the ReacTable is not as fai fiom a geneial puipose piogiamming language as it
may ist appeai.
.e Visual Piogiamming with Textuie
We now intioduce Texture, a visual piogiamming language, based upon the Tidal language foi
the live coding of pauein intioduced in 1.. Te name Texture is intended to accentuate the
iole of text in piogiamming, as a stiuctuie woven into a two dimensional suiface. Tis ieects
Textuies novel use of spatial aiiangement in the notation of a puie functional piogiamming
language. An impoitant design aim foi Textuie is to cieate a piogiamming notation suitable
foi shoit, impiovised sciipts, allowing fast manipulation by an impiovisei, wheie lay audience
membeis may appieciate moie of the stiuctuie behind the code, made explicit thiough cues
that aie both visual and syntactical. Heie we desciibe Textuie as an eaily piototype system
with novel featuies, desciibing the thinking behind it and issues iaised thiough its development
and eaily use.
Te Textuie enviionment and paisei is implemented in C, using the fiee/open souice Clut-
tei giaphics libiaiy (http://clutter-project.org/) foi its usei inteiface. lt compiles its
1c
Cu~v1iv No1~1ioN
piogiams into an inteimediate foim of Haskell (Jones, zccz) code, which is piped stiaight to a
Haskell dynamic inteipietei whenevei a Textuie piogiam is modied. Haskell then takes caie
of the task of evaluating the code and scheduling sound events accoidingly, using Tidal. Much
of Haskells type system is ie-implemented in Textuie, its contiibution not being to piovide a
whole new language, but iathei a complementaiy and novel visual syntax.
5.6.1 Geometric relationships
Apiogiamwiiuen in Textuie is composed of stiongly typed values and highei oidei functions.
loi example the function + takes two numbeis as aiguments (eithei integeis oi ieals, but not
a mixtuie without explicit conveision), and ietuins theii sum. Heie is how + 1 2 looks in
Textuie
Two gieen lines emeige fiom the bouom iight hand side of the function + , both tiavelling
to the uppei lef hand side of its ist aigument 1 , and then one tiavelling on to the second
aigument 2 . Tis visualises Haskells automatic currying of functions (1..1), wheieby each
time a paiametei is applied, a function is ietuined with aiity ieduced by one. Tis visualisation
of aiity makes it easy to peiceive wheie functions aie paitially applied, which we will show
latei in oui example of fmap .
Te piogiammei types in functions and values, but does not manually add the lines con-
necting them, as they would with a Patchei language. Te lines aie instead infeiied and diawn
automatically by the language enviionment, accoiding to a simple iule the closest two type-
compatible function-value paiis connect, followed by the next two closest, and so on. Values
may be moved fieely with the mouse, which may change the topology of the giaph, which
updates automatically.
Textuie has piex notation (also known as Polish notation) wheie the function comes ist,
followed by its aiguments. Teie is no distinction between functions and opeiatois. Te values
can be placed and moved aiound the scieen, with a line diawn linking the function to each of
its aiguments in tuin. loi example + 2 1 may be expiessed as eithei of the following
ln both cases, two paiallel lines tiavel fiom the bouom iight hand coinei of the function
+ to the top lef of 2 , being the closest compatible value. As alieady mentioned, these lines
1cs
Cu~v1iv No1~1ioN
iepiesent the aiity of the function. Te line iepiesenting the paiametei teiminates at its aigu-
ment 2 , and the iemaining line continues to the top lef of 1 . Again, these symbols connect
automatically, closest ist, wheie closest is dened as luclidean distance in two dimensional
space. lf a symbol is moved, the whole piogiam is ie-inteipieted, with connections ie-iouted
accoidingly.
Te functions may be composed togethei as you might expect. Heie is
+ (+ 1 2) (+ 3 4) in Textuie
Te use of gieen in the above examples is signicant, iepiesenting the integei types. Te
following example shows all the types cuiiently suppoited by Textuie at play
Of the basic types, stiings aie golden yellow, integeis aie gieen and oating point numbeis
aie blue. ln geneial, Tidal Pattern s (1..z) aie pink, oi the similai hues of ied foi paueins of
OSC paiameteis oi daik pink foi paueins of OSC messages (1.e). lunctions assume the coloui
of theii ietuin value, oi if they aie not fully applied (i.e. ietuin anothei function) then they
aie white. Lines connecting functions to theii paiameteis aie colouied in the same way, foi
example in the above example every takes an integei, a function and a pauein to apply the
function to, so its lines aie colouied gieen, white and pink iespectively.
Te following shows the output (ie-foimaued heie to aid ieading) fiom Textuie given the
above example.
1cv
Cu~v1iv No1~1ioN
sampler (( )
(sound (every (3)
(< 2)
(lToP [Just "bd", Nothing, Nothing,
Just "sn", Nothing, Nothing])
)
)
(delay (fmap ( 0.500000) (sine1 16)))
)
Te lToP function is of type [Maybe a] Pattern a , and simply tuins a list of events
into a Pattern . Te othei functions aie explained in 1..
Te stiong typing in Haskell places gieat iestiictions on which aiguments may be applied
to which Tidal functions. Tis bondage and discipline woiks out well foi Textuie, as it limits
the numbei of possible connections, making it easiei foi the piogiammei to piedict what will
connect wheie, suppoited by the coloui highlighting. luitheimoie, because Textuie enfoices
type coiiectness, theie is no possibility of syntactically incoiiect code.
5.6.2 User Interface
Te Textuie usei inteiface is centied aiound typing, editing and moving woids. ln fact that is
all you can do theie aie no menus oi key combinations. A new woid is cieated by moving the
cuisoi to an empty pait of the scieen using the mouse, and then typing. Te woid is completed
by piessing the space bai, at which point the cuisoi moves a liule to the iight wheie the next
woid can be begun, mimicking a conventional text editoi. A woid is edited by being given
focus with a click of the mouse, oi moved by holding down the shif key while being diagged
with the mouse. A whole function tiee (the function and its connected aiguments) is moved
by holding down contiol while diagging, although the aiguments may connect dieiently in
the new location accoiding to the new context.
5.6.3 Texture in Practice
Having seen much of the technical detail of Textuie, we tuin to its musical context. By way of
illustiation, a video showing Textuie in use is available on the enclosed DVD.
Textuie is a piototype language that has not yet undeigone full examination thiough HCl
study, howevei pieliminaiy obseivations have been conducted. ln paiticulai a small woikshop
foi six paiticipants was aiianged with the Access Space fiee media lab in Sheeld, and led by
the piesent authoi. A limit of six people was agieed as a good balance foi a guided woikshop,
and the places weie lled on ist-come-ist-seived basis. Te paiticipants weie found by the
aits piogiamme managei at Access Space, who adveitised thiough theii own website, thiough
11c
Cu~v1iv No1~1ioN
public, aits ielated mailing lists, as well as appioaching iegulai Access Space paiticipants di-
iectly. All paiticipants weie male, aged zI, ze, zs, Ic, 1z and 1z yeais of age. loui lived locally
to Sheeld, and two tiavelled fiom Liveipool. All identied as musicians, foui using com-
puteis in theii music, and ve playing tiaditional instiuments. Two had piioi expeiience of
piogiamming, only one of which had expeiience of a functional piogiamming language. Te
woikshop was fiee of chaige, being pait of an aits piogiamme funded by the Aits Council,
lngland. Paiticipants weie fiee to leave at any time with no penalty, but all stayed to the end.
Te woikshop was in the foim of an houi long piesentation suiveying live coding piactice
and othei inuences of Textuie, followed by a thiee houi hands-on woikshop. Te ist half of
the hands-on section intioduced techniques on a piojected display, which paiticipants, while
listening on headphones, copied and adapted in exploiation of theii use. Te second half was
moie fieefoim, wheie each paiticipant had theii own set of steieo speakeis at theii computeis.
Te paiticipants weie playing to a globally set tempo, with accuiate time synchionisation.
I
Tis meant that they weie able to iespond to each otheis paueins, impiovising music togethei,
because of the layout of the ioom, it was only possible to cleaily heai the music of immediate
neighbouis. Recoided video taken fiom this pait of the woikshop is available on the enclosed
DVD.
Te paiticipants weie the ist people to use Textuie besides the piesent authoi, and so
theie was some iisk that the paiticipants would be unable to engage with it due to unfoieseen
technical pioblems oi task diculty. Howevei all showed enthusiasm, weie keen to exploie
the language, and joined in with playing togethei ovei speakeis.
Te paiticipants weie suiveyed foi opinions thiough a questionnaiie answeied via the sui-
veymonkey website, using computeis at Access Space. Tis was done in two paits, immediately
befoie and then immediately afei the woikshop. Te paiticipants weie told that the suivey
was designed to help in the development of the sofwaie used in the woikshop and weie
asked to answei the questions honestly. To encouiage honest iesponses fuithei, the suivey
was conducted anonymously.
Paiticipants weie asked to iate theii agieement with thiee statements both befoie and afei
the woikshop, on a Likeit scale fiom Disagiee Stiongly (1) to Agiee Stiongly (). Although
theie is liule statistical powei foi such a small gioup, feedback fiom these individuals was
encouiaging foi a system at such an eaily stage of development. Te iesults aie visualised
in liguie .. Oveiall, paiticipants tended to agiee with l am inteiested in live coding and
l would like to be a live codei both befoie and afei the woikshop. Tey laigely disagieed
with l am a live codei at the beginning but weie less suie by the end, indicating they had
I
Accuiate time synchionisation was made possible by the netclock piotocol.
111
Cu~v1iv No1~1ioN
Figure 5.7: Responses on a Likert scale from participants (n=6) in a workshop for the Texture visual live
coding language. e graph visualises a straightforward response count for ea question centered on the
neutral responses. Responses to the rst three statements were collected both before and aer the workshop.
achieved some insights and begun to identify with the piactice. At the end, foui out of the six
paiticipants agieed with a nal statement l would like to use Text[uie]
1
again.
Paiticipants weie also given fieefoim questions asking what they liked and disliked about
Textuie, howmuch they felt they undeistood the connection between the visual iepiesentation
and the sound, and soliciting suggestions foi impiovements. Dislikes and suggestions focused
on technical inteiface issues such as the lack of undo, and thiee found the automatic linking
dicult to woik with. On the othei hand, thiee paiticipants iepoited liking how quick and
easy it was to make changes.
5.6.4 Cognitive Dimensions of Texture
Textuie is designed foi the impiovisation of musical pauein, as a visual piogiamming intei-
face to the Tidal pauein DSL. Te iesult is a moie tightly constiained system than many pio-
giamming languages foi music, which geneially include extensive facilities foi low level sound
synthesis. While the ability to compose iight fiom the micio-level of the sound signal oeis
gieat possibilities, it comes with tiade-os, in paiticulai along the hard mental operations and
diuseness (veibosity) cognitive dimensions.
Te visibility of Textuie is high, wheie a complex ihythmcan be notated on a single scieen.
We also aigue that Textuie has high closeness of mapping, as the visual iepiesentation of
tiees within tiees coiiesponds well with the hieiaichical stiuctuie of the pauein that is being
composed. Tis echoes the tiee stiuctuies common in music analysis, and indeed we would
expect signicant coiiespondence between the Textuie stiuctuie and the listeneis peiception
of it. Te extent to which an untiained listenei may ielate the stiuctuie they heai with the
1
At the time of the woikshop, Textuie had the woiking name of Text.
11z
Cu~v1iv No1~1ioN
Textuie piogiam they see is an empiiical question, but we suspect that fuithei development is
needed to suppoit this.
Cieative use of Textuie is aided not only by high visibility but also aspects of provision-
ality. What is meant by cieative use will be outlined in ielation to bricolage programming
in e.I.1. A piogiammei may woik on a section of code and diag it into the main body of the
piogiam when it is ieady. Tey may also diag pait of the code out of the main body and ieuse
it elsewheie latei. Te code must always be syntactically coiiect, but unless it connects to a
function iepiesenting OSC messages sent to a synthesisei, it will have no eect.
Te error-proneness of Textuie is well positioned. lt is impossible to make syntax eiiois
in Textuie, and while the automatic connection can at times have unexpected iesults, the iesult
is at times musically inteiesting, but otheiwise stiaightfoiwaid to ieveise.
5.6.5 Future directions
Textuie is a woiking piototype, in that it is fully functional as a live music inteiface, but is a
pioof of concept of an appioach to piogiamming that biings many fuithei ideas to mind.
ln teims of visual aiiangement, Textuie tieats woids as squaie objects, but peihaps the
individual maiks of the symbols could be biought into the visual notation, thiough expeiiments
in typogiaphy. loi example, a cuisive font could be used wheie the tiajectoiy of the nal stioke
in a woid is continued with a cuive to ow into the leading stioke of the woid it connects to.
Tis suggestion may tuin the stomach of haidened piogiammeis, although Textuie is alieady
unusual in using a piopoitional font, complete with ligatuies.
Cuiiently theie is no piovision in Textuie foi making named abstiactions, so a piece of
code cannot be used moie than once without being iepeated veibatim. Visual syntax foi single
assignment could symbolise a section of code with a shape deiived fiom the aiiangement of
its components. Tat shape would become an ideogiaphic symbol foi the code, and then be
duplicated and ieused elsewheie in the piogiam using the mouse.
Textuie is inspiied by the ReacTable, but does not featuie any of the ReacTables tangible
inteiaction. Such tabletop inteifaces oei a numbei of advantages ovei keyboaid-and-mouse
inteifaces, in paiticulai multitouch, allowing movement of moie than one component at once.
Multi-touch tablet computeis shaie this advantage while avoiding some of the tiadeos of tan-
gible inteifaces. Much of the ReacTable technology is available as an open ieseaich platfoim,
and could be highly useful in this aiea of expeiimentation.
Cuiiently the only output of Textuie is music iendeied as sound, with no visual feedback.
Teie is gieat scope foi expeiimentation in visualising the paueins in the code, making it easiei
foi live codeis and audience membeis to connect musical events and tiansfoimations with
11I
Cu~v1iv No1~1ioN
paiticulai sections of the code. One appioach would be to visualise pauein owing between
nodes, again inspiied by the ReacTable, howevei as Textuie is based upon puie functions iathei
than dataow giaphs, it piesents a iathei dieient design challenge. Textuie could be adapted
to allow any Haskell piogiam to be wiiuen, and so could be applied to othei domains, such as
digital signal piocessing and visual animation. Tis would again place dieient challenges on
the visualisation of iesults.
Most geneially, and peihaps most impoitantly, we look towaids piopei analysis of lay
audience code compiehension, giounding fuithei development with beuei undeistanding of
what the design challenges aie.
. Discussion
We have consideied piogiamming languages as iich notations with both visual and linguis-
tic aspects. Many computei musicians wiite woids to desciibe theii music, foi computeis to
tianslate to sound. Te computei musicians have become comfoitable with this iathei odd
piocess in piivate, but peihaps found it dicult to explain to theii paients. Simply by pioject-
ing theii inteifaces, live codeis have biought this oddity out in public, and must deal with the
consequences of biinging many issues undeilying computei music to the suiface.
Live coding may be undeistood as a dual activity of language and spatial peiception. We
have seen howhumans have the capacity to integiate both simultaneously, showing that the act
of live coding, and peihaps the audience ieception of it, can be iealised and felt simultaneously
as both musical language and musical movement.
By placing the design of live coding languages in a psychological and analytical context,
we have aimed to suppoit futuie diiections in live coding design. Te intioduction of Textuie
takes a step in this diiection, and we hope demonstiates the exciting giound waiting foi futuie
language designeis to exploie.
111
Cu~v1iv e
Creativity
liom symbols up, each of the pieceding thiee chapteis has built a layei of abstiaction on top of
the pievious one. Te next layei takes us above the activity of notation, to the bioadei context
of cieative aitistic activity. How can piogiamming t into a cieative piocess`
e.1 Piogiammei cultuie
liom eaily beginnings piogiammeis have pulled themselves up by theii bootstiaps, cieating
languages within languages in which gieat hieiaichies of inteiacting systems aie expiessed.
Much of this activity has been towaids militaiy, business oi scientic goals. Howevei theie
aie numeious examples of alteinative piogiammei subcultuies foiming aiound fiinge activity
without obvious piactical application. Te Hackei cultuie at MlT was an eaily example (Levy,
zccz), a gioup of male model-iailway enthusiasts and phone netwoik hackeis who dedicated
theii lives to exploiing the possibilities biought by the new, digital computeis. Many othei pio-
giamming cultuies have since ouiished. Paiticulaily stiong and long-lived is the demoscene,
a youth cultuie engaged in pushing computei video animation to the limits of available haid-
waie, using novel algoiithmic techniques to dazzling ends. Te demoscene spans much of the
globe but is paiticulaily stiong in Noidic countiies, wheie annual meetings auiact thousands
of paiticipants (Polgai, zcc). Anothei, peihaps loosei piogiammei cultuie is that of lsoteiic
Piogiamming Languages discussed in .. Te authois of these languages push against the
boundaiies of piogiamming, pioviding insight into the constiaints of mainstieam piogiam-
ming languages.
Membeis of the demoscene and esoteiic language cultuies do not necessaiily self-identify
as aitists, despite theii ielentless seaich foi novel appioaches. Howevei theie aie cultuies
of piogiammeis who do call themselves aitists, now extending into theii second and thiid
geneiations. laily on, communities of expeiimental aitists looking foi new means of expies-
11
Cu~v1iv e Cvi~1ivi1v
Figure 6.1: e robots of the Al-Jazari language by Dave Griths (McLean et al., 2010). Ea robot has a
thought bubble containing a small program, edited through a game pad.
sion giew aiound computeis as soon as access could be gained. ln Gieat Biitain inteiest dui-
ing the 1vecs giew into the foimation of the Computei Aits Society (CAS, Biown et al., zccv,
www.computer-arts-society.org). Howevei afei a cieative boom CAS enteied a peiiod
of doimancy in the mid 1vscs, peihaps diowned out by extieme commeicial giowth in the
computei industiy at that time. CAS has howevei been ievived in moie iecent yeais, encoui-
aged by a majoi iesuigence of sofwaie as a medium foi the aits. Tis iesuigence has seen
a wealth of new piogiamming enviionments designed foi aitists and musicians, such as Pio-
cessing (Reas and liy, zcc), SupeiCollidei (McCaitney, zccz), ChucK (Wang and Cook, zcc1),
VVVV (http://vvvv.org) and Openliamewoiks (http://openframeworks.cc), joining
moie established enviionments such as the Patchei languages PuieData and Max (.I.1).
Te cieatois of piogiamming languages foi the aits aie ofen aitist-piogiammeis them-
selves, motivated to suppoit theii own woik. Tis iesults in expeiimental featuies which ie-
semble those of esoteiic languages, foi example unique iepiesentations of time aie cential
featuies of ChucK and SupeiCollidei (.z). Languages cieated by aitist-piogiammeis have
themselves been exhibited as inteiactive woiks of ait, such as the Al-Jazari music piogiam-
ming enviionment shown in liguie e.1 (McLean et al., zc1c).
Te design of piogiamming languages foi the aits is an active ieseaich domain, with new
appioaches still emeiging, biinging impoitant psychological issues to the foie. As computeis
11e
Cu~v1iv e Cvi~1ivi1v
entei almost eveiy aspect of oui lives, few would now deny that people can be cieative using
computei tools. Against all this context it is cleai to see the activity of piogiamming as poten-
tially being highly cieative. Befoie we discuss the cieative piocesses of piogiamming in depth
howevei, we should desciibe what we mean by cieative behavioui.
e.z Concepts in cieative behavioui
Cieativity is assumed to be a maik of the human species, and something that we all do as pait
of a healthy appioach to life. Te alteinative to a cieative life is a mechanistic one, lacking in
intiospection and contiol ovei the self-imposed iules we live by. Howevei, the woid creative
is used to dene a wide iange of behavioui, foi example a cieative peison in the adveitising
industiy is something iathei moie specic than a cieative peison in the aits. So what exactly
do we mean by creativity` We might tiy to dene it in teims of qualities of the aitefacts that
cieativity pioduces. Howevei we judge the cieativity of an aitefact not only by its intiinsic
value, but also its novelty in a cultuie, and peihaps how much it suipiises us. Tat is, we judge
an aitefact not just by any physical manifestation, but piimaiily the concepts behind it. To
undeistand cieativity then, we should focus on the conceptual activity behind cieative woiks.
ln z.z., we took the denition of a concept as a mental iepiesentation of a class of things
(Muiphy, zccz, p.). By asseiting that cieativity is a conceptual piocess, we theiefoie imply
that cieativity is not in the pioduction of things, but iathei in the oiganisation of things into
novel classes. lf we iecognise cieativity in an aitefact, it is because its piopeities allow us to
infei a novel conceptual class of which it is a membei.
lf concepts aie the piimaiy output of the cieative piocess, we should dene howand wheie
we think they aie iepiesented. ln oui ieview of conceptual iepiesentation (z.z) we shaied
the view that concepts aie stiuctuied in ielation to peiception, that basic concepts aiise fiom
iecuiient states in sensoiimotoi systems, which in tuin foimthe building blocks of highei level
abstiact thought. When we cieatively geneiate novel, valued concepts, we aie liteially alteiing
oui peiception of the woild and of ouiselves.
lxactly how a concept is iepiesented in the human mind is an open question. Heie we
take the view that a conceptual piopeity is iepiesented by a single best possible example, oi
prototype. ln accoidance with the theoiies ieviewed in chaptei z, these piototypes aiise thiough
peiceptual states, within the geometiy of quality dimensions. To giound the discussion in
music, considei a piece of jazz, wheie jazz is the concept and the paiticulai composition is
an instance of that concept. Te musician, in exploiing the boundaiies of jazz, then nds a
piece beyond the usual iules of jazz. Tiough this piocess, the boundaiies of a music genie
11
Cu~v1iv e Cvi~1ivi1v
may be iedened to some degiee, oi if the piece is in paiticulaily feitile new giound, a new
sub-genie of jazz may emeige. lndeed a piece of music which does not bieak boundaiies in
some way could be consideied uncieative. Tese changes in conceptual stiuctuie ist happen
in an individual, which in the case of music is the composei oi impiovisei. Anothei individuals
conceptual stiuctuies may be modied to accoid with a composeis new concept by listening
to the instance of the new musical concept, although success is only likely if the individual
alieady shaies some of the music cultuial noims of the cieatoi.
Wiggins (zccea,b) foimalises the Cieative Systems liamewoik (CSl) based on the woik of
Boden (zccI), in oidei to piovide a fiamewoik foi discussing cieativity and compaiing cieative
systems. Te CSl chaiacteiises cieativity as a seaich in a conceptual space, accoiding to a mu-
table set of iules. Tieating cieativity as a seaich is much the same as tieating it as constiuction,
but implies that cieativity takes place within dened boundaiies. Cieativity is ofen discussed
using the metaphoi of exploiation, aiound wheie boundaiies aie dened and bioken, and the
CSl allows us to talk about such behavioui within its well dened teims.
As a compaiative fiamewoik, the CSl is agnostic to issues of iepiesentation, and so may be
applied to both analogue and disciete conceptual iepiesentations. ln taking the Gaidenfoisian
appioach to conceptual iepiesentation (z.z.), we aigue that cieativity involves employing
visuospatial cognitive iesouices to navigate an analogue seaich space with geometiic stiuctuie.
We will discuss cieative piocesses of piogiamming in teims of the mechanisms of the CSl latei
in e..
Te subject of cieativity within the computei aits eld is miied in confusion and miscon-
ception. Te peiennial question of authorship is always with us if a computei piogiamoutputs
ait, who has made it, the human oi the machine` Positions on cieativity thiough computei
piogiamming tend towaids opposite poles, with outiight denials of cieativity at one end and
outlandish claims of unbound cieativity in geneiative ait at the othei. Heie we look foi claiity
thiough oui anthiopocentiic view, with focus on piogiamming as the key activity behind com-
putei ait. We view the aitist-piogiammei as engaged in innei human ielationships between
peiception, cognition and computation, and ielate this to the notation and opeiation of theii
algoiithms, paiticulaily in the context of live coding.
e.I Cieative Piocesses
Cieative piocesses aie iathei moie mysteiious than computei piocesses, which we like to
think of as well dened, piedictable and subseivient to human contiol. loi aitist-piogiammeis
though, the computei piocess is a component of theii cieative piocess. What then is the iela-
11s
Cu~v1iv e Cvi~1ivi1v
tionship between an aitist, theii cieative piocess, theii piogiam, and theii aitistic woiks` We
will look foi answeis against the backgiounds of psychology, cognitive linguistics, computei
science and computational cieativity, but ist fiom the peispective of an aitist.
Te paintei Paul Klee desciibes a cieative piocess as a feedback loop
Alieady at the veiy beginning of the pioductive act, shoitly afei the initial
motion to cieate, occuis the ist countei motion, the initial movement of iecep-
tivity. Tis means the cieatoi contiols whethei what he has pioduced so fai is
good. Te woik as human action (genesis) is pioductive as well as ieceptive. lt is
continuity. (Klee, 1vI, p. II, oiiginal emphasis)
Tis is cieativity without planning, a feedback loop of making a maik on canvas, peiceiving
the eect, and ieacting with a fuithei maik. Being engaged in a tight cieative feedback loop
places the aitist close to theii woik, guiding an idea to unfoieseeable conclusion thiough a ow
of cieative peiception and action. Klee wiites as a paintei, woiking diiectly with his medium.
Aitist-piogiammeis instead woik using computei language as text iepiesenting theii medium,
and it might seemthat this extia level of abstiaction could hindei cieative feedback. We will see
howevei that this is not necessaiily the case, beginning with the account of Tuikle and Papeit
(1vvz), desciibing a bricolage appioach (afei Levi-Stiauss, 1ves) to piogiamming by analogy
with painting
Te biicoleui iesembles the paintei who stands back between biushstiokes,
looks at the canvas, and only afei this contemplation, decides what to do next.
Biicoleuis use a masteiy of associations and inteiactions. loi planneis, mistakes
aie missteps, biicoleuis use a navigation of mid-couise coiiections. loi planneis,
a piogiam is an instiument foi piemeditated contiol, biicoleuis have goals but set
out to iealize them in the spiiit of a collaboiative ventuie with the machine. loi
planneis, geuing a piogiam to woik is like saying ones piece, foi biicoleuis, it
is moie like a conveisation than a monologue.
(Tuikle and Papeit, 1vvc, p. 1Ie)
Tuikle and Papeit desciibe the biicolage piogiammei as foiming ideas while woiking in
the text editoi, making edits in ieaction to edits, iathei than planning theii woik in advance.
Tis accoids with Klees account, and may also be ielated to that of Reective Practice fiom
piofessional studies (Schon, 1vs1). Reective piactice distinguishes the noimal conception of
knowledge, as gained thiough study of theoiy, fiom that which is leaint, applied and ieected
upon while in the woik. As such, piactice is not iuled by theoiy, but embedded in activity.
Reective piactice has stiong inuences in piofessional tiaining, paiticulaily in the educational
and medical elds. Tis suggests that the piesent discussion could have ielevance beyond oui
focus on the aits.
11v
Cu~v1iv e Cvi~1ivi1v
Although Tuikle and Papeit addiess gendei issues in computei education, this quote should
not be misiead as dividing all piogiammeis into two types, while associating biicolage with
feminine and planning with male tiaits (although foi ciitique of theii method see Blackwell,
zccea), they aie caieful to state that these aie extiemes of a behaviouial continuum. lndeed,
piogiamming style is cleaily task specic foi example a pioject iequiiing a laige team needs
moie planning than a shoit sciipt wiiuen by the end usei.
Biicolage piogiamming is paiticulaily applicable to oui theme of the aitist-piogiammei,
wiiting sofwaie to woik with media such as music and video animation. To giound the fol-
lowing discussion, we biing an image of a visual aitist to mind, piogiamming theii woik using
the Piocessing language enviionment (.z). Oui aitist begins with an uige to diaw supeiim-
posed cuived lines, having been inspiied by something they saw fiom the tiain the pievious
day. Tey quickly come up with the following piogiam, shown with its output below
float rx() { return(random(width)); }
float ry() { return(random(height)); }
void draw() {
background(255);
for (int i = 0; i < 20; ++i) {
bezier(rx(), ry(), rx(), ry(),
rx(), ry(), rx(), ry());
}
}
On seeing the output of the ist iun, oui aitist is immediately stiuck by how haiiy it looks.
Tanks to aibitiaiy placement of the cuives thiough use of pseudo-iandomnumbeis, each time
the aitist iuns the piogiamit comes up with a dieient foim. Ovei a fewiuns, the aitist notices
the occasional suggestion of a handwiiuen sciibble. lntiigued, they change theii piogiam to
join the cuives togethei, in oidei to iemove the haiiiness and accentuate the sciibble
void draw() {
background(255);
float x = rx(); float y = ry();
for (int i = 0; i < 5; ++i) {
float x1 = rx(); float y1 = ry();
bezier(x, y, rx(), ry(),
rx(), ry(), x1, y1);
x = x1; y = y1;
}
}
Te end-points of the cuives aie still placed aibitiaiily, but they now begin at the point
wheie the pievious cuive ended, iesulting in a continuous line. Afei a few moie iuns, oui
1zc
Cu~v1iv e Cvi~1ivi1v
aitist begins to peiceive a leuei-like quality in the sciibble. Tey decide to tiy wiiting them
acioss the page, giouped into woid-like foims
float letterSpace = 30;
float rx() { return(random(letterSpace + 10)); }
float ry() { return(random(height - 10)); }
int rWordlen() { return(3 + int(random(4))); }
void draw() {
background(255);
int letters = (int) (width / letterSpace) - 4;
int wordLen = rWordlen();
int word = 0;
float x = rx(); float y = ry();
for (int letter = 0; letter < letters; ++letter) {
float ox = letter letterSpace + word
letterSpace;
if (wordLen-- == 0) {
wordLen = rWordlen();
word++;
}
for (int i = 0; i < 3; ++i) {
float x1 = rx() + ox; float y1 = ry();
bezier(x, y, rx() + ox, ry(),
rx() + ox, ry(), x1, y1);
x = x1; y = y1;
}
}
}
Te output has a handwiiuen quality, appeaiing almost ieadable, a quality of automatic
wiiting used by mystics to supposedly channel the spiiit woild. Tis may biing fuithei con-
ceptual development to oui aitists mind, but at this point we will leave them pondeiing.
Oui case study is somewhat simplistic, and is not intended to illustiate eithei gieat ait oi
gieat code. Howevei it does tiace a cieative piocess of soits, as caiiied out by the piesent
authoi. We aie not suggesting that the algoiithms themselves aie cieative, any moie than we
would suggest that paint is cieative. Multiple executions helped oui aitist peiceive qualities
in the output, but it was the aitists peiception, and not the algoiithm that discoveied value.
lt is cleai that oui piogiammei, like a paintei, could not undeistand theii code until they had
peiceived its output, that the act of peiception was itself cieative, and that the concept they
weie tiying to encode was continually changing in iesponse to theii peiception of the iesults.
We seek to undeistand this piocess in gieatei depth in the following sections.
1z1
Cu~v1iv e Cvi~1ivi1v
Figure 6.2: e process of action and reaction in bricolage programming
6.3.1 Creative Process of Bricolage
Te souice of the exhilaiation associated with computei piogiamming is the
continual unfolding within the mind and on the computei of mechanisms ex-
piessed as piogiams and the explosion of peiception they geneiate. Alan Peilis
(foiewoid to Abelson and Sussman, 1vve)
liguie e.z chaiacteiises biicolage piogiamming as a cieative feedback loop encompassing
the wiiuen algoiithm, its inteipietation, and the piogiammeis peiception and ieaction to its
output oi behavioui. Cieative feedback loops aie fai fiom unique to piogiamming, but the
addition of the algoiithmic component makes an additional innei loop explicit between the
piogiammei and theii text. At the beginning, the piogiammei may have a half-foimed con-
cept, which only ieaches inteinal consistency thiough the piocess of being expiessed as an
algoiithm. Te innei loop is wheie the piogiammei elaboiates upon theii imagination of what
might be, and the outei wheie this tiajectoiy is giounded in the piagmatics of what they have
actually made. Tiough this piocess both algoiithm and concept aie developed, until the pio-
giammei feels they accoid with one anothei, oi otheiwise judges the cieative piocess to be
nished.
Te lack of foiwaid planning in biicolage piogiamming means the feedback loop in liguie
e.z is self-guided, possibly leading the piogiammei away fiom theii initial motivation. Tis
stiaying is likely, as the possibility foi suipiise is high, paiticulaily when shifing fiom the
innei loop of implementation to the outei loop of peiception. Te output of a geneiative ait
piocess is iaiely exactly what we intended, and we will latei aigue in e. that this possibility
of suipiise is an impoitant contiibution to cieativity.
1zz
Cu~v1iv e Cvi~1ivi1v
Repiesentations in the computei and the mind aie evidently distinct fiom one anothei.
Computei output evokes peiception, but that peicept will both exclude featuies that aie ex-
plicit in the output and include featuies that aie not, due to a iange of eects including auen-
tion, knowledge and illusion. lqually, a human concept is distinct fiom a computei algoiithm.
Peihaps a piogiam wiiuen in a declaiative iathei than impeiative style is somewhat closei to a
concept (1.I). But still, theie is a cleai line to be diawn between a stiing of disciete symbols in
code, and the moiass of both disciete and continuous iepiesentations which undeilie cognition
(z.z).
Teie is something cuiious about how the piogiammeis cieative piocess spawns a sec-
ond, computational piocess. ln an appaient tiade-o, the computational piocess is lacking in
the bioad cognitive abilities of its authoi, but is nonetheless both fastei and moie accuiate at
ceitain tasks by seveial oideis of magnitude. lt would seem that the piogiammei uses the pio-
giamming language and its inteipietei as a cognitive iesouice, augmenting theii own abilities
in line with the extended mind hypothesis (Claik, zccs).
e.1 Symbols and Space
We have seen aigued thioughout this thesis that human conceptual iepiesentation centies
aiound peiception. Algoiithms on the othei hand aie iepiesented in disciete symbolic se-
quences, as is theii output, which must go thiough some foim of digital-to-analogue convei-
sion befoie being piesented to oui sensoiy appaiatus, foi example as light fiom a monitoi
scieen oi sound piessuie waves fiom speakeis, tiiggeiing a piocess we call obseivation. Recall
the aitist-piogiammei fiom oui case study (e.I), who saw something not iepiesented in the
algoiithm oi even in its output, but only in theii own peiception of the output, obseivation is
itself a cieative act.
Te iemaining component to be dealt with fiom liguie e.z is that of piogiammeis concepts
(z.z.). liguie e.z shows concepts mediating between spatial peiception and disciete algo-
iithms, leading us to ask, aie concepts iepiesented moie like spatial geometiy, like peicepts, oi
symbolic language, like algoiithms` Oui focus on metaphoi leads us to take the foimei view,
that conceptual iepiesentation is giounded in peiception and the body. Tis view is taken fiom
Conceptual Metaphoi Teoiy (z.z.e), in paiticulai that concepts aie piimaiily stiuctuied by
metaphoiical ielations, the majoiity of which aie oiientational, undeistood ielative to the hu-
man body in space oi time. ln othei woids, the conceptual systemis giounded in the peiceptual
system.
Gaidenfoisian conceptual spaces (z.z.) aie compelling when applied to concepts ielated
1zI
Cu~v1iv e Cvi~1ivi1v
to bodily peiception, emotion and movement. Howevei, it is dicult to imagine taking a similai
appioach to computei piogiams. What would the quality dimensions of a geometiical space
containing all computei piogiams be` Teie is no place to begin to answei this question,
computei piogiams aie linguistic in natuie, and cannot be coheiently mapped to a geometiical
space giounded in peiception. Again we ietuin to the point that spatial iepiesentation is not
in opposition to linguistic iepiesentation, they aie distinct but suppoit one anothei. Tis is
cleai in computing, haidwaie exists in oui woild of continuous space, but thanks to ieliable
electionics, conjuies up a linguistic woild of disciete computation. Oui minds aie able to do the
same, foi example by computing calculations in oui head, oi encoding concepts into phonetic
movements of the vocal tiact oi alphabetic symbols on the page. We can think of ouiselves as
spatial beings able to simulate a linguistic enviionment to conduct abstiact thought and open
channels of communication. On the othei hand, a piece of computei sofwaie is a linguistic
being able to simulate spatial enviionments, peihaps to cieate a game woild oi guide iobotic
movements, both of which may include some kind of model of human peiception.
e. Components of cieativity
We now have giounds to chaiacteiise how the cieative piocess opeiates in biicolage piogiam-
ming. loi this we employ the Cieative Systems liamewoik (CSl, e.z), intioducing its teims
as we go.
Within the CSl, a cieative seaich has thiee key aspects the conceptual sear space it-
self, traversal of the space and evaluation of concepts found in the space. ln othei woids,
cieativity iequiies somewheie to seaich, a mannei of seaiching, and a means to judge what
is found. Howevei, cieative behavioui may make use of intiospection, self-modication and
need boundaiies to be bioken. Tat is, the constiaints of seaich space, tiaveisal and evaluation
aie not xed, but aie examined, challenged and modied by the cieative agent following (and
dened by) them. Te CSl chaiacteiises paiticulai kinds of aberration fiom a conceptual space,
and appioaches to addiessing them.
Using the teiminology of Gaidenfois (zccc), the seaich spaces of the CSl aie themselves
concepts, dening iegions in a univeisal space dened by quality dimensions. Tiansfoima-
tional cieativity then is a geometiical tiansfoimation of these iegions, motivated by a piocess
of seaiching thiough and beyond them. Tis means that a cieative agent may cieatively push
beyond the boundaiies of the seaich as we will see. While acknowledging that cieative seaich
may opeiate ovei linguistic seaich spaces, we focus on geometiic spaces giounded in peicep-
tion. Tis follows oui focus on aitistic biicolage desciibed in e.I, but foi an appioach unifying
1z1
Cu~v1iv e Cvi~1ivi1v
linguistic and geometiic spaces see loith et al. (zc1c).
We may now claiify the biicolage piogiamming piocess intioduced in e.I.1 within the
CSl. As shown in liguie e.I, the seaich space denes the piogiammeis concept, being theii
cuiient aitistic focus stiuctuied by leaint techniques and conventions. Te tiaveisal stiategy is
the piocess of auempting to geneiate pait of the concept by encoding it as an algoiithm, which
is then inteipieted by the computei. linally, evaluation is a peiceptual piocess in ieaction to
the output.
Algorithm
Interpret
Percept
Elaborate
Encode
Concept
Observe
React
Output
Evaluation Traversal
Sear
space
Figure 6.3: e process of action and reaction in bricolage programming from Figure 6.2, showing the three
components of the Creative Systems Framework, namely sear space, traversal strategy and evaluation.
ln e.I, we alluded to the extended mind hypothesis (Claik, zccs), claiming that biicolage
piogiamming takes pait of the human cieative piocess outside of the mind and into the com-
putei. Te above makes cleai what we claim is being exteinalised pait of the tiaveisal stiat-
egy. Te piogiammeis concept motivates a development of the tiaveisal stiategy, encoded
as a computei piogiam, but the piogiammei does not necessaiily have the cognitive ability
to fully evaluate it. Tat task is taken on by the inteipietei iunning on a computei system,
meaning that tiaveisal encompasses both encoding by the human and inteipietation by the
computei.
Te tiaveisal stiategy is stiuctuied by the techniques and conventions employed to conveit
concepts into opeiational algoiithms. Tese may include design paerns, a standaidised set
of ways of building that have become established aiound impeiative piogiamming languages.
lach design pauein identies a kind of pioblem, and desciibes a geneialised stiuctuie towaids
1z
Cu~v1iv e Cvi~1ivi1v
a solution.
1
Te cieative piocess is constiained by the piogiammeis concept of what is a valid end
iesult. Tis is shaped by the piogiammeis cuiient aitistic focus, being the peiceptual qualities
they aie cuiiently inteiested in, peihaps congiuent with a cultuial theme such as a musical
genie oi aitistic movement. Aitists ofen speak of self-imposed constiaints as pioviding cie-
atively feitile giound. ln teims of a cieative seaich such constiaints foim the boundary of a
seaich space. lt is possible foi a seaich to tiaveise beyond that boundaiy, thus nding invalid
concept instances, a scenaiio called aberration (Wiggins, zccea, .z.z). ln such instances trans-
formational creativity can be tiiggeied. loi example, if an invalid yet (accoiding to evaluation
iules) valued instance is found, then the concept should be enlaiged to include the instance.
An invalid concept instance which is not valued indicates that oui tiaveisal stiategy is awed
and should be modied to avoid such instances in the futuie. A single tiaveisal opeiation may
iesult in both valid and invalid instances being found, indicating that both the tiaveisal iules
and the denition of the concept should be modied.
Te aitist in oui eailiei case study was woiking within the concept of beziei cuives, but
when cuive endpoints happened to join, they peiceived some things outside that concept a
squiggle and some haii. Tey made a value judgement, and decided to change theii concept
in iesponse, which we considei as a case of tiansfoimational cieativity. Tey then made edits
to theii souice code (the tiaveisal stiategy), in oidei to tiy to geneiate output which evoked
peiceptual expeiiences closei to theii concept. Tis may seema minoi case of tiansfoimational
cieativity, but indeed we contend that much cieativity is quite oidinaiy human behavioui,
humans apply cieativity at all levels of life.
Oui case study shows wheie a piogiammei may set themselves up foi being suipiised by
the iesults. Tis is not only due to the use of pseudo-iandom numbeis, afei all noise is iaiely
a souice of suipiise, in infoimation theoietic teims noise has maximal infoimation content,
but in piactice the lack of foim quickly iesults in fatigue oi auention shif. lt is iathei due
to the linguistic abstiaction of an idea that consists of fiagments of peiceptual symbols. ln
othei woids, because the tiaveisal stiategy of a piogiammei includes exteinal notation and
computation, they aie likely to be less successful in wiiting sofwaie that meets theii piecon-
ceptions, oi in othei woids moie successful in being suipiised by the iesults. Acieative piocess
that includes exteinal computation will follow a less piedictable path as a iesult. Nonetheless
the piocess as a whole has the focus of a concept, and is guided by value in ielation to a iich
peiceptual fiamewoik, and so while unpiedictable, this inuence is fai fiom iandom, being
meaningful inteiplay between language and peiceptual expeiience. Te human concepts and
1
Tis stiuctuial heuiistic appioach to pioblemsolving is inspiied by woik in the eld of uiban design (Alexandei
et al., 1v).
1ze
Cu~v1iv e Cvi~1ivi1v
algoiithm aie continually tiansfoimed in iespect to one anothei, and to peiceptual aect, in
cieative feedback.
Accoiding to oui embodied view, not only is peiception ciucial in evaluating output within
biicolage piogiamming, but also in stiuctuiing the space in which piogiams aie conceptualised.
lndeed if the embodied view of conceptual metaphoi theoiy (z.z.e) holds in geneial, the same
would apply to all cieative endeavoui. liom this we nd a message foi the eld of compu-
tational cieativity a pieiequisite foi an aiticial cieative agent is in acquiiing computational
models of peiception sucient to both evaluate its own woiks and stiuctuie its conceptual sys-
tem. Only then will the agent have a basis foi guiding changes to its own conceptual system
and geneiative tiaveisal stiategy, able to modify itself to nd aitefacts that it was not pio-
giammed to nd, and place value judgements on them. Such an agent would need to adapt to
human cultuie in oidei to inteiact with shifing cultuial noims, keeping its conceptual system
and iesultant cieative piocess coheient within that cultuie. loi now howevei this is wishful
thinking, and we must accept geneiative computei piogiams which extend human cieativity,
but aie not cieative agents in theii own iight.
e.e Piogiamming in Time
She is not manipulating the machine by tuining knobs oi piessing buuons.
She is wiiting messages to it by spelling out instiuctions leuei by leuei. Hei
painfully slow typing seems laboiious to adults, but she caiiies on with an ab-
soiption that makes it cleai that time has lost its meaning foi hei. Sheiiy Tuikle
(zcc, p. vz), on Robin, aged 1, piogiamming a computei.
Having investigated the iepiesentation and opeiation of biicolage piogiamming we now
examine howthe cieative piocess opeiates in time. Dijkstia might aigue that consideiing com-
putei piogiams as opeiating in time at all, iathei than as entiiely abstiact logic, is itself a foim
of the anthiopomoiphism examined in z.I. Howevei fiom the above quotation it seems that
Robin stepped out of any notion of physical time, and into the algoiithm she was composing,
enteiing a timeless state. Tis could be a state of optimum expeiience, the ow investigated
by Csikszentmihalyi wheie duiation of time is alteied, houis pass by in minutes, and minutes
can stietch out to seem like houis (Csikszentmihalyi, zccs, p. 1v). Peihaps in this state a pio-
giammei is thinking in algoiithmic time, auending to contiol ow as it ieplays ovei and ovei
in theii imagination, and not to the woild aiound them. Oi peihaps they aie not auending
to the passage of time at all, thinking entiiely of abstiact logic, in a timeless state of building.
ln eithei case, it would seem that the human is enteiing time ielationships of theii sofwaie,
iathei than the opposite, anthiopocentiic diiection of sofwaie enteiing human time. While
1z
Cu~v1iv e Cvi~1ivi1v
piogiammeis can appeai detached fiom physical time, theie aie ways in which the time-lines
of piogiam development and opeiation may be united, as we saw in oui discussion of notation
in time (.z).
Tempoial ielationships aie geneially not iepiesented in souice code. When a piogiammei
needs to do so, foi example as an expeiimental psychologist iequiiing accuiate time mea-
suiements, oi a musician needing accuiate synchionisation between piocesses, they iun into
pioblems of accuiacy and latency. With the wide piolifeiation of inteiacting embedded sys-
tems, this is becoming a bioad concein (Lee, zccv). ln commodity systems time has been de-
centialised, abstiacted away thiough layeis of caching, wheie exact tempoial dependencies
and inteivals between events aie not deemed woithy of geneial inteiest. Piogiammeis talk
of piocessing cycles as a valuable iesouice which theii piocesses should conseive, but they
geneially no longei have piogiammatic access to the high fiequency oscillations of the cential
piocessing units (now, fiequently pluial) in theii computei. Te allocation of time to piocesses
is oiganised top-down by an oveiseeing schedulei, and piogiammeis must woik to achieve
what timing guaiantees aie available. All is not lost howevei, iealtime keinels aie now avail-
able foi commodity systems, allowing psychologists (linney, zcc1) and musicians (e.g. via
http://jackaudio.org/) to get closei to physical time. luithei, the iepiesentation of time
semantics in piogiamming is undeigoing active ieseaich in a sub-eld of computei science
known as reactive programming (llliou, zccv), ideas fiom which have inspiied iepiesentation
of time in the Tidal language (1.).
e. lmbodied piogiammeis
What we have seen piovides stiong motivation foi addiessing the paiticulai needs of aitist-
piogiammeis. Tese include conceins of woikow, wheie elapsed time between souice code
edits and piogiam output slows the cieative piocess. Conceins of piogiamming enviionment
aie also impoitant, which should be optimised foi the piesentation of shoitei piogiams in theii
entiiety to suppoit biicolage piogiamming, iathei than hieiaichical views of laigei codebases.
Peihaps most impoitantly, we have seen motivation foi the development of new piogiamming
languages, pushing the boundaiies to gieatei suppoit aitistic expiession.
liom the embodied view we have taken, it would seem useful to integiate time and space
fuithei into piogiamming languages. ln piactice integiating time can mean on one hand in-
cluding tempoial iepiesentations in coie language semantics, and on the othei uniting devel-
opment time with execution time, as we have seen with inteiactive piogiamming. Tempoial
semantics and live coding both alieady featuie stiongly in some piogiamming languages foi
1zs
Cu~v1iv e Cvi~1ivi1v
the aits, as we saw in e.e, but how about analogous developments in integiating geometiic
ielationships into the semantics and activity of piogiamming` lt would seem the appioaches
shown in Nodal, the ReacTable and Textuie desciibed in chaptei aie showing the way towaids
gieatei integiation of computational geometiy and peiceptual models into piogiamming lan-
guage. Tis is alieady seiving aitists well, and piovides new giound foi visual piogiamming
language ieseaich to exploie.
lailiei we quoted Paul Klee (e.I), a paintei whose pioduction was limited by his two
hands. Te aitist-piogiammei has dieient limitations, but shaies what Klee called his limita-
tion of ieception, by the limitations of the peiceiving eye. Tis is peihaps a limitation to be
expanded but not oveicome, iathei celebiated and fully exploied using all we have, including
oui new computei languages. We have chaiacteiised a biicolage appioach to aitistic piogiam-
ming as an embodied, cieative feedback loop. Tis places the piogiammei close to theii woik,
giounding disciete computation in oiientational and tempoial metaphois of theii human ex-
peiience. Howevei the computei inteipietei extends the piogiammeis abilities beyond theii
own imagination, making unexpected iesults likely, leading the piogiammei to new cieative
possibilities.
e.s Live Coding in Piactice
Two live coding systems have been intioduced in this thesis, the Tidal pauein DSL (1.) and
the ielated visual language Textuie (.e). We have discussed technical aspects of live coding in
the piocess, but until now have not discussed the cieative piactice of live coding. Te teim live
coding emeiged aiound zccI, to desciibe the activity of gioup of piactitioneis and ieseaicheis
who had begun developing new appioaches to making computei music and video animation
(Collins et al., zccI, Waid et al., zcc1, Blackwell and Collins, zcc, Rohihubei et al., zcc). lt
is dened by Waid et al. (zcc1) as the activity of wiiting a computei piogiam while it iuns,
wheie changes to the souice code aie enacted by the iunning piocess without bieaks in musical
oi visual output. Te aichetypal live coding peifoimance involves piogiammeis wiiting code
on stage, with theii scieens piojected foi an audience, theii code dynamically inteipieted to
geneiate music oi video.
Closely ielated teims aie interactive, on-the-y (Wang and Cook, zcc1), conversational
(Kupka and Wilsing, 1vsc), just-in-time (Rohihubei et al., zcc) and with-time (Soiensen and
Gaidnei, zc1c) piogiamming. Many of these teims aie inteichangeable, although theie aie dif-
feiences of technique and emphasis, foi example live coding is most ofen used in the context
of impiovised peifoimance of music oi video animation. Tis context of impiovised computei
1zv
Cu~v1iv e Cvi~1ivi1v
music is adopted heie, and although much of the following could be ielated to woik in live
video animation, focus on computei music is kept foi bievity.
ln live coding the peifoimance is the process of sofwaie development, iathei than its out-
come. Te woik is not geneiated by a nished piogiam, but thiough its jouiney of development
fiomnothing to a complex algoiithm, geneiating continuously changing musical oi visual foim
along the way. Tis is by contiast to generative art, populaiised by the geneiative music of Biian
lno (1vve). Geneiative ait may be undeistood by a gaidening analogy, wheie piocesses aie
composed as seeds, planted in a computei and lef to giow. Te iandom numbei geneiatois
ofen used to piovide vaiiation in geneiative piocesses have led to theii being likened to the
constiuction of wind chimes, in that they aie stiuctuies that aie played by souices of noise.
Like wind chimes, while geneiative ait may constantly vaiy, geneiative systems which pio-
duce qualitative changes aie iaie. Output moie oi less follows the same style, with only a few
peimutations giving an idea of the qualities of the piece. Tis is well illustiated by oui case
study of an aitist-piogiammei (e.I), who ian theii piogiam a few times not to pioduce new
woiks, but to get dieient peispectives on the same woik.
With live coding, hands-on human involvement is essential to the development of a piece.
Metaphoiically speaking, iathei than sowing seeds, live codeis metaphoiically constiuct plants
fiom sciatch by splicing dieient plants togethei, modifying theii DNA while they giow, and
expeiimenting with dieient ways of destioying them foi aitistic eect. With geneiative ait,
onlookeis aie ofen lef to question whethei the piogiammei oi computei is the cieative agent
in the aitistic piocess. Howevei with live coding theie is no question, the piogiammei veiy
visibly piovides all the iules, the human act of piogiamming pioviding all cieative impetus,
and the computei piocess extending the human iange of exploiation.
Live coding allows a piogiammei to examine an algoiithm while it is inteipieted, taking
on live changes without iestaits. Tis unites the time ow of a piogiam with that of its de-
velopment, using dynamic inteipietation oi compilation. Using techniques outlined in (.z),
live coding makes a dynamic cieative piocess of test-while-implement possible, iathei than the
conventional implement-compile-test cycle. Te cieative piocesses shown in liguies e.z and
e.I still apply, but aie fieed fiom the constiaints of time, with the aiiows now iepiesenting
concuiient inuences between components iathei than time-oideied steps.
Live coding not only piovides an ecient cieative feedback loop, but also allows a pio-
giammei to connect sofwaie development with time based ait. Tis is biicolage piogiamming
(e.I.1) taken to a logical and aitistic conclusion, paiticulaily with aichetypal blank slate live
coding. Heie iisk is embiaced and pie-planning eschewed, the aim being to design a piogiam
in the moment wheie it is implemented and executed in the expectant atmospheie cieated by
1Ic
Cu~v1iv e Cvi~1ivi1v
an audience.
Te piimaiy ieseaich focus aiound live coding piactice has been upon the integiation of
peifoimance time with development time, foi example in the live coding papeis alieady cited.
Tis is impoitant woik, as human peiception of the piogiession of time duiing the evaluation
of an algoiithm has ofen been delibeiately ignoied in computei science (e.e). Tis line of
ieseaich is ceitainly not complete, but theie aie now seveial woiking appioaches to impiovis-
ing music thiough live code development. Some ieseaich emphasis has theiefoie moved fiom
time to space, that is, to the consideiation of visuospatial peiception within the activity and
spectacle of live coding peifoimance.
6.8.1 Obscurantism is dangerous. Show us your screens.
Te piesent section title is taken fiomthe manifesto diafed by the Tempoiaiy Oiganisation foi
the Piomotion of Live Algoiithm Piogiamming (TOPLAP, Waid et al., zcc1), a gioup set up by
live codeis to discuss and piomote live coding. lt neatly encapsulates a pioblem at the heait of
live coding, live codeis wish to showtheii inteifaces so that the audience can see the movement
and stiuctuie behind theii music, howevei in positioning themselves against the computei
music tiadition of hiding behind laptop scieens (Collins, zccI), they aie at iisk of a chaige of
gieatei obscuiantism. Most people do not know how to piogiam computeis, and many who
do will not know the paiticulai language in use by a live codei. So, by piojecting scieens, do
audience membeis feel included by a gestuie of openness, oi excluded by a gibbeiish of code
in an obscuie language` Do live coding peifoimances fostei melding of thoughts between
peifoimei and audience, oi do they cause audience membeis to feel stupid` Audiences have
not yet been foimally suiveyed on this issue, but anecdotal expeiience suggests both ieactions
aie possible. Anon-piogiammei inteiviewee in a BBCnews item(Piogiamming, meet music,
zsth August zccv) iepoited ignoiing piojected scieens and just listening to the music, and less
ambiguous negative ieactions have been iumouied. On the othei hand, a populai live coding
tale has it that afei enjoying a live coding peifoimance by Dave Giiths in Biussels (loAM
studios, 1th Decembei zcc), a non-piogiammei tuined to theii lovei and was oveiheaid to
exclaim Now l undeistand! Now l undeistand why you spend so much time piogiamming.
Paitly in ieaction to the issue of inclusion, a new diiection of ieseaich into visual program-
ming has emeiged fiom live coding piactice, evident in the systems ieviewed and intioduced
in the pievious chaptei. Te challenge is to nd new ways of notating piogiams suitable not
only foi containing the expiessions of a well-piacticed live codei, but foi doing so in a way
meaningful to an audience.
1I1
Cu~v1iv e Cvi~1ivi1v
6.8.2 Cognitive Dimensions of Live Coding
Blackwell and Collins (zcc) have examined live coding with iespect to the Cognitive Dimen-
sions of Notation (CDN), using it to compaie the ChucK language foi piogiamming on-the-y
computei music (Wang and Cook, zcc1) with the commeicial Ableton Live pioduction sof-
waie. ChucK, and by implication live coding in geneial, does not come o paiticulaily well.
lt has low on the dimensions of visibility, closeness of mapping and role-expressiveness, is
error-prone and iequiies hard mental operations in pait to deal with its high level of ab-
straction. lt would seem that the progressive evaluation and iepiesentational abstraction
oeied by ChucK come at a cost. Nonetheless, these aie costs that many aie willing to ovei-
come thiough iigoious piactice iegimes ieminiscent of instiumental viituosos (Collins, zcc).
Tey aie willing to do so because abstiaction, while taking the impiovisei away fiom the diiect
manipulation that instiumentalists enjoy, allows them to focus on the compositional stiuctuie
behind the piece. Being able to impiovise music by manipulating compositional stiuctuie in
theoietically unbound ways is too auiactive a piospect foi some to ignoie.
lstablished noims place the live codei in a stage aiea sepaiate fiom theii audience mem-
beis
z
, who depending on the situation, may listen and watch passively oi inteiact enthusiasti-
cally, peihaps by dancing, shouting oi scieaming. We theiefoie have two gioups to considei,
the peifoimeis needing to woik in the moment without technical inteiiuptions that may bieak
cieative ow (Csikszentmihalyi, zccs), and the audience membeis needing to feel included in
the event, while engaged in theii own cieative piocess of musical inteipietation (1.1). Teie
is a challenge then in ieconsideiing live coding inteifaces, cieating new languages positioned
at a place within the CDN well suited foi a bioadei base of musicians and audiences who may
wish to engage with them. Te question is not just how musicians can adapt to piogiamming
enviionments, but also the inveise, how may piogiamming enviionments, ofen designed to
meet the needs of business and militaiy institutions, be iethought to meet the paiticulai needs
of aitists` liist, we should considei what those needs might be.
An inteiesting cognitive dimension with iespect to live coding is error-proneness. Teie
aie dieient avouis of eiioi, some of which aie much celebiated in electionic music, foi
example the glit genie giew fiom an inteiest in mistakes and bioken equipment (Cascone,
zccc). ln impiovisation, an unanticipated outcome can piovide a cieative spaik that leads a
peifoimance in a new diiection. We would classify such desiiable events as semantic eiiois,
in contiast with syntactic eiiois which lead to ciashes and hasty bug-xing.
ln teims of the CDN, biicolage piogiamming iequiies high visibility of components, in
z
Peifoimance noims aie of couise extensively challenged both inside (Rohihubei et al., zcc) and outside (Small,
1vvs) live coding piactice.
1Iz
Cu~v1iv e Cvi~1ivi1v
paiticulai favouiing shoitei piogiams that t on a single scieen, and avoiding unnecessaiy
abstraction. Heie is a conict as noted above abstiaction sets live coding apait fiom othei
appioaches to impiovisation in computei music, but also acts as an obstacle to biicolage pio-
giamming. We aie pulled in dieient diiections, and so look foi the happiest medium, a com-
mon iesult fiom taking a CDN peispective. Some piogiammeis, known in some quaiteis as
aichitectuie astionauts, enjoy intioducing many layeis of abstiaction that only seive to ob-
fuscate (Spolsky, zcc1). Biicolage piogiammeis aie the opposite in wanting to be as close to
theii woik as possible. Tis is not howevei a case of iemoving all abstiaction, but nding the
iight abstiaction foi the woik. Piogiamming afei all is an activity that takes place somewheie
between electiic tiansistois and lambda calculus the tiick is nding the iight level of abstiac-
tion foi the pioblem domain (1.I). Accoidingly a computei musician may nd having to deal
with individual notes a distiaction, and that a layei of abstiaction above them piovides the
cieative suiface wheie they can feel closest to theii composition.
e.v Live codeis on computational cieativity
Cieativity is ofen touched upon in the study of music, but iaiely appioached in detail. Mu-
sicians may woiiy that analysing theii cieative piocesses may somehow spoil them, as if self
ieection can be destiuctive if appioached in too foimal a mannei. Te many points of view
as to the natuie of cieativity, with common disagieement, may lead a scholai to lose inteiest
and look foi a beuei dened eld of ieseaich. Howevei such an impoitant subject deseives
auention, and light is being thiown by woik within sub-elds of philosophy, psychology and
aiticial intelligence, which each contiibute to foim the cioss-disciplinaiy eld of computa-
tional cieativity.
A suivey was caiiied out with the bioad aim of gatheiing ideas foi study of computational
cieativity fiom a live coding peispective. An on-line discussion gioup foi live codeis hosted
by the Tempoiaiy Oiganisation foi the Piomotion of Live Algoiithm Piogiamming (TOPLAP,
Waid et al., zcc1) was asked to ll out an on-line suivey. To encouiage honest iesponses the
suivey was anonymous, and demogiaphic infoimation was not collected. Discussion of cie-
ativity can ievolve aiound ill-dened teims and invite piejudice. loi this ieason, the woid
cieativity was not used in the invitation oi suivey text, and questions addiessing issues of
cieativity weie mixed with geneial questions about live coding. A total of Iz completed the
suivey (although not all answeied eveiy question), and Ic of whom indicated at the end that
they weie happy foi theii answeis to be published undei a cieative commons auiibution li-
cense. Te suivey and iesponses weie in lnglish as the language used by the discussion gioup,
1II
Cu~v1iv e Cvi~1ivi1v
although foi seveial this was a second language.
6.9.1 e subjects
Te iespondents weie a bioad cioss-section of live codeis, with useis of the six pie-eminent
live coding enviionments iepiesented, between ve and fouiteen foi each system (many had
used moie than one). A laige piopoition (zz/Iz) had used one of the listed classes of tiaditional
musical instiuments, with comments suggesting a highei peicentage would have iesulted if
electionic instiuments weie included. liom this we can assume a gioup with a geneially iich
musical backgiound. Relatedly, almost all (Ic/Iz) of subjects live coded music, wheieas only a
small piopoition (e/Iz) live coded video animation, live coding would appeai to cuiiently be a
music led cultuie. Teie weie a diveise iange of appioaches to the question of how to dene
live coding in one sentence, the iesults aie iathei unquantiable but the ieadei is iefeiied to
Appendix A.1 to enjoy the iesponses. While the iesponses show some diveisity of appioach
to live coding, because the subjects had all used at least one of the main languages it is safe to
assume that they aie woiking to laigely the same technical denition.
6.9.2 Creating language
Computei useis ofen discuss and undeistand computei piogiams as tools, helping them do
what they need eciently and without geuing in the way. loi a piogiammei it would instead
seem that a computei language is an immeisive environment to cieate woik in. lndeed a suite
of souice code editing sofwaie is collectively known as an lnteiactive Development lnviion-
ment (lDl). lt is inteiesting then to considei to what extent live codeis adapt theii computei
languages, peisonalising theii enviionments, peihaps in oidei to aid cieativity. Ovei two thiids
(z1/Iz) collected functions into a libiaiy oi made an extensive suite of libiaiies. Tis is analo-
gous to adding woids to a language, and shows the extent of language customisation. Asmallei
piopoition (e/Iz) had gone fuithei to implement theii own language inteipietei and smallei
numbei still (/Iz) had designed theii own language. Tat these aitists aie so engaged with
making fundamental changes to the language in which they expiess theii woik is impiessive.
6.9.3 Code and style
liom the peispective of computational cieativity, it is inteiesting to examine the ielationship
that live codeis have with theii code. An auempt at quantifying this was made by asking,
When you have nished live coding something you paiticulaily like, how do you feel towaids
the code you have made (as opposed to the end iesult)` Ovei half (1/Iz) indicated that code
iesulting fiom a successful live coding session was a desciiption of some aspect of theii style.
1I1
Cu~v1iv e Cvi~1ivi1v
Tis suggests that many feel they aie not encoding a paiticulai piece, but howto make pieces in
theii own paiticulai mannei. Aiound the same numbei (1/Iz) agieed that the code desciibes
something they would piobably do again, which is peihaps a iephiasing of the same question.
A laige numbei, (z1/Iz) answeied yes to eithei oi both questions. Teie aie many ways in
which these questions can be inteipieted, but oveiall this suggests that many subjects feel they
have a stylistic appioach to live coding that peisists acioss live coding sessions, and that this
style is somehow iepiesented in the code they make.
6.9.4 Live coding as a novel approa
Te subjects weie asked the open question What is the dieience between live coding a piece
of music and composing it in the sequencei (live coding an animation and diawing one)` ln
othei woids, howdoes live coding aect the way you pioduce youi woik, and howdoes it aect
the end iesult` Te answeis aie dicult to summaiise, and so again the ieadei is diiected to
Appendix A.z to iead the full iesponses. Some inteiesting points ielevant to computational
cieativity aie selectively quoted foi comment heie.
l have all but [abandoned] live coding as a iegulai peifoimance piactice, but
l use the skills and condence acquiied to modify my sofwaie live if l get a new
idea while on stage.
Tis admission, that geuing new ideas on stage is infiequent, makes an impoitant as well
as humble point. ln teims of the Cieative Systems liamewoik, we can say that live coding
is useful in peifoimance if you need to modify youi conceptual space (the kind of woik you
want to make), oi youi tiaveisal stiategy (the way you tiy to seaich foi oi make it). lf you aie
content with having both xed in advance, then live coding is not waiianted. ln othei woids,
live coding is useful foi invoking tiansfoimational cieativity, although as with this test subject,
tiansfoimational cieativity is not always desiiable in fiont of a paying, iisk-aveise audience.
When l woik on wiiting a piece l can peifect each sound to be piecisely as
l intend it to be, wheieas [when] live coding l have to be moie geneialised as to
my intentions.
Tis iespondent is making the point that live codeis woik at least one level of abstiaction
away fiom enacting individual sounds.
Peihaps most impoitantly the highei pace of livecoding leads to moie impul-
sive choices which keeps things moie inteiesting to cieate. Not suie how ofen
that also cieates a moie inteiesting end iesult but at least sometimes it does.
1I
Cu~v1iv e Cvi~1ivi1v
Tis is inteiesting with iefeience to the cieative feedback loop of biicolage piogiamming
(e.I.1). Live coding allows a change in code to be heaid oi seen immediately in the output,
with no foiced bieak between action and ieception. Tis is in staik contiast to those whose
expeiience of sofwaie development as slow and aiduous.
Live coding has fai less peifection and the pioduct is moie immediate. lt
allows foi impiovisation and spontaneity and discouiages ovei-thinking.
Tis come as a suipiise, as live coding has a ieputation foi being ceiebial and ovei technical.
ln ieality, at least when compaied to othei sofwaie based appioaches, the immediacy of iesults
fosteis spontaneous thought.
Live coding is iiskiei, and one has to live with [unt decisions]. You cant just
go one step back unless you do it with a nice piioueue. Teiefoie the end iesult is
not as clean as an oine-composition, but it can lead you to places you [usually]
nevei would have ended.
Tis comment is paiticulaily incisive, the peculiai ielationship that live codeis have with
time does indeed give a ceitain element of iisk. Riskiei ways of making music aie moie likely
to pioduce abeiiant output, not of the type you weie looking foi. Howevei, wheie such out-
put tuins out to be valuable, then you have the oppoitunity to iedene what you aie looking
foi, tiansfoiming youi conceptual space. lf the output tuins out to be pooi, then you can at
least change the way you woik to avoid similaily pooi output in the futuie, mechanisms of
tiansfoimational cieativity.
while live coding is a peifoimance piactice, it also oeis the tantalising
piospect of manipulating musical stiuctuie at a similai abstiact level as defeiied
time composition. To do this eectively in peifoimance is l think an entiiely dif-
feient skill to the standaid one-acoustic-event-pei-action physical instiumental
peifoimance, but also quite dieient to compositional methods which typically
allow foi iewoik.
Tis ieally gets to the nub of what live coding biings to the impiovising aitist an alteied
peispective of time, wheie a single edit can aect all the events which follow it.
6.9.5 Computational creativity
ln using piogiamming languages to make music, live codeis have a unique peispective on the
question of computational cieativity. lt is inteiesting then to measuie the extent of optimism
foi computeis taking a gieatei iole in the cieative piocess than they alieady do. Towaids this
the subjects weie given a seiies of statements and asked to guess when each would become
1Ie
Cu~v1iv e Cvi~1ivi1v
tiue, with the peitinent iesults in liguie e.1. Regieuably theie was a conguiation eiioi eaily
on in the suiveyed peiiod, iequiiing the answeis of two subjects to be discaided.
Optimism foi the statement Live coding environments will include features designed to give
artistic inspiration to live coders was veiy high, with just ovei half (11/z) claiming that was
alieady tiue, and almost all (z/z) agieeing it would become tiue within ve yeais. Tis in-
dicates stiong suppoit foi a weak foim of computational cieativity as a cieative aid foi live
codeis.
Somewhat suipiisingly, optimism foi the stiongei foim of cieativity suggested by Live
code will be able to modify itself in an artistically valued manner was also high, with two fhs
(11/zs) claiming that was alieady possible. lf that is the case, it would be appieciated if the
live code in question could make itself known. Peihaps they aie iefeiiing to feedba.pl, a live
coding editoi foi the Peil piogiamming language. Tis editoi does indeed allowself-modifying
code, but we aie some way o fiom seeing an aitistic computational agent emeige fiom it.
Moie pessimism is seen in iesponse to A computer agent will be developed that produces a
live coding performance indistinguishable from that of a human live coder, with a thiid (v/z)
saying that this will nevei happen. Tis question is posed in iefeience to the imitation game
detailed by Tuiing (1vc), howevei oui veision involving musical iathei than language based
imitation seems iathei easiei to full. As one subject commented, the test indistinguishable
fiom a human is veiy loose and theie can be some veiy bad human live coding music. Tat
would peihaps explain why half (1I/z) of iespondents thought the statement was eithei al-
ieady tiue oi would become so within ve yeais.
6.9.6 Discussion
What if a computational appioach to the musicology of live coding weie to develop, wheie ie-
seaicheis deconstiuct the code behind live coding impiovisations as pait of theii woik` Coiie-
lations between expiessions in foimal languages and musical foimin sound could be identied,
and the development of newways of expiessing newmusical foims could be tiacked. lf success-
ful, the iesult need not be a new kind of music, but could be music undeistood in a novel way.
Peihaps this new computational appioach to undeistanding music that could piove invaluable
in the seaich foi a musically cieative sofwaie agent.
ln looking at cieativity thiough the eyes of live codeis, we can see some piomise foi com-
putational cieativity even at this eaily stage of development of both elds. Live codeis feel
theii musical style is encoded in the code they wiite, and that theii language inteifaces pio-
vide them with inspiiation. Tey aie actively developing computei languages to beuei expiess
the music they want to make, cieating computei language enviionments that fostei cieativity.
1I
Cu~v1iv e Cvi~1ivi1v
s4 s5 s7
Never
Over 10 years
510 years
15 years
In the next year
That's true already
When do you think the following will become true?
R
e
s
p
o
n
s
e
s
0
5
1
0
1
5
2
0
2
5
3
0
Figure 6.4: Responses to statements s4: Live coding environments will include features designed to give
artistic inspiration to live coders, s5: Live code will be able to modify itself in an artistically valued manner
and s7 A computer agent will be developed that produces a live coding performance indistinguishable from
that of a human live coder.
liom heie it is easy to imagine that live coding enviionments could become moie involved in
the cieation of highei oidei conceptual iepiesentations of time-based ait that live codeis aie
conceined with. Peihaps this will piovide the language, enviionment and application in which
the cieative piocesses of a computational agent will one day thiive.
e.1c Slub
Te band Slub begun in the yeai zccc, as a collaboiation between Adiian Waid (http:
//www.adeward.com/) and the piesent authoi. Tey both shaied a desiie to make music and
enthusiasm foi piogiamming, and iesolved to combine them.
So if youve got piogiamming skills and enjoy making music, it makes sense
to combine them - just like combining poueiy and integial mathematics can be
stimulating. (Adiian Waid in inteiview with Shulgin, zccI)
Slub established a cleai aim eaily on, to make people dance to theii algoiithms. Tey ist
met this aim in the Paiadiso club duiing the zcc1 Sonic Acts festival in Amsteidam, the ist of
1Is
Cu~v1iv e Cvi~1ivi1v
many appeaiances at majoi luiopean festivals.
Figure 6.5: A screenshot showing a typical performance desktop used by the present author circa 2003.
An eaily Slub system is desciibed in detail by Collins et al. (zccI). ln biief it featuied a
synthesisei and eaily live coding system wiiuen by Waid, and a numbei of beat and bass-
line geneiating piogiams wiiuen by McLean. Although theii piimaiy aim was musical, Slub
enjoyed being faced with the challenge of being accepted as piogiammeis who make music.
To this end they began piojecting theii scieens, confionting audiences with the conceptual
oveilap between theii hand-ciafed sofwaie and the music they pioduced using it.
Tis whole netwoik of sofwaie was wiiuen by slub to full theii individual
needs, foiming an enviionment ideal to theii methods of woiking. Tis biings
us to an impoitant point that the code is not just iunning in the computei. To
explain, when piogiammeis aie watching a computei execute theii own piogiams,
the code is also executing in theii minds. Tey have intimate knowledge of the
piocess, and so can imagine it iunning. ln this way, the code is alive in slub and
in theii computeis, and hopefully also in theii sound and the audience too. Te
music is also alive in these foui places. As fai as slub aie conceined, code is music.
(Collins et al., zccI, pp. IzIIz1)
Slub weie not just challenging theii audiences, but also the use of computeis in the aits
in geneial. Waid was instiumental in challenging the concept of sofwaie as a passive tool
with his sofwaie Auto-Illustrator (Waid et al., zccz). By diawing a satiie on the widely used
Adobe lllustiatoi sofwaie, Auto-lllustiatoi confounded expectations, making it dicult to foi
1Iv
Cu~v1iv e Cvi~1ivi1v
example diaw a stiaight line. Tis biought auention to the aesthetic decisions built into all
sofwaie, leading the Tiansmediale Sofwaie Ait juiy to awaid it ist piize in zcc1. Tis point
was fuithei exploied by McLean with forkbomb.pl, a simple piogiam which pioduced paueins
while pushing a computei system to its limits, in eect visualising low-level inteiactions com-
posed by human system piogiammeis. Tis woik was also awaided the Tiansmediale Sofwaie
Ait awaid, and both aitwoiks weie exhibited in the touiing zccz/zccI Generator gioup show
alongside woiks by Sol Lewiu and Yoko Ono.
I
lollowing the foimation of TOPLAP, Waid focused on live coding (cieating Pure Events,
based on the Traer paiadigm, Nash and Blackwell, zc11), and McLean cieated his own live
coding enviionment, the feedba.pl editoi (McLean, zcc1) foi self-modifying code (.z). liom
then on, Slub peifoimed only using live coding inteifaces. Having developed his own live
coding piactice in paiallel, Dave Giiths (http://pawfal.org/dave/) joined Slub follow-
ing a key joint peifoimance at Sonai festival in zcc. Dave took the Slub aesthetic in a new,
audio/visual diiection, by developing game-like live coding enviionments foi music (McLean
et al., zc1c).
6.10.1 Reections following a Slub performance
Slub peifoimed at the Maison Rouge, Paiis on the Icth Septembei zc11, invited by the Sony
Computei Laboiatoiy in Paiis as pait of theii 1th anniveisaiy celebiations. Desk iecoidings of
the peifoimance aie included on the enclosed DVD, which includes a mix of all thiee peifoim-
eis, and sepaiate iecoidings of each peifoimei. Te piesent authoi inteiviewed Giiths and
Waid in the days following the peifoimance, to ieect upon the peifoimance and the jouiney
that Slub had taken to theii cuiient piactice.
Dave Giiths (DG) [Live coding] seems to dene us at the moment, although
it wasnt always that way. l think peihaps it will giow less impoitant with time.
Te code we base peifoimances on seems veiy unstable in that its constantly be-
ing iewiiuen in dieient languages, we individually switch between visual and
text based piogiamming and back again - lots of liule expeiiments. Despite this
the music seems to giow in a independent mannei, we somehow ietain the knowl-
edge of wheie we aie going duiing a live peifoimance and what we aie doing as
peifoimeis.
Teie aie two stiands of development then, musical and technological. But in Slub peifoi-
mances the code is displayed, and the two stiands aie inteitwined. How impoitant is this`
Adiian Waid (AW) lts impoitant to display oui scieens. But lmnot convinced
piojections aie the best way to do this as they come with quite a bit of baggage
I
http://www.generative.net/generator/
11c
Cu~v1iv e Cvi~1ivi1v
(a) Tidal (4.5)
(b) Texture (5.6)
Figure 6.6: Slub at the Maison Rouge, Paris, 2011. Dave Griths and his SemeBris (McLean et al.,
2010) interface are on house le, Adrian Ward and his Pure Events soware are shown house right. e
present authors screen is displayed in the centre, using soware named in the above captions.
theyie too passive, they ieinfoice the authoi/audience hieiaichy, theyie boiing.
Howevei, technical and piactical means make anything else unfeasible.
DG loi me the piojection is moie impoitant than the sound. Te passivity is
a pioblem, my favouiite gigs aie when people come up and me what lm doing. l
like the bieakdown of the hieiaichy veiy much, and make a point to explain and
talk to them.
Te iole that piojections play in live coding is pioblematic (e.s.1), ieected heie in Waids
unease, indicating that it is impoitant to show scieens, but that in doing so deep issues aie
uneaithed. ln staik contiast Giiths, who has a backgiound in ne ait and computei games
ieseaich, sees the piojection of code as moie impoitant than the sound. Tis is a suipiising
statement, but when asked to expand upon his point, Giiths showed fundamental agieement
with Waid, what is of piime impoitance is not the piojection itself, but that peifoimances
should centie aiound activity.
111
Cu~v1iv e Cvi~1ivi1v
DG l would like to tiy being in the middle of the audience and piojecting
on the ooi aiound us. Oi maybe in small peifoimances, make use of moie of a
physical mateiial electionics, tangible computing oi some use of computei vision
could be a way to iemove the iole of the scieen, and piojection. l think woikshop-
peifoimances aie something to follow up (thinking of youi Textual woikshop at
Access Space, Alex) if you have zc oi so people joining in the piojection is not
impoitant any moie.
ln moie usual situations, peihaps we think about what the absolute minimum
we need to pioject would be - peihaps just the chaiacteis as we piess the keys,
oi the expiession the inteipietei is cuiiently executing. l have a feeling if we
distill it down it could end up being moie meaningful to an audience than seeing
eveiything, all the time.
Slub ensuie they peifoim side-by-side, allowing intei-peifoimei communication which
they deem impoitant
AW Yes. And [we] use gestuies, too - pointing, laughing, dancing etc.
DG Agieed - lm not suie what we talk about, it can be quite minimal like
shall we stop soon oi why am l the only one making sound` but l like it a lot
and l think its ieally impoitant to be able to do that.
ln live coding cultuie, fiom-sciatch coding is held as an ideal, howevei Slub desciibe a
moie ielaxed appioach, with impiovisation taking a staiting point fiom pie-piepaied oi piac-
ticed elements
AW My piepaiation usually takes the foim of piepaiing sound samples, and
ie-leaining what lve foigouen (awkwaid JavaSciipt syntax, mostly). Te actual
peifoimance is a lot of impiovisation but lmnot stiict enough with myself to deny
using pieviously wiiuen code, which l know is a bad habit.
DG l geneially will have a couple of dieient staiting points ieheaised, a set of
possible things l will build and a familiaiity with the inteiface built up befoie a gig.
l think ieheaisal in livecoding is ieally about making suie you can cieate a situation
wheie you have lots of bianching points foi impiovising fiom. lts impoitant to
be comfoitable enough that youi code will piovide some piedictable iesults when
you need them, but exible enough that you can dive into the unknown and see
what happens.
Tis ts with the well established view of impiovisation as being an exploiation of pie-
established musical schemata, which aie not necessaiily changed oi deviated fiom duiing a
peifoimance (Piessing, 1vs).
e.11 Discussion
Live codeis such as Slub aie shifing the discouise aiound the use of computeis in music pei-
foimance, foi example taking the leading ieseaichei in electionic music Simon lmmeison by
11z
Cu~v1iv e Cvi~1ivi1v
suipiise
Te most unexpected inteiface foi live sound contiol to have ieemeiged in ie-
cent yeais is the lexical, ieading and wiiting. Te incieasing dominance of giaphic
inteifaces foi music sofwaie obscuied the continuing piesence of the command-
line tiadition, the code wiitei, the hackei. (lmmeison, zccb, p. 11)
ln an on-line ieseaich statement Live electionics and the acousmatic tiadition (zcc1),
Andiew Deakin ieacted to a comment made by lmmeison duiing a iadio discussion
lmmeison suggested that now we have aoidable and iealtime computei-
based systems theie is ieally only one iemaining aiea needing development in-
teiface and/oi instiument design. l could not agiee moie, a 1cc yeai-old keyboaid
layout is suiely not the answei.
What was missing fiom this discussion is iecognition of the impoitance of language, which
the computei allows us specialised access to. We must not allow the diiectness of embodied
inteifaces to mask the impoitance that the development of language, including computei lan-
guage, has to the development of humanity. By natuie, human cultuie continually ie-cieates
itself, and as computeis play an incieasing iole in society, including in music cultuie, piogiam-
meis have found themselves in an incieasingly piivileged position. Douglas Rushko puts this
well
Digital technology is piogiammed. Tis makes it biased towaid those with the
capacity to wiite the code. ln a digital age, we must leain howto make the sofwaie,
oi iisk becoming the sofwaie. lt is not too dicult oi too late to leain the code
behind the things we use oi at least to undeistand that theie is code behind theii
inteifaces. Otheiwise, we aie at the meicy of those who do the piogiamming, the
people paying them, oi even the technology itself. (Rushko, zc1c, p. 1zs)
But even in zc11, the impoitance of piogiamming languages aie still not felt in music in-
teiaction ieseaich. While deliveiing a papei on live coding at the New lnteifaces foi Musical
lxpiession confeience (Aaion et al., zc11), Sam Aaion expiessed shock that the woid language
was not amongst the top zcc confeience papei keywoids.
1
We aie still developing piogiam-
ming as a cieative piocess, theie is still moie to be done, and much moie to be gained.
1
A video iecoiding of Sam Aaions piesentation is available at http://vimeo.com/26905683/.
11I
Cu~v1iv
Future directions
Te piesent ieseaich has been speculative and integiated with ongoing piactice, and so as
endings aie beginnings, conclusions also seive as intioductions. ln the following we take stock
of the piesent cycle of ieseaich, considei its impacts, and wheie the following cycle may take
us next.
.1 Te lieedom of lnteipietation
Wheie computation is at times hidden and foigouen in computei ait, we foiget some of its
iadical natuie. Tiough piogiamming, aitists aie able to cieate puie, linguistic abstiactions,
then giound them in human senses thiough actuatois in stiiking ways. We have seen how a
piogiamming language inteipietei can t into a human cieative piocess, peifoiming the lin-
guistic stiuctuies we give it, to geneiate output to evoke human peiception. lt seems howevei
that the fieedom to inteipiet piogiams is undei thieat, wheie consumei computeis designed
foi enteitainment aie taking the place of geneial puipose machines. Just as undeilying com-
putei languages aie ofen hidden in digitally-iealised aits, they aie being made incieasingly
inaccessible in end-usei computing in geneial.
Without inteipieteis, we would not have sofwaie, but yet inteipieteis aie also sofwaie.
Tis is why we talk about bootstrapping, wheie sofwaie pulls itself fiom the ooi by its boot-
stiaps, a paiadox seuled by the existence of haidwaie miciocode. Tis is also why Naui (1vvza)
piefeis the woid Dataology to the phiase Computei Science, piogiams aie data, which op-
eiate ovei data.
Any piece of sofwaie exists as a combination of two paits, some instiuctions in a com-
putei language and an inteipietei of that language. Alone they do nothing, put them togethei
and they can notionally do anything. Ofen theie aie inteimediaiy steps, commonly compi-
lation into bytecode, but these aie just tianslations into anothei language, which still iequiie
111
Cu~v1iv lU1Uvi bivic1ioNs
inteipieting as a sequence of instiuctions foi the magic to happen.
lnteipieteis allow us to tiy out ideas beyond oui imaginations, adding some instiuctions,
inteipieting them to get output iendeied as sound oi light to oui senses, peiceiving otheiwise
impossible woilds, and ietuining to the souice code to twist the encoded stiuctuies into new
contoitions inspiied by the iesults so fai. We expand the iealms of peiception thiough compu-
tation, not cieating things but wiiting about ideas in oidei to tiy to invoke them. We aie only
sciatching the suiface of what is possible, aitistic and otheiwise, fiom maiiying high speed
computation with embodied human expeiience.
lt is of concein then that the fieedom of thought given by inteipieteis happens to thieaten
business models of laige companies, who aie accoidingly seaiching foi the powei to make fiee
access to them illegal on the computeis they pioduce. Games consoles aie computeis wheie
the end usei is not allowed access to an inteipietei, unless they pay foi an ofen piohibitively
expensive license. You aie not otheiwise allowed to modify code, ceitainly not allowed to mod-
ify the inteipietei, and so must be satised with using whatevei piogiams the manufactuiei
allows you to.
luitheimoie these business models aie spieading, fiomcomputei games, to handheld com-
puteis and now to tablet computeis. iOS, the opeiating system foi the iPhone and iPad, was
a paiticulai shock as a device coming fiom Apple Computei, lnc., a company pioducing haid-
waie tiaditionally maiketed at the cieative. Howevei to develop and distiibute sofwaie foi
these platfoims you must pay an annual license. Te iOS teims have now ielaxed to the point
that Apple have allowed the distiibution of Codea (http://twolivesleft.com/Codea/), a
tablet based piogiamming enviionment foi the Lua language. Tis is a welcome development,
but still only exists within the teims of Apples business model, and so the ability to shaie pio-
giams with otheis iequiies the puichase of Codea, and cumbeisome tiansfei of souice code
listings.
Tis issue is ielated to the notion of generativity intioduced by Ziuiain (zccv) to desciibe
components of computei systems which aie fundamentally incomplete.
1
Ziuiain gives the
opeiating system as one example, as a system which need not be tied to paiticulai undeily-
ing systems (i.e. the haidwaie layei) oi constiain systems iunning on it (i.e. the application
layei). ln othei woids, geneiative systems aie designed to be used in ways which the oiiginal
designeis do not necessaiily anticipate. Ziuiain (zccv) sees geneiativity as undei thieat due
to secuiity issues which geneiative systems aie pione to, but the thieat also comes fiom aes-
thetic and moial contiol, foi example Apple have blocked sofwaie fiom distiibution on the
basis of aesthetic appeaiance, depiction of sexuality and foi political commentaiy in the foim
1
Tis use of the woid generative is distinct fiom the use in generative art (1.1.z).
11
Cu~v1iv lU1Uvi bivic1ioNs
of iidicule of public guies. As Lessig (zcce) aigues, those in contiol of code eectively set the
iules foi eveiyone else. lt is no suipiise then to see that in following theii vision foi a haidwaie
platfoim, vendois seek to limit and contiol the ability to piogiam it.
lnteipietation is of cential concein in cieative use of computeis, and so the cieep towaids
centialised, coipoiate contiol ovei inteipietation is deeply woiiying. Te home computei
ievolution in the 1vscs biought BBC Miciocomputeis and Sinclaii Spectiums into schools and
the home, which encouiaged piogiamming fiom the moment they weie switched on. Using
these computeis empoweied childien to cieate with language undei theii own teims (Tuikle,
zcc). lt would be a gieat shame if this exposuie to piogiamming was lost. One way to piotect
the fieedom of inteipietation may simply be thiough the development of novel appioaches to
piogiamming language design. lf newways of piogiamming engage end-useis, then consumei
piessuie may be enough to pieseive theii fieedom to piogiam. Tis can alieady be seen in the
development of Codea, which is hopefully just the beginning of a sofwaie genie, engaging end
useis in piogiamming in and of these new platfoims.
.z Sofwaie engineeiing standaids
As sofwaie takes an evei incieasing iole in stiuctuiing life in the developing woild, considei-
able woik has been put into the piocesses of sofwaie engineeiing. Methodologies of sofwaie
design piomote methods to stieamline the specication, planning, development, testing, and
deployment of sofwaie. Howevei like the design of piogiamming notations (.1), standaid
development piactices (such as the application of lSO vccc to sofwaie engineeiing) aie de-
signed within paiticulai constiaints of commeicial development which may not necessaiily
apply to moie expeiimental oi cieative situations. Biicolage piogiamming (e.I.1) is a case in
point, wheie specications aie dened by the act of piogiamming, iathei than vice-veisa. As
the piogiammei is engaged in end-usei piogiamming, theie is no othei usei to satisfy, and
foimal discussions with audience membeis aie haidly piactical. Te design goals aie theiefoie
inteinal and highly changeable.
Piactices such as paii piogiamming and test-diiven development aie intended to make
development both ieliable and iesponsive to iequiiements, and aie collectively known as agile
piogiamming. Howevei built-in assumptions of ieaction to exteinal iequiiements, iathei than
an individuals exploiation and expeiimentation, mean they aie inappiopiiate foi the piocesses
of aitist-piogiammeis, who may quite iightly iesist foimalisation of theii peisonal cieative
piocesses, which may caiiy impoitant aspects of theii aitistic style.
Biicolage piogiamming is in staik opposition to the test-diiven development movement,
11e
Cu~v1iv lU1Uvi bivic1ioNs
wheie a suite of small piogiams aie wiiuen to test new featuies and xes before they aie im-
plemented. A piogiam is then judged to be featuie complete when all the tests pass. Tis is
akin to desciibing a piogiams behavioui twice, istly to outwaidly test it and secondly to
inwaidly implement it. Rathei than developing piogiammatic denitions of behavioui befoie
implementation, biicolage piogiammeis make decisions on the basis of behavioui while imple-
menting it. Tese aie dieient ways of navigating a pioblem space, test-diiven development
is useful foi aiiiving at a pie-aiianged taiget, wheieas biicolage piogiamming is useful foi
exploiing, looking foi novel outputs that aie valuable in potentially suipiising ways.
Anothei point of diveision is the mannei in which piogiammeis collaboiate. ln industiy,
theie is stiong emphasis on gioup woik in teams, with gioup oveiiiding individual identity,
paiticulaily wheie individuals aie employed on xed teim contiacts. ln the aits piogiammeis
may be employed on laigei piojects undei similai teims, as technical, aitist assistants. Howevei
the aitist-piogiammei ofen woiks alone, to iealise theii own individual woiks. Piactices such
as pair programming, wheie two piogiammeis woik togethei on one computei, aie iathei alien
to this situation. While aitists and musicians do woik togethei on laige, fiee/open souice
piojects, the development is usually centied aiound a paiticulai individual. Tis is even the case
in laige piojects, such as the development of piogiamming languages foi cieative woik. Tis
was the case both foi lmpiomptu and SupeiCollidei, wheie development followed a pauein
wheie the ist yeais of development is conducted by a single individual. Only when theii
vision has been fully scoped out is the codebase opened, and a widei fiee/open souice sofwaie
development community foimed.
.I Cyclic ievision contiol
We have seen how aitist-piogiammeis may have unconventional sofwaie design piocesses
(e.I), placing paiticulai demands on theii languages and tools. Revision contiol systems aie
among the most impoitant membeis of a piogiammeis toolbox, allowing the histoiy of a pio-
giam to be annotated and managed. Modein ievision contiol systems allow dieient pio-
giammeis to woik on theii own bianches of code, which aie then meiged back into the main
ow at a latei date. Wheie pioblems aiise in development, ievision contiol systems can help
developeis undeistand the souice of the pioblem, by pioviding histoiical iefeience. Tese sys-
tems alieady have much to oei aitist-piogiammeis, but could ievision contiol be iethought
to beuei meet theii paiticulai needs`
Considei a live codei, wiiting sofwaie to geneiate a music peifoimance. ln teims of ie-
vision contiol they aie in an unusual situation. Noimally we think of piogiammeis making
11
Cu~v1iv lU1Uvi bivic1ioNs
ievisions towaids a nal iesult oi milestone, at which point they ship, packaging and ieleas-
ing theii code foi otheis to use. loi live codeis, eveiy ievision they make is pait of the nal
iesult. ln this case nothing gets shipped, as they aie alieady the end useis. We might somewhat
ciassly think of live coding in teims of shipping a pioduct to an audience, but ieally what is
being shipped is not sofwaie, but a sofwaie development piocess, as musical development.
Also unusual in live coding ievisions is that wheieas conventional sofwaie development
may begin with nothing, and nish with a complete, complex stiuctuie, a live codei both begins
and ends with nothing. Rathei than aim foi a lineai path towaids a piedened goal, musicians
instead aie conceined with how to ietuin back to nothing in a satisfying aic. A live codei may
begin theii peifoimance with a blank editoi, and be faced with an immediate decision about
what to build, and how. As they impiovise by encoding and modifying musical ideas, they
eventually piogiess towaids a conclusion. Teii nal challenge is how to end, some inciease
complexity to a ciescendo and nish abiuptly, and otheis deciease complexity to the minimum,
befoie nal ieduction back to an empty editoi and silence.
Teie aie two ways of thinking about time, eithei as a lineai piogiession, oi as a iecuiient
cycle oi oscillation, as shown in liguie .1. Tese appioaches aie not mutually exclusive, they
iathei piovide dieient ways of looking at the same tempoial piocesses. Conventional sofwaie
design piocesses aie chaiacteiised in teims of cycles of development, with iepeating paueins
between milestones. Nonetheless, it is not conventional to think of the code itself ending up
back wheie it staited, while duiing music peifoimance, we ofen do ietuin to piioi states.
We aie all familiai with choius and veise stiuctuie foi example, and peifoimances necessaiily
begin and end at silence.
lt may be that if we ieconsidei code development in teims of time cycles iathei than lineai
piogiession, then we could nd new ways of suppoiting sofwaie development of music and
othei time based aits. Without fuithei speculative ieseaich it is dicult to visualise what the
outcome of this might be, but theie is alieady some active, iadical woik in the iepiesentation
of development time. loi example ievision contiol is a cential pait of the lield language en-
viionment (http://openendedgroup.com/field/), and allows the piogiammei to sciub
the development timeline. lt seems howevei that in such expeiimental systems, the ievision
contiol timeline has so fai been tieated as a lineai stiuctuie, with occasional paits bianching
and ie-meeting the main ow latei on. lt is not unheaid of foi timelines to feed back on them-
selves in conventional sofwaie development, a piocess called baporting, but this is geneially
avoided, only done in uigent ciicumstances such as in applying secuiity xes to old sofwaie
veisions.
What if instead of being lineai, sofwaie development timelines weie of cycles within cy-
11s
Cu~v1iv lU1Uvi bivic1ioNs
Figure 7.1: Below gure and caption reproduced from Buzsaki (2006, pg .7) with permission
period
time
phase
a
m
p
l
i
t
u
d
e p
h
a
s
e
amplitude
Pantha rei Circle of life
Oscillations illustiate the oithogonal ielationship between fiequency and time
and space and time. An event can iepeat ovei and ovei, giving the impiession of
no change (e.g., ciicle of life). Alteinatively, the event evolves ovei time (pantha
iei). Te foiwaid oidei of succession is a main aigument foi causality. One peiiod
(iight) coiiesponds to the peiimetei of the ciicle (lef).
cles, with ievision contiol designed not to aid piogiession towaids futuie featuies, but help
the piogiammei wiestle theii code back towaids the state it was in ten minutes ago, and ten
minutes befoie that` We leave this as a question foi futuie ieseaich, but suggest the iesult
could be a system that beuei suppoits thematic development in music and video animation.
.1 Conclusion
My view is that todays computei woild is based on techie misundeistandings
of human thought and human life, and the imposition of inappiopiiate stiuctuies
on things we want to do in the human woild. Ted Nelson (zccs, http://
youtu.be/zumdnI4EG14)
Day to day, it does not paiticulaily mauei that we do not know the mechanisms behind
oui own actions, we can simply leain thiough doing. Te need foi theoiy is sometimes deiided
in music cultuie, the long-lived phiase wiiting about music is like singing about economics
tiaced back neaily 1cc yeais (H. K. M., 1v1s), moie iecently taking the foim of compaiison with
dancing about aichitectuie. Te sentiment is that music is about activity, and not theoiising.
Teoiising is dicult foi aitist-piogiammeis to avoid howevei, including those making
music. ln oidei to wiite a geneiative music piogiam, an aitist-piogiammei must necessaiily
theoiise and encode musical stiuctuie in language. lndeed wiiting computei piogiams to make
music is a foim of wiiting about music it iequiies the intiospection, abstiaction and foimal-
isation that many have deiided by compaiison with aichitectuial dance. Not only is wiiting
11v
Cu~v1iv lU1Uvi bivic1ioNs
about the piocesses of ait necessaiy, but in the piocess, coming to gieatei undeistanding of
ouiselves allows us to ieach beyond what would otheiwise be possible.
With intiospection as oui motivation, we have tiied to chaiacteiise the innei piocesses of
aitist-piogiammeis, in teims of inteitwined analogue and digital iepiesentations. Tis leaves
many questions unansweied and indeed unposed, but piomotes a balanced view of the aitist-
piogiammei engaged closely both with theii taiget medium, and the meta-medium of the
souice code. We piopose that this cieative appioach should be embiaced with the full iange
of human faculties available.
Te ideas of the aitist-piogiammei aie wiapped in both analogue and digital packages, and
as they aie unwiapped, the piogiams that spiing foith fiom ngeis at keyboaid have a tiace
of analogue ielations in conceptual space, as well as high level, stiuctuial ieections cast fiom
the language of the mind. Souice code may be oiganised into disciete tiees, but those tiees
sway in an analogue bieeze fiom the activity of peiception. We hope the aitwoiks, languages
and notations intioduced thiough this thesis demonstiate the feitile giound available to the
aitist-piogiammei, and inspiie gieatei woiks to follow.
1c
References
Aaion, S., Blackwell, A. l., Hoadley, R., and Regan, T. (zc11). A piincipled appioach to devel-
oping new languages foi live coding. ln Proceedings of New Interfaces for Musical Expression
2011, pages Is1Ise.
Abelson, H. and Sussman, G. J. (1vve). Structure and Interpretation of Computer Programs. Te
MlT Piess, second edition.
Agawu, K. (zccI). Representing African Music: Postcolonial Notes, eries, Positions. Routledge,
ist edition.
Alexandei, C., lshikawa, S., and Silveistein, M. (1v). A Paern Language: Towns, Buildings,
Construction. Oxfoid Univeisity Piess, ist edition.
Ains, l. (zcc1). Read me, iun me, execute me. code as executable text Sofwaie ait and its focus
on piogiam code as peifoimative text. ln Goiiunova, O. and Shulgin, A., editois, Read me
Soware Art and Cultures, pages 11ss.
Baisalou, L. W. (1vvv). Peiceptual symbol systems. e Behavioral and brain sciences, zz(1).
Baisalou, L. W., Santos, A., Simmons, W. K., and Wilson, C. D. (zccs). Language and Simulation
in Conceptual Processing, chaptei 1I, pages z1zsI. Oxfoid Univeisity Piess, USA, ist
edition.
Beckwith, L. and Buineu, M. (zcc1). Gendei An impoitant factoi in lnd-Usei piogiamming
enviionments` ln Proceedings of IEEE Symposium on Visual Languages and Human Centric
Computing 2004, pages 1c111.
Bel, B. (zcc1). Rationalizing musical time syntactic and symbolic-numeiic appioaches. ln
Bailow, C., editoi, e Ratio Book, pages se1c1. leedback Studio.
Bijmolt, T. H. A. and Wedel, M. (1vv). Te eects of alteinative methods of collecting similaiity
data foi multidimensional scaling. Resear in Marketing, 1z(1).
11
Cu~v1iv lU1Uvi bivic1ioNs
Blackwell, A. (zccea). Gendei in domestic piogiamming liom biicolage to seances dessayage.
ln CHI Workshop on End User Soware Engineering 2006.
Blackwell, A. (zcceb). Ten yeais of cognitive dimensions in visual languages and computing.
Journal of Visual Languages & Computing, 1(1)zszs.
Blackwell, A. and Collins, N. (zcc). Te piogiamming language as a musical instiument. ln
Proceedings of 17th Psyology of Programming Interest Group 2005. Univeisity of Sussex.
Blackwell, A. and Gieen, T. (zccz). Notational systems the cognitive dimensions of nota-
tions fiamewoik. ln Caiioll, J. M., editoi, HCI Models, eories, and Frameworks: Toward a
Multidisciplinary Science, pages 1cI1I1. Moigan Kaufmann.
Blackwell, A. l. (zccec). Metacognitive theoiies of visual piogiamming what do we think we
aie doing` ln Proceedings of IEEE Symposium on Visual Languages, pages z1cz1e.
Blackwell, A. l. (zcced). Metaphois we piogiam by Space, action and society in java. ln
Proceedings of the 18th Psyology of Programming Interest Group 2006.
Blackwell, A. l. (zccee). Te ieication of metaphoi as a design tool. ACM Transactions on
Computer-Human Interaction, 1I(1)1vcIc.
Boden, M. A. (zccI). e Creative Mind: Myths and Meanisms. Routledge, second edition.
Boei, B. (zc1c). Modelling vocal anatomys signicant eect on speech. Journal of Evolutionary
Psyology, s(1)I1Iee.
Boulez, P. (1vvc). Orientations: Collected Writings. Haivaid Univeisity Piess.
Biegman, A. S. (1vv1). Auditory Scene Analysis: e Perceptual Organization of Sound. Te MlT
Piess.
Biinghuist, R. (zcc1). e Elements of Typographic Style. Haitley & Maiks Publisheis, thiid
edition.
Biown, P., Geie, C., Lambeit, N., and Mason, C., editois (zccv). White Heat Cold Logic: British
Computer Art 1960-1980 (Leonardo Books). Te MlT Piess.
Buzsaki, G. (zcce). Rhythms of the Brain. Oxfoid Univeisity Piess, USA, ist edition.
Caclin, A., McAdams, S., Smith, B. K., and Winsbeig, S. (zcc). Acoustic coiielates of timbie
space dimensions A conimatoiy study using synthetic tones. e Journal of the Acoustical
Society of America, 11s(1)111sz.
1z
Cu~v1iv lU1Uvi bivic1ioNs
Cage, J. (1vev). Ait and technology. ln John Cage: Writer. Coopei Squaie Piess.
Calvo-Meiino, B., Glasei, D. l., Gi`ezes, J., Passingham, R. l., and Haggaid, P. (zcc). Action
obseivation and acquiied motoi skills An fMRl study with expeit danceis. Cerebral Cortex,
1(s)1z1I1z1v.
Campbell, J. l. (1ssc). Canntairead : articulate music. Aichibald Sinclaii.
Caipentei, l. and Lacceui, J. (zcce). Open souice embioideiy (inteiview).
Cascone, K. (zccc). Te aesthetics of failuie Post-Digital tendencies in contempoiaiy com-
putei music. Computer Music Journal, z1(1)1z1s.
Chambeis, C. K. (1vsc). Non-lexical vocables in Scoish traditional music. PhDthesis, Univeisity
of ldinbuigh.
Chomsky, N. (1ve). Tiee models foi the desciiption of language. IRE Transactions on Infor-
mation eory, z(I)11I1z1.
Chong, T. T. J., Cunnington, R., Williams, M. A., Kanwishei, N., and Mauingley, J. B. (zccs).
fMRl adaptation ieveals miiioi neuions in human infeiioi paiietal coitex. Current biology :
CB, 1s(zc)1e1sc.
Chuich, A. (1v11). e Calculi of Lambda Conversion. Piinceton Univeisity Piess, Piinceton, NJ,
USA.
Chuich, L. and Gieen, T. (zccs). Cognitive dimensions - a shoit tutoiial. ln Proceedings of 20th
Psyology of Programming Interest Group 2008.
Claik, A. (zccs). Supersizing the Mind: Embodiment, Action, and Cognitive Extension (Philosophy
of Mind Series). Oxfoid Univeisity Piess, USA.
Clayton, M. (zccs). Time in Indian Music: Rhythm, Metre, and Form in North Indian Rag Perfor-
mance (Oxford Monographs on Music). Oxfoid Univeisity Piess, USA.
Collins, N. (zccI). Geneiative music and laptop peifoimance. Contemporary Music Review,
zz(1)ev.
Collins, N. (zcc). Live coding piactice. ln Proceedings of New Interfaces for Musical Expression
2007.
Collins, N., McLean, A., Rohihubei, J., and Waid, A. (zccI). Live coding in laptop peifoimance.
Organised Sound, s(cI)Iz1IIc.
1I
Cu~v1iv lU1Uvi bivic1ioNs
Cox, G., McLean, A., and Waid, A. (zccc). Te aesthetics of geneiative code. ln International
Conference on Generative Art.
Cox, G., McLean, A., and Waid, A. (zcc1). Coding piaxis Reconsideiing the aesthetics of code.
ln Goiiunova, O. and Shulgin, A., editois, read me Soware Art and Cultures, pages 1e111.
Aaihus Univeisity Piess.
Cioss, l. and Tolbeit, l. (zccs). Music and meaning. ln e Oxford Handbook of Music Psyology.
Oxfoid Univeisity Piess.
Csikszentmihalyi, M. (zccs). Flow: the psyology of optimal experience. HaipeiCollins.
Cupchik, G. (zcc1). Shaied piocesses in spatial iotation and musical peimutation. Brain and
Cognition, 1e(I)IIIsz.
de Rijke, V., Waid, A., Stockhausen, K., Dievei, J. L., and Abdullah, H. (zccI). Qack pioject CD
covei notes.
Dean, R., Whitelaw, M., Smith, H., and Woiiall, D. (zcce). Te miiage of ieal-time algoiithmic
synaesthesia Some compositional mechanisms and ieseaich agendas in computei music and
sonication. Contemporary Music Review, z(1)I11Ize.
Deleuze, G. and Guauaii, l. (1vs). 111c Te smooth and the stiiated. ln ousand Plateaus:
Capitalism and Sizophrenia, chaptei 11. Univeisity of Minnesota Piess, ist edition.
Deutsch, D. and leioe, J. (1vs1). Te inteinal iepiesentation of pitch sequences in tonal music.
Psyological Review, ss(e)cIzz.
di Pellegiino, G., ladiga, L., logassi, L., Gallese, V., and Rizzolaui, G. (1vvz). Undeistanding
motoi events a neuiophysiological study. Experimental brain resear. Experimentelle Hirn-
forsung. Exp erimentation c er ebrale, v1(1)1e1sc.
Dijkstia, l. W. (1vs). On anthiopomoiphism in science.
Dijkstia, l. W. (1vss). On the ciuelty of ieally teaching computing science.
Dinstein, l., Tomas, C., Behimann, M., and Heegei, D. J. (zccs). A miiioi up to natuie. Current
biology : CB, 1s(1).
Douglas, K. M. and Bilkey, D. K. (zcc). Amusia is associated with decits in spatial piocessing.
Nature Neuroscience, 1c()v1vz1.
11
Cu~v1iv lU1Uvi bivic1ioNs
du Sautoy, M. (zccs). Finding Moonshine: A Mathematicians Journey rough Symmetry.
HaipeiCollins Publisheis.
llliou, C. (zccv). Push-pull functional ieactive piogiamming. ln Proceedings of 2nd ACM SIG-
PLAN symposium on Haskell 2009.
lmmeison, S. (zcca). Te human body in electioacoustic music Sublimated oi celebiated` ln
Living Electronic Music, pages e1s. Ashgate Pub Co.
lmmeison, S. (zccb). Postsciipt the unexpected is always upon us live coding. ln Living
Electronic Music, page 11. Ashgate Pub Co.
lno, B. (1vve). Geneiative music. In Motion Magazine.
lssingei, J. (zcc1). Jacquards Web: How a Hand-Loom Led to the Birth of the Information Age.
Oxfoid Univeisity Piess, USA, ist edition.
lellbaum, C., editoi (1vvs). WordNet: An Electronic Lexical Database (Language, Spee, and
Communication). Te MlT Piess, illustiated edition.
lilesystem Hieiaichy Standaid Gioup (zccI). lilesystem hieiaichy standaid.
linney, S. A. (zcc1). Real-time data collection in linux Acase study. Behavior ResearMethods,
Instruments, & Computers, II(z)1e1I.
llanagan, D. and Matsumoto, Y. (zccs). e Ruby Programming Language. OReilly Media, ist
edition.
lodoi, J. A. (1vsc). e Language of ought (Language & ought Series). Haivaid Univeisity
Piess, ist edition.
lontana, l. and Rocchesso, D. (1vv). A new foimulation of the zD-waveguide mesh foi peicus-
sion instiuments. ln Proceedings of XI Colloquium on Musical Informatics 1995, pages zIc.
loith, J., Wiggins, G., and McLean, A. (zc1c). Unifying conceptual spaces Concept foimation
in musical cieative systems. Minds and Maines, zc(1)cIIz.
lieed, A. and Schmedei, A. (zccv). leatuies and futuie of open sound contiol veision 1.1 foi
NlMl. ln Proceedings of New Interfaces for Musical Expression 2009.
liibeig, A., Biesin, R., and Sundbeig, J. (zcce). Oveiview of the KTH iule system foi musical
peifoimance. Advances in Cognitive Psyology, Special Issue on Music Performance, z(z-
I)111e1.
1
Cu~v1iv lU1Uvi bivic1ioNs
liiedl, J. l. l. (zcce). Mastering Regular Expressions. OReilly Media, thiid edition.
liitz, T., Jentschke, S., Gosselin, N., Sammlei, D., Peietz, l., Tuinei, R., liiedeiici, A. D., and
Koelsch, S. (zccv). Univeisal iecognition of thiee basic emotions in music. Current Biology,
1v()Ie.
Galantei, P. (zccI). What is geneiative ait` complexity theoiy as a context foi ait theoiy. ln
In GA2003 6th Generative Art Conference.
Galantucci, B., lowlei, C. A., and Tuivey, M. T. (zcce). Te motoi theoiy of speech peiception
ieviewed. Psyonomic Bulletin & Review, 1I(I)Ie1I.
Gallaido, D., Juli`a, C. l., and Joid`a, S. (zccs). TuiTan a tangible piogiamming language foi cie-
ative exploiation. ln ird annual IEEE international workshop on horizontal human-computer
systems (TABLETOP).
Gaidenfois, P. (zccc). Conceptual Spaces: e Geometry of ought. Te MlT Piess.
Gieenewalt, M. H. (1v1e). Nourathar, the Fine Art of Light Color Playing. Philadelphia. Pa.
Westbiook.
Giey, J. M. (1v). Multidimensional peiceptual scaling of musical timbies. e Journal of the
Acoustical Society of America, e1()1zc1z.
Giiths, D. and Baiiy, P. (zccv). Head First Programming: A Learners Guide to Programming
Using the Python Language. OReilly Media, ist edition.
H. K. M. (1v1s). Te unseen woild. e New Republic, 11eI-.
Hajda, J. M., Kendall, R. A., Caiteieue, l. C., and Haischbeigei, M. L. (1vv). Methodological
issues in timbie ieseaich. ln Deli`ege, l. and Sloboda, J., editois, Perception and Cognition of
Music, pages zIIc. Psychology Piess.
Hendeison, J. (zccI). Human gaze contiol duiing ieal-woild scene peiception. Trends in Cog-
nitive Sciences, (11)1vsc1.
Hickok, G. (zccv). light pioblems foi the miiioi neuion theoiy of action undeistanding in
monkeys and humans. Journal of Cognitive Neuroscience, z1()1zzv1z1I.
Hudak, P. (zccc). e Haskell Sool of Expression: Learning Functional Programming through
Multimedia. Cambiidge Univeisity Piess, ist edition.
1e
Cu~v1iv lU1Uvi bivic1ioNs
Janei, J. (zccs). Singing-driven Interfaces for Sound Synthesizers. PhD thesis, Univeisitat Pompeu
labia, Baicelona.
Jes, C. (zcc). Cylob music system. http://durftal.com/cms/cylobmusicsystem.html.
Jones, D. l. (1vvc). Speech extiapolated. Perspectives of New Music, zs(1)11z11z.
Jones, S. P., editoi (zccz). Haskell 98 Language and Libraries: e Revised Report.
hup//haskell.oig/.
Joid`a, S., Geigei, G., Alonso, M., and Kaltenbiunnei, M. (zcc). Te ieacTable lxploiing the
syneigy between live music peifoimance and tabletop tangible inteifaces. ln Proceedings of
Tangible and Embedded Interaction 2007.
Kaiplus, K. and Stiong, A. (1vsI). Digital synthesis of plucked stiing and dium timbies. Com-
puter Music Journal, (z)1I.
Keinighan, B. W. and Ritchie, D. M. (1vss). C Programming Language. Pientice Hall, second
edition.
Kippen, J. (1vss). e Tabla of Lunow - A cultural analysis of a musical tradiation. Cambiidge
Univeisity Piess.
Klee, P. (1vI). Pedagogical sketbook. labei and labei.
Kohlei, W. (1vIc). Gestalt Psyology. Camelot Piess.
Kokol, P. and Kokol, T. (1vve). Linguistic laws and computei piogiams. Journal of the American
Society for Information Science, 1s1s.
Kowalski, R. (1vv). Algoiithm logic - contiol. Communications of the ACM, zz()1z11Ie.
Kiumhansl, C. L. (1vsv). Why is musical timbie so haid to undeistand` ln Nielzen, S. and
Olsson, O., editois, Structure and Perception of Electroacoustic Sound and Music, Proceedings
of the Marcus Wallenberg symposium 1998, pages 1II. lxceipta Medica.
Kupka, l. and Wilsing, N. (1vsc). Conversational Languages. John Wiley and Sons.
Ladefoged, P. (1vvc). Te ievised inteinational phonetic alphabet. Language, ee(I)cz.
Laiid, J. A. (zcc1). e Physical Modelling of Drums using Digital Waveguides. PhD thesis,
Univeisity of Biistol.
Lako, G. (1vv). Women, Fire, and Dangerous ings. Univeisity Of Chicago Piess, 1vv edition.
1
Cu~v1iv lU1Uvi bivic1ioNs
Lako, G. and Johnson, M. (1vsc). Metaphors We Live By. Univeisity of Chicago Piess, second
edition.
Lee, l. A. (zccv). Computing needs time. Communications of the ACM, z()cv.
Leman, M. (zcc). Embodied Music Cognition and Mediation Tenology. Te MlT Piess.
Leidahl, l. and Jackendo, R. (1vsI). A Generative eory of Tonal Music. MlT Piess, Cambiidge,
MA.
Lessig, L. (zcce). Code: And Other Laws of Cyberspace, Version 2.0. Basic Books.
Levenshtein, V. l. (1vee). Binaiy codes capable of coiiecting deletions, inseitions and ieveisals.
Soviet Physics Doklady., 1c(s)c1c.
Levi-Stiauss, C. (1ves). e Savage Mind (Nature of Human Society). Univeisity Of Chicago
Piess.
Levy, S. (zccz). Haers: Heroes of the Computer Revolution. Penguin Putnam.
Lewis, G. (1vvI). Max in a musical context. Computer Music Journal, 1(z).
Libeiman, A. M. and Mauingly, l. G. (1vs). Te motoi theoiy of speech peiception ievised.
Cognition, z1(1)1Ie.
Ling, L. l., Giabe, l., and Nolan, l. (zccc). Qantitative chaiacteiizations of speech ihythm
syllable-timing in singapoie english. Language and spee, 1I(1)I1c1.
Listei, R., Adams, l. S., litzgeiald, S., lone, W., Hamei, J., Lindholm, M., McCaitney, R.,
Mosti om, J. l., Sandeis, K., Seppala, O., Simon, B., and Tomas, L. (zcc1). A multi-national
study of ieading and tiacing skills in novice piogiammeis. ln Working group reports from
ITiCSE on Innovation and tenology in computer science education, lTiCSl-WGR c1, pages
11v1c, New Yoik, NY, USA. ACM.
London, J. (zcc1). Hearing in Time: Psyological Aspects of Musical Meter. Oxfoid Univeisity
Piess, USA.
Lyons, W. l. (1vse). e disappearance of introspection. MlT Piess.
Magnusson, T. (zccv). Of epistemic tools musical instiuments as cognitive extensions. Organ-
ised Sound, 11(z)1es1e.
Maitin, G. N. (zcce). Human Neuropsyology. Pientice Hall, second edition.
1s
Cu~v1iv lU1Uvi bivic1ioNs
Maitino, D. (1vee). Notation in Geneial-Aiticulation in paiticulai. Perspectives of New Music,
1(z)1s.
McAdams, S. (1vvv). Peispectives on the contiibution of timbie to musical stiuctuie. Computer
Music Journal, zI(I)s1cz.
McCaitney, J. (zccz). Rethinking the computei music language SupeiCollidei. Computer Music
Journal, ze(1)e1es.
McGuik, H. and MacDonald, J. W. (1ve). Heaiing lips and seeing voices. Nature, ze1(z1e-z1s).
Mcilwain, P., Mccoimack, J., Doiin, A., and Lane, A. (zcc). Composing with nodal netwoiks. ln
Opie, T. and Biown, A., editois, Proceedings of the Australasian Computer Music Conference
2005, pages ve1c1.
McLean, A. (zcc1). Hacking peil in nightclubs. hup//www.peil.com/pub/a/zcc1/cs/I1/livecode.html.
McLean, A. (zcc). lmpiovising with synthesised vocables, with analysis towaids computa-
tional cieativity. Masteis thesis, Goldsmiths College, Univeisity of London.
McLean, A., Giiths, D., Collins, N., and Wiggins, G. (zc1c). Visualisation of live code. ln
Proceedings of Electronic Visualisation and the Arts London 2010.
McLean, A. and Wiggins, G. (zc1c). Petiol Reactive pauein language foi impiovised music. ln
Proceedings of the International Computer Music Conference 2010.
Melaia, R. D., Maiks, L. l., and Lesko, K. l. (1vvz). Optional piocesses in similaiity judgments.
Perception & Psyophysics, 1(z)1zI1II.
Millei, G. A. (1ve). Te magical numbei seven plus oi minus two some limits on oui capacity
foi piocessing infoimation. Psyological review, eI(z)s1v.
Milnei, R., Tofe, M., and Haipei, R. (1vvc). e denition of Standard ML. MlT Piess, Cambiidge,
MA, USA.
Mole, C. (zccv). Te motoi theoiy of speech peiception. ln Sounds and Perception: New Philo-
sophical Essays. Oxfoid Univeisity Piess.
Muiphy, G. L. (zccz). e Big Book of Concepts (Bradford Books). Te MlT Piess.
Myeis, B. (1vvc). Taxonomies of visual piogiamming and piogiam visualization. Journal of
Visual Languages & Computing, 1(1)v1zI.
Nake, l. (1v1). Teie should be no computei ait. PAGE, 1s.
1v
Cu~v1iv lU1Uvi bivic1ioNs
Nash, C. and Blackwell, A. l. (zc11). Tiacking viituosity and ow in computei music. ln
Proceedings of International Computer Music Conference 2011.
Naui, P. (1vvza). Human knowing, language and disciete stiuctuies. ln Computing: A Human
Activity, pages 1sI. ACM Piess.
Naui, P. (1vvzb). Piogiamming languages aie not languages why piogiamming language is
a misleading designation. ln Computing: A Human Activity, pages cI1c. ACM Piess.
Okabe, A., Boots, B., Sugihaia, K., and Chiu, S. N. (zccc). Spatial Tessellations: Concepts and
Applications of Voronoi Diagrams. Seiies in Piobability and Statistics. John Wiley and Sons,
lnc., second edition.
Olivei, J., Savicic, G., and Vasiliev, D. (zc11). Te ciitical engineeiing manifesto.
hup//ciiticalengineeiing.oig/.
Paivio, A. (1vvc). Mental Representations: A Dual Coding Approa (Oxford Psyology Series).
Oxfoid Univeisity Piess, USA.
Patel, A. D. (zcc). Music, Language, and the Brain. Oxfoid Univeisity Piess, USA, ist edition.
Patel, A. D. and lveisen, J. R. (zccI). Acoustic and peiceptual compaiison of speech and dium
sounds in the noith indian tabla tiadition An empiiical study of sound symbolism. ln
Proceedings of the 15th International Congress of Phonetic Sciences (ICPhS).
Peaice, J. M. (zcc). Selected obseivations on amusia. European neurology, 1(I)1111s.
Petie, M. and Blackwell, A. l. (1vvv). Mental imageiy in piogiam design and visual piogiam-
ming. International Journal of Human-Computer Studies, 1Ic.
Polgai, T. (zcc). Freax: e Brief History of the Computer Demoscene. CSW-Veilag.
Piessing, J. (1vs1). Cognitive piocesses in impiovisation. ln Cioziei, R. and Chapman, A., edi-
tois, Cognitive Processes in the Perception of Art, pages I1IeI. llseviei Science Publisheis.
Piessing, J. (1vs). lmpiovisation Methods and models. ln Sloboda, J. A., editoi, Generative
Processes in Music, pages 1zv1s. Oxfoid Univeisity Piess.
Puckeue, M. (1vss). Te patchei. ln Proceedings of International Computer Music Conference
1988.
Pullum, G. K. and Gazdai, G. (1vsz). Natuial languages and context-fiee languages. Linguistics
and Philosophy, 1(1)11c1.
1ec
Cu~v1iv lU1Uvi bivic1ioNs
Pylyshyn, Z. W. (zcc). ings and Places: How the Mind Connects with the World (Jean Nicod
Lectures). Te MlT Piess, ist edition.
Ramachandian, V. S. (zccc). Miiioi neuions and imitation leaining as the diiving foice behind
the gieat leap foiwaid in human evolution. ird Culture.
Ramachandian, V. S. and Hubbaid, l. M. (zcc1a). Psychophysical investigations into the neuial
basis of synaesthesia. Proceedings. Biological sciences / e Royal Society, zes(11c)vvvsI.
Ramachandian, V. S. and Hubbaid, l. M. (zcc1b). Synaesthesia a window into peiception,
thought and language. Journal of Consciousness Studies, s(1z)II1.
Raynei, K. and Pollatsek, A. (1vv1). Woid peiception. ln e Psyology of Reading, chaptei I.
Routledge.
Reas, C. and liy, B. (zcc). Processing: A Programming Handbook for Visual Designers and
Artists. Te MlT Piess.
Remez, R. l., Paido, J. S., Pioikowski, R. L., and Rubin, P. l. (zcc1). On the bistability of sine
wave analogues of speech. Psyological Science, 1z(1)z1zv.
Rodet, X., Potaid, Y., and Baiiieie, J. B. (1vs1). Te CHANT pioject liom the synthesis of the
singing voice to synthesis in geneial. Computer Music Journal, s(I).
Rohihubei, J., de Campo, A., and Wiesei, R. (zcc). Algoiithms today Notes on language design
foi just in time piogiamming. ln Proceedings of International Computer Music Conference
2005.
Rohihubei, J., de Campo, A., Wiesei, R., van Kampen, J.-K., Ho, l., and H olzl, H. (zcc). Pui-
loined leueis and distiibuted peisons. ln Music in the Global Village Conference 2007.
Rosenblum, L. D. and Salda na, H. M. (1vve). An audiovisual test of kinematic piimitives foi
visual speech peiception. Journal of experimental psyology. Human perception and perfor-
mance, zz(z)I1sII1.
Rowe, R. (zcc1). Maine Musicianship. Te MlT Piess.
Rushko, D. (zc1c). Program or Be Programmed: Ten Commands for a Digital Age. OR Books,
ist edition edition.
Saiamago, J. (zccc). e Stone Ra (Panther). Te Haivill Piess.
1e1
Cu~v1iv lU1Uvi bivic1ioNs
Schon, D. A. (1vs1). e Reective Practitioner: How Professionals ink In Action. Basic Books,
1 edition.
Schwiueis, K. (1vIz). Uisonate. Merz, z1.
Shepaid, R. (1vez). Te analysis of pioximities Multidimensional scaling with an unknown
distance function. i. Psyometrika, z(z)1z11c.
Shepaid, R. N. and Metzlei, J. (1v1). Mental iotation of thiee-dimensional objects. Science (New
York, N.Y.), 11(vz)c1cI.
Shulgin, A. (zccI). Listen to the tools, inteiview with Alex McLean and Adiian Waid. ln
read me 2.3 reader. NllCA.
Simon, H. A. and Sumnei, R. K. (1vvz). Pauein in music. ln Schwanauei, S. and Leviu, D.,
editois, Maine models of music, pages sI11c. MlT Piess, Cambiidge, MA, USA.
Small, C. (1vvs). Musiing: e Meanings of Performing and Listening (Music Culture). Wes-
leyan, ist edition.
Smalley, D. (1vv1). Dening timbie iening timbie. Contemporary Music Review, 1c(z)I1s.
Soiensen, A. (zcc). lmpiomptu An inteiactive piogiamming enviionment foi composition
and peifoimance. ln Proceedings of the Australasian Computer Music Conference 2005, pages
11v1I.
Soiensen, A. and Gaidnei, H. (zc1c). Piogiamming with time cybei-physical piogiamming
with impiomptu. ln Proceedings of ACM OOPLSA, pages szzsI1.
Spiegel, L. (1vs1). Manipulations of musical paueins. ln Proceedings of the Symposium on Small
Computers and the Arts, pages 1vzz.
Spiegel, L. (1vs). A shoit histoiy of intelligent instiuments. Computer Music Journal, 11(I).
Spolsky, J. (zcc1). Dont let aichitectuie astionauts scaie you. ln Joel on Soware, pages 111111.
Apiess.
Steiling, L. and Shapiio, l. (1vv1). e Art of Prolog, Second Edition: Advanced Programming
Teniques (Logic Programming). Te MlT Piess, second edition.
Stewait, D. and Chakiavaity, M. M. T. (zcc). Dynamic applications fiom the giound up. ln
Proceedings of ACM SIGPLAN workshop on Haskell 2005, pages zIs, New Yoik, NY, USA.
ACM.
1ez
Cu~v1iv lU1Uvi bivic1ioNs
Stewait, L. and Walsh, V. (zcc). Music peiception sounds lost in space. Current biology : CB,
1(zc).
Stockhausen, K. and Maconie, R. (zccc). Stohausen on Music. Maiion Boyais Publisheis Ltd.
Stowell, D. (zccs). Chaiacteiistics of the beatboxing vocal style, C1DM-TR-cs-c1. Technical
iepoit, Qeen Maiy, Univeisity of London.
Suuon-Spence, R. and Woll, B. (1vvv). e Linguistics of British Sign Language: An Introduction.
Cambiidge Univeisity Piess.
Taube, H. K. (zcc1). Notes from the Metalevel: Introduction to Algorithmic Music Composition.
Swets & Zeitlingei, Lisse, Te Netheilands.
Tiaube, C. and DAlessandio, N. (zcc). Vocal synthesis and giaphical iepiesentation of the
phonetic gestuies undeilying guitai timbie desciiption. ln 8th International Conference on
Digital Audio Eects (DAFx05), pages 1c11cv, Madiid, Spain.
Tuiing, A. M. (1vc). Computing machineiy and intelligence. Mind, LlX1II1ec.
Tuiing, A. M. (1vvz). lntelligent machineiy. iepoit, national physics laboiatoiy. ln lnce, D. C.,
editoi, Collected Works of A. M. Turing: Meanical Intelligence, pages 1c1z. llseviei, Am-
steidam.
Tuikle, S. (zcc). e Second Self: Computers and the Human Spirit. Te MlT Piess, twentieth
anniveisaiy edition.
Tuikle, S. and Papeit, S. (1vvc). lpistemological pluialism Styles and voices within the com-
putei cultuie. Signs, 1e(1)1zs1.
Tuikle, S. and Papeit, S. (1vvz). lpistemological pluialism and the ievaluation of the conciete.
Journal of Mathematical Behavior, 11(1)III.
Tyte, T. (zccs). Standaid beatbox notation. online, http://www.humanbeatbox.com/tips/
p2_articleid/2.
Usselmann, R. (zccI). Te dilemma of media ait Cybeinetic seiendipity at the lCA london.
Leonardo, Ie()IsvIve.
van Deuisen, A., Klint, P., and Vissei, J. (zccc). Domain-specic languages an annotated bib-
liogiaphy. SIGPLAN Not., I(e)zeIe.
1eI
Cu~v1iv lU1Uvi bivic1ioNs
Van Duyne, S. A. and Smith, J. O. (1vvI). Te z-D digital waveguide mesh. ln Applications of
Signal Processing to Audio and Acoustics, pages 11sc.
Vogel, J. (zccI). Ceiebial lateialization of spatial abilities Ameta-analysis. Brain and Cognition,
z(z)1vzc1.
Wang, G. and Cook, P. R. (zcc1). On-the-y piogiamming using code as an expiessive musi-
cal instiument. ln Proceedings of New interfaces for musical expression 2004, pages 1Is11I.
National Univeisity of Singapoie.
Waid, A., Levin, G., Lia, and Meta (zccz). 4x4 Generative Design (with Auto-Illustrator, Java,
DBN, Lingo): Life/Oblivion. Apiess, ist edition.
Waid, A., Rohihubei, J., Olofsson, l., McLean, A., Giiths, D., Collins, N., and Alexandei, A.
(zcc1). Live algoiithm piogiamming and a tempoiaiy oiganisation foi its piomotion. ln
Goiiunova, O. and Shulgin, A., editois, read me Soware Art and Cultures.
Waid, J., Tompson-Lake, D., lly, R., and Kaminski, l. (zccs). Synaesthesia, cieativity and ait
What is the link` British Journal of Psyology, vv(1)1z111.
Wellei, S. C. and Romney, A. K. (1vss). Systematic Data Collection (alitative Resear Methods
Series 10). Sage Publications, lnc, ist edition.
Wessel, D. L. (1vv). Timbie space as a musical contiol stiuctuie. Computer Music Journal,
I(z)1z.
Whitehead, A. N. (zcc1). Dialogues of Alfred North Whitehead (A Nonpareil Book). David R
Godine.
Wiggins, G. A. (1vvs). Music, syntax, and the meaning of meaning. ln Proceedings of the First
Symposium on Music and Computers, pages 1szI.
Wiggins, G. A. (zccea). A pieliminaiy fiamewoik foi desciiption, analysis and compaiison of
cieative systems. Journal of Knowledge Based Systems, 1v()11v1c.
Wiggins, G. A. (zcceb). Seaiching foi computational cieativity. New Generation Computing,
z1(I)zcvzzz.
Wiugenstein, L. (zccv). Philosophical Investigations. Wiley-Blackwell, fouith edition.
Wolfiam, S. (zccz). A New Kind of Science. Wolfiam Media, ist edition.
Youngblood, G. (1vc). Expanded Cinema. l P Duuon, ist edition.
1e1
Cu~v1iv lU1Uvi bivic1ioNs
Zhang, N. and Chen, W. (zcce). A dynamic fMRl study of illusoiy double-ash eect on human
visual coitex. Experimental Brain Resear, 1z(1)ee.
Ziuiain, J. (zccv). e Future of the InternetAnd How to Stop It. Yale Univeisity Piess.
1e
AvviNbix A
Live coding survey
Te following contains fieefoim answeis given in iesponse to the suivey discussed in e.v.
A.1 Denitions of live coding
Te following aie the twenty nine answeis to the question How would you dene live coding,
in one sentence`, piesented heie unedited.
Live coding is instiument building, composing and peifoiming in one peifoimative act.
lxposing, foi an audience, the thought piocesses behind making music algoiithmically.
Coding as a peifoimance ait. (lm not veiy good at it yet)
a way to enteitain, the geeks and the noobs.
An audio visual peifoimance piactice wheie computei sofwaie that geneiates the audi
visuals is wiiuen as pait of the peifoimance.
A live peifoimance piactice which uses computei piogiamming languages and enviion-
ments as an inteiface foi the inteiactive manipulation of media iich sofwaie iuntime
systems.
Live coding is peifoiming a woik by wiiting and modifying computei piogiamming code
which is iesponsible foi cieating the iesultant woik.
Live coding is a peifoimance piactice of constiucting and inteiacting with algoiithmic
piocesses to cieate ait.
An exchange of knowledge and expeiience between two oi moie peifoimeis/codeis.
Cieating logic, stiuctuie and meaning as pait of a peifoimance instead of befoiehand.
1ee
AvviNbix A Livi cobiNc sUvviv
maing ieal-time changes to sound geneiation
coding stu .. live
listening to changes
Build fiom sciatch, make it tiandpaient.
live coding is impiovisation on the instiument computei
lm a bit haid coie, and piefei live coding to be on the y, and fiom sciatch - none of this
executing pie-built patches / code. l also have a tendency to think it should be code and
not giaphic apps such as MAX/ PD.
Piogiamming as cieative jouiney oi iitual.
electionic impiovising fieedom
Coding music/animation with diiect sounding/viewable iesults.
l ofen dene it as an impiovisional way of doing music with computeis, thus being to
electionic music what fiee jazz is to jazz. lts also a way of consideiing the computei as
an instiument. (soiiy two sentences)
Te use of instiuctions and/oi iules foi the contiol of computei(s) (oi peison(s)) as a
method of cieative expiession.
seveial iteiations of coding/executing within the span of a song/piece
a moie natuial inteiface to the cieative piocess to those of us that think moie like com-
puteis than bipedal meatbags
peifoimace as in peifoimance ait
building youi instiument while playing it
the ait of cieating music fiom a pioceduie yet to be staited
Refoimalising an active foimal system.
Cieating and shaiing algoiithms live.
An impiovised peifoimance of music oi animation by using inteiactive piogiamming to
desciibe and contiol piocess.
1e
AvviNbix A Livi cobiNc sUvviv
A.z Compaiing live coding
Te following aie the twenty eight fieefoim, unedited iesponses to the question What is the
dieience between live coding a piece of music and composing it in the sequencei (live coding
an animation and diawing one)` ln othei woids, how does live coding aect the way you
pioduce youi woik, and how does it aect the end iesult`
lt is veiy much live. Tempoiality comes stiongly in. Te sequenced piece has its own
foimal stiuctuie dened in the sequencei. Te foimal stiuctuie of the live coded piece
is dened by the complexity of the language. Live coding pieces can theiefoie be boiing
to listen to. ) But they can be fun to watch being piogiammed. Teiefoie, l nd
livecoding moie inteiesting as a peifoimative piactice than a musical piactice (if this
distinction makes any sense).
(Tis question might be too open to answei in a meaningful way.) Live coding is pei-
foiming, composing is not.
To date, my live coding has only been a piocess of assembling code alieady wiiuen, when
to tuin this on oi o, tweak gain/pan, etc.
it is moie about the expeiience of the audience than about my own concepts. yes, some-
times you tiy to integiate them, but if it doesnt appeal it wont help you in the contest.
while making xed media music, one can choose foi a ceitain amount of extiemity, al-
though one might not sell a lot of cds than, but hey, thats up to you.
Live coding is a meta compositional piocess that emphasises paueins and how they play
out, while sequencing is ofen captuiing of peifoimed gestuies oi sound.
Live coding is about abstiactions. Tis is ieally the only dieience between me live cod-
ing and me using a sequencei. Amoie inteiesting question is maybe whats the dieience
between using impiomptu and using a guitai. Ten it becomes about the symbolic vs the
gestuial as well as the many vs the few (concuiient activities that is).
Live coding foices a woik to be peifoimed, and possibly (ideally`) impiovised. l nd this
a veiy dieient piospect to composing music in a sequencei (oi even wiiting notation
on papei) foi a latei peifoimance. Howevei, while Live Coding is a peifoimance piac-
tice, it also oeis the tantalising piospect of manipulating musical stiuctuie at a similai
abstiact level as defeiied time composition. To do this eectively in peifoimance is
1es
AvviNbix A Livi cobiNc sUvviv
l think an entiiely dieient skill to the standaid one-acoustic-event-pei-action physi-
cal instiumental peifoimance, but also quite dieient to compositional methods which
typically allow foi iewoik.
Live coding encouiages cieative thinking at a moie abstiact level, foi example, the
paiametiisation of laige scale foimal stiuctuies. Sequencing can piovide a composei
with a highly detailed iepiesentation of the music, which can be useful when tiying to
iealise a veiy conciete idea. Howevei, l would aigue that sequencing is, on the whole,
a moie passive medium, in that it geneially contiibutes less to the cieative piocess than
a live coding enviionment. Peihaps a ieason foi this is that sequencing aims to piovide
an analogical iepiesentation of music, wheie as theie is much moie scope foi the iepie-
sentation of musical ideas in live coding. Seiendipity is also moie likely to guie in live
coding, wheie subtle iandom piocesses oi plain bugs might iesult in inteiesting unin-
tended ideas. Plus the live aspect imposes its own set of constiaints that aie not piesent
in o-line composition, as well as opens up possibilities foi ieal-time collaboiation.
Live coding connects me with the music oi sound the same way playing guitai does, it
gives me a feeling of contiol and makes me moie conscious of the now. My ist concein is
to pioduce a sound, much the same way as a jazz impioviseis ist concein is a melody.
Stiuctuie is something to stait woiking on once you have suitable sounds (oi scales,
melodies, haimonies, ihythms, whatevei). Note l am a beginning livecodei, so my view
on livecoding could veiy well change ovei the yeais.
Sequenced music isnt peifoimed, its just playback. lollowing this logic, and assuming
l want to impiovise in a peifoimance, theies no way to piopeily peifoim using a com-
putei if l had to sequence eveiything. Te othei pait of me thinks that wiiting code is
the same as sequencing, its just a shoithand. To answei youi question, this all hinges on
the need to be impiovising at a peifoimance. lf this wasnt necessaiy, and it was meiely
about a playback, then theie would be no need foi livecoding.
lve only played aiound with this at home, but Live coding has fai less peifection and
the pioduct is moie oie immediate. lt allows foi impiovisation and spontaneity and
discouiages ovei-thinking.
As someone who isnt veiy apt in the manual woild of iealtime composition with phys-
ical, tiaditional instiuments l piobably stiuggle with both appioaches. Live coding is
just anothei instiument to me, just one which has a moie signicant setup time and lin-
guistic inteiface. When l do use sequenceis l fiequently use a lot of embedded algo type
1ev
AvviNbix A Livi cobiNc sUvviv
piocesses and code exteinal apps (pd etc) to piocess alongside. ls the distinction moie
appaient when its live to a live audience` as that implies a deadline, expectations etc. lf
you compose music at home on youi own geai, including a day long woikout with se-
quenceis and sofwaie (geneially with liuel pause in the actual loop) is that livecoding`
semantics schemantix..
theie is only a iecoiding lef, but i cant iepeat the coding.
Coding live intuitive decisions with no pie-built sequences. lieedom, dangei and punk.
geneially i nd composing in sequenceis veiy boiing as a piocess (so i haidly do it nowa-
days), wheieas live coding (LC) adds moie excitement to the piocess (will it actually
woik at all`). whenevei i do sequence-like stiuctuies in LC, they aie usually geneiated
(using e.g. iandomness (with constiaints to t them into the musical context)), so they
aie not totally piedictable. sequences will eventually be ie-geneiated. howevei, i ofen
dont use sequences at all, but iathei tiy geneiate all the non-live stiuctuies algoiithmi-
cally.
Well, the live-coding sofwaie piovides gieatei scope foi sound manipulation than othei
audio editing sofwaie (eg pio tools etc), but doing it on the y (live coding) vs wiiting
code in my own time is moie of a psychological huidle - lts haidei to be satised with
the outcome with live coding. When l woik on wiiting a piece, in my own time (and
taking a long time) l can peifect each sound to be piecisely as l intend it to be, wheieas
doing it on the y , i.e. live coding l have to be moie geneialised as to my intentions.
lssentially the main dieience is the ability to impiovise. Te live aspect becomes moie
impoitant than the end iesult. Collaboiation becomes easiei, and moie natuial - as its
easiei to adapt the woik to dieient people and new situations.
l use live coding to develop ideas and test them then l would use them in composed
pieces, but the piocess and impivising is almost moie inteiesting as the end iesult.
Mostly the fact that while livecoding l always found sounds that l could nevei had found
otheiwise. Tus, when l nd a sound that seems inteiesting to me, l tiy to tweek eveiy
paiametei to see how beuei it could be. l am used to keep tiack of eveiy session l am
doing, but l ofen feel a bit disappointed by the static feeling of the last sound l get.
But most impoitantly, even if sometimes l make sessions that is totally ciap, l always
leain something new. And as l come back to livecoding, all the knowledge l got fiom all
sessions is heie ieadily available to use in my head. Anothei advantage is the fact that
1c
AvviNbix A Livi cobiNc sUvviv
even if you use iathei the same stiuctuies eveiy nowand then, it always sounds dieient.
On a mental piocess way of thinking, livecoding gives the oppoitunity to always fostei
new ideas while not coding. lts a ieal cieative way of making music.
Live Coding is iiskiei, and one has to live with unuing decissions. You cant just go one
step back unless you do it with a nice piioueue. Teiefoie the end iesult is not as clean
as an oine-composition, but it can lead you to places you usally nevei would have
ended.
lm not suie they need be dieient at all, as a basic sequencei is faiily easy to implement.
Mostly l use laigei gestuies to place a whole seiies of notes at once, this is fastei than
clicking them all in and leads ofen leads to unexpected iesults. ln homebiew sofwaie
we can have a tightei link between the pauein geneiation and the signal geneiation
piocesses but in livecoding cieating this link may become a foim of expiession itself,
which is of couise quite exciting. Peihaps most impoitantly the highei pace of livecoding
leads to moie impulsive choices which keeps things moie inteiesting to cieate. Not suie
how ofen that also cieates a moie inteiesting end iesult but at least sometimes it does.
Te shoitei setup time befoie we can get staited with making music (less geai to tuin
on, no patch cables) makes it moie suitable foi impulsive playing as well.
i dont think of it as dieient to a sequencei, its just a dieient usei inteiface. i think in
the same way using both. i just ofen nd the piogiammatic inteiface to t my thought
piocesses beuei.
You dont have the oppoitunity to do ie-takes
the whole piocess becomes moie impiovisational. like sketching. nevei nished compo-
sitions (on a timeline / with a xed foim). foi me its moie about having fun coding than
the end iesult anyway.
Live coding foices me to expeiiment and move fastei within the compositional/ impio-
visational piocess. Peisonally l considei live coding to be the impiovisation of the com-
putei music woild.
lve nevei ieally done much with sequenceis, so l dont know the dieience. l like the
fact that live coding is moie sculptuial and passive.
Live coding exposes the beauty of piogiamming languages (oi piogiammatic piactices)
and the invention of inteiface live. Tough you could do live coding with an established
tool, l believe pait of the inteiest is seeing the building of tools, even if they aie meta
11
AvviNbix A Livi cobiNc sUvviv
tools. l have all but abandonned live coding as a iegulai peifoimance piactice, but l use
the skills and condence acquiied to modify my sofwaie live if l get a new idea while
on stage.
You have dieient kind of canvas, and it favois iesults within youi technique.
1z

También podría gustarte