Machine Perception and Creativity:

Building on

Image Schema, Conceptual Metaphor and Blending

Zaheer Ahmad PhD Scholar ahmad.zaheer@yahoo.com Department of Computer Science University of Peshawar

Agenda
Research Question New AI, Cognition ( Imagery, Embodiement )

Some Theory About Image Schema, Conceptual Metaphor and Conceptual Blending

Computational models of image schemas and reasoning

Gestalt Principle Nash Equilibrium Research Proposal Building on: Concepts to Think and Reason Creatively Emotions, Reasoning and Image Schema---an idea

Research Question
How to develop complex concepts by mapping simple ones onto each other, to let the computer Perceive, think and reason creatively

From New AI to GOFAI, A Flashback

Second AI winter 19871993 In the 80s and 90s, many cognitive scientists and AI researchers also rejected the symbol processing model of the mind and argued that the body was essential for reasoning, a theory called the embodied mind thesis Another encouraging event in the early 1980s was the revival of connectionism in the work of John Hopfield and David Rumelhart, from this AI had achieved success but not for too long Dates back to 1956 when John McCarthy of MIT invited many leading researchers of the time to a workshop where he introduced the term articial intelligence. Among the participants were Marvin Minsky , Herbert Simon, and Allan Newell, the founding fathers, so to speak, of articial intelligence What originated from this meeting, and what came to be the guiding principles until the mid-1980s, was what is now known as the classical, symbol-processing paradigm, also known as the cognitivistic paradigm

Cognitive Linguistics
In linguistics, cognitive linguistics (CL) refers to the branch of linguistics that interprets language in terms of the concepts, sometimes universal, sometimes specific to a particular tongue, which underlie its forms. It is thus closely associated with semantics cognitive linguists view meaning in terms of conceptualization Aspects of cognition that are of interest to cognitive linguists include:
Construction grammar and cognitive grammar. Conceptual metaphor and conceptual blending. Image schemas and...

Thought is impossible without an image.

Aristotle, On Memory and Recollection

Cognitive Linguistics and Imagery

Imagery is visualizing, seeing in the mind's eye, hearing in the head , imagining the feel of, thought to be caused by the presence of picture-like representations (mental images) in the mind / brain. Over the course of the past 2 decades, imagery has once again become increasingly more interesting to cognitive scientists. A number of studies have shown that humans automatically and unconsciously engage perceptual and motor imagery when performing high-level cognitive tasks ( hearing, speaking, seeing)

Cognitive Linguistics and Imagery

Human beings, rationality is embodied That is their thinking is in visible form containing one inside another. two types of imaginative structures are presented to explain the embodied experience functions in our life,
image schema and Conceptual metaphorical projection

Nouvelle AI and Embodied Cognition

Philosophers, cognitive scientists and artificial intelligence researchers who study embodied cognition and the embodied mind believe that
the nature of the human mind is largely determined by the form of the human body. They argue that all aspects of cognition, such as ideas, thoughts, concepts and categories are shaped by aspects of the body. These aspects include the perceptual system, the intuitions that underlie the ability to move, activities and interactions with our environment and the native understanding of the world that is built into the body and the brain.

 Johnson argues that :

Meaning cannot be separated from the structures of our embodied perceptual interactions and movements. If we did not have the bodies we do, or if they were somehow radically different than they are now, then we would not create, understand, and communicate meaning in the way we do.
Meanings in natural language come with gurative(metaphor) and multivalent patterns that cannot be reduced to literal concepts and propositions; the patterns and their connections are embodied and cannot be reduced to literal concepts and propositions.

 The embodied mind thesis is opposed to other theories of cognition such as cognitivisim, computationalism and Cartesian dualism.
George Lakoff (a cognitive scientist and linguist) and his collaborators (including Mark Johnson, Mark Turner, and Rafael E. Nez) have written a series of books promoting and expanding the thesis based on discoveries in cognitive science, such as conceptual metaphor and image schema. Robotics researchers such as Rodney Brooks, Hans Moravec and Rolf Pfeifer have argued that true artificial intelligence can only be achieved by machines that have sensory and motor skills and are connected to the world through a body.

Definitions as Reminder of Journey

Cartesian dualism----I think, therefore I am Cognitivisim/Symbolism---symbols and logic , Words, Sentence, Discourse, Pragmatics Connectionism---NN---- Embodiement -----Image schema---conceptual metaphor---blending Statistical/ Empirical --Corpora other stats machine learning tech.. Hybrid Approaches

Embodiement as
Strong AI / AGI (may be)
Embodiement is generally presented as Strong AI Models/Simulation of actual thinking, is called weak AI, Programs that are actually thinking is known as the strong AI weak AI position is unproblematic and generally accepted: The nature of the simulation model is clearly different from the thing it simulates as in a simulation of rain the computer does not get wet, the model of thinking is different from the thinking process itself. It is the strong AI stance with which people often take issue. This is not surprising. It is unsettling for many people to believe that a computer is actually thinking, rather than just simulating the process. the classical paradigm has had its definite successes, but it has failed to make clear the nature of intelligence Where as Emobodied Cognition has shown some potentials in the area of Strong AI

Image Schema

Linguistic interpretation of image schema

In Cognitive Linguistics, the term image schema implicates perception in all acts of conceptualization. Concepts develop from representation of a perceptual conglomeration(merging) of multiple senses, including visual, auditory, haptic, motoric, olfactory, and gustatory(taste) experiences. Schemas are structures of the imagination [Kant] schemas are xed templates superimposed onto perceptions and conceptions to render meaningful representations * Image Schema is a structures that allow us to store and categorize our basic physical and sensory experiences in dierent meaningful ways**

*Towards Computational Model of Image Schema

**The Body in the Mind(1990)

Image Schema
Image schemas are relatively simple structures that constantly recur in our everyday bodily experience: CONTAINERS, PATHS, LINKS, FORCES, BALANCE ,and in various orientations and relations: UP-DOWN, FRONT-BACK, PART-WHOLE, CENTER-PERIPHERY * According to Johnson, image schemas are structures that organize our mental representations at a level more general and abstract. An image schema is a recurring, dynamic pattern of our perceptual interactions and motor programs that gives coherence and structure to our experience It is universally agreed that image schemas are mental patterns associated with broad classes of concepts or experiences**
*Lakoff 1987: 267, emphasis added **Perception to Meaning by Beat Hamper

 dierent kinds of image schemas are containment (such as houses, cars, and boxes); force (which causes a door to stay open or results from contact with others); up-down orientation (which arises because we live in a gravitational eld) structure of our conceptual schemais primarily determined by practical criteria, rather than abstract or logical ones*
Embodied Cognition: A eld guide, by Andrson

Evidence of Image Schema Existence

Neuroscientists have found evidence indicates activation patterns like image schemas exist in both animal and human brains

How Image Schemas are Formed

Based on our physical features, we create concepts as right and left, front and back, near and far and as human beings able to walk, so we create concepts as source, path, and goal*, all these are image schemas On account of being confronted by forces that can pull or push us (wind, animals, and other human beings) we created concepts as compulsion, attraction and blockage of movement. As a consequence of our orthostatic upright position, we incorporate concepts as balance and verticality.
*Learning phrasal verbs through image schemas: a new approach by Sarah Barbieri Vieira

The main image schemas are:

SOURCE-PATH-GOAL CONTAINER LINK BALANCE PART-WHOLE CENTER-PERIPHERY

Extensions to Image Schema

Some linguists have added new image schemas to the original list proposed by its forerunners. E.g. The BOUNDED REGION If this region has only one dimension, we have the SOURCE-PATH-GOAL schema. If it has two dimensions, we have the SURFACE schema. Finally, if it has three dimensions, we have the CONTAINER schema. A room, for instance, would be the prototypical example of a CONTAINER, but a table would be also a CONTAINER. According to Peas framework, a table would be a SURFACE.

 From the CONTAINER image we can raise two subsidiary images:

FULL-EMPTY and EXCESS as in sentences She is full of confidence or She has an excess of confidence.

Linked to the SOURCE-PATH-GOAL schema, we can get the FRONT-BACK image as in a sentence
I think we need to step back from this situation. Linked to the SURFACE schema, we can get still NEARFAR and CONTACT. As shown in the following diagram

We have placed some image schemas under SURFACE/SOURCEPATH-GOAL because they are subsidiary to both of them.

Typical Image Schema

An image schema will have parts and relations. The parts represent a set of entities, and the relations represent the connection among dierent parts, like
causal relations, temporal sequences, part-whole patterns, etc. Figure 1 can be thought of as a PATH schema.

It contains three parts, a source point A, a terminal point B, and a trace between them.

 The basic meaning of this schema can be understood as moving from A to B in a physical environment. i.e. representing a physical connection.. and It can be used to understand the sentence like the melting of ice into water,

 The IN-OUT schema shown in gure below. The Schema has two parts as landmark (LM) and trajectory(TR). This schema clearly represents the visual situation of the rst sentence. 1. Mary got out of the car.
Here the circle represents the car, and Mary moves along the arrow out of the car. Since the car cant be circular, Mary cant move along a straight line in leaving the car, so this schema gives us only one idealized image.

In-Out Schema

Other Image Schemas

 From the above examples, we can see that image schemas are dynamic patterns rather than xed and static images, as their visual diagrams represent them. They are dynamic in two important respects: 1. schemas are structures of an activity by which we organize our experience in ways that we can comprehend; 2. unlike templates, schemas are exible in that they can take on any number of specic instantiations in varying contexts

Properties of image schemas

Image schemas are pre-conceptual in origin. However once the recurrent patterns of sensory information have been extracted and stored as an image schema, sensory experience gives rise to a conceptual representation. This means that image schemas are concepts, but of a special kind: they are the foundations of the conceptual system, because they are the first concepts to emerge in the human mind,
image schemas are so fundamental to our way of thinking that we are not consciously aware of them, we take our awareness of what it means to be a physical being in a physical world very much for granted because we acquire this knowledge so early in life, certainly before the emergence of language.

Image schema transformations: the body within the brain

Image schemas do not simply exist as single entities, rather We can perform mental operations on image schemas that are analogs of spatial operations. For example, we can rotate mental image both in 2D and 3D situation, and that also at a xed rate of approximately 60 degree per second, it indicates primary schematic operations, called image-schema transformations (i.e. IS are linked together to form natural relationships) Image schema transformations have been shown to play a special role in linking perception and reason. (Lakoff 1987) image schemas motivate important aspects of how we think, reason, and imagine [cognitive linguisticsGeeraerts Al Eds 2006]

Conceptual Metaphor

Metaphorical Projection
Metaphorical interpretation is the bridge by which we can understand and structure one domain of experience in terms of another domain of a dierent kind. Metaphor is thus defined both as a linguistic phenomenon in which vocabulary is shared among different domains, and as a conceptual one in which different conceptual domains are linked by metaphoric mappings Image Schema has thus shows to lie at the basis of numerous metaphorical construction(mapping)*
* Metaphor We Live By

 Metaphorical projection, allows us to project patterns from one domain of experience into the structure of a dierent domain Here, metaphor is not only a linguistic mode of expression; rather, it is one of the chief cognitive structures by which we are able to have coherent, ordered experiences that we can reason about and make sense of

Conceptual Metaphors*
Conceptual metaphor defines an understanding of one conceptual domain in terms of another; for example, using one persons experience of life to understand a different persons experience. A conceptual domain can be any coherently represented experience. Conceptual domains play one of the following two roles in conceptual metaphors: (a) Source domain: the conceptual domain from which we draw metaphorical expressions (b) Target domain: the conceptual domain that we try to understand or explain Metaphorical expressions present themselves in the form of TARGET is the SOURCE where target is the concept to be characterized by the mapping of constituent elements from the source concept.

Metaphor A Practical Introduction

Model for metaphor learning

When we are exposed to a new idea or concept, the brain tries to understand it. we try to ground the concept by building some mental models to capture it. This model may be built either directly in its own right or as composed of other models and symbols. Direct grounding is likely for spatial or temporal phenomenon only. Image schemas such as PATH, SUPPORT, CONTAINMENT etc, have been proposed to model this basic understanding of the physical world. More complex and abstract domains like nance, relationships, politics etc need to be grounded in terms of other grounded symbols.

 examples can demonstrate the importance of metaphorical projection in helping people to connect knowledge from dierent domains. More is up is the most typical one. More is up implies that we understand quantity in terms of a verticality schema in our everyday experience. Examples like Prices keep going up

 and Turn down the heat suggest that we understand MORE as being oriented UPWARD. The two schemas MORE and UP are therefore correlated in our experience that provides a physical basis for our abstract understanding of quantity.

Current Computational models of image schemas and reasoning

One explicit attempt to model image schemas can be found from Regiers work [1996]. Regier focused on spatial relations concepts, like in, out, from, to, on, o, front, back, above, below. Like. the book is on/above /below/in front. He used a connectionist network and the following input: 1. project on a screen a set of movies consisting of a sequence of pictures of a geometrical gure either stationary or moving relative to others; 2. pair each movie with a spatial relation term in some natural language that correctly characterizes the spatial relation depicted on the input screen. 3. Given such pairs of movies and terms, Regier has the network learn spatial relations concepts

Differentiation of Conceptual and Linguistic Metaphor Through Example

Here color-capitals are used for the statement of conceptual metaphors and italics for metaphorical linguistic expressions. AN ARUGMENT IS A WAR Your claims are indefensible. He attacked every weak point in my argument. His criticisms were right on target. I demolished his argument. Ive never won an argument with him. You disagree? Okay, shoot!

If you use that strategy, hell wipe you out. He shot down all of my arguments. LOVE IS A JOURNEY Look how far weve come. Were at a crossroads. Well just have to go our separate ways. We cant turn back now. I dont think this relationship is going anywhere. Where are we? Were stuck. Its been a long, bumpy road.

This relationship is a dead-end street. Were just spinning our wheels. Our marriage is on the rocks. Weve gotten off the track. This relationship is foundering. THEORIES ARE BUILDING Is that the foundation for your theory? The theory needs more support. We need to construct a strong argument for that. We need to buttress the theory with solid arguments.

 the linguistic expressions (i.e., ways of talking) make explicit, or are manifestations of, the conceptual metaphors (i.e., ways of thinking). To put the same thing differently, it is the metaphorical linguistic expressions that reveal the existence of the conceptual metaphors. The terminology of a source domain that is utilized in the metaphorical process is one kind of evidence for the existence of conceptual metaphor

Mapping of Metaphors
There is a set of systematic correspondences between the source and the target in the sense that constituent conceptual elements of b correspond to constituent elements of a. Technically, these conceptual correspondences are often referred to as mappings. Let us look at some cases where elements of the source domain are mapped onto elements of the target domain. Lets take the love is a journey conceptual metaphor first. When we use the sentence We arent going anywhere,the expression go somewhere indicates traveling to a destination, in this particular sentence, a journey which has no clear destination.

 The word we obviously refers to the travelers involved. This sentence then gives us three constituent elements of journeys: the travelers, the travel or the journey as such,and the destination. The relationship is foundering suggests that somehow relationships are conceptually equated with the vehicles used in journeys. The sentence Its been a bumpy road is not about the physical obstacles on the way but about the difficulties that the lovers experience in their relationship.

Metaphor Mapping
Consider the reversed conceptual metaphors mapping, source- target (concrete to the more abstract concept) to make it easily Understandable

Metaphor Mapping Graphically

Christmas is two days away (Time is Space)
two days away recasts an event (Christmas) as a location with respect to the speakers current location in time by specifying a temporal interval (two days) as a distance*

Fig shows: Concept correspondence between source and target domains

*Towards a cognitive model of conceptual blending by Markus, School of Informatics, University of Edinburgh

Metaphor as Knowledge
A significant amount (perhaps the vast majority) of what cognitive psychologists consider knowledge resides in the human mind as a collection of conceptual metaphors that are built up and categorized over years of experience*

* Lakoff

Metaphor, Reason and Imagination*

The reason we have focused so much on metaphor is that it unites reason and imagination. Reason, at the very least, involves categorization, entailment, and inference. Imagination, in one of its many aspects, involves seeing one kind of thing in terms of another kind of thingwhat we have called metaphorical thought. Metaphor is thus imaginative rationality

*Metaphor We Live by

Imagination and Reasoning

Johnson concurs with the nineteenth-century view that imagination connotes creativity, invention and artistic expression. However, he makes the case that imagination also plays a critical role in human reasoning and understanding. Imagination involves the human capacity to organize mental representations into meaningful, coherent unities that are comprehensible. Johnson presents compelling evidence that embodied or image schematic structures are metaphorically extended to direct and constrain peoples networks of meanings.
* Designing to Support Reasoned Imagination through Embodied Metaphor by Alissa Antle

 These acts are ones of reasoned imagination and are used to creatively structure reasoning about novel concepts, perceptions and experiences Hegarty points out that Mental simulation may involve motor representations as well as visual representations.

Blending
metaphor comprehension can be explained in terms of conceptual blending, which involves not mapping from a source to a target but rather combining and fusing two or more mental spaces in order to create a new, blended mental space, This blended space includes both projections from the constituent input domains and inferences not present in any of the inputs.

The meeting Point of Imagery, Schema, Metaphor and Blending

Consider: birth of an economy and growth of economy

The hypothesis is that subconsciously, our understanding of the abstract ideas like of economy" is that of a living entity. When we wish to talk about it, our subconscious mind invokes mental imagery on this living entity, allowing us to think and reason about it.
*On The Computational Modeling of Metaphors thesis by Nitish, Mukerjii 2010

 Since this imagery uses similar neural pathways as the real physical imagery, the language center of the brain might receive similar activations, leading to same choice of words for both the physical and the abstract concepts. In other words we can say: As grounding takes place In the childhood more and more image schemas are created in the mind, but with the age, old image schema are used to model new ideas. Because the brain tries to represent the new knowledge using minimum infrastructure*

Computational Approaches so far to Image Schema and Metaphor

Computational Metaphor Identification to Foster Critical Thinking and Creativity, Phd Thesis by ERIC BAUMER, UNIVERS ITY OF CALIFORNIA, IRVINE,

Working on Computational Metaphor Identification and To model Metaphors to understand how human thinks. And to Not well written Thesis

IS Language
Build by St.Amant et al. [2006]. It is computational formalization , called Image Schema Language, a language in which image schemas can be modeled computationally, The target of designing ISL is to let image schemas have a relational structure with compositional semantics that admits operations of interpretation and permits cross domain transfer of image schema structure. In another words, ISL represents image schemas in a general purpose syntactic forms with the property that syntactic operations on them are equivalent to semantic operations in an indenitely large number of domains.

 For each image schema, ISL represents it as an object with internal slots and set of operations. Slots are used to represent the relations with other objects. Operations determine the capabilities of an image schema object. For example, a container object may take one object as its content, one object as its boundary. It also has operations like put into and get out. Not Much Explained Paper

JEAN System
Event, Action System, by the developer of IS Language

Nash Equilibrium
A fundamental concept in game theory and economics most widely used method of predicting the outcome of a strategic interaction/Decision Making

Nash Equilibrium
If we all go for the blonde and block each other, not a single one of us is going to get her. So then we go for her friends, but they will all give us the cold shoulder because no on likes to be second choice. But what if none of us goes for the blonde? We won't get in each other's way and we won't insult the other girls. It's the only way to win. It's the only way we all get laid [2].

[2]

A Beautiful Mind. Dir. Ron Howard. Universal Pictures, 2001

Prisoner's Dilemma
A game frequently displayed in television police dramas. Two partners in crime are separated into separate rooms at the police station and given a similar deal. If one implicates the other, he may go free while the other receives a life in prison. If neither implicates the other, both are given moderate sentences, and if both implicate the other, the sentences for both are severe. Each player has a dominant strategy to implicate the other, and thus in equilibrium each receives a harsh punishment, but both would be better off if each remained silent. In a repeated or iterated prisoner's dilemma, cooperation may be sustained through trigger strategies such as tit for tat.

Coordination game
The coordination game is a classic (symmetric) two player, two strategy game, with an example payoff matrix shown to the right. The players should thus coordinate, both adopting strategy A, to receive the highest payoff; i.e., 4. If both players chose strategy B though, there is still a Nash equilibrium. Although each player is awarded less than optimal payoff, neither player has incentive to change strategy due to a reduction in the immediate payoff (from 3 to 1).
Player 2 adopts strategy A Player 1 adopts strategy A Player 1 adopts strategy B 4, 4 3, 1 Player 2 adopts strategy B 1, 3 3, 3

Research Proposal Building on: Concepts to Think and Reason Creatively

In the proposed system , we will develop complex concepts by mapping simple ones into each other, to let the computer think and reason creatively Our system will be having some basic level of concepts in the form of simple Image Schemas about world objects like updown, containers., movements/actions etc. Image Schema will lie at the basis of numerious metaphorical constructions to let them project from one domain to other domain.

 That is the basic set of schema making a concept about something will be used as metaphor ( transfer basic idea into a new idea using conceptual metaphor) to create a new idea or concept. There might be conceptual blending in which a new concept will be created. ( If I were you, I would have done that) ( Example of Conceptual Blending) The system will be able to think of its own to perceive language, and reason in a controlled situation/environment (with Basic Image Schema and other inputs/stimuli)

Structure of the System

There will be a number of basic schemas. All these basic schemas will be well defined in terms of meaning, its usage, its compositional structure, its integrational/compositional properties with other schemas, i.e. what is the necessary properties of a schema to be combined with any other schema. in what circumstances, and After integration or combining with other schema what will be the criteria to accept the newly born schema as family, or part of the domain or to decide a new domain.

 Following are different Basic Schemas with different meaning for each schema in different situations. Simple situation is presenting the standard scenario, Accepting situation presents the values/action/events it can receive or get from any other schema. Whereas provider situation shows the conditions in which a schema can provide some properties of its own to any other schema to make a new compound schema to create more complex meaning or concepts or perception about things or events.

Some Rules for IS

Stands for Simple Situation aa 1 Arrow aaa Schema aaaa aaaaaa Cc Circle Ccc 2 Schema Cccc Ccccc Fff Fffff Force 3 Fffffff Schema Fffffffff Ffffffffffff Going Going Going Going Accepting Situation Out Up Down Back Accepting Properties

Provider Provider Marri Situation Properties age to With something in hand something 1,2,3 in hand Over all other stuff The hill With success in hand Throwing 2 something Riding on 3 bike 2,3

 Marriage show which Schema can be combined with which one like, 1 with 2 and which properties it can give and take to form a third schema, what will be the properties of 3 schema they will make, what properties the 3 schema can share and take to other schema when it will combine with any other schema.. what can be the situation in which a schema will combine with another schema to produce good schema and what can be the situation to produce a bad or ill schema.. what is an ill schema .what is a good schema How Nashs Equilibrium will play its role to achieve optimum position.

IS-A
G
Eq

IS-B

IS-C
G
Eq

IS-D

G
Eq
Com IS-II

G e s t a l r t C y c l e

Eq

Where the Flag might show: National Pride Pakistan A country An Asian Country

Concept

Still To Work On: What Inputs to Perceive on

Words Concepts How NN

Emotions as driving force for Intelligence and Image Schema

I Always believed in numbers, in equations and logic that lead to reason, but after a lifetime of such pursuit I asked for what truly is logic? Who decides reason? My quest has taken me to the physical, the metaphysical, the delusional, and back. I have made the most important discovery of my career - the most important discovery of my life. It is only in the mysterious equations of love that any logic or reason can be found. I am only here tonight because of you. You are the only reason I am. You are all my reasons*

Thank You
* A Beautiful Mind. Dir. Ron Howard. Universal Pictures, 2001

Supporting Material

Creativity
creativity to be exhibited needs atleast domain knowledge. no doubt creativity is gifted property of human mind but it doesnt work like Miracle Creativity is the peak of intelligence, it finds out a way out no way using some unrelated relashionships among previous knowledge and experience.

 Creativityany thing unusual with some value is creativity.all human with average intelligence are creative even with the small things they do in their every day lifeI go to Karachi using direct flight all go like this another go through 55 stops because he has some other tasks to do his way to karachi so he plans his route using 55 stops in the way.these stops never routed/follwed by any one else so he is creative.but creativity is assumed something of great importance like a great invention or a great discoverybut for machines we can try to follow the simple form of creativity

 Evolution is an imperfect and often violent process, a battle between what exists and what yet to be born ( Heroes S01E05) For years scientist stumps by the mystery of whales migration. In such a big ocean how do they they find each other. Then some scientists recorded their songs in the wild. They dont sing in captivity ( Heroes S04E08) The Universe isnt random, there are moves and forced moves(Alphas S01E01)

 Context and Culture help to understand natives to understand their language well as compare to well trained non native speakers.