Journal of Mixed Methods Research

http://mmr.sagepub.com/

Mapping Mixed Methods Research: Methods, Measures, and Meaning

J. Wheeldon Journal of Mixed Methods Research 2010 4: 87 DOI: 10.1177/1558689809358755 The online version of this article can be found at: http://mmr.sagepub.com/content/4/2/87

Published by:
http://www.sagepublications.com

Additional services and information for Journal of Mixed Methods Research can be found at: Email Alerts: http://mmr.sagepub.com/cgi/alerts Subscriptions: http://mmr.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations: http://mmr.sagepub.com/content/4/2/87.refs.html

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

Mapping Mixed Methods Research: Methods, Measures, and Meaning

J. Wheeldon1

Journal of Mixed Methods Research 4(2) 87102 The Author(s) 2010 Reprints and permission: http://www. sagepub.com/journalsPermissions.nav DOI: 10.1177/1558689809358755 http://jmmr.sagepub.com

Abstract This article explores how concept maps and mind maps can be used as data collection tools in mixed methods research to combine the clarity of quantitative counts with the nuance of qualitative reflections. Based on more traditional mixed methods approaches, this article details how the use of pre/post concept maps can be used to design qualitative interviews. As a more novel contribution, this article also explores how qualitative participant-generated mind maps can be used in multistage data collection to develop a mixed methods salience score. Although in many ways preliminary, this approach offers an initial means to consider on what basis a mixed methods measure could be conceived. Keywords mixed methods measure, pragmatism, abduction, data collection, concept maps, mind maps

Introduction
The long-standing argument between qualitative and quantitative researchers has underscored broader political differences and research agendas (Jick, 1979). Once useful in creating insular research fiefdoms, for a new generation of researchers the either/or approaches of the past must give way to more nuanced, relevant, and socially useful considerations (Rorty, 1999). Based on the complexity of emergent research problems and the acknowledgment that multiple paths to meaning exist, researchers face what is best described as a series of choices (Palys, 1992). Mixed methods research provides a variety of choices, options, and approaches to consider. (Johnson & Onwuegbuzie, 2004). Although it appears to have emerged as the third methodological movement (Creswell & Plano Clark, 2007, p. 13), its development is still ongoing. In the past few years, scholars in this journal and elsewhere have debated whether mixing multiple methods was possible and explored the theoretical basis for this integration. More recent lines of inquiry have focused on the construction of mixed approaches and how this may vary by discipline and the needs of individual researchers (Tashakkori & Creswell, 2008). Another question that might be explored is whether and how data collection procedures from other disciplines can be used in

Heritage University, Toppenish, WA, USA

Corresponding Author: J. Wheeldon, Heritage University, 3240 Fort Road, Toppenish, WA 98948, USA Email: jwheeldon@gmail.com

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

88

Journal of Mixed Methods Research 4(2)

mixed methods research and how these tools may influence and inform methods, measures, and meaning. One trend in social science research is the development of new means of data collection that are either more explicitly user-generated or that are less influenced by what may be sui generis research/participant interactions. In fields such as health, education, and engineering, participant-generated concept maps and mind maps are of resurgent interest (Nesbit & Adescope, 2006). Although existing models of mixed methods research exist and remain viable, maps offer a unique way for research participants to represent their experiences while assisting researchers to make better sense of gathered data. For mixed methods research, maps can provide an accessible and innovative means to combine the clarity of quantitative counts with the nuance and perspective of qualitative reflections. Indeed, maps appear flexible enough to be used both in established mixed methods approaches and in the construction of unique and novel mixed method measures. This article is organized to present a short overview of existing approaches to mixed methods research and review how pragmatism and abductive reasoning are connected to the relationship between theory and data. By introducing and differentiating concept maps, this article provides specific examples of how concept and mind maps can be used, alone and in combination, to contribute to a broader discussion about data collection in social science research.

Mixed Methods Approaches: Theory and Practice

As mixed methods research has grown in practice and recognition, the combination of quantitative and qualitative strategies has required new thinking about the theoretical basis for integrative research. Greene and Caracelli (1997) have highlighted a number of purposes or justifications for mixing methods. These include the following: to test the consistency of findings obtained through different instruments, to clarify and build on the results of one method with another, and to show how the results from one method shape subsequent methods or research decisions. Four major types of mixed methods designs have been presented. These include the triangulation design, embedded design, explanatory design, and exploratory design (Creswell & Plano Clark, 2007). The number and variety of research designs and approaches has required the consideration of an alternative epistemological framework. A variety of views on this paradigmatic issue have been suggested (Greene, Benjamin, & Goodyear, 2001; Tashakkori & Teddlie, 1998), and pragmatism has emerged as a common alternative to the either/or choice of positivism and constructivism (Creswell & Plano Clark, 2007). Thus, instead of relying on deductive reasoning and general premises to reach specific conclusions, or inductive approaches that seek general conclusions based on specific premises, pragmatism allows for a more flexible abductive approach. By focusing on solving practical problems, the debate about the existence of objective truth, or the value of subjective perceptions, can be usefully sidestepped. As such, pragmatists have no problem with asserting both that there is a single real world and that all individuals have their own unique interpretations of that world (Morgan, 2007). The mixed methods approach is perhaps of most interest to the postparadigm generation of scholars because it can produce more robust measures of association while explicitly valuing the depth of the experiences, perspectives, and histories of research participants (Ritchie & Lewis, 2003, p. 3). Abduction is crucial in this respect. Through an innovative combination of existing knowledge, one can both generate possible research solutions and at the same time attempt to integrate various theories and approaches (Tomiyamal, Takeda, Yoshioka, & Shimomura, 2003). In this way, abductive reasoning allows for tentative explanations and hypotheses to emerge through the research process based on the expertise, experience, and intuition of researchers (Schurz, 2002). Through this iterative approach, these tentative explanations can be tested both

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

Wheeldon
Table 1. Key Issues in Social Science Research Quantitative Approach Connection of theory and data Relationship to research process Inference from data Source: Morgan (2007, p. 71). Deductive Objectivity Generality Qualitative Approach Inductive Subjectivity Context

89

Pragmatic Approach Abductive Intersubjectivity Transferability

theoretically and empirically. As Morgan (2007, p. 71) suggests by moving back and forth between induction and deduction, one can convert observations into theories and then assess those theories through action. In this way, understanding based on shared meaning can benefit from the different approaches to research outlined in Table 1. Studies that employ mixed methods may do so in a variety of ways. Researchers may gather data through quantitative surveys and qualitative interviews (Baumann, 1999; Way, Stauber, Nakkula, & London, 1994), use different data procedures though multiple data collection phases (Rogers, Day, Randall, & Bentall, 2003), or collect both qualitative and quantitative data at the same time (Gogolin & Swartz, 1992; J. E. Jenkins, 2001). Collecting different sorts of data may be challenging, given the approach and types of data traditionally associated with quantitative and qualitative research. Quantitative data are often based on instruments that measure individual performance and attitudes or use interview or observational data based on clearly predefined categories. In mixed methods, researchers have usefully shown how to better explain quantitative findings with qualitative data (Aldridge, Fraser, & Huang, 1999). Yet additional approaches are needed to meaningfully embed and bring to light qualitative data within quantitative designs (Creswell & Plano Clark, 2007). In contrast, qualitative data are generally based on themes that emerge through open-ended interviews, observations, or the review of various documents and audiovisual materials such as videotapes, photographs, sounds, and even email and text messages (Creswell, 2005). Past mixed methods approaches have sought to use qualitative data to develop quantitative data collection tools (Myers & Oetzel, 2003) and explored new ways to quantify qualitative research (Sandelowski, 2001). Although many advance a broad definition of mixed methods that includes a wide variety of research methods and means of data collection, more often traditional data collection means are employed in mixed method research. It may be time to consider other forms of data collection that provide a means for researchers to see more easily participant meaning and use practical data analysis strategies through the flexibility abductive reasoning affords. One approach to the focus on intuition, shared meaning, and transferability of data might be usefully connected to the idea that knowledge and understanding are themselves based on patterns (Kaplan, 1964). These patterns might be better identified, recognized, and understood through more graphic representations of knowledge, experience, and perception (Wheeldon, 2009). Despite the wide variety of types, tools, and approaches since the 1970s, visual depictions of data analysis have not become widely popular. One reason may be that, according to Ahlberg (2008), too few are willing to acknowledge the shoulders of those on which they stand. Indeed, it often appears this is an area in which many different types of graphic organizers are pitted against one another. Examples of various approaches include box-and-whisker charts and stem-and-leaf diagrams (Tukey, 1977), concept maps (Novak, 1981), vee diagrams (Gowin, 1981), mind maps (Buzan, 1974), clustering (Rico, 1983), knowledge mapping (Dansereau, Sells, & Simpson, 1979), and spider diagrams (Hanf, 1971). Despite these numerous examples, visual representations in current research are generally limited to the presentation of data and explanation of findings. A notable exception in this regard is Creswell

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

90

Journal of Mixed Methods Research 4(2)

Concept Maps

connect

are organized

Related Ideas through different kinds of

assists recall of

Memories/ Experiences/ Perceptions/ Reflections

hierarchically demonstrate

are related to

guide

Links and Cross links

Individual Understanding

Meaning Making

Figure 1. Concept map of a concept map

and Plano Clarks (2007) mixed methods text that uses a variety of diagrams, figures, and maps to visually present mixed methods approaches, strategies, and procedures. Of immediate interest is how concept maps and mind maps can be used to collect data in social science research. Based on the notion that people attempt to make meaning of the world around them (Novak & Gowin, 1984) and the assumption that this meaning is based on the recognition of patterns (Miles & Huberman, 1994), tools that graphically represent knowledge and experience may be especially useful. Whereas concept maps provide a more structured approach to explore connections between and among concepts using linking words, mind maps are more flexible tools in which a central governing concept is explored using groupings or areas. By asking participants to complete concept or mind maps, these maps and the ways in which concepts are presented, ordered, and connected can serve as another means for researchers to collect data. These data can be used alone or to design additional data collection strategies, to better explore and assess meaning.

Concept Maps and Mind Maps in Social Science Research

In general terms, mapping knowledge is a technique that can demonstrate how people visualize relationships between various concepts (Lanzig, 1996). Both concept maps and mind maps focus on the meaning associated with visual representations of experience, knowledge, perception, or memory. Yet important differences exist. These are related to the way the maps are structured, what they may represent, and the ways in which their meaning might be interpreted. A concept map can be defined as a graphical tool for organizing and representing knowledge. First presented formally by Novak (1981), this approach was usefully expanded and developed in the seminal book Learning How to Learn (Novak & Gowin, 1984). Concept maps include unique concepts, usually enclosed in circles or boxes. Lines and linking words between and among concepts suggest hierarchical relationships as presented in Figure 1.

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

Wheeldon

91

Beliefs

Examples

Relationships

Linkages

Related Concept(s)

Feelings CENTRAL THEME/ FOCUS Connections

Personal Experience

Examples

Memories

Figure 2. Constructing a mind map

A mind map, in contrast, is a diagram used to represent words, ideas, tasks, or other items linked to and arranged around a central key word or idea. First described by Buzan in 1974, they were further defined and developed over the next decade. Mind maps are less formal and structurally more flexible. They may present ideas in a variety of ways (Buzan & Buzan, 2000). Figure 2 provides an example of how to construct a mind map. As is clear it begins with a central idea and builds outward. Interest in the use of maps in social science research is clearly increasing. Nesbit and Adescope (2006) discovered, through selective searches of education and psychology databases, that more than 500 peer-reviewed articles have been published on the application of concept or knowledge maps and that most of these publications came after 1997 (Nesbit & Adescope, 2006). In the past, concept maps have most commonly been used as part of quantitative research designs, perhaps owing to their first use in science education (Novak, 1981). More recently, they have been used to facilitate knowledge integration and explore student understanding (Turns, Atman, & Adams, 2000). In education, they have been shown to be more effective to assist knowledge retention than attending class lectures, reading, or participating in class discussion (Poole & Davis, 2006). They can affect concentration and overall test performance by promoting interaction and engagement between the student and material (Hall & ODonnell, 1996). Maps may be an easier way to communicate ones knowledge when compared with text writing (Czuchry & Dansereau, 1996). Mind maps have been used in nursing education (Rooda, 1994), care planning (Kern, Bush, & McCleish, 2006; Mueller, Johnston, & Bligh, 2001), and evaluation (A. Jenkins, 2005), but in general, there is less specific research on mind maps than concept maps (Farrand, Hussain, & Hennessy, 2002; Pressley, VanEtten, Yokoi, Freebern, & VanMeter, 1998). One reason may be that many appear to refer to concept, knowledge, cognitive, and mind maps as though they are interchangeable (Nesbit & Adescope, 2006). This oversight belies the important differences

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

92

Journal of Mixed Methods Research 4(2)

MAIN CONCEPT

Links

Links

Scoring Links/Connectors 1pt Examples 2 pts Related Concepts 5pts Cross links 10 pts

Related Concept connects

Cross Link

Related Concept

connects

connects connects

example

example

example

Figure 3. How to score a concept map

that exist and undermine their utility in research design. In short, concept maps often are more structured and include multiple concepts, whereas mind maps usually are less formal and generally focus on one topic at a time. These inherent differences provide some flexibility for researchers interested in using maps in different sorts of mixed methods research designs (Wheeldon, 2009). One way to think about the differences between concept maps and mind maps is to consider how they have been used in past research.

Concept Maps: Quantitative Research and Mixed Method Designs

Concept maps are most commonly used in quantitative research. This may be because earlier versions of concept maps were used to explore science education (Stewart, Van Kirk, & Rowell, 1979), and were often quantitatively scored by an expert to assess how understanding was demonstrated through the structure of the map itself. A focus on structure remains an integral feature for many concept map researchers who insist that a number of steps are required for the creation of a good concept map. These include the generation of an initial ranked list of concepts, a hierarchical presentation in the map itself of general concepts on top and specific concepts on bottom, and the use of linking words to show how the various concepts are related (Novak & Caas, 2008). Once created in this manner, maps can be assessed, scored, and/or compared to assess an individuals understanding of a topic. Novak and Gowin (1984) describe the utility of maps to assess understanding in education. They argue that by having students complete concept maps on certain topics, structured interview questions can be posed to the student to explore areas of student misunderstanding or confusion based on the students map. To score a concept map, Novak and Gowin (1984) suggested that maps be assessed by a subject matter expert based on the number of valid propositions, levels of hierarchy, and number of branchings, cross-links, and specific examples provided in the map. Figure 3 provides an example of how a concept map might be scored based on its structure. Since then, various refinements have been presented (Besterfield-Sacre, Gerchak, Lyons, Shuman, & Wolfe, 2004; Rye & Rubba, 2002). Similar structural scoring systems have been

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

Wheeldon

93

proposed based on the physical characteristics of the maps (Turns et al., 2000). These approaches score the quality of concepts and propositions within the maps (Rye & Rubba, 2002) or identify different levels of concepts and subconcepts (Bayram, 1995). Other approaches rely on maps as a means to compare experts and learners (McGaghie, McCrimmon, Mitchell, & Thompson, 2000) and develop scoring systems based on the convergence of concepts between a novice and expert map (Ruiz-Primo, Shavelson, Li, & Schultz, 2001). However one chooses to score a map, a research design of immediate interest uses concept maps as a pre/post data collection tool to quantitatively suggest how understanding, views, and/or perceptions can change over time (Kilic, Kaya, & Dogan, 2004). This approach can usefully be applied to mixed method research. By first comparing the pre/post maps of participants, specific interview questions can be designed to explore either of the pre/post maps as presented by the participant or the space and time in between the completion of the maps. A variant of this approach was used in a pilot study to assess different teaching strategies for internship students related to values and ethics in criminal justice (Wheeldon, 2008). Forty-five students were assigned a unique identifier code and were asked to complete a map during the first pre-internship class and again during the last pre-internship class. Based on a predefined structure, the maps identified important individual values and ethics for each student along with their perceived origin such as school, family, and religion among others. To assess how views had changed over the course, the pre/post maps were scored based on the number of individual concepts and on the complexity based on the number of hierarchies within the maps. The results were statistically significant. Students not only identified more concepts, values, ethics, and origins on average but also provided greater detail within their post maps about the connections between events, experiences, and the development of their ethical beliefs (Wheeldon, 2008). In addition to scoring and comparing the pre/post maps, of additional interest was how and why these views changed over time. Within the literature disagreement exists about the means used to guide instruction on values and ethics (Cederblom & Spohn, 1991; Heffernan & Stroup, 1985). To explore student perceptions about these questions, data were collected from students during the course of their 4-month criminal justice internship. Through an open-ended survey and focus groups, they provided their views on the importance of ethics to their placements and the value of the different approaches, exercises, and scenarios used to teach ethical decision making during the pre-internship course (Wheeldon, 2008). This approach is in some ways similar to existing mixed method approaches that use quantitative surveys and qualitative interviews (Way et al., 1994). In this case, the concept map simply served in place of another survey instrument. Although not appropriate for all types of research, this approach provides an innovative strategy for mixed methods based on pre/post data collection that may allow the researcher to further explore the graphic data within the maps through subsequent qualitative interviews.

Mind Maps: Qualitative Research and Mixed Methods Measures

Despite the successes of mixed method researchers in demonstrating how different approaches to data collection and analysis can lead to a more vigorous interrogation of meaning, doubts persist about the combination of approaches based on what are very different foundational assumptions (Elliott, 2005). This concern may also be relevant in a discussion of the utility of maps in social science research more broadly. As Wheeldon and Faubert (2009) note, some researchers have explored how to use concept maps as part qualitative research. A useful example is Anne Raymonds (1997) work in math education. Yet complications can arise when qualitative researchers adhere to the formal requirements of concept maps (Daley, 2004). This may result in research that favors quantitative comparisons over unique participation representations. More flexibility may be required if maps are to become a more commonly used data collection tool in social

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

94

Journal of Mixed Methods Research 4(2)

science research (Wheeldon & Faubert, 2009). Indeed, for qualitative researchers, capturing unique and individual perceptions, reflections, or experiences is prized, and thus, the flexibility of mind maps may be more appropriate for qualitative research designs. Less structured and formal than traditional definitions of concept maps, mind maps can be more unsolicited, free form, and user generated (Buzan & Buzan, 2000). It has also been suggested that mind maps may provide a valuable means to collect more personalized and individualistic data from research participants (Tattersall, Watts, & Vernon, 2007). Thus, instead of focusing on how well the maps adhere to a set of predetermined rules, researchers can consider how the construction of a participants map is connected to the meaning associated with a particular topic or experience. This approach was used in a pilot study about how selected international justice trainers who worked abroad understood their role as experts (Wheeldon, 2007). Based on the structure of each map, qualitative interviews were designed to explore the connections identified within the maps. Instead of quantitatively scoring the maps, the identification of themes within the maps led to another more complex research design that combined the frequency of individual variables identified over a variety data collection stages. In that study (Wheeldon, 2009), the construction of a novel mixed methods measure, the salience score, was developed to assess the effectiveness of various elements of training and capacity development in a legal technical assistance project in the Baltic State of Latvia. Based on emergent views within organizational and international development (Chodosh, 2005), and legal technical assistance literature (Shaw & Dandurand, 2006), this study explored how participants viewed various components of capacity building and training during a 20-month justice reform project. In total, 19 participants provided data over 1 year through a multistage research design. In the first stage of data collection, participants were asked to complete a mind map about their experience of a legal technical assistance project. In the second stage of data collection, participants were asked more general interview questions. These general questions were open-ended and probed positive and negative experiences, perceived results and challenges, as well as previously identified concepts, gathered through a literature review. In addition to the general questions in the first and second stages, in the third stage, specific questions were asked of participants based on each participants map to clarify meaning, probe connections, and explore the maps construction. Finally, in the fourth stage, conclusionary and more reflective open-ended questions followed the more directive data collection stages. By providing participants an opportunity to identify areas not previously addressed, it was hoped they would reflect on their experience as whole, restating aspects of particular significance, or provide additional clarifying commentary. By combining the maps with the different stages of follow-up interviews, the frequency with which individual variables were identified through the multiple data collection stages was recorded. To analyze the interview data in a more meaningful way a mixed methods measure called a salience score was developed and tested. The construction of a mixed methods salience score may involve a number of separate yet rather simple operations. In the first step, unique, individual concepts, elements, and activities identified by participants in different stages of data collection can be recorded as variables. These variables can then be quantified through the use of a concept counting system that records the frequency or presence of individual variables throughout data collection. Individual variables might be identified in mind maps, through general or specific interviews, or in summative and reflective statements. They also may be identified in one, multiple, or all stages of data collection. These initial calculations provide a sense of the number of times individual variables were identified in total across the data collection stages and the number of times each participant identified a variable across multiple data collection stages. Of some interest, these frequency measures can only provide a sense of whether and how often these variables were identified. An assumption of this study was that the way in which

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

Wheeldon
Table 2. Tabulating the Salience Scoring Data Collection Stage 1. Mind map 2. General interview 3. Specific interview 4. Reference statement Total Salience score Frequency of Concept 0 1 0 1 2 5 Weighted Measure 4 3 1 2 Percentage 0 50.0 0 50.0 100 Valid Percentage 0 50.0 0 50.0 100

95

Cumulative Percentage 0 50.0 0 100.0

Source: Wheeldon (2009). [AQ: 12 ]

the variables were identified might more usefully demonstrate the relevance or legitimacy of a proposed association between an individual and a concept (Cash et al., 2002). For each variable identified in multiple stages of data collection, a salience score or weighted measure was developed using a weighted count system (Stillwell, Winterfeldt, & John, 1987). This strategy allows the researcher to assign participants a score for each individual variable they identify depending on the stage(s) at which these variables were recorded. For example, individual variables that emerged from user-generated, open-ended, and unsolicited data collection procedures can be treated as more valuable and given more weight in the overall measure. User-generated concepts gathered through the maps might be deemed worth 4 points, along with responses to general, nonspecific questions worth 3 points. Concepts identified by respondents based on specifically designed questions by the researcher could be worth the least at 1 point, whereas conclusionary questions asked at the end of both the general and specific question sets could be worth 2 points, given that participants came back to these concepts after several other data collection stages. To tabulate a score for each individual variable, common unique variables were identified in each mind map (Turns et al., 2000) and within the qualitative interviews (Sandelowski, 2001) and then combined for each individual. By adding the points assigned through each stage of the data collection process, salience scores for identified variables can produce values ranging from 0 (not salient) to 10 (extremely salient). Table 2 presents an example of how a salience score of 5 might be tabulated for a concept identified in two out of four stages of data collection. By repeating this process, a mixed method salience score can be tabulated for each individual variable in which both the frequency of the concepts identified and the way in which the data were collected can be combined. To get an overall variable salience score (OVSS) for the total sample, each participants individual variable salience score (IVSS) can be added together and the result divided by the total sample (n). This operation can be represented by the formula OVSS IVSS 1 IVSS 2 IVSS n =n: Although in some ways this approach is novel, it is based on the use of qualitative data to develop a quantitative data collection tool (Creswell & Plano Clark, 2007). In this study, instead of using qualitative findings to create and validate a quantitative data collection survey (Myers & Oetzel, 2003), the salience score as a measure is based on a quantification of qualitative data. By explicitly privileging, through a weighted scoring scheme, variables that emerge through more unsolicited data collection stages, the salience score can incorporate both overall variable frequencies while accounting for variables identified throughout multiple data collection stages. When combined with the more nuanced qualitative data gathered through interviews, this approach may provide a strengthened means

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

96

Journal of Mixed Methods Research 4(2)

to clarify and build on the results of one method with the perspective of another (Greene & Caracelli, 1997).

Mapping Mixed Methods: Discussion and Limitations

Although the use of maps as data collection means provides a flexible approach for researchers and a novel experience for participants, it is based on an assumption that maps and other graphic tools are a useful means to capture experience. Although it seems clear that how people visualize relationships between various concepts or ideas can suggest more dynamic schemes of understanding (Mls, 2004), the use of maps in social science research may require a more in-depth theoretical justification. Beyond the purview of this article, this justification might be built on the notion that individuals think more clearly and effectively if they can avoid the assumptions and inherent limitations built into language (Korzybski, 1933). Indeed, because people learn in different ways and think using a combination of words, graphics, and images, the focus on language as the sole means of communication may be arbitrary (Ausubel, 1963). One problem with verbal interactions is that individual human experience often gets filtered and mediated by linguistic constructions (Whorf, 1956). For data collection in social science research, this may mean that surveys and interviews may focus on recording only one sort of data, based on the psycholinguistic assumptions about the role of syntax, semantics, and context in the construction of meaning (Cassirer, 1946). Maps may provide a means to break out of conventional representations of experience through language and allow participants and researchers to access other kinds of information (Legard, Keegan, & Ward, 2003). Although more study is needed, it does appear as though maps provide a means of data collection that can be used as part of multiple mixed method designs. By scoring pre/post maps and basing interview questions on the results, maps provide an alternative to existing approaches such as those that rely on surveys and other more traditional means of data collection. Various map scoring systems exist and are still being developed (Ahlberg & Ahoranta, 2004). The pre/post mixed method concept map approach outlined above is perhaps a rather obvious example of how maps can fit within established mixed method approaches (Way et al., 1994). Less obvious is the approach taken to develop a salience score based on multistage data collection designs that include mind maps. A core assumption here is that the more user-generated the data collection, the more weight should be assigned to the variables that emerge. By using mind maps, by definition more flexible and participant focused, the variables in the maps may take on new significance. To ensure this significance is self-sustaining, the salience score ensures that variables identified in multiple stages of data collection emerge as central quantitative findings. By reviewing the qualitative data that emerges from the interviews, this numeric finding can be explored and considered in more detail (Greene & Caracelli, 1997). Despite the potential for both of these approaches to contribute to new means, methods, and measures in mixed methods research, important limitations exist. The first relates to the observed resistance among some participants when asked to create maps. It appears that whereas the graphic representation of experience may serve certain populations, it may alienate others. Some participants have suggested that completing the map was difficult, strange, and hard to complete (Wheeldon, 2009). Indeed, there is evidence that important differences between people, groups, personalities, and learning styles can limit the utility of concept maps to gather data (Rohm, 1994). A related concern is how the maps themselves are constructed. This is important in two ways. For many researchers concept maps need to be created based on strict criteria, and some suggest participants must be taught to create them properly (Novak & Caas, 2008). This can create a variety of practical complications for researchers and

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

Wheeldon

97

participants related to the comparability of concept maps. To address this, researchers interested in using concept maps might investigate the use of computer software, such as CMap among others, to create a defined interface for participants. This can better ensure consistency and facilitate concept quantification and map comparisons (Bayram, 1995; Safayeni, Derbentseva, & Caas, 2005). Another challenge is specific to mind maps. Whereas a general challenge in qualitative research is the difficulty in analyzing the data generated (Patton, 2002), described variously as an attractive nuisance (Miles, 1979) and an embarrassment of riches (Chenail, 1995, p. 2), maps may be especially difficult to read and/or interpret. One solution may be to limit maps to one page and provide clear instructions along with an exemplar map to participants when using mind maps to collect data (Wheeldon, 2007). An additional concern relates to the salience score itself. As a means to quantify qualitative findings, this measure can be justified as providing a useful means to highlight the value of data collected through stages that are less directive and more user generated. Thus the maps and the responses to the general interview questions are treated as most important. Although defensible, there are fewif anystudies that have attempted to weigh data in this way. In addition, the combination of individual scores without accounting for demographic differences within the sample is problematic. One approach is to collect demographic data during data collection. Before combining the individual variable salience scores, researchers can assess the statistical significance of demographic differences among groups in the sample using an independent t test (adjusted for multiple tests). If no significant differences are observed between demographic groups, the salience score can be assumed to be a useful measure to assess the value of concepts that emerged through data collection. A final consideration is related to the problem of validity and reliability for the data gathered from maps. Identified by Trochim (1989), reliability can be understood here to mean the degree to which the data in the map are repeatable, whereas validity refers to the degree to which the data in the map accurately reflect reality. The suggestions provided in this article may offer a partial reply to some of these concerns. Issues related to the quality of the data derived from maps nonetheless require more investigation. Although it may be tempting to argue that reliability and validity are addressed if data from participant maps also emerges through other sorts of data collection, the quality of the data within a map ought not be confused with the credibility of conclusions drawn from data gathered through multiple stages. Indeed, although maps appear to offer a means to assist researchers to abductively infer meaning, this article has argued that more attention should be paid to how researchers collect, analyze, and derive meaning from data in general. Mixed methods may offer a useful platform for further and more in-depth exploration.

Conclusion
As mixed methods research continues to grow and demonstrate its viability, integrity, and flexibility, scholars would do well to reflect on a challenge posed by Morgan (2007). To be sustainable mixed methods requires a dialogue about which aspects of the work of researchers from different traditions can be practically used in combination. Mixed methods research can no doubt continue to rely on traditional data collection means and ordering, combining, or embedding findings through existent approaches. Yet other models may exist. This article has explored how renewed interest in alternative forms of data collection may be based on a pragmatic understanding of intuitive and abductive connections between theory and data. Concept maps and mind maps may be important tools in this regard. By providing a visual record of understanding,

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

98

Journal of Mixed Methods Research 4(2)

they offer what might be seen as unique data that can be assessed both quantitatively and qualitatively. The combination of these different research approaches and associated data analysis strategies may provide a means to gain a better understanding of phenomena under investigation, yet it is far from certain that multiple poorly designed data collection strategies are always better. With this in mind, it may be time for a renewed effort and examination of what sorts of data collection are best suited to research problems, designs, and analyses. Although concept maps and mind maps offer a novel means of data collection, more study, reflection, and research examples are needed to assess whether and to what extent maps can be usefully employed by mixed methods researchers. Indeed, more of a focus on the choices researchers make in considering research problems may illicit valuable and unconsidered aspects of research and assumptions about the viability of various data collection tools. As a methodological contribution, this article has attempted to provide a theoretical justification for the use of concept maps and mind maps in mixed methods research, present associated methods and measures, and consider some specific examples of how maps might be used as part of multistage data collection strategies. This article has argued that the more structured concept map may be better suited to designs in which quantitative scoring and comparisons of pre/post maps are used to shape and guide subsequent qualitative data collection. The more flexible mind map might be best used in designs in which more open-ended qualitative data collection data are combined to tabulate a quantitative mixed method salience score. By explicitly valuing data collected through more unsolicited means, these approaches can combine the reliability of counts with the credibility of participant perception. By combining data gathered through participant mind maps with other sorts of collected data, a weighted scoring systems may be a useful way to think about quantifying qualitative data. The simple construction of the presented mixed measure salience score should allow it to be tested, challenged, revised, and perhaps validated. Acknowledgments
I would like to extend my thanks to Simon Verdun-Jones, Bill Glackman, and Stephen Easton of Simon Fraser University for their support and assistance. I would also like to acknowledge the reviewers at JMMR whose comments and suggestion have significantly strengthened this article.

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the authorship and/or publication of this article.

Funding
The author received no financial support for the research and/or authorship of this article.

References
Ahlberg, M. (2008, September). Concept mapping as an innovation: Documents, memories and notes from Finland, Sweden, Estonia and Russia 1984-2008. Paper presented at the proceedings of the Third International Conference on Concept Mapping, Tallin, Estonia. Ahlberg, M., & Ahoranta, V. (2004, September). Concept maps: Theory, methodology, technology. Paper presented at the First International Conference on Concept Mapping, Pamplona, Spain.

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

Wheeldon

99

Aldridge, J. M., Fraser, B. J., & Huang, T. (1999). Investigating classroom environments in Taiwan and Australia with multiple research methods. Journal of Educational Research, 93(1), 48-62. Ausubel, D. (1963). The psychology of meaningful verbal learning. New York: Grune & Stratton. Baumann, C. (1999). Adoptive fathers and birthfathers: A study of attitudes. Child & Adolescent Social Work Journal, 16, 373-391. Bayram, S. (1995). The effectiveness of concept and software mapping for representing student data and process schema in science. Unpublished masters thesis, University of Pittsburg, PA. Besterfield-Sacre, M., Gerchak, J., Lyons, M., Shuman, L. J., & Wolfe, H. (2004). Scoring concept maps: An integrated rubric for assessing engineering education. Journal of Engineering Education, 93, 105-116. Buzan, T. (1974). Use of your head. London: BBC Books. Buzan, T., & Buzan, B. (2000). The mind map book. London: BBC Books. Cash, D., Clark, W. C., Alcock, F., Dickson, N., Eckley, N., & Jäger, J. (2002). Credibility, legitimacy and boundaries: Linking, assessment and decision making. Cambridge, MA: Kennedy School of Government. Cassirer, E. (1946). Language and myth. New York: Harper & Brothers. Cederblom, J., & Spohn, C. (1991). A model for teaching criminal justice ethics. Journal of Criminal Justice Education, 2, 201-218. Chenail, R. J. (1995, December). Presenting qualitative data. The Qualitative Report, 2(3). Retrieved December 11, 2009, from http://www.nova.edu/ssss/QR/QR2-3/presenting.html Chodosh, H. (2005). Global justice reform: A comparative methodology. New York: New York University Press. Creswell, J. W. (2005). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Pearson. Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: SAGE. Czuchry, M., & Dansereau, D. F. (1996). Node-link mapping as an alternative to traditional writing assignments in undergraduate psychology courses. Teaching of Psychology, 23(2), 91-96. Daley, B. (2004, September). Concept maps: Theory, methodology, technology. Paper presented at the proceedings of the First International Conference on Concept Mapping, Pamplona, Spain. Dansereau, D., Sells, S. B., & Simpson, D. D. (1979). Evaluation of treatment for youth in the Drug Abuse Reporting Program. In G. Beschner & A. S. Friedman (Eds.), Youth drug abuse: Problems, issues, and treatments (pp. 571-628). Lexington, MA: Lexington Books. Elliott, J. (2005). Using narrative in social research: Qualitative and quantitative approaches. London: SAGE. Farrand, P., Hussain, F., & Hennessy, E. (2002). The efficacy of the mind map study technique. Medical Education, 36(5), 26-31. Gogolin, L., & Swartz, F. (1992). A quantitative and qualitative inquiry into the attitudes toward science of nonscience college students. Journal of Research in Science Teaching, 29, 487-504. Gowin, D. B. (1981). Educating. Ithaca, NY: Cornell University Press. Greene, J. C., Benjamin, L., & Goodyear, L. (2001). The merits of mixing methods in evaluation. Evaluation, 7(1), 25-44. Greene, J. C., & Caracelli, V. J. (1997). Defining and describing the paradigm issue in mixed-method evaluation. In J. C. Greene & V. J. Caracelli (Eds.), Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms (New Directions for Program Evaluation, No. 74). San Francisco: Jossey-Bass, 1-18. Hall, R. H., & ODonnell, A. (1996). Cognitive and affective outcomes of learning from knowledge maps. Contemporary Educational Psychology, 21, 94-101. Hanf, M. B. (1971). Mapping: A technique for translating reading into thinking. Journal of Reading, 14, 225-230. Heffernan, W. C., & Stroup, T. (1985). Police ethics: Hard choices in law enforcement. New York: John Jay Press. Jenkins, A. (2005). Mind mapping. Nursing Standard, 20(7), 85.

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

100

Journal of Mixed Methods Research 4(2)

Jenkins, J. E. (2001). Rural adolescent perceptions of alcohol and other drug resistance. Child Study Journal, 31, 211-224. Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, 602-611. Johnson, R. B., & Onwuegbuzie, A. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26. Kaplan, A. (1964). The conduct of inquiry: Methodology for behavioral science. San Francisco: Chandler Press. Kern, C. S., Bush, K. L., & McCleish, J. M. (2006). Mind-mapped care plans: Integrating an innovative educational tool as an alternative to traditional care plans. Journal of Nursing Education, 45, 112-119. Kilic, Z., Kaya, O. N., & Dogan, A. (2004). Effects of students pre- and post-laboratory concept maps on students attitudes toward chemistry laboratory in university general chemistry. Paper presented at the 18th International Conference on Chemical Education, Istanbul, Turkey. Korzybski, A. (1933). Science and sanity: An introduction to non-Aristotelian systems and general semantics. Fort Worth, TX: Institute of General Semantics. Lanzig, J. W. A. (1996). Everything you always wanted to know about . . . concept mapping. Retrieved October 16, 2006, from http://utto1031.to.utwente.nl/artikel1/ Legard, R., Keegan, J., & Ward, K. (2003). In-depth interviews. In J. Ritchie & J. Lewis (Eds.), Qualitative research practice: A guide for social research students and researchers (pp. 138-169). Thousand Oaks, CA: SAGE. McGaghie, W., McCrimmon, D., Mitchell, G., & Thompson, J. (2000). Quantitative concept mapping in pulmonary physiology: Comparison of student and faculty knowledge structures. Advanced Physiological Education, 23(1), 72. Miles, M. B. (1979). Qualitative data as an attractive nuisance: The problem of analysis. Administrative Science Quarterly, 24, 590-600. Miles, M. B., & Huberman, M. A. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.) Newbury Park, CA: SAGE. Mls, K. (2004). From concept mapping to qualitative modeling in cognitive research. Retrieved August 19, 2008, from www.cmc.ihmc.us/papers/cmc2004-159.pdf Morgan, D. L. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1), 48-76. Mueller, A., Johnston, M., & Bligh, D. (2001). Mind-mapped care plans: A remarkable alternative to traditional nursing care plans. Nurse Educator, 26(2), 75-80. Myers, K., & Oetzel, J. (2003). Exploring the dimensions of organizational assimilation: Creating and validating a communication measure. Communication Quarterly, 51, 436-455. Nesbit, J. C., & Adescope, O. O. (2006). Learning with concept and knowledge maps: A meta-analysis. Review of Educational Research, 76, 413-448. Novak, J. D. (1981). Applying learning psychology and philosophy of science to biology teaching. The American Biology Teacher, 43(1), 12-20. Novak, J. D., & Caas, A. J. (2008). The theory underlying concept maps and how to construct and use them (Tech. Rep.). Pensacola: IHMC Florida Institute for Human and Machine Cognition. Novak, J. D., & Gowin, J. B. (1984). Learning how to learn. Cambridge, UK: Cambridge University Press. Palys, T. (1992). Research decisions: Quantitative and qualitative perspectives (3rd ed.). Toronto, Ontario: Thompson Canada. Patton, M. Q. (2002). Qualitative evaluation and research methods. Thousand Oaks, CA: SAGE. Poole, D., & Davis, T. (2006). Concept mapping to measure outcomes in a study abroad program. Social Work Education, 25(1), 61-77. Pressley, M., VanEtten, S., Yokoi, L., Freebern, G., & VanMeter, P. (1998). The metacognition of college studentship: A grounded theory approach. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in theory and practice (pp. 347-367). Mahwah, NJ: Erlbaum. Raymond, A. M. (1997). The use of concept mapping in qualitative research: A multiple case study in mathematics education. Focus on Learning Problems in Mathematics, 19(3), 1-28.

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

Wheeldon

101

Rico, G. L. (1983). Writing the natural way: Using right-brain techniques to release your expressive powers. New York: Penguin Putnam. Ritchie, J., & Lewis, J. (Eds.). (2003). Qualitative research practice: A guide for social science students and researchers. Thousand Oaks, CA: SAGE. Rogers, A. E., Day, J., Randall, F., & Bentall, R. P. (2003). A patients understanding and participation in a trial designed to improve the management of anti-psychotic medication. Social Psychiatry and Psychiatric Epidemiology, 38, 720-727. Rohm, R. (1994). Positive personality profiles. Atlanta, GA: Personality Insights. Rooda, L. (1994) Effects of mind mapping on student achievement in a nursing research course. Nurse Educator, 19(6), 25-27. Rorty, R. (1999). Philosophy and social hope. London: Penguin. Ruiz-Primo, M., Shavelson, R. J., Li, M., & Schultz, S. E. (2001). On the validity of cognitive interpretations of scores from alternative concept-mapping techniques. Educational Assessment, 7, 99-141. Rye, J. A., & Rubba, P. A. (2002). Scoring concept maps: An expert map-based scheme weighted for relationships. School Science & Mathematics, 102(1), 33-45. Sandelowski, M. (2001). Real qualitative researchers do not count: The use of numbers in qualitative research. Research in Nursing & Health, 24, 230-240. Safayeni, F., Derbentseva, N., & Canas, A. J. (2005). A Theoretical Note on Concept Maps and the Need for Cyclic Concept Maps. Journal of Research in Science Teaching, 42(7), 741-766. Schurz, G. (2002). Models of abductive reasoning (TPD Preprints Annual 2002 No. 1). In G. Schurz & M. Werning (Eds.), Philosophical prepublication series of the Chair of Theoretical Philosophy. Dusseldorf, Germany: University of Dusseldorf. Shaw, M., & Dandurand, Y. (Eds.). (2006). Effective technical assistance in crime prevention and criminal justice. Maximizing the effectiveness of the technical assistance provided in the fields of crime prevention and criminal justice (HEUNI Report Series No. 49). Helsinki, Finland: HEUNI. Stewart, J., Van Kirk, J., & Rowell, R. (1979). Concept maps: A tool for use in biology teaching. The American Biology Teacher, 41, 171-175. Stillwell, W., Winterfeldt, D. V., & John, R. S. (1987). Comparing hierarchical and nonhierarchical weighting methods for eliciting multiattribute value models. Management Science, 33, 442-450. Tashakkori, A., & Creswell, J. W. (2008). Mixed methodology across disciplines. Journal of Mixed Methods Research, 2(1), 3-6. Tashakkori, A., & Teddlie, C. (1998). Mixed methodology combining quantitative and qualitative approaches. Thousand Oaks, CA: SAGE. Tattersall, C., Watts, A., & Vernon, S. (2007). Mind mapping as a tool in qualitative research. Nursing Times, 103, 32-33. Tomiyamal, T., Takeda, H., Yoshioka, M., & Shimomura, Y. (2003). Abduction for creative design (AAAI Tech. Rep. No. SS-03-02). Menlo Park, Ca: AAAI Press. Trochim, W. (1989). Concept mapping: Soft science or hard art? Evaluation and Program Planning, 12, 87-110. Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley. Turns, J., Atman, C., & Adams, R. (2000). Concept maps for engineering education: A cognitively motivated tool supporting varied assessment functions. IEEE Transactions on Education, 43, 164-173. Way, N., Stauber, H. Y., Nakkula, M. J., & London, P. (1994). Depression and substance use in two divergent high school cultures: A quantitative and qualitative analysis. Journal of Youth and Adolescence, 23, 331-357. Wheeldon, J. (2007). Bringing British Columbia to Latvia: Canadians reflect on expert designation in an international criminal justice project. Burnaby, British Columbia, Canada: School of Criminology, Simon Fraser University. Wheeldon, J. (2008, November). Mapping ethics, values and problem solving among criminal justice students. Paper presented at the American Society of Criminology Annual Meeting, St. Louis, MO. Wheeldon, J. (2009). Mapping international knowledge transfer: Latvian-Canadian cooperation in justice reform. Unpublished doctoral dissertation, Simon Fraser University, Burnaby, British Columbia, Canada.

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010

102

Journal of Mixed Methods Research 4(2)

Wheeldon, J., & Faubert, J. (2009). Framing experience: Concept maps, mind maps, and data collection in qualitative research. International Journal of Qualitative Methods, 8(3), 68-83. Whorf, B. L. (1956). Language, thought and reality: Selected writings of Benjamin Lee Whorf (J. B. Carroll, Ed.). Cambridge: MIT Press.

Downloaded from mmr.sagepub.com at UNIV AUT CIUDAD JUAREZ on August 3, 2010