Research Methods - Mar 2013

Can the concept of reliability, validity and generalizability have any utility within a research context using semi-structured interviews? Draw on one published research paper for how any or all of these concepts are dealt with within the qualitative tradition
The concepts of reliability, validity and generalizability have their origins in the quantitative tradition. AERA, APA, NCME (1999) define the reliability of a test as the consistency of measurements when the testing procedure is repeated. Validity refers to the appropriateness, meaningfulness, and usefulness of the specific inferences made from test scores (AERA, APA, NCME 1999). Generalizability in turn is related to a specific form of validity called external validity that addresses the ability to generalize an inference derived from a specific sample to a wider general population. Embedded within these definitions is the concept of repeatability of measurements o r tests and the existence of an objective true value against which the tests are compared against. However it is a challenge to apply these concepts in the context of qualitative studies especially those that use semi-structured interviews as the measurement device is the interviewer him/herself. Furthermore, the objective in qualitative studies can sometimes be more focused on the development of an explanatory framework rather than the measurement of the causal relationships between variables. Therefore the very concepts of reliability and validity need to re-framed in qualitative research. Davies & Dodd (2002) (As quoted in Golafshani 2003) argue that the notion of research rigour often appears in reference to discussions related to reliability and validity. Curry et al (2004) suggests that rigour is associated with the systematic collection , organisation, and interpretation of data in accordance with rigourous and widely accepted techniques for research strategy , sampling, data collection and analysis. An example of how these concepts are practically applied can be observed in research conducted by Parker & Mobey (2004). This research paper focuses on the identification of the factors related to perception of risk in project management. The authors utilised of a combination of focus group workshops and semi-structured interviews with key members of a major UK company in relation to a planned roll out of a electronic document management system within the organization. From a research strategy and sampling perspective, the authors engaged the use of established phenomenological methods which seek to generate a deeper understanding of the meaning of a particular phenomenon from the individuals own perspectives through the collection of extensive narrative data (Patton 2002 as quoted in Curry et al 2009) This fits with research strategy and methodology described (Parker & Mobey 2004 p.19-21) where the initial framework developed from a literature review was refined through the use of two focus group sessions followed by a seventeen semi-structured interviews of participants who were drawn from the focus group sessions. The interview

questions themselves were initially structured based on the research propositions but later modified to include experiential information from the focus group sessions. From the data collection perspective, the researchers utilized a codification system on all interview transcripts to develop a possible explanation to the research propositions where open coding was used to ascertain underlying concepts and themes (Parker & Mobey 2004 p.24). This aligns with the concept of the constant comparative method that requires data to be reviewed line by line in detail and where a concept is apparent, a code is assigned to catalogue the concept whilst preserving the context in which it occurred (Strauss & Corbin 1998 as quoted in Curry et al 2009) From a data analysis perspective, even though the information collected in the semi-structured interviews was largely qualitative, a section of the interview required interviewees to apply a Likert scale ranking of perceived factors of project failure derived from the focus group sessions. This mixed-method approach allowed for quantifiable comparisons between largely emotional and attitudinal input and observation of the emergent themes from the research to be linked to the literature hence, in the authors own words, increasing the validity of the research framework (Parker & Mobey p. 24) As for the generalisability of the results, the authors do not make any explicit claims of applicability of the findings to all other IT projects but instead focus on the fact that the framework developed can be used to support a better understanding of risk assessment and project management within the academic community and practitioners .Therefore in that sense , the work of Parker & Mobey (2004) can be best categorized as being thematic which is defined as recurrent unifying concepts about a subject of inquiry that can be used to generate hypotheses for subsequent investigation (Curry et al 2004 quoting Boyatzis, 1998). Therefore even within the research context of semi-structured interviews the concepts of reliability , validity and generalisability do have utility where a demonstration of systematic research that is based on established techniques contributes towards enhancing the credibility of the research outcomes. [773 vs Target 500 Words] References : American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME)(1999). Standards for educational and psychological testing. Washington, DC: AERA Boyatzis R. Transforming Qualitative Information: Thematic and Code Development. Thousand Oaks, Calif: Sage; 1998.

Curry, Leslie A., Ingrid M. Nembhard, and Elizabeth H. Bradley. "Qualitative and mixed methods provide unique contributions to outcomes research." Circulation 119.10 (2009): 1442-1452 http://circ.ahajournals.org/content/119/10/1442.full.pdf+html

David Parker, Alison Mobey, (2004) "Action research to explore perceptions of risk in project management", International Journal of Productivity and Performance Management, Vol. 53 Iss: 1, pp.18 32 Davies, D., & Dodd, J. (2002). Qualitative research and the question of rigor. Qualitative Health research, 12(2), 279-289 Gubashani, N. (2003) Understanding reliability and validity in qualitative research. The Qualitative Report, 8(4), 597-606. Retrieved [ 14 Mar 2013] from http://nova.edu/ssss/QR/QR8-4/golafshani.pdf Patton MQ. Qualitative Research and Evaluation Methods. 3rd ed. Thousand Oaks, Calif: Sage; 2002.

Strauss A, Corbin J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Newbury Park, Calif: Sage; 1998

Bibiliography Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard Educational Review, 62(3), 279300 Winter, G. (2000, March). A comparative discussion of the notion of 'validity' in qualitative and quantitative research. [58 paragraphs]. The Qualitative Report [On-line serial], 4(3/4). Available: http://www.nova.edu/ssss/QR/QR4-3/winter.html