Integrated Engineering Physics Laboratory Assessment Amanda Skuriat, Dr. Daniela Buna, Dr.

Caroline Brisson, Engineering Physics School of Theoretical and Applied Science, Ramapo College of New Jersey, Mahwah, NJ, 07430 Abstract The goal of this research project is to develop a set of Physics Education Research tools to help assess the student learning outcomes in an Engineering Physics laboratory program. The data collected highlights the significant progress of skills and knowledge made through the introduction of innovative laboratory equipment and teaching/testing methods. However, evaluative feedback provided by the students on their work was overly optimistic in comparison to the feedback of their professors. Several reasons for this disconnect have been investigated and the findings are reported. It is imperative to further analyze what aspects can be improved upon in the curriculum as well as how to improve students self-evaluation skills. In addition, it is essential to investigate ways to propagate increased interest in undergraduate research. Methods In order to conduct a study based on student progress, we created formative and summative questionnaires to collect data from both students and faculty. In addition, we used the Ramapo College course assessment forms, which are used for program assessment college-wide. The students self evaluated their skills before and after taking the laboratory while the faculty evaluated their performance at the end of the course. Currently, the engineering physics major offers the following sequence of laboratories: Physics with Calculus (100-level) Modern Physics (200-level)

Electronics I and II (200 level) Experimental Research Methods (ERM, 400-level and capstone of major)

Evaluation plan: a. Evaluate the integration of the curriculum b. Evaluate student experimental skills before and after the laboratory is completed (student selfevaluations and faculty evaluations) c. d. Evaluate student interest in independent research projects Evaluate the quality and number of laboratory set-ups/experimental procedures

The goal of an integrated curriculum is to provide growing experimental skills across a sequence of laboratories. The skills selected through the goals and outcomes of the syllabus of each laboratory have been aligned as shown in Table 1 and evaluated through a Before and After student questionnaire as well as faculty post-laboratory evaluations. The results are summarized in Table 2 (students) and Table 3 (students versus faculty for the ERM course only). Data Analysis: The student data contained in Table 2 demonstrates a general trend of overall improvement. Additionally, faculty assessment forms reflected this trend. In the three laboratories that followed Physics with Calculus, students became more aware of what was expected of them through analysis of course requirements, skills to be mastered, and related writing samples. After completion of Modern Physics, all areas were rated at or above the 70% target value in every area except technical writing skills, aptitude to clearly describe laboratory procedures, and level of understanding of theoretical concepts. In the Electronics course, all students rated themselves above the target value. In the

Experimental Research Methods course, students were given a pre-laboratory manual and questionnaire that tested the extent of their grasp on theoretical information and calculations involved in the experiment. After each experiment, students filled out a formative evaluation summary in addition to completing their laboratory reports. Table 3 shows the student/faculty comparison evaluation of skills. Results were similar between both parties for the level of understanding of theoretical concepts, aptitude to clearly describe laboratory procedures and technical writing skills in the Before areas. In the After assessment for students, eight of the ten categories were rated at 100% at or above the B level (the other two categories were rated at 90%). Although the 70% target was reached for nearly all of the categories, there is a distinct discrepancy between the After student selfevaluations in Table 2 and the instructor evaluation. As demonstrated in Table 3, a small increased improvement in skills has been reached, but the discrepancies between student and faculty evaluations range from 18 to 33%. The largest difference is in the category of technical writing skills. In the summative evaluations for the ERM course, the student results were highly positive as well (in fact, they could be called overly optimistic). A particularly interesting finding was that only 45% of the students were interested in pursuing research projects on average throughout the course of the formative evaluations, but this number increased to 100% in the summative survey.

Table 1. Summary of skills to be acquired during a sequence of four laboratory offerings.

Laboratory/Skills
Perform measurements and analyze the precision, accuracy and error of the measurements Independently connect a computerized system, set up the interface, select and calibrate the sensors, acquire and analyze data Apply knowledge of classical physics principles to simple conservation laws measurements Learn how to keep a laboratory notebook and develop technical writing skills Distinguish between experiments that reveal the particle vs. the wave nature of light Apply classical and modern physics concepts to measure fundamental constants in physics Introduce linear and non-linear detectors and understand the concepts of calibration and detector response. Learn to use advanced measuring devices Develop problem-solving skills

Phys w/Calc

Electronics I and II

Modern Phys

ERM

Develop a practical understanding of the principles of electronics and interfacing with electronic instrumentation Independent theoretical preparation for laboratory measurements; Test the acquired knowledge by completing a pre-lab questionnaire Develop a formal laboratory report that mimics the format and demands of a formal research paper in physics Design a small research project to be completed in the following semester as an independent study; present a scientific poster summarizing the work

Table 2. The percentage of students that marked a Good (B) or an excellent (A) grade.
Skill to be evaluated PHYS-115 S-11 Before 80 63 82 82 82 45 72 After 100 91 100 91 100 91 100 Modern Physics F-11 Before 67 78 67 87 100 55 55 After 77 100 89 100 88 100 100 Electronics S-12 Before 86 87 90 93 93 80 80 After 90 90 90 100 90 94 90 Experimental Research Methods F-12 Before After 90 100 80 90 80 70 90 60 100 100 100 90 100 100

Structure / organization of the report Correctness of the retrieved data Analysis of the collected values Interpretation of graphical information Error analysis Statistical analysis of the data Level of understanding of the theoretical concepts Aptitude to clearly describe laboratory procedures Aptitude to analyze outcome of experiments Technical writing skills

91

100

100

100

87

90

60

100

90 100

100 100

44 55

55 55

90 93

90 90

90 50

100 90

Table 3. Faculty assessment versus student self-assessment. The percentage of students that were marked a Good (B) or an excellent (A) grade on their lab report and activity in class.
Skills to be evaluated Experimental Research Methods F-12 Structure/organization of the report Correctness of the retrieved data Analysis of the collected values Interpretation of graphical information Error Analysis Statistical analysis of the data Level of understanding of the theoretical concepts Aptitude to clearly describe laboratory procedures Aptitude to analyze outcome of experiments Technical writing skills Student Self-Assessment Before 90 80 90 80 70 90 60 60 90 50 After 100 100 100 100 90 100 100 100 100 90 Faculty Assessment of student skills Before After 70 80 70 80 50 70 70 80 50 70 60 70 50 70 60 60 50 80 70 60

Findings: A specific goal based on the findings was to research and develop solutions across the three areas that needed the most improvement (the level of understanding of theoretical concepts, aptitude to clearly describe laboratory procedures and technical writing skills). The instructors provided the students with sample laboratory reports as examples of poorly written or edited work as well as exemplary work. However, many of the students seemed to be unable to perceive the difference in quality. Research shows that this is an overwhelming issue in our educational system: students appear to have inadequate reading and writing skills across the board. In order to correct these disparaging trends, its essential that undergraduate learning is made a priority in institutions. Undergraduate courses should be given a more challenging structure and emphasize written/communication skills, in addition to more rigorous instruction at the pre-college level (Arum). Additionally, purposeful review of previous work according to a highly detailed rubric and the restructuring of teaching methods may prove to be helpful. Perhaps revising our curriculum to include more reading and writing intensive aspects can improve learning, and implementing more evaluative standards for written work will allow for students to realize their errors more readily. The very high rankings the students provided for their own performance may be explained by several factors. Some of these include: incompetency, lack of metacognitive ability, the above-average effect, a dearth of critical feedback given by professors, and the lack of self-evaluative ability among the students. A mix of these factors in varying degrees can most likely explain the results obtained. According to Kruger and Dunning, incompetent people suffer a dual burden: Not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it (Kruger, 1121). In this study, they predict that these students will be extremely overconfident in judging their skills and performance regarding evaluative outcomes in relation to their

more knowledgeable classmates. They will also exhibit metacognitive deficiency which severely impacts their ability to realize that they are lacking competence. In effect, besides being unable to recognize their incompetency, their judgment of the competency of others is likewise impaired and therefore cant be taken into account when assessing their own skillsets (Kruger, 1122). Interestingly enough, it has been proven that incompetent students can learn how to recognize this flaw; however, this comes (paradoxically) by making them more competent, thus providing them the metacognitive skills necessary to be able to realize that they have performed poorly (Kruger, 1122). Additionally, the above-average effect often complicates evaluation results: in test results, the students with the lowest ability tended to feel that they were above average and greatly overestimated their abilities, and conversely, the highest achieving students ranked their abilities much lower in respect to their classmates (Kruger, 1123-4). In high achieving groups of students, there is a tendency to automatically assume that they must be the best, regardless of other factors and the respective abilities of other students (Austin). Competent students will often assume that everyone else performed comparably, whereas if they were shown how their performances ranked they would be able to judge more accurately (Kruger,1126). A major flaw in todays educational system is the lack of accurate criticism or feedback students receive from their instructors. Teachers hesitate to give negative feedback for various reasons, and as a result, students dont understand the reasons behind their poor performance. Therefore, it is imperative that teachers provide both positive and negative feedback, along with illustrative examples for each category so that students can improve their skills rather than just looking at assessment as a mere grade. The degree of interest in the topic being evaluated and the willingness to complete the evaluation are also key factors to take into consideration when analyzing results. For example, the study of selfassessment is frequently confounded by the ability of participants to obfuscate their true opinions, or to simply not engage fully in the task and lazily report to investigators due to a perception that the task of

self-assessment is itself not worthwhile (Austin). A factor that is often overlooked is that students simply may not understand how to self-assess their skills or employ its results for improvement in the future. It is taken for granted as an inherent skill, and many students dont understand how to analyze their performance correctly (Austin). Conclusions The results show that the improvement of the curriculum of upper level classes, the new technology, setups and laboratory manuals that were introduced in the laboratories all served to increase student learning outcomes. In addition, it was discovered that in order to improve the laboratory experience, more setups should be purchased to provide students with essential hands-on experience. The incorporation of some of the suggested tactics in alignment with research may also lead to further improvement in the future.
References: 1. Arum, Richard, Josipa Roksa and Esther Cho. Improving Undergraduate Learning: Policy Recommendations from the SSRC-CLA Longitudinal Project. Social Science Research Council. Web. 13 Mar 2013. http://highered.ssrc.org/files/SSRC_Report.pdf Kruger, Justin, and David Dunning. "Unskilled And Unaware Of It: How Difficulties In Recognizing One's Own Incompetence Lead To Inflated Self-Assessments." Journal Of Personality And Social Psychology 77.6 (1999): 11211134. PsycARTICLES. Web. 10 Feb. 2013. Austin, Zubin, and Paul A. Gregory. "Evaluating the Accuracy of Pharmacy Students' Self-Assessment Skills." American Journal of Pharmaceutical Education 89th ser. 71.5 (2007): 1-8. National Center for Biotechnology Information. American Association of Colleges of Pharmacy, 15 Oct. 2007. Web. 10 Feb. 2013. <http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2064887/>.

2.

3.