Influence of Written Explanations on Multiple Choice Answer Selection in Energy Balances

This study investigates the development of conceptual understanding by having students engage in active learning, answer conceptual questions, and write explanations to communicate their reasoning.  A lack of conceptual understanding can severely hinder a student’s ability to solve novel problems both in the classroom and in industry.  Engineering courses are still largely lecture-based, which promotes rote memorization over conceptual understanding. However, the instructional design of these courses is beginning to change to include more active learning (e.g. peer instruction). Conceptual questions are an integral part of many active learning pedagogies and can help target and improve conceptual understanding.  It follows that with better understanding students should also have better scores.  High quality conceptual questions require little to no calculation to obtain an answer so that students cannot rely solely on memorizing equations.  Using these high quality conceptual questions extends assessment beyond “What does a student remember?” to “What does a student understand?”

The constructivist model of learning can help explain why students benefit from the use of active learning pedagogies.  Constructivism sees knowledge as actively “constructed” by the student rather than transmitted for the student to passively absorb.  From this perspective, students enter the classroom with prior knowledge and preconceptions which they use to make sense of new experiences.  Effective teaching, therefore, must explicitly confront a student’s pre-existing knowledge structures and provide students activities that lead to their modification and reorganization.  In this study, we specifically posit that formulating a written explanation to a conceptual question provides students greater opportunity to engage with and reorganize their knowledge structures. Based on this perspective, we hypothesize that requesting written explanations of student answer choices to conceptual questions will increase the frequency that they choose the correct response.

To determine the effect of requesting written explanations on student performance, data was collected from two cohorts comprised of a total of 302 students. Students in each cohort were enrolled in a required undergraduate sophomore-level energy balances course and attended the same lecture, but attended different bi-weekly recitations.  The instructor and room remained consistent across both cohorts.  Concept questions were posed to students during recitation through the AIChE Concept Warehouse, a tool that affords answer submissions via laptops or smartphones.  Each recitation section alternated between answering the multiple choice concept questions along with a written explanations (treatment condition) and simply answering the questions with no written explanation (comparison condition).  Statistical significance of student answer correctness was determined using non-parametric sign tests and chi-squared tests.  Students’ written explanations were also categorized by correctness using an emergent, iterative coding process.  Codes ascend from 1 (incomplete reasoning) to 4 (complete and correct reasoning).  Cohen’s Kappa was used to compare inter-rater reliability after coding.  Reliability ranged from 0.87 to 0.89, indicating reasonable agreement, for the two questions  highlighted in detail.

Data indicated that writing explanations significantly changed the distribution of student answers about half of the time. Many questions showed a positive treatment effect (where students who wrote explanations outperformed students who did not).  The positive effect aligns with Chi et al.’s assertion (1994) that the active process of explaining encourages students to integrate new knowledge, leading to better conceptual understanding.  However, some questions had a negative treatment effect, allowing students to attain the correct answer by guessing or using less developed (and faulty) reasoning.

To better illustrate the differences between questions with a positive and negative treatment effect, we analyzed an isomorphic question pair.  Isomorphic questions are those on which students need to apply the same concept to attain the correct answer, but the questions have different surface features, or “cover stories.” Both isomorphs had a similar distribution of performance associated with understanding coded as 1 and 2 (i.e., they were likely to answer correctly or incorrectly with faulty reasoning).  For better reasoning (i.e., codes 3 and 4) the performance on the isomorphs became quite distinct.  Code 3 explanations always led to an incorrect answer, regardless of question. Code 4 explanations almost always lead to a correct answer. The isomorph where there was a positive treatment effect had a higher percentage of students providing a code 4 explanation (showing complete and correct reasoning) to the exclusion of code 3.  For the isomorph where there was a negative treatment effect, a greater percentage of students provided a code 3 explanation instead of code 4.  We conjecture that there is a shift from faulty reasoning (codes 1 and 2) to more correct reasoning (codes 3 and 4) prompted/facilitated by the written explanation, even though scores are the only indicator we have for the no written explanation condition. The shift to correct and fully developed reasoning in one case and correct but inadequately developed reasoning in the other explains both treatment effects related to performance. We believe question surface features that provide context for the question are the factor responsible for the relative magnitude of the shift in understanding.

Regardless of the possibility of a negative effect, students should write written explanations when answering conceptual questions.  Written explanations allow the instructor to examine student reasoning and identify questions where students can apply faulty reasoning to attain the correct answer (through use of a tool like the AIChE Concept Warehouse).  This study also illustrates the necessity for high-quality conceptual questions.  Many attributes of “good” conceptual questions have been outlined in literature: these questions should target a specific learning goal, uncover student misconceptions, and elicit a range of responses.  Because both of the isomorphs we studied possess these characteristics, an important attribute to add would be that the correct answer would have a low probability of being obtained by faulty reasoning.