The Cognition of Counterexample in Mathematics Students
aa r X i v : . [ m a t h . HO ] F e b THE COGNITION OF COUNTEREXAMPLE IN MATHEMATICSSTUDENTS
SHANNON EZZAT AND SCOTT RODNEY
Abstract.
Studying Mathematics requires a synthesis of skills from a multitude ofacademic disciplines; logical reasoning being chief among them. This paper exploresmathematical logical preparedness of students entering first year university mathemat-ics courses and also the effectiveness of using logical facility to predict successful courseoutcomes. We analyze data collected from students enrolled at the University of Win-nipeg in a pre-service course for high school teachers. We do find that, being able tosuccessfully answer logical questions, both before and after intervention, are significantin relation to improved student outcomes.
Keywords.
Counterexample, mathematical thinking, logical inference, student success Introduction
This paper explores mathematical logic preparedness and its effect on success in thefirst year university mathematics classroom. It is well known that skill sets amongstudents entering courses in university mathematics vary widely from student to studentand thus an understanding of the abilities and effects of logical aptitude is a worthwhileendeavour to help improve student success in mathematics.This work focuses on two themes: student facility in reasoning about counterexample,and how well assessment of this logical skill can predict successful course outcomes inintroductory mathematics courses.To address these questions, a 3-month study of university students was initiated inMATH-2903, Math for Early/Middle Year Teachers during the winter semester of the2017-18 academic year. As indicated by its name, this class is designed for pre-serviceelementary- and middle-school teachers. The prerequisite for the course is any university-level math class from Grade 12 in Manitoba (with a grade above 65% for applied math orany passing grade in precalculus) and does not count towards any math major course, oras a required math course for Bachelor of Science students. The students in this coursepredominantly have poor to adequate arithmetic and algebraic skills and a substantiallynegative attitude towards mathematics as a subject.Our methods are inspired by several works on student ability with logic and counterex-ample including [BF] which discusses undergraduate students understanding of variousconcepts in calculus using the tool of diagnostic tests, intervention/primers, and quizzes;[L], which studies the cognition of counterexample of high school students in a similarfashion; and [IS], which studies undergraduate (and staff) success in the Wason SelectionTask. We use these tools to measure students understanding of logical statements, andspecifically counterexamples to these statements.A careful description of our study is found in Section 2. Section 3 contains statisticalanalysis discussing evidence of predictive assessment question types. A summary of ourresults together with concluding remarks is found in Section 4. Note that in a futurenote the authors will report results for a similar study from MATH-1105, Differentialand Integral Calculus I .Reasoning logically using counterexample is linked to the idea of conditional state-ments. We begin with a discussion about conditional statements and the notion of Date : University of Winnipeg (UW), Manitoba, Canada Cape Breton University (CBU), Nova Scotia, Canada
SHANNON EZZAT AND SCOTT RODNEY counterexample itself and its importance in mathematical pedagogy, including a partic-ular instance of a question type designed by Peter Wason [W] called the Wason SelectionTask. We refer to this as the Wason question in our tests.1.1.
Conditional Statements.
A conditional statement consists of two parts, an an-tecedent (if-part) and a consequent (then-part). In general, these are of the form “If Pthen Q” , for some statements P and Q. An example would be the following:
If an integer is even then the rightmost digit of the number is either a , , or “If P then Q” becomes “All P are Q” and specific to the example above: All even integers have the rightmost digit being , , or counterexample to the conditional statement.In fact, a conditional statement is false exactly when such an object exists. For moredetails, the reader is encouraged to refer to the introductory chapters of most textbooksin logic or discrete mathematics. It is important to see that the idea of counterexampleis not necessarily intuitive. We note one more example from our diagnostic study toillustrate this point:If two shapes have the same perimeter then they have the same area.On the diagnostic test, 31 of 62 responded correctly indicating that the statement is falsewhile only 8 of those justified their conclusion using a counterexample. Counterexamplesprovided by correct respondents most often used rectangles with few referring to circlesor triangles.1.2. The Importance of Counterexample.
In mathematical reasoning, the notionof counterexample is fundamental. Counterexamples allow us to understand the limitsof mathematical results and why each condition in a mathematical logical statementis needed to justify the conclusion. In [K], Klymchuck writes that, for undergraduatestudents, understanding by way of counterexamples “can improve students’ conceptualunderstanding in mathematics, reduce their common misconceptions, provide a broaderview on the subject and enhance students’ critical thinking skills”. For students that planon a career in education Vinsonhaler and Lynch [VL] believe that primary and secondaryschool teachers “ support student reasoning and thinking and promote productive strug-gle by incorporating counterexamples into the classroom.” Zazkis and Chernoff [ZC]suggest that counterexamples can be particularly convincing when they are “in accordwith individuals’ example spaces”.1.3.
The Wason Question.
The card selection question was originally reported byWason [W]. In his experiment, there are cards on a table and the participant is told thatthere is a number on one side and a letter on the other. The participants are told of arule, for example “If a card has a D on one side then it has a 3 on the other side” or,reworded as a universally quantified statement “All cards with a D on one side have a3 on the other side” . Four cards are presented that correspond to each logical case (Ptrue, P false, Q true, Q false) and the participant is asked “to select all those cards, butonly those cards, which you would have to turn over in order to discover whether or notthe rule has been violated.”
To answer the question, the participant is required to assess the conditional statementfour times in relation to each card presented. The correct answer is to select the cardswhere either P is true or Q is false since these conditions are necessary (but not sufficient)for the card to be a counterexample or exception to the rule. Interestingly, most of the
HE COGNITION OF COUNTEREXAMPLE IN MATHEMATICS STUDENTS population does not make this connection. Indeed, Inglis and Simpson in [IS] found froman online, self-selected sample at the University of Warwick, only 29 percent of mathstudents, 8 percent of history students and 49 percent of mathematics staff respondedcorrectly. During our study described below, only 2 students answered this style ofquestion correctly on the pre-test; see § Proving a conditional true.
More difficult perhaps is the skill of reasoning thejustification of a true conditional. Consider the following sentence:If the sum of two positive whole numbers is odd then their product iseven.To justify the truth of this statement the participant must covey that, since the sum ofthe numbers is odd, one number must be even while the other odd. This requires morealgebraic ability, and usually a considerable amount of lecturing to convey; indeed, manyuniversities have a course in the first or second year of the program (usually DiscreteMathematics or a standalone Introduction to Proofs course) that spend considerable timeteaching the concept of correct proof writing. We note that the diagnostic test did notinclude any questions of this type, however as part of the course material, some lecturetime was spent teaching how to prove a conditional statement for simple examples; e.g.“If a, b, c are three consecutive whole numbers then their sum has a factor of 3”.As we will discuss in more detail, the ability to understand and construct counterex-amples to a conditional statement is not inherent in most students entering Math 2903.These results are not surprising in the context of similar statistical results found in [IS]and [L]. However, perhaps more interesting is that on average, students who are ableto respond well to these problems have better course outcomes. We also observe thatstudents who learn these skills through in-course intervention experience better courseoutcomes on average. As such, our study suggests that logical skill assessment can beused as a tool to help improve course outcomes for students. Such assessment is alsoeffective in identifying struggling students early on so that they can find effective help.Lastly, it is also quite interesting that a simple intervention can on average have a mea-surable impact on student success. 2.
The Study
The data used for this study comes from Math 2903, Math for Early/Middle YearTeachers at the University of Winnipeg. We chose students in Math 2903 since it was,most likely, the students’ first university mathematics class. Indeed, the likelihood thatour student populations in the class had been exposed to logical reasoning on an academiclevel (for example, a course on critical thinking) was quite low. This is borne out in theresults.The study was conducted in four phases. In the first week of classes (Phase I), stu-dents were given a diagnostic test to benchmark their understanding of logical thinking.In the third week (Phase II), students were given an intervention (75-minute lab/lectureon the idea of conditional statements and counterexamples) complete with a worksheetfor practice. During the 6th week of classes (Phase III), a post test was given as part ofthe in-class midterm exam. Final grades were tallied (Phase IV) and small sample sizestatistical methods were used to analyze aggregate data collected. In what follows, wegive detailed descriptions of each phase of the study but leave Phase IV to § I - The Diagnostic Test.
The Diagnostic test was designed to gauge incomingstudent ability to reason by counterexample and was given during the first week of class.The problems appearing on the test were selected from 2 categories similar to those foundin [L]. We describe them now. We also note that specific questions can be found in § SHANNON EZZAT AND SCOTT RODNEY
Type I: The Wason selection task.
As described in the introduction, this exerciseconcerns rule verification for a finite set of objects. A complete description of the Wasonselection task can be found in both of [L] and [IS]. For this study, we depicted an arrayof 6 cards each with a number on one side and a letter on the other, see (1). (1)The student was told: “Each card has a number and letter on opposite sides. Everycard with a vowel on one side has an even number on the other. Of the six cards below(it can be more than one) which must you flip over in order to determine whether ornot the rule has been violated? Try your best to explain your reasoning.” Note thatthe correct answer to this problem is to flip the A, 9, and 3, as these could be possiblecounterexamples; for example, the A could have a 3 on the other side, and the 3 or 9could have an A on the other side, and these cards would violate the rule).2.1.2.
Type II: Provide a counterexample to a given conditional statement.
Here the stu-dent is asked to falsify a conditional statement by constructing a counterexample. Forexample,“If two shapes have the same perimeter then they have the same area”is seen to be false by considering two rectangles with dimensions 2 × ×
1. Thisforms the required counterexample.2.2.
II - The Intervention.
The intervention consisted of a 75 minute primer on con-ditional statements that occurred during the second week of classes. During the inter-vention, students were introduced to the concept of a conditional statement, initially viaa truth table, and the relationship between universally quantified statements and con-ditionals. Students were then shown an applied example of a liquor inspector enforcingthe rule described by the conditional statement “If you are drinking alcohol then youmust be 18 years of age or older” to reinforce the idea of when a conditional statementis false. This was then connected to the notion of counterexample. The interventionlecture closed with a discussion of the Wason selection task where students were shownthe relationship between the truth values of the conditional sentence defining the ruleand the values on each of the cards. This was followed by a group work session wherestudents reinforced these ideas by completing a practice worksheet with their peers.In addition to this 75 minute primer, as part of the course work students were givenone lesson on verifying a conditional statement via an algebraic proof. The studentswere also given a small number of beginner practice problems, and one problem on anassignment to complete. Note that this type of question was not included on the originaldiagnostic test as algebraic verification is a technique that is more specialized and theinstructor of the course (the first author) believed that most students would not havebeen able to answer the question appropriately.2.3.
III - The Post Test.
The post test was given as part of the MATH-2903 classmidterm test. The students were given one Wason Selection Task question, as well as onequestion where the students would have to identify which of two conditional statementshad a counterexample, provide a counterexample for that statement, and algebraicallyjustify the other conditional statement.
HE COGNITION OF COUNTEREXAMPLE IN MATHEMATICS STUDENTS Data and Results
The aim of the data analysis found in this section is to compile and compare resultsof the diagnostic test (DT) and post test (PT). Our results, presented in tables 1-6,analyze relationships between success/retention/difficulty with specific problems on ourtests with mean final course grades. In order to present this efficiently, we group andname specific problems of interest together with our raw data. Our data consists ofevaluated student performance in terms of the following categories.(i) CR : Correct answer and reasoning.
This evaluation is given if the student providesa clear, complete solution to the problem providing a counterexample if necessary.(ii) C : Correct answer with incorrect reasoning.
The student was able to answer theproblem correctly but lacked sufficient justification such as, for example, an erro-neous or missing counterexample.(iii) I : Incorrect answer.
The student has answered incorrectly with or without justifi-cation.(iv) B : The student left the problem blank.
The Diagnostic Test.
The diagnostic test consisted of 5 questions of varying typethat we state now followed by student response data. • DT:1
If a number n is less than 1, then the expression n − • DT:2
If two shapes have the same perimeter, then they have the same area. • DT:3
If the sum of two numbers is odd, then their product is even. • DT:4
If two numbers x, y have positive sum, then both x and y are positive. • DT:5
This is the Wason selection task problem exactly as given in section § Table 1.
Summary of student answers to counterexample questions ondiagnostic testFor the Wason selection task problem DT:5, we analyze student responses more finelyusing common observed card selections. Category “A93” refers to the student selectingthe correct cards, the ace, nine, and three. The other categories, “A”, “A4”, “All”,“Blank” and “Other” are self evident.Answers A93 A A4 All Blank OtherQuestion 2 3 2 26 12 10 9
Table 2.
Summary of student answers to Wason Selection Task questionon diagnostic test
Remark 3.1.
We note that very few students correctly answered the question (A93) onthe diagnostic test. Furthermore the most common answer was to select the A and four,(A4). This choice is the common fallacy that the conditional “if P then Q ” is equivalent SHANNON EZZAT AND SCOTT RODNEY to “ P and Q ”. This result aligns with previous work by Inglis and Simpson [IS] , andWason [W] . The Post Test:
The post test consisted of three questions PT:1, PT:2, PT:3 onthe first midterm of Math 2903. Question PT:1 was the Wason selection task problemas in § One of the following conditional statements is always true and the other has a coun-terexample. ***Circle (a) or (b)*** of the the true statement and justify that it is true(using the direct method). Find a counterexample for the other statement (and explainwhy it is a counterexample). • PT:2 (a) If m and n are whole numbers and m × n is even, then both m and n areeven. • PT:3 (b) If n is an even number then n is a multiple of . The results of the post-test questions are shown in the table below. Students aregrouped into two categories: correct, and otherwise.Question Correct Answer Correct IncorrectPT:1 Selected (93KQ) 21 38PT:2 Provided a counterexample 41 18PT:3 Provided a correct proof 12 47
Table 3.
Number of students answering post-test questions correctly
Remark 3.2.
In contrast with Remark 3.1 and noting that this specific question type wascovered in the stage II intervention, it is very encouraging that a far larger proportion ofstudents answered correctly. However, checking the truth of “ P and Q ” is still frequent. Relationship between counterexample knowledge and final grade.
Beloware tables with mean final grades in Math 2903 grouped by their results on the diagnosticand post-tests, respectively. We used independent two-sample t-tests (equal variancesnot assumed) to determine if these groups means differ significantly. We note that thisanalysis is moot for problem PT:5 as most students answered incorrectly.Question MFG (Correct/Justified) MFG (Otherwise) p-valueDT:1 73.9 66.5 0.072DT:2 81.2 66.7 0.019DT:3 75.4 66.1 0.029DT:4 70.0 67.9 0.582DT:5 * * *
Table 4.
Comparison of mean final grade (MFG) of students by diag-nostic question success
HE COGNITION OF COUNTEREXAMPLE IN MATHEMATICS STUDENTS Question MFG (Correct/Justified) MFG (Otherwise) p-valuePT:1 78.0 64.2 < < Table 5.
Comparison of mean final grade (MFG) of students by post-test question successTo aid in answering the question of whether learning logic and counterexample affectsstudent success, for each of the parts of DT:1-DT:4 on the Math 2903 diagnostic testwe looked at the mean final grades for the students who did not answer the questioncompletely correctly (not both correct and justified). We then compared the mean finalgrades of those answering the PT:2 on the post-test question correctly (CR: correct andjustified) and those that did not. We again performed an independent two-sample t-test(equal variances not assumed) to determine if these groups means differ significantly.The results are in the table below.Incorrect MFG (PT:2 CR) MFG (Otherwise) p-valueDT:1 69.1 61.6 0.104DT:2 67.2 65.9 0.76DT:3 68.7 60.2 0.053DT:4 68.9 66.1 0.610
Table 6.
Comparison of mean final grade (MFG) of students with in-correct diagnostic test answers by post-test question success4.
Conclusions
We set out to study the effect of minimal logical training on student outcomes. Ourdata shows interesting trends associated to the effect of understanding the idea of coun-terexamples in logic. Through our investigations we found that certain problems typescan be used as checks for students less likely to find success in their mathematics course.We now examine these ideas specifically in relation to our data and then once again inthe context of our perspective on learning mathematics in general.From the data above, we can reasonably conclude two important things: first, thatvery few students have a strong idea of counterexample entering Math 2903, thoughmany do have some intuition about whether a statement is true or false, without theskills to definitively show why. Indeed, from Table 2 it can be seen that only 2 out of60 students chose the correct answer, and the plurality of students checking both P trueand Q true. This is in line with the findings presented in [IS]. Additionally, only 8 of 62students answered DT:2 correctly.From the midterm data, it is worth noting that these pre-service teachers can learnlogical reasoning and counterexample. Given the data for the post-test, it is clear thatstudents that do successfully learn the idea of logic and counterexamples have moresuccess in the course overall than students that do not. From Table 5 we note that whilethe Wason question (PT:1) was worth approximately 1 per cent of the students’ finalgrade, students that do answer the question correctly do, on average, 13.8 percentagepoints better in the course than those that do not. We emphasize that this question testsexactly one logical idea, without any computational or algebraic skills needed. Indeed,the Wason Selection Task seems to be a highly discriminant question. More research is
SHANNON EZZAT AND SCOTT RODNEY needed to determine whether this logical knowledge is a driver of higher course grades,or whether stronger students simply learn these ideas more quickly than other students.Question PT2, where the student needed to supply a counterexample, was not verypredictive. We believe this is the case since the question may have been too intuitive,though students who did not understand counterexamples in the beginning of the courseand did indeed answer this question correctly seemed to have a significant, or near-significant, difference in mean grades as opposed to students who did not learn theidea of counterexample. We conjecture that it is important to understand the idea ofcounterexample at its core and not just intuitively for easy cases, as a central idea ofmathematics is using logical and mathematical techniques to understand non-intuitiveproblems. Indeed, if we chose a slightly more challenging counterexample problem it ispossible that we would have seen more significant differences in Table 6.Question PT3 was highly predictive. We are not surprised by this data; the skills towrite a correct proof mean that you understand the statement and the associated objectsit deals with in generality, and have the computational and algebraic skills necessary tocomplete this proof.In our comparison of mean final grades of students that did not answer the first fourdiagnostic questions correctly, we find that, while the mean final grades of the studentswho did answer PT:2 correctly was higher for all diagnostic test questions, we find thatat α = 0 .
05 none of these means were significantly higher than the groups of studentsthat answered PT:2 incorrectly. However, we would conjecture that we may have founda significant difference in mean final grade if PT:2 was a more difficult question.Putting this all together, we believe that the data supports the current research thatcounterexample plays a very strong role in mathematical understanding in early under-graduate mathematics. Indeed, a student gaining understanding of the idea significantlyimproves student success in their respective course.
References [BF] C. Bardelle and P. L. Ferrari, Definitions and examples in elementary calculus: the case of mono-tonicity of functions, ZDM Mathematics Education 2011 43:233-246, DOI 10.1007/s11858-101-0303-4[W] P. C. Wason, Reasoning about a rule, Quarterly Journal of Experimental Psychology 1968 Vol. 20Is. 3:273-281[K] S. Klymchuk, Using Counter-Examples to Enhance Learners’ Understanding of UndergraduateMathematics 2009, Good Practice Publication Grants New Zealand: Ako Aotearoa National Centreon Tertiary Teaching Excellence (Online)[VL] R. Vinsonhaler and A. G. Lynch, Students’ Understanding of Counterexamples 2020, MathematicsTeacher: Learning and Teaching PK-12 MTLT, Vol. 113 Iss. 3:223-228[DO] B. Divjak and D. Oreˇski, Prediction of Academic Performance Using Discriminant Analysis, Pro-ceedings of the ITI 2009 31st Int. Conf. on Information Technology Interfaces, June 22-25, 2009,Cavtat, Croatia[IS] A. Inglis and M. Simpson, Mathematics and the selection task, Proceedings of the 28th Conferenceof the International Group for the Psychology of Mathematics Education, 2004[KBT] F. Bagheri, O. Kara, and T. Tolin, Factors Affecting Students’ Grades In Principles Of Economics,Amer. J. Bus. Ed. 2009 Vol. 2 No.7:25-34[L] K. Lee, Student’s logical reasoning and mathematical proving of implications, PhD thesis, MichiganState University, 2011[ZC] R. Zazkis, E.J. Chernoff, What makes a counterexample exemplary?, Educational Studies in Math-ematics, Volume 68:195-208
Department of Mathematics and Statistics, University of Winnipeg, 515 Portage Av-enue,, Winnipeg, Manitoba, Canada R3B 2E9
Email address : [email protected] (Shannon Ezzat) Department of Mathematics, Physics, and Geology, Cape Breton University, Box 5300,1250 Grand Lake Road, Sydney, Nova Scotia, Canada B1P 6L2
Email address ::