The complex relationship between students’ critical thinking and epistemological beliefs in the context of problem solving
Heidi Hyytinena, Katariina Holmab, Auli Tooma, Richard J. Shavelsonc, Sari Lindblom-Ylännea
a University of Helsinki, Finland
b University of Eastern Finland, Finland
c SK Partners, LLC & Graduate School of Education, Stanford University, USA
Article received 2 May 2014 / revised 14 June 2014 / accepted 27 July 2014 / available online 24 September 2014
The study utilized a multi-method approach to explore the connection between critical thinking and epistemological beliefs in a specific problem-solving situation. Data drawn from a sample of ten third-year bioscience students were collected using a combination of a cognitive lab and a performance task from the Collegiate Learning Assessment (CLA). The cognitive-lab data were analysed using thematic analysis. The findings showed that students’ epistemological beliefs were interwoven into their critical thinking: students used critical thinking as a tool (1) for enhancing understanding and (2) for determining truth or falsehood. Based on this classification, students could be placed in one of two qualitative profiles, either (1) thorough processing or (2) superficial processing. The results indicated that students who showed superficial processing palmed off justification for knowing on authoritative figures. In contrast to previous studies these students did not consider knowledge to be absolutely certain or unquestionable. The findings also show that students with thorough processing believed knowledge to be tentative and fallible, but did not share the relativist view of knowledge where any claim counts because all knowledge is relative. All ten students shared a fallibilist view of knowledge.
Keywords: Critical Thinking; Epistemological Beliefs; Cognitive Lab; Relativism; Fallibilism
Critical thinking has been singled out as one of the most important skills for citizens of the twenty-first century (Halpern, 2014). Mastering critical thinking is thus a goal that can be found in almost every higher education curriculum today. However, recent studies have raised concerns that even though most students make significant progress in learning concepts and procedures during their university studies, some students show little if any growth in critical thinking (Arum & Roksa, 2011a, 2011b; Bok, 2006; Pascarella, Blaich, Martin & Hanson, 2011).
In the field of higher education, research on critical thinking has generally focused on the development of critical thinking skills (e.g. Arum & Roksa, 2011a; Heijltjes, van Gog, Leppink & Paas, 2014). Researchers have also highlighted the importance of understanding critical thinking as a social activity (e.g. Arum & Roksa, 2011b; Kuhn, 2005; Moore, 2004; 2013). In this exploratory study we provide a multidimensional framework for analysing critical thinking by combining theoretical aspects from philosophical, educational and psychological approaches. In our view the concept of critical thinking is closely connected to the concepts of ‘knowledge’ and ‘knowing’. Furthermore, we assume that critical thinking cannot be formulated by referring to skills alone, but also always involves a disposition to use these skills adequately (see Bailin & Siegel, 2003; Holma, 2014; Siegel, 1988).
Previous research on critical thinking and personal epistemology has frequently applied quantitative multiple-choice tests, questionnaires or qualitative interviews (see e.g. Australian Council of Education Research, 2001; Heijltjes, van Gog, Leppink & Paas, 2014; Greene & Yu 2014; Lahtinen & Pehkonen, 2013; Tremblay, Lalancette & Roseveare, 2012). Recently, many researchers have questioned the reliability and adequacy of self-report questionnaires (Greene & Yu, 2014; Elby & Hammer, 2001). As a result, researchers have stated that there is a need for studies that assess the performance of students directly (e.g. Elby & Hammer, 2001; Hofer, 2004; Stes, Min-Leliveld, Gijbels and van Petegem 2009). At the same time researchers have also assumed that one assessment method is not enough to evaluate complex cognitive processes such as reasoning (e.g. Baartman, Bastiaens, Kirschner & Vleuten, 2007; Dierick & Dochy, 2001; Maclellan, 2004). This study responds to current concerns by exploring students’ critical thinking as well as their epistemological beliefs, as elaborated upon below, in a problem-solving situation to which we applied a multi-method qualitative approach. A think-aloud method was used as the students worked through an open-ended performance task. Our aim is to identify and understand qualitative differences in the critical thinking of students and in their beliefs about knowledge, as well as in their personal relationships.
2. Critical thinking in university-level studies
Critical thinking is often ‘regarded as fundamental aim of education’ (Bailin & Siegel, 2003, p.188; cf. Dewey, 1910). In a university context critical thinking has an essential role and is an important component of the learning outcomes (Bok, 2006). Critical thinking is defined as a process that enables an individual to make an informed decision about conflicting claims (Ennis, 1991; Fisher, 2011; Bailin & Siegel, 2003). It is purposeful, reasoned and reflective thinking (Ennis, 1991; American Philosophical Association, 1990). A critical thinker knows how to assess the strength of evidence and the reasons that are relevant to the particular context or type of task, and also shows the disposition to draw on these skills (Bailin & Siegel, 2003; Scheffler, 1965, Halpern, 2014).
Critical thinking is seen as a skilful activity in which a person may be more or less proficient (Fisher, 2011; Scheffler, 1965). Definitions of critical thinking typically include a list of the thinking skills that characterise an ideal critical thinker. For example, Fisher (2011) lists the following: the ability to identify the elements in a reasoned case, especially reasons and conclusions; the abilities to identify and evaluate assumptions; the abilities to clarify and interpret expressions and ideas; to be able to judge the acceptability, especially the credibility, of claims; to evaluate arguments, analyse, evaluate and produce explanations; to be able to analyse, evaluate, and make decisions; to draw inferences and produce arguments (see also Halpern, 2014). University studies require all of these abilities.
However, many philosophers have argued that critical thinking cannot be conceptualised merely by referring to a prescribed set of skills (Bailin & Siegel, 2003; Holma, 2014; Fisher, 2011; Siegel, 1988, Scheffler, 1965; see also Halpern, 2014). It may be that a person has acquired the skills, but does not use them (Fisher, 2011). As Holma (2014) has pointed out, it is not enough for students to have critical thinking skills; they also need to use these skills effectively. Thus, critical thinking always involves both the essential skills or abilities and the disposition to use them (Bailin & Siegel, 2003, Holma, 2014; Siegel, 1988).
Previous studies have called attention to the fact that students’ critical thinking skills do not always develop during university studies (Arum & Roksa, 2011a; Bok, 2006; Pascarella, Blaich, Martin & Hanson, 2011). Arum and Roksa (2011b) demonstrated in their longitudinal study that a large number of university students showed no significant improvement in a range of critical thinking skills, such as reasoning and problem solving. However, a recent study by Heijltjes and colleagues (2014) has shown that the combination of explicit instruction and practice has proven successful in improving students’ performance in reasoning skills.
3. Knowledge and knowing in critical thinking
Critical thinking demands a comprehensive use of different types of knowledge (Bok, 2006; Ennis, 1991). There is a reciprocal relationship between ‘critical thinking’, ‘knowledge’ and ‘knowing’; on the one hand, students need knowledge about a phenomenon before they can think about it critically (Halpern, 2014); on the other hand, students must have the necessary skills to evaluate that knowledge. The concepts of ‘knowledge’ and ‘knowing’ are thus substantial aspects of conceptualising critical thinking.
There are several different definitions and classifications of the concept of knowledge. For example, philosophical epistemologists usually differentiate amongst three types of knowledge: propositional knowledge, procedural knowledge and knowledge by acquaintance (Everitt & Fisher, 1995; Ichikawa & Steup, 2012), although there is no consensus on the interpretation of knowledge or on the number of types of knowledge (Fenstermacher, 1994). For our purposes the distinction between propositional and procedural knowledge has theoretical importance.
Propositional knowledge is defined as knowing that ‘such-and-such is the case’. This is sometimes referred to as factual or declarative knowledge. Propositional knowledge (i.e. ‘knowing that’) is usually distinguished from procedural knowledge (i.e. ‘knowing how’) (Ryle, 1949). In philosophical discussions propositional knowledge is related to such epistemological concepts as truth, justification, reason and evidence (Ryle, 1949; Scheffler, 1965, see also Niiniluoto, 1999; Shope, 2004). Scheffler (1965) argued that the ‘knowing that’ attributes of a person may reveal his epistemological orientations, such as the criteria for justifying knowing. Empirical research on personal epistemology focuses particularly on these personal orientations.
Procedural knowledge, meaning ‘knowing how’ to do something (knowing how to analyse, knowing how to swim, etc.; see Everitt & Fisher, 1995; Shope, 2004), is related to possessing a skill (Scheffler, 1965). In this sense critical thinking represents procedural knowledge, which is consistent with the other aspect of critical thinking mentioned above. However, several researchers have assumed that procedural knowledge always involves some propositional knowledge (i.e. Everitt & Fisher, 1995; Smith 2002; Markowitsch & Messerer, 2007). For example, if a person knows how to play chess, he will probably know certain facts (e.g. rules) about playing chess. Smith (2002) has emphasized that an individual has a certain skill only when his performance reflects both procedural and propositional knowledge.
In sum, critical thinking involves a disposition to think critically, having the necessary propositional knowledge about a phenomenon and having the thinking skills (i.e. procedural knowledge) to evaluate that knowledge (cf. Halpern, 2014).
4. Students’ epistemological beliefs as premises of critical thinking
The term ‘personal epistemology’ or, alternatively, ‘epistemological belief’ is defined as an individual’s views of the nature of knowledge and knowing. The term also includes a view of one’s personal beliefs as a knower (Pintrich, 2002; Hofer, 2004). The concept of ‘personal epistemology can be described along a continuum from less sophisticated to more sophisticated’ ways of knowing (Kaartinen-Koutaniemi & Lindblom-Ylänne, 2012, p. 2) or a progress ‘from a state of simple, absolute certainty into a multifaceted, evaluative system’ (West, 2004, p. 61). During this process the individual changes from a passive recipient of knowledge to an active participant in constructing and evaluating knowledge (Hofer & Pintrich, 2002; Kuhn, 2005; King & Kitchener, 2004). Over time epistemological beliefs develop more and more toward relativistic beliefs (Hofer & Pintrich, 1997, 2002).
Previous research on personal epistemology has found that the ability to think critically is embedded in a progression of epistemological beliefs (i.e. King & Kitchener, 2004; Kuhn & Weinstock, 2002; Kuhn, 1999; 2005). Several researchers have hypothesised that students with weak critical thinking skills have an absolute view of knowledge. When students move on to the most developed epistemological level, their critical thinking tends to improve as well (Bok, 2006; Kuhn, 1999; Kuhn & Weinstock, 2002). It has also been demonstrated that students’ epistemological beliefs play an important role in their ability to evaluate the credibility of competing claims (Barzilai & Zohar, 2012).
Whether instruction has any influence on the development of epistemological beliefs is currently under discussion (e.g. Valanides & Angeli, 2005; Lahtinen & Pehkonen, 2013). However, there is evidence that not all university students reach the most highly developed level of personal epistemology (Kuhn & Weinstock, 2002; Kaartinen-Koutaniemi & Lindblom-Ylänne, 2012; King & Kitchener, 2004; Perry, 1970). King and Kitchener (2004) have found that only advanced doctoral students consistently show the highest level of epistemological beliefs. Furthermore, Kaartinen-Koutaniemi and Lindblom-Ylänne (2008, 2012) have shown that there is a considerable variation in personal epistemology among final-year master’s students. Their results also showed variations between students in different age groups, study phases and disciplines (see also Hofer, 2006; Muis, Bendixen & Haerle, 2006). In addition, researchers have assumed that students’ epistemological beliefs may vary within the same discipline or domain (Hammer & Elby, 2003; Greene & Yu, 2014).
5. Critical thinking and different conceptions of knowledge
As the brief review above indicates, the literature of personal epistemology makes a distinction between a lower level of epistemological beliefs, in which knowledge is perceived as consisting of unchanging facts and is acquired directly from external authorities, and higher level epistemological beliefs, in which knowledge is seen as uncertain and constructed by the individual himself (Kuhn & Weinstock, 2002; Hofer, 2005; Valanides & Angeli, 2005). Several researchers have stated that students with higher-level epistemological beliefs have better critical thinking skills than students with lower level epistemological beliefs (King & Kitchener, 2004; Kuhn & Weinstock, 2002; Kuhn, 1999; 2005). Recently, Holma and Hyytinen (2014) have argued that there are several conceptual problems in this kind of hierarchical theory of knowledge (see also Elby & Hammer, 2001). In this section we focus on three conceptions of knowledge identified in the review of the literature on epistemology. These conceptions, specifically relativism, metaphysical realism and fallibilism, have theoretical importance for conceptualising critical thinking.
A relativist position implies that all knowledge is relative to the person who believes or that all interpretations, theories and beliefs are equally right. Because all beliefs are equally right, there is no reason to compare and evaluate different beliefs—all beliefs are equally justified (Holma, 2012; Holma & Hyytinen, 2014). The problem of relativism becomes clear when it is related to the concept of critical thinking (Holma & Hyytinen, 2014). Given that relativism allows people to construct their own ‘personal truths’, critical thinking turns out to be unnecessary (Bleazby, 2011). For example, there is no need to evaluate ideas or search for alternatives, because all ideas are equally trustworthy and justifiable (Bleazby, 2011; Holma & Hyytinen, 2014). Therefore, the idea that critical thinking presupposes the relativist view of knowledge is untenable.
Metaphysical realism is an epistemological position that assumes that ‘our knowledge and symbol systems [i.e. theories] directly reflect the structure of reality’ (Holma, 2004, p. 421; Putnam, 1981). The literature of personal epistemology seems to understand realism as metaphysical realism (see e.g. Kuhn 2005; Kuhn & Weinstock, 2002; see also Holma & Hyytinen 2014), and furthermore, it appears to connect with metaphysical realism the assumption of the possibility of the certainty of human knowledge. As King and Kitchener (2004) put it, knowledge is ‘obtained with certainty by direct observation’ (p. 7). In the context of metaphysical realism, critical thinking turns out to be pointless.
Fallibilism is an epistemological position that implies that all our beliefs are liable to error (Reed, 2002; Niiniluoto, 1999; Holma, 2012). Contrary to relativism, fallibilism does not assume that all beliefs or theories are equally right. It presumes the possibility of improving our current conceptions, theories or beliefs. As Holma (2012, p. 399) aptly states of fallibilism, ‘this position, like the belief that all human knowledge is uncertain, coheres with the evolutionary understanding of knowledge: the bodies of knowledge we now have may be mistaken and thus [are] possible subjects for revision, but they have, nevertheless, survived the process of evolution to this point; as such, they provide the best available starting point for choices and action of the present moment concerning further inquiry’ (see also Peirce, 1934). From this point of view, epistemological fallibilism fits the presumption of critical thinking. Previous research on personal epistemology lacks the notion of epistemological fallibilism.
Summary of the key concepts of this study
Process that enables an individual to make an informed decision between conflicting claims. It involves skills and dispositions (e.g. attitude and motivation) to evaluate the reliability and relevance of evidence, to identify arguments, to analyse, interpret and synthesise data from a variety of sources, to draw valid conclusions and address opposing viewpoints).1 Critical thinking also involves ‘knowing how to do something’ (procedural knowledge) and ‘knowing that’ (propositional knowledge).2
Students’ thoughts/beliefs about the nature of knowledge and the nature of knowing, including personal beliefs about themselves as knowers.3
- metaphysical realism
The idea that human beliefs are direct copies of reality. The belief that all human knowledge is certain is connected to this epistemological position.4
The view that all knowledge is relative to the person who believes or that all interpretations/beliefs are equally correct. Because all beliefs are equally correct, there are no means for comparing different beliefs.5
- epistemological fallibilism
The view that human knowledge is uncertain. In contrast to relativism, it presumes the possibility of improving our current conceptions, theories or beliefs, seeking criteria for evaluating, comparing and justifying these beliefs or theories.5
1Based on Bailin & Siegel (2003); Ennis (1991); Fisher (2011); Fisher & Scriven (1997), Siegel (1988).
2Based on Scheffler (1965); cf. also Ryle (1949).
3Based on Pintrich (2002).
4Based on Holma (2004); Putnam (1981).
5Based on Holma (2012); Holma & Hyytinen (2014); Peirce (1934).
Table 1 provides a summary of the definitions of the key concepts in this study. With this broader framework we are able to pin down different areas in critical thinking and epistemological beliefs, which have been shown to be vital for conceptualising these phenomena in prior studies or theorizations. Although the conventions of critical thinking and epistemological beliefs are commonly embodied in social practices (e.g. Arum & Roksa, 2011b; Elby & Hammer, 2001; Kuhn, 2005), the underlying dimensions (i.e. evaluating the reliability and relevance of evidence, identifying arguments, analysing information, addressing opposing viewpoints, reasoning) are relevant in each scientific discipline. Moreover, in line with previous studies we expected that students’ epistemological beliefs and critical thinking might vary within the same discipline (see Greene & Yu, 2014; see also Bailin & Siegel, 2003).
In our study we focused on the qualitative differences in critical thinking and personal epistemological beliefs by examining ten third-year university students’ thinking and performance in a cognitively-demanding authentic problem-solving situation. The aims of this study are twofold: to identify and describe qualitative differences in third-year university students’ critical thinking skills and epistemological beliefs in a problem-solving situation, and to analyse the interconnections between students’ personal epistemologies and critical thinking skills. To achieve these aims, we formulated the following research questions: (1) How are critical thinking and epistemological beliefs presented in a problem-solving situation in a specific group of third-year university students? (2) How do critical thinking and epistemological beliefs vary from one individual to the next?
6. Research methods and materials
This study was conducted with ten third-year bioscience students drawn from the fields of biological and environmental sciences in a research-intensive university in Finland. The target population consisted of all third-year bioscience students in this particular university. First, we selected 40 students at random (approximately one-half of the target population). Then we invited all students selected to participate in our study. Ten out of 40 students volunteered. Seven of the participants were female and three male. The students’ ages varied from 22 to 29, the mean age being 24. All came from a homogeneous cultural background, and all shared the same first language (Finnish). In addition, the students had the same national high school certificate and had enrolled in the same bachelor’s study programme. The participants were at the same phase of their studies, that is, near the end of their bachelor’s studies, with the exception of one student whose study pace had been slower. During their university careers, the students had participated in lectures, practical laboratories, seminars, field courses and web-based teaching. We are aware that the sample size is too small for generalization. However, the purpose of this study is to deepen understanding of critical thinking and epistemological beliefs, for example, so as to describe how these phenomena vary across individuals in this specific group of students.
For this study we collected a large body of data for each participant using a multi-method approach (Johnson, Onwuegbuzie & Turner, 2007), including think-aloud protocol, interviews and a Collegiate Learning Assessment (CLA) performance task. The data collection was carried out in the spring of 2010 and consisted of ten cognitive labs. The students came to a classroom and were given the details of the study. The students spent two to three hours reading and responding to the performance task. In responding to the task, the students were asked to verbalise their thoughts (to ‘think aloud’). In the course of carrying out the task while thinking aloud, the students were also asked to write a memorandum addressing critical issues in the task and recommending —and justifying— a course of action. Following the task, the students were interviewed about their processes in carrying out the task. Students were also asked questions about critical thinking, knowledge and knowing. Details of the procedures are provided below in appropriate sections.
6.2.1 Collegiate Learning Assessment (CLA)
The Collegiate Learning Assessment (CLA) instrument for assessing college-level critical thinking skills used in this study was developed by the Council for Aid to Education (CAE). The CLA is a standardised, open-ended test and it measures analytical reasoning, problem solving and written communication. Unlike most standardised tests used in measuring critical thinking, the version of the CLA used here did not include any multiple-choice questions (Klein, Benjamin, Shavelson & Bolus, 2007). The CLA consists of two elements: a set of performance tasks and a set of analytical writing tasks (Shavelson, 2010). Only the performance task was used in this study. Recent studies have found that open-ended problems with no obvious solution provide an opportunity for students to reflect on their beliefs about knowledge (Barzilai & Zohar, 2012; Ferguson & Bråten, 2012). For example, in a problem-solving situation students would need to determine the trustworthiness, and relevance, of different types of information presented to them, co-ordinate various pieces of information related to the problem and consider the underlying assumptions and claims (Shavelson, 2010).
The CLA performance task presents a realistic situation or problem and includes directions, open-ended questions and a document library containing reading material. In order to respond to the task, the students need to read, organise, synthesise and analyse information (which might be reliable/unreliable; relevant/irrelevant to the completion of the task; see Shavelson, 2010) from multiple documents (for example letters, memos, summaries of research reports, articles, diagrams, graphs, maps, interview notes). In doing these activities the students need to assess their confidence in information taken from various sources, including the relevance of the source, and thereby deal with conflicting information. They then need to decide on a course of action and provide a reasoned explanation and justification for their course, drawing on supporting information from the document library (Klein et al., 2007; Shavelson, 2010). They also have to argue for and against alternative explanations. The specific performance task used in this study is proprietary and consequently cannot be described here. An example of a representative CLA performance is presented in Figure 1.
Figure 1. (see pdf) An example of a CLA performance task. Adapted from R. Shavelson, 2010, Measuring College Learning Responsibly: Accountability in a New Era. Stanford, CA: Stanford University Press, p. 38.
6.2.2 Cognitive labs
The purpose of cognitive labs is to study the cognitive processes that students use when they complete different tasks. Students are asked to report their thoughts verbally as they carry out a task (see Johnstone, Bottsford-Miller & Thompson, 2006). In this study cognitive labs were divided into three parts: (1) instruction and training, where the researcher explained what the cognitive lab was about and trained the students to think aloud with a short warm-up task; (2) ‘think-aloud’, where the students talked aloud while completing the CLA performance task; and (3) a follow-up interview. The cognitive lab for each student was video-recorded and lasted two to three hours. To ensure the consistency of cognitive labs, a script of directions and the same training task and the interview questions for each student were used. The videos were recorded with two cameras and a table microphone. The cognitive workshop produced the following materials: video data, content logs (see below), written test answers and transcribed interview data.
The neutral type of think-aloud protocol conducted by Ericsson and Simon (1993) in which students were not interrupted while they were performing a task was used in this study. The think-aloud method makes it possible to collect data about a student’s ongoing thinking processes whilst he or she is working on a task (Ericsson & Simon, 1993; Cotton & Gresty, 2006; van Someren, Barnard & Sandberg, 1994). We assume that students’ ‘knowing-that’ attributions (e.g. ‘scientific knowledge is true’) may reflect their epistemological orientations and reveal their criteria for justifying beliefs (see Scheffler, 1965). Moreover, in some cases the think-aloud method makes it possible to explore critical thinking in action, especially in situations that simulate real-world circumstances.
Immediately after the task was performed, a follow-up interview was conducted. The aim of the interview was to gain more detailed information about the processes and knowledge that the students used to complete the task and to probe students’ beliefs about knowledge and knowing. For example, the students were asked questions about how they dealt with conflicting information, how they decided which information to use, what sources of information in documents from the documents library they trusted and why, and how they usually evaluate knowledge.
7. Data analysis
The data were analysed using a qualitative thematic analysis with an abductive approach (Timmermans & Tavory, 2012; Haig 2005). An abductive strategy means that the themes identified from the data were linked to the theoretical understanding based on previous studies. Abduction is a process that combines things which one had not previously associated by creating a new interpretation, that is, the relationship of a new combination of study features (Timmermans & Tavory, 2012). Hence, the analysis process was nonlinear, moving back and forward amongst all the data, data items, analysed qualities and understanding of the phenomenon based on prior studies. The first and fifth authors were responsible for the analysis, but the final results were obtained through a thorough discussion with all authors. The data were processed in such a way that the participants could not be identified.
The analysis included four phases (Figure 2) that represented the unique combination of data-grounded and theory-driven phases, as well as phenomenon and individual-level analyses. During the first phase, video recordings were initially indexed with the ELAN program, which allows the addition of as many tiers and annotations on the video stream as needed (see Lausberg & Sloetjes, 2009; Max Planck Institute for Psycholinguistics, 2012). The purpose of indexing was to make the large video data set easier to handle. In this study the indexing tiers corresponded to the parts of cognitive labs including training, think-aloud methods and interviews. In addition, students’ interviews from the videos were transcribed.
*Based on Braun & Clarke (2006, p. 87).
Figure 2. (see pdf) A visualisation of the analysis process.
After the indexing, content logs were created for each video in which accurate descriptions and summaries of events were systematically recorded. Transcriptions of relevant sections of verbalisations of students’ critical thinking and epistemological beliefs (e.g. whenever a student evaluated the quality and reliability of the information in a document or where a student reached a conclusion based on her or his analysis) and nonverbal acts (e.g. a student did not read in detail or skipped over the document) were also included in the log (cf. Table 1).
The second phase of the analysis was the data coding (see Table 2 for definitions). This phase was theory-driven, meaning that the features guiding the coding were based on prior studies (see Table 1). The coding focused on the following qualities: the process by which the student approached the task and solved the problem, the knowledge that the student used to carry out the task, the critical thinking exhibited, and epistemological beliefs. These different qualities were coded systematically across the entire data set and within the data items such as the transcribed interviews and the think-aloud videos of each person. By this means, all the data items from one student, including the video data, content log, written test answers and transcribed interviews, were coded and analysed separately, after which data from all students were combined and compared (see Table 3 for an example of the codes). All extracts were labelled with a student code (S1-S10) and a method code (I= interview, T=think aloud, W= written test answer). The data examples were translated into English.
Data sources and focal points of coding
Video data, content logs, transcribed interviews
1. The process: how does the student approach the task and solve the problem?
Video data, students’ written answers, content logs, transcribed interviews
2. What knowledge/information does the student use to solve the task?
2.1 What kind of knowledge/information did the student use?
2.3 How does the student use that knowledge/information?
Video data, students’ written answers, content logs, transcribed interviews
3. Critical thinking
3.1 How does the student identify, analyse and evaluate information, ideas and arguments?
3.2 How does the student judge the acceptability (especially the credibility) of documents?
3.3 How does the student interpret data/ graphs/ maps?
3.4 How does the student recognise the relationship between assumptions?
3.5 How does the student evaluate background information?
3.6 How does the student make a decision?
3.7 How does the student identify reasons and come to a conclusion?
3.8 How does the student produce explanations and arguments?
Video data, students’ written answers, content logs, transcribed interviews
4. Epistemological beliefs
4.1 What does the student think about knowledge, knowing and the credibility of knowledge?
4.2 How does the student determine the trustworthiness, acceptability and justification of different types of information?
4.3 How does the student describe herself or himself as a knower?
An example of codes
You could consider this a good argument; the expert has gone [to the place where events took place] to see for himself (S9T)
4.1 What does the student think about knowledge, knowing and the credibility of knowledge?
4.2 How does the student determine the trustworthiness, acceptability and justification of the different types of information?
- - yeah, I don’t believe [the chair of the stakeholder group] is completely off the mark either. [Reliability]isjust always case-specific. (S8I)
4.1 What does the student think about knowledge, knowing and the credibility of knowledge?
This just seems scientific somehow. (S6T)
4.1 What does the student think about knowledge, knowing and the credibility of knowledge?
In the third phase the codes and coded extracts were grouped under potential themes, and all the relevant data were gathered under each theme (Braun & Clarke, 2006). We identified a variety of preliminary themes on the basis of the codes. During the analysis, the preliminary themes were defined and combined several times. In the end two main themes and two subthemes remained (see Figure 2). The final themes were refined, labelled and cross-checked to see if they worked in relation to the coded extracts and the entire data set. The focus of the thematic analysis was the variation of study features on the phenomenon level.
After completing the thematic analysis, we found that the students could be placed in different profiles based on our themes as well as on patterns of behaviour and cognition observed. This phase focused on the variation of study features at the individual level. Thereafter, we conducted final descriptions, interpretations and revisions of the results. The results of thematic analysis show how critical thinking and epistemological beliefs manifested themselves in this particular group of students, whereas the student profiles describe how these phenomena vary across individuals.
In the thematic analysis two main themes were identified: (1) flexibility in critical thinking and (2) variation in critical thinking and epistemological beliefs. The two themes emerged from exploring the students’ critical thinking from different perspectives. The ways in which the themes were related differed amongst the participants, which further allowed us to identify student profiles. We identified two main profiles, and on the basis of their characteristic features we labelled them as (1) thorough processing and (2) superficial processing. The results are described using a combination of identified themes and student profiles.
8.1 Flexibility in critical thinking
Students showed various skills in their ability to adapt their thinking and their performance flexibility to the demands of the task. There was clear variation in the students’ ability to change their actions or ways of critical thinking, in which we identified both rigidity and flexibility. Flexibility meant that the students could modify their actions and processes and change their behaviours as needed, whereas rigidity refers to situations in which students could not change their processes or look at things from a new perspective or adjust to new evidence in a problem-solving situation. Students who were able to make changes in their actions showed open-mindedness and an inquiring attitude.
In the following extract, one student describes how he adjusted his performance and ended up analysing and interpreting the documents correctly:
I approached[this assignment]maybe a little too much as if I had simply copied what they say here in these papers and put them down [in my answer]. But then when I started thinking, like about my own views on the topics, then right off in [question] number one, it took me a really long time to answer this question. (S8I)
American Association of Colleges for Teacher Education (2007). Preparing STEM teachers: The key to global competitiveness. Selected profiles of teacher preparation programs. Washington: Author
Author, in review [details removed for review]
Baker, L. (1989). Metacognition, comprehension monitoring, and the adult reader. Educational Psychology Review, 1, 3-38.
Bandura, A. (1997). Self-efficacy: The exercise of control. New York: Freeman.
Bell, P., & Lin, M. C. (2002). Beliefs about science: How does science instruction contribute? In B. Hofer & P.R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp.321-346).Mahwah, NJ: Erlbaum.
Dellinger, A. M., Bobbet, J. J., Olivier, D. F., & Ellet, C. D. (2008). Measuring teachers‟ self- efficacy beliefs: Development and use of the TEBS – Self. Teaching and Teacher Education, 24, 751-766.
Enochs, L. G. & Riggs, I. M. (1990). Further Development of an Elementary Science Teaching Efficacy Belief Instrument: A preservice elementary scale. School Science and Mathematics, 90, 695-706.
Hacker, D. J., Bol, L., Horgan, D. D., & Rakow, E. A. (2000). Test prediction and performance in a classroom context. Journal of Educational Psychology, 92, 160-170.
Hacker, D. J., Dunlosky, J., & Graesser, A. C. (2009). Handbook of Metacognition in Education. New York: Routledge.
Hammer, D. & Elby, A. (2002). On the form of a personal epistemology. In B. Hofer & P.R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp.169-190). Mahwah, NJ: Erlbaum.
Jehng, J. J., Johnson, S. D., & Anderson, R. C. (1993). Schooling and student’s epistemological beliefs about learning. Contemporary Educational Psychology, 18(1), 23–35.
Jones, E. (2008). Predicting performance in first-semester college basic writers: Revisiting the role of self-beliefs. Contemporary Educational Psychology, 33, 209-238.
Kardash, C. M., & Scholes, R. J. (1996). Efects of preexisting beliefs, epistemological beliefs, and need for cognition on interpretation of controversial issues. Journal of Educational Psychology, 88, 260-271.
Kennedy, M. M., Ahn, S., & Choi, J. (2008). The value added by teacher education. In M. CochranSmith, S. Feiman-Nemser, & J. McIntyre (Eds.), Handbook of research on teacher education (pp.1249–1273). New York: Macmillan
Keren, G. (1991). Calibration and probability judgments: Conceptual and methodological issues. Acta Pscyhologica, 77, 217–273.
Kramarski, B., & Michalsky, T. (2009). Investigating preservice teachers’ professional growth in self-regulated learning environments. Journal of Educational Psychology, 101, 161-175.
Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121–1134.
Ku, K. Y. L., & Ho, I. T. (2010). Metacognitive strategies that enhance critical thinking. Metacognition and Learning, 5, 251-267.
Kuhn, D. (1989).Children and adults as intuitive scientists. Psychological Review, 96, 674-689.
Lee, O., Hart, J. Cuevas, J., & Enders, C. (2004). Professional development in inquiry-
based science for elementary teachers of diverse student groups. Journal of Research in Science Teaching, 41, 1021-1043.
Leou, M., Abder, P., Riordan, M., & Zollar, U. (2004). Using ‘HOCS-Centered Learning’ as a pathway to promote science teachers’ metacognitive development. Research in Science
Lumpe, A. T., Haney, J. J., & Czerniak, C. M. (2000). Assessing Teachers’ Beliefs about their Science Teaching Context. Journal of Research in Science Teaching, 37, 275-292.
Manlove, S., Lazonder, A. W., & de Jong, T. (2007). Software scaffolds to promote regulation during scientific inquiry learning. Metacognition and Learning, 2, 141-155.
Mansour, N. (2009). Science teachers’ beliefs and practices: Issues, implications and research agenda. International Journal of Environmental & Science Education, 4, 25-48.
Michalsky, T. (2012). Shaping self-regulation in science teachers’ professional growth: Inquiry skills. Science Education, 96, 1106-1133.
Michalsky T., & Schecheter,C. (2013). Preservice teachers’ capacity to teach self-regulated learning: Integrating learning from problems and learning from success. Teaching and Teacher Education, 30, 60-73.
National Academies of Science (2006). Rising above the gathering storm: energizing and employing America for a brighter economic future. Washington: Author.
National Research Council (NRC). (2011). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Council on Conceptual Framework for the New K-12 Science Education Standards, National Research Council. Retrieved on May 13, 2013 from: http://www.nap.edu/catalog.php?record_id=13165.
National Science Teachers Association (NSTA). (2004). NSTA Position Statement: Scientific Inquiry.
Nietfeld, J. L., Cao, L., & Osborne, J. W. (2006). The effect of distributed monitoring exercises and feedback on performance and monitoring accuracy. Metacognition and Learning, 2, 159-179.
Olafson, L. & Schraw, G. (2006). Teachers’ beliefs and practices within and across domains. International Journal of Educational Research, 45, 71-84.
Pajares, F., & Johnson, M. (1996). Self-efficacy beliefs and the writing performance of entering high school students. Psychology in the Schools, 33, 163-175.
Peters-Burton, E. & Frazier, W. M. (2012). Voices from the front lines: Exemplary science teachers on education reform. School Science and Mathematics, 112, 179–190.
Poon, C-L., Lee, Y-J., Tan, A-L., & Lim, S. S. ( 2012). Knowing inquiry as practice and theory: Developing a pedagogical framework with elementary school teachers. Research in Science Education, 42, 303-327.
Pressley, M., Borkowski, J., & Schneider, W. (1987). Cognitive strategies: Good strategy users coordinate metacognition and knowledge. In R. Vasta and G. Whitehurst (Eds.), Annals of Child Development (Vol. 4, pp. 89-129). Greenwich, CN: JAI.
Randi, J. (2004). Teachers as self-regulated learners. Teachers College Record, 106, 1825-1853.
Randi, J., & Corno, L. (2000). Teacher innovations in self-regulated learning. In P. Pintrich, M. Boekaerts, & M. Zeidner (Eds). Handbook of self-regulation (pp.651-685). Orlando, FL: Academic Press.
Richardson, G. M., & Liang, L.L. (2008). The use of inquiry in the development of preservice teacher efficacy in mathematics and science. Journal of Elementary Science Education, 20 (1), 1-16.
Riggs, I.M. & Enochs, L.G. (1990). Toward the development of an elementary teacher's science teaching efficacy belief instrument. Science Education, 74, 625–637.
Rimm-Kaufman, S. E., & Sawyer, B. E. (2004). Primary-grade teachers’ self-efficacy beliefs, attitudes toward teaching, and discipline and teaching practice priorities in relation to the “responsive classroom” approach. The Elementary School Journal, 104 (4), 321-341.
Schiffman, S. S., Reynolds, M. L., & Young, F. W. (1981). Introduction to multidimensional scaling: Theory, methods, and applications. New York: Academic Press.
Schommer-Aikins, M. (2002). An evolving theoretical framework for an epistemological beliefs system. In B. Hofer & P.R. Pinrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp.103-177). Mahwah, NJ: Erlbaum.
Schraw, G. (2009). A conceptual analysis of five measures of metacognitive monitoring accuracy. Metacognition and Learning, 4, 33-45.
Schraw, G., Bendixen, L. D., & Dunkle, M. E. (2002). Development and validation of the Epistemic Beliefs Inventory (EBI). In B.K. Hofer & P.R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 261-275). Mahwah, NJ: Erlbaum.
Schraw, G., Crippen, K. J., & Hartley, K. (2006). Promoting self-regulation in science education: Metacognition as part of a broader perspective on learning. Research in Science Education, 36, 111-139.
Schraw, G., & Olafson, L. (2002). Teachers’ epistemological world views and educational practices. Issues in Education, 8, 99-149.
Schraw, G. & Roedel, T. D. (1994). Test difficulty and judgment bias. Memory and Cognition, 22, 63-69.
Sharp, J., Hopkin, R., James, S., Peacock, G., Kelly, L., Davies, D., & Bowker, R. (2009). Teacher preparation and the national primary science curriculum: A twentieth-anniversary perspective. Research Papers in Education, 24(3), 247-263.
Sharp, J. G., Hopkin, R., & Lewthwaite, B. (2011). Teacher perceptions of science in the National Curriculum. International Journal of Science Education, 33(17), 2407-2436.
Smith, L. K. & Southerland, S. A. (2007). Reforming practice or modifying reforms? Elementary teachers’ response to the tools of reform. Journal of research in science teaching, 44(3), 396-423.
Swars, S., Hart, L. C., Smith, S. Z., Smith, M., & Tolar, T. ( 2007). A longitudinal study of elementary pre-service teachers’ mathematics beliefs and content knowledge. School Science and Mathematics, 107 (8), 325-335.
Thiede, K. W., Anderson, M. C. M., & Therriault, D. (2003). Accuracy of metacognitive monitoring affects learning of texts. Journal of Educational Psychology, 95, 66-73.
Thomas, G. P., & McRobbie, C. J. (2001). Using a metaphor for learning to improve students’ metacognition in the chemistry classroom. Journal of Research in Science Teaching, 38, 222-259.
Thomson, M. M., & Gregory, B. (2013). Elementary Teachers' Classroom Practices and Beliefs in Relation to US Science Education Reform: Reflections from within. International Journal of Science Education. DOI:10.1080/09500693.2013.791956
Utley, J., Moseley, C., & Bryant, R. (2005). Relationship between science and mathematics teaching efficacy of preservice elementary teachers. School Science and Mathematics, 105(2), 82-88.
Waters-Adams, S. (2006). The relationship between understanding the nature of science and practice: The influence of teachers’ beliefs about education, teaching and learning. International Journal of Science Education, 28(8), 919-944.
Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531–566). Orlando, FL: Academic Press.
Yates, J. F. (1990). Judgment and decision making. Englewood Cliffs, NJ: Prentice-Hall.
Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45, 166-183.
Zimmerman, B. J., & Moylan, A. R. (2009). Self-Regulation: Where metacognition and motivation intersect. In Hacker, D. J., Dunlosky, J., Graesser, A. C. (Eds.). Handbook of metacognition in education. New York: Routledge.
Zohar, A. & Ben David, A. (2008). Explicit teaching of meta-strategic knowledge in authentic classrooms situations. Metacognition and Learning, 3, 59-82.