There is no measurement in research about education, but there is plenty of enumeration, rating and ranking

In books of research about education, the definition of measurement offered by F.N. Kerlinger (1910-1991) is often quoted and widely followed:

“The assignment of numerals to objects or events according to rules” (Foundations of Behaviorial Research, 3rd edition, 1986. p. 391).

Unfortunately, this is a woefully inadequate definition of measurement because it is far too inclusive. It allows the inclusion of nominal numerals and ordinal numerals as concepts which can be used in measurement. Nominal and ordinal numerals do not denote cardinal value and thus can not ever represent units of measurement in either their interval or ratio form. There are always rules required in the application of symbols, otherwise the script or figures (alpha, numeric, ideographic, iconic, etc.) can not be symbols. The crucial question is what rule applies in what case. In the case of the Kerlinger definition of measurement, the allowance of any rule to be used with numerical symbols is mistaken, and it fails to delineate the essential characteristics of measurement.

The mathematical and scientific concept of measurement is the appropriate one to use in conceptualizing measurement as a procedure used in conducting research about eduation. Measurement, as a mathematical concept, is the application of a system in the task of assigning a cardinal number to indicate the size, amount or intensity of each subset of a set. Measurement is a comparison of the size, amount or intensity of something with a standard unit of size, amount or intensity of the something. The something being measured is a subset of a set of phenomena. A cardinal number is assigned to the size, amount or intensity of the subset, and the number is a multiple of the standard unit of size, amount or intensity of something. Unequivocal examples of measurement include

(a) using the centimeter (a standard unit of measurement of size) to determine the length (some cardinal number of centimeters) of a stick (a physical thing, i.e. a subset of the set of all physical things denoted by the term stick),

(b) using the gram (a standard unit of measurement of amount) to determine the mass (some cardinal number of grams) of a stick (a physical thing, i.e. a subset of the set of all physical things denoted by the term stick),

(c) using the centimeter and the second (a standard unit of measurement of size and amount) to determine the speed (some cardinal number of centimeters per second) at which a stick is moving in a stream of water (a physical thing, i.e. a subset of the set of all physical things denoted by the term stick).

The essential elements of measurement are

(1) a standard unit of size, amount or intensity of a subset of a set of phenomena,

(2) a size, amount or intensity of a subset of a set of phenomena,

(3) a comparison of the standard unit with the subset of the set of phenomena,

(4) an assignment of a cardinal number to the size, amount or intensity of the subset as a multiple of the standard unit of size, amount or intensity of a subset of the set of phenomena.

In applying a system of measurement, one of the subsets of a set is selected and nominated as the basic unit of the system by which all other subsets of the set of phenomena are to be compared. The number assigned to the basic unit is distributive, associative and commutative, i.e. the unit can be added to itself to form regular intervals of units, and the regular intervals constitute a scale.

A scale is a system of ordered standard units used for comparing the size, amount or intensity of a subset of a set of phenomena. If the scale has an absolute value of zero (0), the multiple units on the scale form ratios. If the scale does not have an absolute zero (0), it is not possible to find ratios on the scale. Such a scale shows intervals, but no ratios.

For example, a degree Centigrade is a standard unit for measuring temperature. Water freezes at 0 degrees and boils at 100 degrees Centigrade at one atmosphere of pressure. But 40 degrees Centigrade is not twice the temperature of 20 degrees Centigrade. It is not possible to establish ratios of intervals on the Centigrade scale because there is no absolute value of zero (0).

On the other hand, the meter is a standard unit for measuring length. The definition of the meter has undergone several historical revisions, and its current definition is the distance traveled by light rays in a specified time (a very small fraction of a second, 1/299,792,459th of a second). On a scale consisting of multiples of meters, there is an absolute value of zero (0). Thus it is possible to calculate ratios of measurements in meters. Two meters is half the length of four meters. The ratio is one to two (1:2).

Thus, in scales of measurement, there can be interval scales and ratio scales. Interval scales assign only values of units to a subset of a set of phenomena (e.g. 20, 40, 80 degrees Centigrade). Ratio scales assign both values of units and ratios of comparison to a subset of a set of phenomena, e.g. 20, 40, 80 meters with ratios of 20:40 (1:2), 40:80 (1:2) and 20:80 (1:4) respectively.

In the natural sciences, standard units of measurement have been established as part of the conceptual theory of natural sciences, e.g. in addition to meters and degrees Centigrade, there are kilograms, degrees Kelvin, volts, amperes, watts, ohms, kilojoules, etc.

In the social sciences, including scientific educology, no such standard units of measurement have been established in conceptual theory. For example, in educological theory, there are no units of intelligence, no units of studying, no units of teaching, no units of knowing, no units of appropriate practice, etc.

While social scientists in general, and scientific educologists in particular, make claims to measuring phenomena, they are false claims. There are no appropriate units of measurement as yet established to conduct measurement of educational phenomena, and without units of measurement, the task of measuring is impossible.

Of course, in discourse about education, it is common to find the expressions measurement and tests and measurements. But the use of the term measurement in relation to scientific educological research is either a mistake, a conceit or a deception.

It is a mistake when educological researchers do not recognize and do not understand that they are operating with an ill conceived definition of the term measurement.

It is a conceit when social scientific researchers know that they are operating with an ill conceived definition of the term measurement, but want recognition as legitimate scientific researchers, and thus claim that they do conduct measurement, but the measurement is of a different nature (i.e. measurement without units) from that used in the natural sciences.

And it is a deception when social scientific researchers know and understand that they are operating with an ill conceived definition of the term measurement and wilfully assert (falsely) that the definition of the term measurement is adequately conceived i.e. conceived as “the assignment of numerals to objects or events according to rules” (F. Kerlinger, 1986, p. 391).

Putting a wine label on a bottle of vinegar does not turn the vinegar into wine. And putting the label of measurement on research instruments which do not provide measurement units does not turn those instruments into measurement instruments.

Without measurement (i.e. without a system of units of measurement to assign cardinal values to amount, size or intensity of subsets of a set of phenomena), there are still many ways and means of expressing data about educational phenomena in cardinal numerical form from observations and tests. The cardinal numerical data can readily be used to test hypotheses about educational phenomena for the purpose of evaluating educological theory. Data expressed in cardinal numerical form from tests and observations facilitate enumeration (counting) of subsets of sets of phenomena, and statistical analysis can be performed on the cardinal numbers generated from the enumeration.

Intelligence tests are a model case of expressing data in numerical form as cardinal numbers (without measurement units) and using that data to enumerate, analyze and establish position of intellectual performance. The process begins with the development of the intelligence test. A set of reasoning processes is conceived as part of exemplifying intelligence. Test items are generated which are judged by test makers to require the use of those reasoning processes. The test is scored as some number correctly answered out of the total items on the test.

A random sample is selected from a population. The individuals in the randomly selected sample are tested. Their scores are analyzed statistically to establish the distribution and mean of the scores. A standard deviation is calculated to establish the position of the individual scores within all of the scores. Different versions of the test are administered and analyzed to establish consistency of results among the tests so that reliability is established for the tests. Any individual test score indicates the position of the individual’s score in relation to all other scores.

The position is a rank of a subset (i.e. one person’s score) of a set (i.e. all person’s scores) of correctly answered test items. But there is no indication of the value between positions. Thus a score of 120 has a higher rank than a score of 60, but the interval between 120 and 60 does not indicate an intelligence which is twice that of the score of 60. Nor does the interval between 120 and 60 indicate a difference of 60 units of intelligence (because there are no units of intelligence as yet established). It is an indication of position, but not an interval measure nor a ratio measure of intelligence.

Thus, with intelligence tests, there is position, but there is no measured unit of intelligence. There is no interval of intelligence. There is no scale of intelligence. Hence there is no measurement. There is no measurement in the sense of making a comparison of the size, amount or intensity of a subset of phenomena with a standard unit of the phenomena so that the subset can be described numerically as some multiple of the standard unit of the phenomena.

And so it is with all other standardized tests (and indeed for all tests, standardized or otherwise) which are used in social scientific (including educological scientific) research. They establish positions for scores within a range of scores on the test. But they do not use any units of measurement to assign cardinal numbers which give value to amount, size or intensity of subsets of a set of phenomena, expressed as some number of standard units of an attribute, property or characteristic. Therefore, they are not instruments for measuring, but they are instruments for enumerating (i.e. counting) and ranking.

After a century of development of observational research instruments for educological scientific research (and for all social scientific research), no scientific educological instrument has been developed which expresses data as measures of educational phenomena (e.g. no measurement units of intelligence, learning or knowing have been developed). No scientific educological instrument has been developed which expresses data as intervals or ratios of units of measurement of educational phenomena. The current state of affairs in instruments for scientific educological research is that there are many observation schedules, behavioral inventories, questionnaires, instruments for expressions of preference, rating guidelines, scoring instructions, aptitude tests, attitude tests, achievement tests, etc., but no instruments measuring educational phenomena and expressing those measures as some interval or ratio unit of measurement.

Scientific educological research instruments over the past century have been developed to express data about educational phenomena as numerals to denote

(1) categories (with nominal numerals),

(2) ratings (with nominal numerals),

(3) rankings (with ordinal numerals) and

(4) enumerations (with cardinal numbers).

Thus in scientific educological research, there is no measurement as yet developed (it may never be developed!) of educational phenomena, but there is plenty of enumeration, ranking and rating of educational phenomena.

Key roles in a unit (faculty, college, school, department) of educology

The proper role of a unit of educology (i.e. a faculty, school, college, department or division of educology) is to provide a favorable and supportive environment in which

(1) educologists may conduct research about the educational process and extend educology,

(2) teachers of educology may successfully guide their students in extending their range of educological knowing and understanding,

(3) users of educology may consult with educologists about ways and means of resolving issues and meeting challenges in the educational process,

(4) meta-educologists may conduct research about discourse about the educational process and extend meta-educology,

(5) teachers of meta-educology may successfully guide their students in extending their range of meta-educological knowing and understanding and

(6) users of meta-educology may use their meta-educological understanding as educologists in the conduct of their educological research.

The guided study of educology is appropriate for those who wish to use educology in the role of teacher, administrator, counselor, tutor, mentor, curriculum developer, educational resource publisher, tests and measurement expert, legislator, decision maker of a funding organization and the like. Educology, i.e. knowledge about education, provides the basis for conducting well informed, intelligent and effective action for and within the educational process.

The guided study of meta-educology is appropriate for those who wish to take on the role of educologist, teacher of educology, meta-educologist and teacher of meta-educology. Meta-educology, i.e. knowledge about knowledge about education, provides the basis for conducting and evaluating disciplined research about education.

Logical absurdity of the concept of alignment of curriculum, pedagogy and assessment

Bertrand Russell famously wrote that

“The fact that an opinion has been widely held is no evidence whatever that it is not utterly absurd; indeed in view of the silliness of the majority of mankind, a widespread belief is more likely to be foolish than sensible” (Marriage and Morals, 1929).

His remark applies to so many situations and long standing practices, especially in the literature about education. A case in point is the oft repeated contention in Queensland (where I live these days) that “research shows” that there needs to be an alignment of curriculum, pedagogy and assessment in order for effective education to occur.  Queensland Education curriculum documents, school documents and even university course descriptions make this assertion and attribute it to Dr. Allan Luke, former Dean of Education at Queensland University, former Deputy Director General of Education for Queensland, and currently Emeritus Professor at Queensland University of Technology, Brisbane.

The assertion that there must be an alignment of curriculum, pedagogy and assessment is a classical case of Chinese Whispers.  It is a misquote and a distortion of what Allan Luke said in an opinion piece (some writers have claimed that it is a research report; it is not a research report!) that he wrote in 1999 (Allan Luke [1999], “Education 2010 and new times: Why equity and social justice still matter, but differently,” a paper presented to the Education Queensland Online Conference, 20 October 1999).

People notoriously misquote the character Rick in the movie Casablanca  (1942). They say that Rick said, “Play it again, Sam.” But Rick never uttered these words (watch the movie!). People misquote the character Harry Callahan in the movie Dirty Harry (1971). They say that Harry said, “Are you feeling lucky, Punk?” But Harry never uttered those words (watch the movie!). People commonly attribute the statement, “Let them eat cake,” to the unfortunate French queen, Marie Antoinette (guillotined in 1793 just short of her 38th birthday). Yet, there is no reliable historical record that verifies she ever uttered or wrote those words (read the historical record!).

As it is with Rick and Harry and Marie Antoinette, so it is with Professor Allan Luke.  People say that he wrote and uttered the words, “curriculum, pedagogy and assessment must be aligned for effective education,” but he never did (read his paper, Allan Luke, [1999], “Education 2010 and new times, Why equity and social justice still matter, but differently,” Education Queensland Online Conference, pp.1-14.

Here is one (there are many) example of a misquote and misinterpretation of Professor Allan Luke:

“Dr Allan Luke, Professor of Education at Queensland University of Technology, argues that the three key elements of education i.e. curriculum, pedagogy and assessment, must be aligned for effective education. Otherwise, there is dysfunction. Further, he argues that if there is dysfunction in one area it spreads to the others, and unless there is full alignment of these systems, reform efforts will be hard to sell to practitioners.” Quoted from John Craig (2007), “Curriculum Synchronicity: Alignment, Improvement and Accountablity,” Australian Council for Educational Leaders (ACEL), October 2007, available at http://www.acel.or.au/conf07/papers/CurriculumSynchronicity.

The author of the misquote is John Craig, one time principal of Blacktown Boys High School (NSW) and Kingsgrove North High School (NSW), a  member of the NSW Principals Council Strategic Futures Group, Executive Team Leader Quality Assurance NSW Department of Education and Training, and Director of Human Resources, Newington College. As a principal, Craig led Blacktown BHS to two Director General’s Awards for curriculum and pastoral care (Craig, 2007).

The truth of the matter is that Allan Luke did not write that “curriculum, pedagogy and assessment must be aligned for effective education.” Read his paper (Luke, 1999)!

Michele Bruniges did a better job of quoting and interpreting Luke’s remarks from his 1999 paper in her presentation to the Curriculum Corporation Conference in Brisbane in June 2005 (Michele Bruniges [June, 2005], “What’s driving curriculum reform in Australia?” a paper delivered to the Curriculum Corporation Conference, Brisbane, June 2005, available at http://www.curriculum.edu.au/verve/_resources/Bruniges_edited.pdf.

“Alan [sic] Luke argues that effective education reform requires alignment of the three key message systems that exist in education – curriculum, pedagogy and assessment. Unless these systems are fully aligned, reform efforts will be hard to sell to practitioners on the one hand, and will be dysfunctional on the other” (Bruniges, 2005).

The most important phrase in the Bruniges quote is “key message systems.”

What Luke wrote about the alignment of curriculum, pedagogy and assessment was

The three message systems – curriculum, pedagogy, assessment – need to be brought into proper alignment for us to get desired educational results and outcomes (Allan Luke, [1999], “Education 2010 and new times, Why equity and social justice still matter, but differently,” Education Queensland Online Conference, p. 4).

A message system is a system of discourse. It is a way of writing (and talking) about a set of entities or a state of affairs. Luke was arguing for coherency and consistency in

  1. the DISCOURSE (“the message systems”) about curriculum,
  2. the DISCOURSE about pedagogy and
  3. the DISCOURSE about assessment.

This makes good sense. The written and spoken discourse about curriculum, pedagogy and assessment is frequently very messy, sloppy, ambiguous, conflated and nonsensical. For people engaged in the day to day task of guiding students in learning something worthwhile, badly written documents urging reform make little sense, offer unsound guidance and present useless distractions from the main game of teaching, studying and learning. It is common sense that documents that purport to be about curriculum, pedagogy and assessment should be using the same terms to talk about the same things in plain language. The language of the documents should be clear, simple and easy to read. The language should be noncontradictory. It should not conflate categories. It should make sense. This is the main message in Luke’s paper (Luke, 1999) with regard to curriculum, pedagogy and assessment. The discourse about them should be clear, consistent and sensible.

But Michele Bruniges moves from that message, i.e. discourse should be clear, unconflated and noncontradictory about curriculum, pedagogy and assessment, to the argument for “aligning assessment and curriculum.” She writes:

“Luke’s argument for alignment [of discourse] is a powerful one. If Australia is to achieve equitable, farreaching high-quality educational standards for all students, just as curriculum is reformed, so too must be the forms of assessment with which it is coupled. A key principle in aligning assessment and curriculum is that the assessment strategy selected must be appropriate to what it purports to measure or describe. The strategy needs to encompass a diagnostic capacity to inform further teacher and learning.”

This is a shift in argument from coherency in the discourse about curriculum, pedagogy and assessment to an argument for logical coherency among the components of a curriculum. And I wish to note that a plan for assessment is not outside of an adequately planned curriculum. On the contrary, it is a necessary component of curriculum. And of course the plan for assessment should be logically coherent with all the other components of the curriculum, e.g. there necessarily must be coherency among

  • the statement of intended learning outcomes,
  • the statement of teaching and study activities,
  • the description of teaching and studying resources,
  • the description of content to be taught and studied,
  • the description of evaluation procedures (including the assessment procedures) for judging the degree to which students achieve intended learning outcomes.

Again, this is common sense. Who would argue for an incomplete, incoherent, contradictory, nonsensical curriculum?

And yet, the call for “an alignment of curriculum, pedagogy and assessment” is incoherent, contradictory and nonsensical. A curriculum is a plan for the teaching and studying of some content with the intention that the students achieve some new degree of knowing, understanding, expertise, improvement of attitudes and embracing of sound values.

A curriculum is a plan that provides the basis for a syllabus (or a set of syllabi). A syllabus is a plan that provides the basis for an instructional unit plan (or a set of unit plans). And an instructional unit plan provides the basis for a set of daily lesson plans. In other words, lesson plans are subsets of a unit plan. Unit plans are subsets of a syllabus. Syllabi are subsets of a curriculum.

The term pedagogy malfunctions in the English language. It is sometimes used to name the process of teaching, and it is sometimes used to name knowledge (warranted assertions) about the process of teaching. Let’s focus on the first common usage meaning of the term pedagogy, viz. the process of teaching. Teaching includes all methods, approaches, strategies and social styles that a person might use in the process of guiding someone to learn something.

The term assessment of learning denotes the process of gathering evidence of a student’s learning. The evidence of learning is the evaluatum. An evaluatum is the thing that is being evaluated, i.e. the thing about which a value judgment is to be made. Assessment is the process of gathering or producing an evaluatum. The evaluatum may be (1) a performance by a student or (2) the product of a performance by a student. The assessment process provides the student the opportunity (1) to display a cognitive performance or (2) to present the product of a cognitive performance.

Assessment is a part of the evaluation process. The evaluation of a student’s learning is the process of making a value judgment about the degree to which someone (a student) has achieved some intended level of knowing, understanding or expertise. The evaluation process is a process of reasoning that includes these steps:

  1. Adoption of a standard or set of standards for evaluating (ranking or grading) the evaluatum.
  2. Operational clarification of the standards. This consists in a set of statements to the effect that if an object O has characteristics C, it fulfills a certain standard S to a certain degree D; that if O has characterisics C’, it fulfills S to a greater (lesser) degree D’; etc.
  3. Specification of the class of comparison.
  4. Determining the good-making and bad-making characteristics of the evaluatum. (In the case of value rankings this would ideally be done for every member of the class or comparison.)
  5. Deducing, from 2 and 4, the degree to which the evaluatum on the whole fulfills or fails to fulfill the standards. (In the case of value rankings this would ideally be done for every member of the class of comparison. The members would then be ranked in order according to the varying degrees to which they fulfilled or failed to fulfill the standards.) [Points 1-5 are quoted from Paul T. Taylor (1961): Normative Discourse. Westport, Connecticut: Greenwood Press, Publishers, pp. 9-10.]

An adequate curriculum includes plans for evaluation of student achievement that are consistent with the other components of the curriculum. To assert that assessment must be aligned with curriculum, is to say something like the part of the curriculum that makes plans for assessment must be aligned (logically coherent) with plans for assessment. Plans for assessment must be aligned with plans for assessment? What kind of tautological nonsense is that?

To make this clear, let’s make some word substitution in the statement,

“Curriculum, pedagogy and assessment must be aligned for effective education.”

  1. Let the term curriculum mean the same as plans for teaching and studying some content with some intended learning outcomes in mind and plans for evaluating student achievement of intended learning outcomes.
  2. Let the term pedagogy mean plans for teaching.
  3. Let the term assessment mean plans for gathering evidence of student achievement of intended learning outcomes.
  4. Let the term aligned mean logically coherent
  5. Let the term effective education mean teaching and studying that leads to student achievement of intended learning outcomes.

Now, make the term substitutions in the original statement,

“Curriculum, pedagogy and assessment must be aligned for effective education.”

The first term substitution produces the statement,

“[Plans for teaching and studying some content with some intended learning outcomes in mind and plans for evaluating student achievement of intended learning outcomes], pedagogy and assessment must be aligned for effective education.”

The second term substitution produces the statement,

“[Plans for teaching and studying some content with some intended learning outcomes in mind and plans for evaluating student achievement of intended learning outcomes], [plans for teaching] and assessment must be aligned for effective education.”

The third term substitution produces the statement,

“[Plans for teaching and studying some content with some intended learning outcomes in mind and plans for evaluating student achievement of intended learning outcomes], [plans for teaching] and [plans for gathering evidence of student achievement of intended learning outcomes] must be aligned for effective education.”

The fourth term substitution produces the statement,

“[Plans for teaching and studying some content with some intended learning outcomes in mind and plans for evaluating student achievement of intended learning outcomes], [plans for teaching] and [plans for gathering evidence of student achievement of intended learning outcomes] must be [logically coherent] for effective education.”

The fifth term substitution produces the statement,

“[Plans for teaching and studying some content with some intended learning outcomes in mind and plans for evaluating student achievement of intended learning outcomes], [plans for teaching] and [plans for gathering evidence of student achievement of intended learning outcomes] must be [logically coherent] for [teaching and studying that leads to student achievement of intended learning outcomes].”

The point of these term substitutions is to demonstrate that the statement,

“Curriculum, pedagogy and assessment must be aligned for effective education,”

is tautological nonsense. It amounts to saying that a plan for C (which already necessarily includes A and B) should include A and B. This is not a useful admonition or guideline for anything, and it certainly is not based on any kind of research.

It would be far more fruitful to say simply that a coherent curriculum plan probably improves the chances for education to be effective. A coherent curriculum plan is certainly necessary for accountability and for rational allocation of resources and effort. If we don’t know what we want to achieve in an educational program, then we have no basis for allocating resources and effort to achieve it. And if we do know clearly what we want to achieve in an educational program, then we need some clear publicly verifiable evidence of our achievement. Otherwise we have no system of accountability and no clear way of knowing whether we are acting wisely and responsibly in allocation of resources and effort.

Back to the statement,

“Curriculum, pedagogy and assessment must be aligned for effective education.”

It just doesn’t make any sense, unless you conceive the term curriculum so narrowly that a curriculum plan excludes plans for teaching activities and plans for assessment and evaluation of students’ achievement of intended learning outcomes. But what sensible teacher or educologist of curriculum would conceive the term curriculum so narrowly?

Yet the statement,

“Curriculum, pedagogy and assessment must be aligned for effective education”

is seemingly etched in stone in Education Queensland curriculum documents, educological publications and university curriculum courses (in Queensland). It is solemnly repeated as received truth and wisdom, even though it is such a silly and patently foolish statement.

Thus, I end in the way I began, with the quote from Bertrand Russell.

“The fact that an opinion has been widely held is no evidence whatever that it is not utterly absurd; indeed in view of the silliness of the majority of mankind, a widespread belief is more likely to be foolish than sensible” (Marriage and Morals, 1929).