Educational Research

There are a variety of important issues in regards to educational research these days. One hot topic right now is that our current federal administration has restricted the definition of exceptable research to only one type of research design. This design is known as the experimental design. Qualitative research, which allows us to look at what actually goes on in classrooms and schools, and with children, as well as at the process of how education is working, is not deemed acceptable. Neither are other designs of quantitative research, which might examine a particular school or setting or situation, without having a matched control group. This decision to only allow this type of research does not come from any consensus within the scientific or educational research community as to what counts as research(1). It is a political decision by the current federal administration. This policy has important implications for our schools. One implication is that it highly influences what research gets done. It does so directly by the fact the government sponsors research. Federal dollars will only sponsor research that fits the administration’s definition. It affects schools secondarily by what research they cite and use for their policy decisions. Researchers who want their research to influence these policies are likely to adhere to those protocols. Third, outside researchers and universities may decide to only fund research that follows that research paradigm, again restricting what research gets done. It also affects in some cases what practices schools may use, as the federal government insists that it only fund “scientifically proven” methods. Federal monies for curriculum and instruction are therefore funneled to areas that are supported by this one particular type of research.

I raise the above issue of what counts as research to make a point about educational research in general. This point is about the limitations of much of the research that is done and has been done in education even before the current policies. The above policies will only exacerbate the one’s I will address below.

Two difficulties that I will address here in regards to interpreting educational research are, one: what was used to measure the effects; and two: over what period were the effects measured.

Most educational research uses standardized tests to measure the success or failure of a particular program, or method or other variable of interest(2). However, the validity and reliability of these tests as actual measures of what they purport to measure is highly controversial(3). I will use the example of reading. I recently went to a talk about the research on learning to read. The presenter argued that the research showed that phonemic awareness was required to learn to read. However, the research cited actually showed that the explicit teaching of phonemic awareness and phonics helped students to score higher on tests of phonemes and phonics! This has been part of the trouble with the debate between whole language versus phonics and “phonemic awareness” advocates. Whole language theorists tend to use measures such as comprehension, reading for pleasure, and quantity of reading as their measures of success. Phonics and phonemic awareness advocates tend to use standardized tests that focus on phonics and phonemic awareness skills as their measures of success. How they define “reading” and how they measure reading end up predicting the outcomes they are looking for! According to Elaine Garan(4), a member of the National Reading Panel, the panel made this error in its recommendations—in limiting its analysis to studies using the experimental design, and focusing on experiments that looked at reading sub-skills, it biased its own conclusions.

Similar scenarios occur across many areas of educational research. It is not that research can say anything, but that one must be careful to examine how the researcher defined and measured success of the variable they claim to be examining. The reader and user of the research must be able to decide if they agree with the researcher’s definition, and whether the tool used to measure it is valid according to that definition.

The second problem is with the short-term aspect of most educational research. Most research is done over a one school year or shorter duration. There is an assumption that if gains are shown, they will persist over time. However, much of what we know from experience and other research contradicts that assumption. I refer us here to the “Three Little Pigs” analogy. Let us say we decide to study what materials are best for building houses. We have three identical pigs, all building houses. We notice one is building his house from straw, another from sticks, and a third from bricks. First, what is our measure of success? It is going to be how far has each pig gotten in building his house. After day one, we look to see how far each has gotten, and we notice the pig who is building his house from straw is already done. The one using sticks has his walls mostly up. The pig building with bricks is just getting his foundation done. We conclude straws much be the best material for house building, and mandate straw—based on research!

As most of us are aware, we often forget what we learned in a class or course soon after the class is over, or even during the class, right after the test! Short terms gains often do not correlate to long-term gains. Sometimes it is just do to lack of use—the knowledge or skills learned are not used again, and therefore we don’t remember them. Sometimes it may be that a strong foundation was not built, and so, like the straw house, our understanding collapses easily when it needs to support more complex use or understanding. Researchers Wayne Thomas and Virginia Collier(5) have shown evidence of this particularly in language learning, where English-only methods show slight gains in early language learning for English language learners, but students in bilingual classes overtake them in later years, due to, according to language theory, a stronger foundation in their primary language. Research on developmental versus skills based approaches to early childhood education have shown similar patterns. Early academic advantages for skills based approaches are lost over the years to longer-term advantages for the developmental approaches(6).

A main reason for this problem is summed up in an old joke I will repeat here:

 It is late evening and a woman sees a man on the street by a lamppost who looks like he is searching for something. She asks him if she can help.

He responds, “Yes, I dropped my keys.” Together they continue to look for a while. Finally, as they are having no luck, she asks him if he can remember, when and where he last had them, so they might narrow their search.

He tells her, “Oh, yes,” points across the street, and says, “I dropped them over there somewhere.”

“Then why are we looking here?”

“The light is better”

Short term designs and standardized test measurement is the lamppost. It is very difficult to carry out long-term research. It is expensive, so funding is difficult. The researcher must commit to the long haul. They may need a team who can also commit this time. The “subjects” are hard to keep track of as years go by. And the variables get more complex as time passes. At the end of a school term, or of our test of the method, we can be fairly sure that the large majority of our subjects will be right there in the same place for us to administer our tests.

Standardized tests are given to virtually all students, can easily be compared across students, classes, schools, even districts or possibly states. Even if the standardized tests are particular to the study, they tend to be quicker, easier and less expensive to administer than other measures. They are also easier to run statistical analyses on.

However, what good does it do for me to know that “such and such” a reading series or teaching method led to higher test scores for these second graders, if there is no evidence that these higher test scores actually lead to an adult who reads, understands what they reads, and knows how to use what they read to better their life and their society?

As they say “Garbage in, garbage out.” All of the advantages of time and money and statistical reliability do not matter if they will not really answer the questions we want answers to. If what I want to know is: will what am studying lead to a better educated citizen?, then I better make sure that the tools I use to measure that really do measure it.

Now I come back to my original discussion of what counts as research by the government. The federal government defines research only as the experimental design. This design lends itself well to short-term research using quantifiable scores, such as those of standardized tests. The second issue—what counts as evidence—is also more restricted. It is especially difficult to get long term research to fit the experimental design, as following exactly matched groups over years becomes more and more difficult as time passes. Many questions cannot be studied using matched samples, as in many instances it would be unethical to randomly assign students to different groups. Should we randomly retain some students and not others to see the effects of this policy? In other cases, it is impossible. For instance we cannot clone a school or district and recreate the exact same situation if we want to understand policy or curriculum decisions made on that scale. What makes for an educated and successful citizen is not always easily quantifiable, and definitions vary. Therefore, the narrow type of research the government allows also restricts what types of questions even get asked by the research.

It is my contention that although the experimental design in research is commendable and valuable where practical, it can never be the only model of research to answer the complex questions about human learning and behavior. To answer such questions we must use the broader definition of research that virtually all scientific disciplines understand.

If we want to answer important questions in education we are going to have to find a way to fund long term research, and use more complex measures of success that are more closely aligned with the actual skills and knowledge that successful members of society need and use.


1. Debra Viadero, “AERA Stresses Value of Alternatives to ‘Gold Standard’,” Education Week, April 18 2007.

2. Deborah W Meier, “Needed: Thoughtful Research for Thoughtful Schools,” in Issues in Education Research, ed. Ellen Condliffe Lagemann and Lee Shulman (San Francisco: Jossey-Bass, 1999).

3. Alfie Kohn, The Case against Standardized Testing: Raising the Scores, Ruining the Schools (Portsmouth, NH: Heinemann, 2000), Deborah W Meier, In Schools We Trust: Creating Communities of Learning in an Era of Testing and Standardization (Boston: Beacon Press, 2002), Susan Ohanian, One Size Fits Few: The Folly of Educational Standards (Portsmouth, NH: Heinemann, 1999).

4. Elaine M. Garan, “What Does the Report of the National Reading Panel Really Tell Us About Teaching Phonics,” Language Arts 79, no. 1 (2001).

5. Wayne Thomas and Virginia Collier, “School Effectiveness for Language Minority Students,”  (Washington, DC: National Clearinghouse for Bilingual Education, 1997), Wayne Thomas and Virginia Collier, “A National Study of School Effectiveness for Language Minority Students’ Long-Term Academic Achievement: Executive Summary,”  (Washington, DC: Center for Research on Education, Diversity & Excellence, 2002).

6. Rebecca A. Marcon, “Moving up the Grades: Relationship between Preschool Model and Later School Success,” Early Childhood Research & Practice 4, no. 1 (2002), Jeanne E. Montie, Zongping Xiang, and Lawrence J. Schweinhart, “Preschool Experience in 10 Countries: Cognitive and Language Performance at Age 7.” Early Childhood Research Quarterly 21, no. 3 (2006): 313-31.

School Reform: Where is the Evidence?

Under the Bush administration, the rhetoric is that the decisions we make in schools should be based on “scientific” evidence. Not only must it be “scientific,” according to the administration, but it must be based on the controlled experimental design, which is actually just one acceptable form of evidence within the scientific paradigm. No actual scientific field relies exclusively on this one form. However, putting that aside, even accepting a broader range of scientific evidence, the basic tenets mandated by the No Child Left Behind Act (NCLB) are not based on any empirical evidence, controlled experiment or not(1), and many, as I will outline, are in contradiction to accepted educational and organizational theory.

What are some of these mandates that I am referring to? High stakes testing, external tutoring programs, state takeover or charter school reform for “failing schools,” are the ones I will discuss in this column.

High Stakes testing has been around for a long time, and each time it is used it tends to show gains in test scores in the early years which quickly flatten out. Long term educational improvement of any sort has never been demonstrated. NCLB is different in that the stakes are quite higher than in previous reforms, so many argue that previous evidence(2) is not valid. However, the best that that leaves us with is an untested experiment on a massive scale, affecting nearly every public school child in the nation. I won’t even discuss here the massive amounts of monies going to the corporations the make these tests. They get money for developing the test, then selling the tests to the schools, and then for scoring the tests. Then they develop curriculum to help students prepare for these very tests that they design so schools can boost their test scores.

Another major feature of NCLB is that schools that do not reach the required test score goals must offer children tutoring that is done by an outside agency. The theory is that if the school failed the children, they are obviously not qualified to help these children. There is some logic to that theory. However, again, there is no evidence that outside agencies, as a generic category, are better equipped to help failing students than the public schools themselves(3). The administration did not first pilot this approach in some places, and test it against in-house support to demonstrate that it was more effective. Therefore, this mandate is another massive untested experiment, moving enormous Federal dollars from the public to the private sector.

If schools continue to fail to reach mandated test score goals (with rising moving targets—every year a larger percentage of students are required to “pass” the test), then they can be taken over by the state or turned over to private charter agencies. What is the record on this? School districts have been taken over by city or state governments in the past. In California, Compton, and recently Oakland have been the targets of such take-overs. In neither case have there been any significant changes in the education students receive. State governments, not surprisingly, have shown no more capability for creating positive educational changes than the local bodies they replaced. In fact, one would be hard pressed to find a theory to support why one would expect them to(4).

Charter schools, which began as an experiment in the early 1990s, and quickly spread across states and cities nationwide, were based on a theory that more freedom from state regulations and forcing local public schools to compete for students would create educational innovations and improvements. This is based on the market theory. This is a reasonable theory, especially in a country whose economy is based on such a theory. In fact many charter schools are exciting places, with innovative pedagogy showing successful results. However, after extensive research, charter schools as a class, have shown no higher test scores than their public school counterparts(5).

Another possibility in some states is the use of vouchers to send children to private schools. However, again, if you hold demographic variables constant, even private schools show no better results on standardized test scores than do public schools(6). If we are supposedly doing this reform in the name of accountability, private schools have no accountability either to state governments nor their local constituencies. There are no public school boards nor open meetings laws required of private schools, nor are their financial records open to public or government scrutiny. Once more, this aspect of NCLB is based on a theory which current evidence does not support. In most states private schools do not have to administer the same standardized tests that NCLB holds public schools accountable to. While public schools, who are answerable to the public directly, are not trusted without such tests, for some reason, private schools do not need to demonstrate any such accountability.

Is there evidence for other ideas? There is something that schools that have made a significant and dramatic difference for students have in common—local control. Some of the most effective schools are those where the people closest to the kids—the teachers, parents, and community—are actively involved in deciding the mission and curriculum of the school. It appears to matter less what that curriculum and vision is than that it was made by those closest to the kids. Virtually all of the reforms being called for at the sate and national level are based on a profound mistrust of those very people. Yet it should be obvious that when people feel coerced they are less likely to work efficiently. When people feel empowered, they are most effective. The evidence bears this out. Find a school that has significantly beaten the odds with low-income and minority students, and I’ll bet it did not happen based on external mandates! Progressive examples such as the Central Park East schools in New York, as well as models based on more conservative ideas, such as the KIPP academy and Core Knowledge demonstrate this. Not only that, but it honors our democratic ideals. Democracy is based on  the absurd idea that all citizens are capable of making the important decisions in the public sphere and should do so an equal basis. While it is absurd, no one yet has devised a better alternative.

In terms of a particular approach to learning, there have been a number of longitudinal studies showing the success of progressive and developmental approaches to teaching and learning. These are forms of teaching and learning that are the opposite of the scripted teacher-centered approaches mandated in schools that fail to meet the standardized test score targets required under NCLB. The famous Eight-year study, started in the 1930s, which followed students from their freshmen year in high school to four years after graduation found that those in the progressive schools did better on all significant measures, both in high school and in college, than their matched counterparts(7). The more extensive the reforms, the more impressive the results. Despite these dramatic findings, the public mood had shifted away from such innovations, and the results were ignored after they were published. Another more recent example is the Central Park East schools (both elementary and secondary schools) in New York City, and their resounding success of working with poor minority students in East Harlem, with 80 to 90% of the graduates getting into and being successful in four year colleges. Yet these schools are under constant attack to discontinue their innovative approaches(8). A couple of recent studies of preschool practices, comparing developmental child-centered approaches against academic skills based approaches have shown better academic and social outcomes in later elementary grades for those in the child-centered developmental programs. One of these was done in Florida(9), and the other an international study covering over 5,000 students in 1,800 preschool setting in 10 different countries(10).

The reforms of NCLB are based on a premise that those closest to the children should not be trusted to make the important decision about their education. The teachers should not be trusted to make the important decisions about how to teach the children, and the parents should not be trusted to govern the schools locally. It is based on a theory that unless coerced, these parties will not act in the best interest of their own children. It is based on a theory that unless coerced, children are not interested in learning. This is in direct contradiction to the basis of democracy. Democracy is based on the theory that no one is better positioned nor has more of a right to make decisions over their own lives than those most directly effected.

As children spend twelve or more years incarcerated in these institutions called schools, which are becoming more and more anti-democratic, our children are losing the one public place where they might learn what it means to be citizens in a democracy, where they might experience democracy in practice.

If you believe as I do that NCLB is counter to the educational needs of our children and the democratic needs of our society, at a minimum let your state and federal representatives know, as NCLB is up for reauthorization very soon. Unless they hear otherwise, these legislatures will take the politically safe course and not make any significant changes. If you would like to be more involved see my links page for some organizations that are working actively on this issue, such as The Forum for Education and Democracy, and FairTest.


1. Gerald W. Bracy, “Things Fall Apart: NCLB Self-Destructs,” Phi Delta Kappan, February 2007.

2. A.L. Armrein and David C. Berliner, “High-Stakes Testing, Uncertainty, and Student Learning,” Education Policy Analysis Archives 10, no. 18 (2002).

3. Bracy, “Things Fall Apart: NCLB Self-Destructs.”

4. Kenneth K. Wong and Francis X. Shen, “Measuring the Effectiveness of City and State Takeover as a School Reform Strategy,” Peabody Journal of Education 78, no. 4 (2003).

5. Katrina Bulkley and Jennifer Fisler, “A Decade of Charter Schools: From Theory to Practice,” Educational Policy 17, no. 3 (2003).

6. Sarah Theule Lubienski and Christopher Lubienski, “A New Look at Public and Private Schools: Student Background and Mathematics Achievement,” Phi Delta Kappan, May 2005.

7. Wilford M. Aiken, The Story of the Eight-Year Study (New York: Harper and Row, 1942).

8. David Bensman, Central Park East and Its Graduates: Learning by Heart (New York: Teachers College Press, 2000).

9. Rebecca A. Marcon, “Moving up the Grades: Relationship between Preschool Model and Later School Success,” Early Childhood Research & Practice 4, no. 1 (2002).

10. Jeanne E. Montie, Zongping Xiang, and Lawrence J. Schweinhart, “Preschool Experience in 10 Countries: Cognitive and Language Performance at Age 7,” Early Childhood Research Quarterly 21, no. 3 (2006).

What is Education For?

Our nation is preoccupied with the issue of improving our schools. Claims of falling standards and achievement abound (these claims are dubious if one actually examines the data. Richard Rothstein’s book The Way We Were is an excellent refutation to that claim). Even if one accepts that the educational quality of our schools is lacking (which I accept, even if not the claim that it is declining), an important point is often left out of this issue: What are we educating for?

There appear to be two main goals that drive current educational reforms. Implicit in current reforms is that higher test scores equal better educated citizens. I come to this conclusion since it is only such test scores that are used to rate states, districts, and schools, and even nations as to their educational success. And at that, it is often only test scores in Language Arts and Mathematics that are examined. By relying exclusively on standardized test scores, it is implicitly saying that those are the only important goals. We see the effect of this when other subjects receive little or no attention, as is true in many elementary schools. Subjects such as arts, music, and even social studies and science are often ignored in elementary schools. This is especially true of schools serving the poor and students of color. It has gotten to the point that some schools are doing away with recess, and even being built without play areas (while we regularly read about the obesity epidemic sweeping our country)! It is not only the subjects that are taught, but how they are taught that this goal affects. The ability to think deeply, critically and creatively, to put the knowledge to use, is often ignored, as those skills are not directly tested, and teaching to such abilities is seen as taking away time that could be used to prepare students for the tests. Again, this is especially true for those not from the dominant culture, or those who are poor—those who are more likely to fail the tests.

The second goal is the rhetoric we hear from government, corporate, private think-tanks and media sources. These groups almost exclusively connect educational attainment to the national economy and to personal economic gain. However, even if we agreed with those aims, the claims as to the cause-effect relationship between education and the economy are somewhat questionable. The claim that increased schooling in a developed country improves the economy is not an excepted theory among educational economists. In fact, it is likely to be the reverse. Schooling responds to needs in the economy, and not the the other way around. When the computer industry boomed, lots of students went into the computer sciences. When the bust years came, they didn’t, and those that had often found themselves unemployed. Training more highly qualified engineers will not help them compete with engineers in India who will work for $7,500 a year, at a time when even highly technical jobs can be outsourced.

Will more schooling at least help one individually? There is strong evidence that years of schooling and degree attainment is highly correlated to income. There is some evidence that at least part of that correlation is not causal—that is, it may be that those who come form higher socioeconomic backgrounds are likely to have both more schooling and higher incomes, and it is their background that is the cause. However, even if we accept the premise that it is causal, at least to some degree, it is not clear that it is the content or quality of schooling that matters. Schooling may simply be a sorting mechanism for employers to screen applicants. This is the meritocracy argument. The best will rise to the station in life that they deserve. It is not clear if the actual knowledge and skills learned in school directly relate to the knowledge and skills needed in the workplace. Ask your doctor how much of medical school was useful? Or a lawyer about law school. Or a teacher about their education courses. Most will tell you that very little was useful, and the real skills were learned on the job. More schooling may be needed in our economy due to the fact that as more people graduate from high school, employers need to up the ante, and require a B.A. to differentiate candidates and reduce the pool. As more people have B.A’s, the same employers start asking for at least an M.A. to apply.

There is considerable debate whether schools do this sorting fairly. There is considerable evidence that again, students of color and the poor suffer discrimination in this system. Even if they don’t, should it be the job of a public institution to sort children for the sake of private employers? Is it the right of the government to force all citizens to take part in this sorting mechanism? Does it serve a compelling public (rather than private) purpose? None come easily to my mind.

Let us now examine the purpose of schooling as giving students the knowledge and skills for employment. Even if schools did do this, should it even be the job of public schools to train workers for private industry? What gives the government the right to compel 13 years of attendance in school during such a precious period in one’s life, if it is solely to meet the needs of the private sector? Why shouldn’t this be an individual choice, paid for either by the families who valued such training or the companies who wanted such workers?

Having read up until now you might think I am questioning the usefulness of public school. You would be wrong. I am questioning the current implied and stated goals. If we look back historically, the arguments for public schooling were made in large part based on two other goals. Going back to Thomas Jefferson, many argued that a democracy required an educated populace that could weigh evidence and make informed decisions. As Dewey said, if we are all the ruling class (which is the assumption of a true democracy) then we all need an education worthy of the ruling class. Anything less would be anti-democratic. Another argument was that in a country made up of people from so many different cultures and backgrounds, we needed a place where they would all become citizens of the United States and learn to accept each other’s differences. As an advocate for democracy, it is these latter goals that I find more convincing.

Such goals imply a different kind of learning and teaching than is common in many public schools. It requires focusing not on the rote learning of basic skills and the memorization of historical and scientific facts, but rather the ability to use those skills and facts to weigh evidence, come to conclusions, better understand one’s world, and even to take action, action that both fulfills one individually but also helps the nation or world improve upon our democratic ideals. We don’t know exactly what knowledge and skills will be needed either for the economy of the future nor to solve the problems that our society will face. We do know it will take the ability to work with others, find the necessary information, and to think both critically and creatively. It will also require tackling consciously and overtly issues of difference; of being able to take on different perspectives, understand other points of view, and to have empathy for those different than oneself. This type of education needs to go beyond the idea that we all should respect each other despite our difference, but to understanding the roots, causes and costs of prejudice and discrimination. It needs to get at not just the past wrongs that have been overcome, but the ongoing problems in our society and the world at large.

It is the goal of educating our youth for their place in a pluralistic democratic society that, for me, is a compelling reason to have a public school system for all of our children.

If you would like to join me in working toward these goals, you might inform yourself about, and join, one or more of the educational reform organizations featured on my Education Links page.