Statistics, Analytics, Program Evaluation, and the Great Sell-Out

A few years ago, people were signalling the death of the University because new statistics proved the jig was up. Of course, when you hear things like “Thirty-six percent of the students saw no statistically significant gains in their CLA (Collegiate Learning Assessment) scores between their freshman and senior years”, it does sound pretty bad. The CLA is a “widely-used essay test that measures reasoning and writing skills.” So, in other words, college students are not learning to read or write.

But what if we are looking at the numbers in the wrong way?

Take this real life scenario into consideration: a certain college near where I work increased the student body population from 10,000 students to 25,000 students in less than two decades. A 250% increase. Now consider this: has the quality of the average high school graduate really increased by 250% over the past two decades? Has the quality of the average high school graduate really increased at all? Or have colleges lowered their entrance standards? You will find similar statistics in many of your “growing” colleges and universities today. If you do the math, 60% of those students may have not been ready at all for college when they entered.

I’m not trying to say that they shouldn’t be there – I want to see people educated. But in light of the consideration that maybe up to (at least) 60% of the students in any given college may not have even been prepared to be there, a statistic that says that 36% of them didn’t show statistically significant gains might be a small miracle.

Or maybe not – but if we aren’t looking at bigger picture factors in all this data we are gathering… how do we really know what we are looking at?

I know this is old news to people that are really into research and data… but some of this still seems to be shocking people out there. Today we read that “Students Might Not Be ‘Academically Adrift’ After All, Study Finds.” One of the many interesting points in this article is that the authors’ of Academically Adrift might have been incorrectly “translated by some people in politics to say, ‘College doesn’t matter.'”

Some people? Try a lot of people in key positions. People that are now rushing into certain crazes without having a bit of evidence that they work. Which is not a bad idea at an experimental level, but when entire degrees and millions of dollars are thrown at untested ideas just because people didn’t take a more nuanced look at the numbers? That is very dangerous territory for a field that is already teetering on the edge of obscurity, unable to afford another big blow to credibility.

The time of “chilling out” and just “being happy that there is attention to new ideas” is long past. The whole idea of “don’t look behind the curtain, don’t think critically, just ignore the negative because we all need to be shiny happy people that hold hands and sing kumbaya” never really worked for, well, anyone. It sounds cool to say “don’t bash the movement, just move it in the right direction using positive energy” before the sell-out happens… but it is impossible to accomplish that (and it sounds down-right Great Wizard of Oz-ish to say it) once the sell-out has already happened. And I just don’t see how anyone can look at education now and not see that the Great Sell-Out has already started to happen en masse.

I get that people don’t like that the haters are now tossing out the baby with the bathwater.. but is the only other option to keep the bathwater because we like the baby?

Debunking the Industrial Age School Myth

Something about the argument that our “current school system was created in an industrial era and hasn’t changed since” has always not set right with me. A lot of this has to do with the fact that my high school looked nothing like these videos of students on an assembly line with graduation dates stamped on our foreheads. A decade later, the school that I taught at in a  different city didn’t look like that, either. And even schools that I worked with recently in a different part of the state did not look like that.

But maybe that was just my experience. Maybe I was just hitting the same outlier school every time?

Or maybe it really was that the argument itself was invalid and I just didn’t know enough about history to know why.

Mike Caufield posted today about the problems he has with the current ideas of Sugata Mitra, and in doing so he also shed some light on the real problems with the “schools are industrial-age relic” arguments.

Caufield’s summary of his disagreement really tells the whole picture, but it is worth reading the whole article to look at specific historical reasons why he comes to his conclusions:

The history Mitra narrates is this. There once was a race of Victorians. They built a can opener called education, and nobody has changed that can opener since, even though we no longer eat from cans. But we no longer eat from cans! Give me a million dollars, please.

This is what we hear from so many educational reformers today.  No one would deny that our systems are not working and need to be fixed, changed, or re-built from the ground up. But how can we really know what needs to be changed if we are so ignorant of where we have come from and where we currently stand right now? To extend Caufield’s summary into the world of educational reformers, what we are basically getting is this:

But we no longer eat from cans! We eat from plastic containers. So we are going to change the can opener to a plastic container opener thingee (whatever that is – don’t ask us to explain ourselves okay?). Don’t listen to these people saying they don’t need can openers or plastic containers.  Ignore the man behind the curtain and give me a million dollars, please.

More Useless 2012 Predictions

  • Everything from email to libraries to blogs to universities will be declared dead. Again. For the 10th year in a row.
  • People will continue to call for educational reform. Ignoring, of course, the fact that education is constantly reforming and changing and that there are people out there exploring new ideas and concepts.
  • “Experts” will continue to claim that the lecture model is still dominant at universities, even if they can’t quote any evidence to back this claim up. I counted up all of the courses I took in college in the early 1990s that were lecture based it came out to be less than half. I have heard from current students that, at least at this college, that number has gone way down even since then.
  • Several new LMS options will be labeled “Blackboard killers.” But none will make a dent because labeling any tech a “killer” usually dooms its existence.
  • Even more “experts” will claim that colleges are now irrelevant, despite the numerous studies showing that everyone from employers to future students still think they are highly relevant and necessary. Who needs facts and figures when you just want to grind an ax with a society that won’t pay English majors a seven figure salary right after graduation?
  • Despite overwhelming evidence of the educational value of hybrid or fully online courses, many organizations will develop a case of amnesia and claim there isn’t any evidence. I’m looking at you, Idaho.
  • All of us will suddenly remember that we haven’t logged in to Second Life in over a year and then collectedly feel guilty for letting such a great tool slip away.
  • The American people will get so tired of hearing about new technology lawsuits every day that they will write really extra terse Tweets about the big companies. But of course not do anything to stop the insanity of this whole patent lawsuit mess. Really Google, Apple, Motorolla, and others… its getting old.