Month: January 2015

The Mirage of Measurable Success

The last post that I wrote on measuring success in MOOCs created some good, interesting conversation around the idea of measurable success. The most important questions that were asked had to deal with “why even offer dalmooc if you don’t know what measurable success would look like?”

That’s a good question and one that I think can be answered in many ways. Honestly, the best answer to that question is “because four world-renowned experts wanted to teach it and a lot of people signed up to take it.” To me, especially in the informal realm of education where dalmooc existed, that is one of the biggest measurable signs of success. We live in a world that is so full of compulsory education and set degree plans that we forget that choosing to sign up for an informal voluntary learning experience is measurable success in itself. Over 19,000 people initially said “that sounds interesting, sign me up,” with over 10,000 signing in at one point or another to view the materials. Hundreds of participants were active on Twitter, Facebook, EdX forums, ProSolo, Google Hangouts, and other parts of the course. All voluntarily.  To me, that is measurable success.

Another area of measurable success, although definitely more on the qualitative side, is what I covered in the last post:

So maybe when the participants use words like “blended” or “straddled” or “combined,” those statements in themselves are actually signs of success. In the beginning, some claimed that DALMOOC was more cMOOC than xMOOC, but by the end others were claiming it was more xMOOC than cMOOC. Maybe all of these various, different views of the course are proof that the original design intention of truly self-determined learning was realized

To clarify this a bit more, there are those that thought that dalmooc was more instructivist / xMOOC:

https://twitter.com/cpjobling/status/555074861320921088

And then there were those that thought it was much more connectivist / cMOOC (myself included)

So to me, that is another realm of measurable success – learners came out of the experience with vastly different views on what happened. That was a goal we had.

However, I know that when people talk about “measurable success,” they are usually referring more to standardized test results, student satisfaction, completion rates, and – the holy grail of education – grades! The elephant in the room that many people won’t deal with, but we all know is true, is that these measures of success are often a mirage.

Standardized tests are probably the biggest mirage of all. The problem is that a score of 90% on a test really only means that a learner was able to mark 90% of the questions correctly, but not necessarily that they actually understood 90% of the material. They may have only understood 60% of it and guessed the next 30% correctly. The fact that the right answer is somewhere in a list of multiple choice answers should negate their usefulness as a way to measure success, but our society still chooses to ignore this problem. Then you can add into this mix that most multiple choice questions are poorly written in ways that give away the answers to people that are taught how to game them (like I was).

Then there is the problem of coming up with questions for tests. Some tests contain, say, two questions about the core knowledge that learners should have gained and then a whole lot of related trivia that they could just Google if needed. Yet they could still get the two essential questions wrong and all the rest correct and will be labeled as “mastering” the concept. Rubrics for papers or projects often do the same thing – giving most points to grammar and following instructions and few to actual content mastery. Someone could write a great paper that shows no knowledge of the topic at hand but still pass because they got all other areas perfect.

Add to this that we would compare two children to each other based on this false sense of “success.” One child could have tanked a test based on the trivia but got all of the core content correct and still be labeled as less successful than the one that got the trivia right and core knowledge wrong…. just because its all on the same test. Oh, and let’s not forget the practice of giving similar or equal weights to all questions on a test when not all questions are really equal. Again, two learners could get the same score, but one only answered the easy questions correctly while the other answered all of the challenging ones correctly.

And speaking of different learners, there is always the oft-ignored problems of cultural bias in testing and learning.  Are learners not testing well because they didn’t learn, or were there cultural references on the test they didn’t get? Did a learner really learn the content, or were they just able to quickly memorize some factoids because of some weird thing Aunt Ida said about planets that helped them connect the new information to this weird family quirk? Are they being labeled as smarter because they are or because their weird Aunt Ida gave them a memory that helped them memorize?

Most of what we call “measurable success” in education is really just a mirage of numbers games. For those like me that fell on the privileged side of those games, it was a great system that we probably want to fight to keep. And we are probably most likely the ones now in control, so….

Now, of course, this is not to say that learning isn’t happening. This is more about how most institutions measure learning and success. I believe people are always learning formally and informally, even if its not always what they had intended to learn. It just takes a lot of time, effort, and money (yes – money!) to truly assess learning, and the educational field in general is being tasked with the opposite. “Do better assessment with less money, less time, and less effort (ie people power)!” No real easy answers, but there is a problem with the system and the culture that drives that system that needs to be addressed before “measurable success” becomes a trustworthy idea.

Measuring Success in MOOCs (or More Specifically, DALMOOC)

I was asked last week how we knew whether or not DALMOOC was successful. Seems a fair question since “success” in MOOCs seems to be measured by everything from completion rates to enrollment numbers to certificate numbers to alignment of the stars on the Wednesday of the first week the MOOC is offered. I had to be honest about that question and say that since everyone that work on DALMOOC lived and are still on speaking terms (at least as far as I know), then we were successful. Running a MOOC can be far more intensive and stressful than many people realize. We almost didn’t make it out the other side with everyone still happy.

In some ways, when I see people saying that we were blending xMOOCs and cMOOCs, or combining the two, I think we might have failed in the communication of what we were doing. Maybe I can blame that somewhat on our current educational system that only thinks in linear ways; therefore any course with more than one layer is not scene as complex, but “blended” or “combined.” Words like “straddled” seem closer, but to be honest we didn’t feel the need to straddle. We just had two layers that were not walled in, allowing learners to choose one or the other or both or to move back and forth as they felt. Infinite possibilities. A Multiverse Open Online Course maybe even. But not really a linear mix of the two from the design side.

Of course, the learner experience was linear even if they skipping back and forth between both layers. So maybe when the participants use words like “blended” or “straddled” or “combined,” those statements in themselves are actually signs of success. In the beginning, some claimed that DALMOOC was more cMOOC than xMOOC, but by the end others were claiming it was more xMOOC than cMOOC. Maybe all of these various, different views of the course are proof that the original design intention of truly self-determined learning was realized. At the very least, DALMOOC feedback was an interesting study in how bias and ontologies and epistemologies and all those -ologies affect participant perception of design. Maybe it doesn’t matter than participants can’t articulate the basics of dual layer architecture because the point all along should have been to hide the architecture and let learners learn how to learn.

So, at the end of the day, I will be able to come up with some philosophical jargon to “prove” that DALMOOC was a success to the powers-that-be who ask – all of which will be true. But to be honest, the only thing I really want to do is shrug my shoulders and say “beats me – go look the participant output itself and see if that looks like success to you.”

Looks like success to me.

The Underlying Barrier to Education Reform

The last few weeks Dave Cormier has been on a roll addressing some of the bigger issues in education (the system of education, as opposed to “learning” that is always happening because of life). When Cormier writes about the need for learners to care about learning, this idea is at the core of why I support heutagogy (learning how to learn) so much. One of the foundations of learning how to learn is to care about learning. But then Cormier asks some questions that I think highlight a glaring problem that is hindering education reform today:

How do we make a minister of education happy about that idea? How do we convince parents that the way a kid feels about learning is more important than what they learned? How would we teach learning? Oh my gosh… how would we assess it? How, inevitably, do we bureaucratize it?

Do you see the big fear-driven problem glaring at us all from behind these questions? Let me take a look at this from another angle then. Often we hear education critics say that “students make it out of a degree without learning anything” or that we have to “prepare students for jobs that don’t exist yet.” But when I hear these statements, I feel they show that we are still not fully understanding what is happening with education right now. What we really mean by these statements is that students graduate without being able to score the same passing scores on tests that they scored while in school, or that we did not fill their heads with the factoids that will only exist in the future once new job titles are created. We really have no idea if they learned or were prepared or not – we just know they don’t test as well as we want them to.

In other words, we’re still critiquing education based on the problem of education in 1870 that Cormier explains in his post linked above… but not based on where the world is today. We say that our schools are not modern, but then we say that our educational systems are failing because “students can’t regurgitate factoids on a test or in a paper.”

I still remember when I fired up my first blog and the first blog post rolled out with ease. And then another, and another. No one taught me how to write a blog post, and I certainly don’t remember ever being tested on this kind of writing… so where did it come from? Then I remembered a high school English teacher that had us write hundreds of words daily on each chapter we read in each book we read. Then I got it – that tedious lesson was not about those books, but practicing how to write freely about whatever came to mind. I was never tested on that skill…. so I never counted it as “learning” since it was never on the test or final paper.

Then there was the first time I tried PHP. I cruised through the basic and intermediate lessons with ease, realizing that all that time digging into Algebra problems solving for X was not about finding a number, but digging through problems mathematically to figure out what was missing. They just had to give me the Algebra tests to prove that I had that skill because they needed numbers to prove it.

However, I could also give you pages worth of educational activities that did not work well at all for me – I don’t want to pretend like there are no problems or that our systems are mostly okay. The point I am getting at here is that learning may or may not occurring in formal education… but we would never really know either way because our extreme focus on standardized testing is making us lose focus on what is really happening. What is driving this extreme focus?

I would submit that one major factor is fear of failure. “Failure is not an option!” Well, actually its a part of life, and a great learning experience on top of that. But we are decades into a system that breeds a fear of failure at all levels.

This fear starts with the classroom level, where good lessons on self reflection are just side activities to build writing skills (in order reach a level of passing a standardized test on parts of language, of course) rather than solid activities in and of themselves. Because we so fear failure as a culture, we have to let all learners know how they measure up against the norm. We have to grade everything. And we rarely stop to ask if these grades really mean anything. Of course, why a 70 is passing and a 69 is failing is as much a mystery as anything else. I would like to say standard deviations and all of that, but seeing that we try to get all learners to cross that passing line – so much for that.

Let’s not even go into how one child scoring a 95 on one test might actually not know the content as well as one scoring an 89. They could have just gotten lucky that day.

But here’s where it gets worse: we have to prove that a failing student is really failing to the student, the student’s parents, and the school system in general. We are so afraid of failure that we have to get who passes and who fails right… so more standardization. Not only that, we identify the quality of teachers based on pass/fail scores. Oh, and we also compare school to school, state to state, and country to country with these numbers… all because we are so afraid of failing that we have to have good enough numbers to prove that we aren’t.

So, back to Cormier’s questions: how do we get the system on board with true education reform? We have to end this fear of failure and the ensuing drive that creates a system where everything is compared against a standard of failure. At some point I would love to see a system of personalized learning that embraces failures as just another learning opportunity (and only considers failures based on personal learning goals and not the ability to get a certain score on an assignment). But this fear of failure that creates the need to set up a system that unfairly pushes all learners into the same mold (or one of six variations on this mold that we currently label “personalization”) is going to be a significant barrier. How do we undo decades and centuries worth of ingrained “grades” and “failures” and “top ten percents” and all of that kind of stuff that our systems are built on? There’s an inherent power structure there that favors the quick and early bloomers… who might not want to give up the power that this system affords them.

I have no answers there, but I do foresee some messy fights as the old system erodes and the new one raises. I don’t believe in disruption of systems as much as I believe in evolution, so here is hoping that the new system that evolves is not a regurgitated 1870s system that we got back when the system of the 1980s started evolving in the late 1990s.