EduGeek Journal

Proud Sponsor of Your Future

Tuesday, January 20, 2015 (2:17 pm)

Matt CrosslinThe Mirage of Measurable Success

Posted by: Matt Crosslin In: Open Learning|Systemic Change

The last post that I wrote on measuring success in MOOCs created some good, interesting conversation around the idea of measurable success. The most important questions that were asked had to deal with “why even offer dalmooc if you don’t know what measurable success would look like?”

That’s a good question and one that I think can be answered in many ways. Honestly, the best answer to that question is “because four world-renowned experts wanted to teach it and a lot of people signed up to take it.” To me, especially in the informal realm of education where dalmooc existed, that is one of the biggest measurable signs of success. We live in a world that is so full of compulsory education and set degree plans that we forget that choosing to sign up for an informal voluntary learning experience is measurable success in itself. Over 19,000 people initially said “that sounds interesting, sign me up,” with over 10,000 signing in at one point or another to view the materials. Hundreds of participants were active on Twitter, Facebook, EdX forums, ProSolo, Google Hangouts, and other parts of the course. All voluntarily. To me, that is measurable success.

Another area of measurable success, although definitely more on the qualitative side, is what I covered in the last post:

So maybe when the participants use words like “blended” or “straddled” or “combined,” those statements in themselves are actually signs of success. In the beginning, some claimed that DALMOOC was more cMOOC than xMOOC, but by the end others were claiming it was more xMOOC than cMOOC. Maybe all of these various, different views of the course are proof that the original design intention of truly self-determined learning was realized

To clarify this a bit more, there are those that thought that dalmooc was more instructivist / xMOOC:

And then there were those that thought it was much more connectivist / cMOOC (myself included)

So to me, that is another realm of measurable success – learners came out of the experience with vastly different views on what happened. That was a goal we had.

However, I know that when people talk about “measurable success,” they are usually referring more to standardized test results, student satisfaction, completion rates, and – the holy grail of education – grades! The elephant in the room that many people won’t deal with, but we all know is true, is that these measures of success are often a mirage.

Standardized tests are probably the biggest mirage of all. The problem is that a score of 90% on a test really only means that a learner was able to mark 90% of the questions correctly, but not necessarily that they actually understood 90% of the material. They may have only understood 60% of it and guessed the next 30% correctly. The fact that the right answer is somewhere in a list of multiple choice answers should negate their usefulness as a way to measure success, but our society still chooses to ignore this problem. Then you can add into this mix that most multiple choice questions are poorly written in ways that give away the answers to people that are taught how to game them (like I was).

Then there is the problem of coming up with questions for tests. Some tests contain, say, two questions about the core knowledge that learners should have gained and then a whole lot of related trivia that they could just Google if needed. Yet they could still get the two essential questions wrong and all the rest correct and will be labeled as “mastering” the concept. Rubrics for papers or projects often do the same thing – giving most points to grammar and following instructions and few to actual content mastery. Someone could write a great paper that shows no knowledge of the topic at hand but still pass because they got all other areas perfect.

Add to this that we would compare two children to each other based on this false sense of “success.” One child could have tanked a test based on the trivia but got all of the core content correct and still be labeled as less successful than the one that got the trivia right and core knowledge wrong…. just because its all on the same test. Oh, and let’s not forget the practice of giving similar or equal weights to all questions on a test when not all questions are really equal. Again, two learners could get the same score, but one only answered the easy questions correctly while the other answered all of the challenging ones correctly.

And speaking of different learners, there is always the oft-ignored problems of cultural bias in testing and learning. Are learners not testing well because they didn’t learn, or were there cultural references on the test they didn’t get? Did a learner really learn the content, or were they just able to quickly memorize some factoids because of some weird thing Aunt Ida said about planets that helped them connect the new information to this weird family quirk? Are they being labeled as smarter because they are or because their weird Aunt Ida gave them a memory that helped them memorize?

Most of what we call “measurable success” in education is really just a mirage of numbers games. For those like me that fell on the privileged side of those games, it was a great system that we probably want to fight to keep. And we are probably most likely the ones now in control, so….

Now, of course, this is not to say that learning isn’t happening. This is more about how most institutions measure learning and success. I believe people are always learning formally and informally, even if its not always what they had intended to learn. It just takes a lot of time, effort, and money (yes – money!) to truly assess learning, and the educational field in general is being tasked with the opposite. “Do better assessment with less money, less time, and less effort (ie people power)!” No real easy answers, but there is a problem with the system and the culture that drives that system that needs to be addressed before “measurable success” becomes a trustworthy idea.

Wednesday, January 14, 2015 (1:41 pm)

Matt CrosslinMeasuring Success in MOOCs (or More Specifically, DALMOOC)

Posted by: Matt Crosslin In: Open Learning

I was asked last week how we knew whether or not DALMOOC was successful. Seems a fair question since “success” in MOOCs seems to be measured by everything from completion rates to enrollment numbers to certificate numbers to alignment of the stars on the Wednesday of the first week the MOOC is offered. I had to be honest about that question and say that since everyone that work on DALMOOC lived and are still on speaking terms (at least as far as I know), then we were successful. Running a MOOC can be far more intensive and stressful than many people realize. We almost didn’t make it out the other side with everyone still happy.

In some ways, when I see people saying that we were blending xMOOCs and cMOOCs, or combining the two, I think we might have failed in the communication of what we were doing. Maybe I can blame that somewhat on our current educational system that only thinks in linear ways; therefore any course with more than one layer is not scene as complex, but “blended” or “combined.” Words like “straddled” seem closer, but to be honest we didn’t feel the need to straddle. We just had two layers that were not walled in, allowing learners to choose one or the other or both or to move back and forth as they felt. Infinite possibilities. A Multiverse Open Online Course maybe even. But not really a linear mix of the two from the design side.

Of course, the learner experience was linear even if they skipping back and forth between both layers. So maybe when the participants use words like “blended” or “straddled” or “combined,” those statements in themselves are actually signs of success. In the beginning, some claimed that DALMOOC was more cMOOC than xMOOC, but by the end others were claiming it was more xMOOC than cMOOC. Maybe all of these various, different views of the course are proof that the original design intention of truly self-determined learning was realized. At the very least, DALMOOC feedback was an interesting study in how bias and ontologies and epistemologies and all those -ologies affect participant perception of design. Maybe it doesn’t matter than participants can’t articulate the basics of dual layer architecture because the point all along should have been to hide the architecture and let learners learn how to learn.

So, at the end of the day, I will be able to come up with some philosophical jargon to “prove” that DALMOOC was a success to the powers-that-be who ask – all of which will be true. But to be honest, the only thing I really want to do is shrug my shoulders and say “beats me – go look the participant output itself and see if that looks like success to you.”

Looks like success to me.

Welcome to EduGeek Journal

Welcome to EduGeek Journal, proud sponsor of your future. Our goal is to promote educational technology by helping educators stay one step ahead of Joneses. We like to pour over new ideas and dream about what could possibly happen in the future in the world of education.

Email Updates

Enter your email address to get the latest blog posts sent directly to your inbox:

Delivered by FeedBurner

Also want get all comments emailed to your inbox? Enter your email address in this box:

Delivered by FeedBurner

EduGeek Journal on Facebook

ClustrMap + Badges