ChatGPT Can Ace an Entire Master’s Level Course? So What?

Every once and a while, you see a new “shocking” video that shows some Ai tool passing a test, a module, a course, etc. This is the latest one on TikTok, where they talk about how ChatGPT was enrolled in a Master’s level Health Administration course:

https://www.tiktok.com/@chatedupodcast/video/7473271959862824238

Of course, the way they did this experiment sounds like it could have been fraud, and we should be concerned that people have to break the law to prove these points. And yes, this is also another anecdote that shows that AI detection is not really a thing. But to those saying “if your course can be passed by a bot” or even “this proves how advanced Ai has become”… not so fast.

This video should make one concerned for how society views a “course.” Is it just a thing to tick off a list and get a grade for? Is it the be-all end-all of learning? Is it maybe just one piece in the overall educational journey?

For those that are programming Ai systems, of course it is a big deal that they are getting to the point that the bots can take an entire course. That took a lot of work to get to that point. For the rest of society that has to wrestle with what to do with all this Ai that is being forced on us? Its totally irrelevant that an Ai system can ace a course.

The point of any course is to teach a certain concept to a human, who will then make that learning one part of their overall knowledge of their degree or program. Many courses (maybe even most) are already teaching things that are out there online. Before the Internet, most of the knowledge in courses could be found in books at a decent-sized library. For centuries now, students didn’t just need access to knowledge to learn something. They didn’t even need an external organizing and sensemaking system like a book or bot to help them do something with that information. They needed an expert to guide them in how to engage with the knowledge in a way that will help them use that knowledge to do something in the future (usually work a job in a specific field).

Is it really surprising that an Ai system that has the entire internet stored in its memory can pull up that information when needed? Is it really that impressive that a digital system that can index and search all of that knowledge in the blink of an eye and make the right matches to pass a course? Again, to those that designed said system – of course it is impressive. To the rest of us, if we want to recognize that courses are something more than just a grade… that the reason we structure them the way we do has meaning for something beyond the course… then, no, it doesn’t matter.

The important part about any one single course is not necessarily the passing grade, but the organization of content and activities that are already out there into a structure that teaches the learner a specific way of knowing those random facts and activities in meaningful way for their future use. It was never a system designed to pattern match a bunch of “if/then” statements with instant access to all of the knowledge online.

Again, from a programming perspective – of course that is impressive. From a learning perspective? This is all very unimpressive. Are you smarter than a bot? Of course you are. The bot just has to do a search and match through all the data it has been trained on. You, as a human, had to learn that information and it is mostly not “stored” in your memory very well. In fact, we know that our brains don’t work like computers, so its a bad comparison to even ask if we are “smarter” than a computer. Are “smarts” only based on information we can recall and pattern match? I would argue that once computing power reached the point that it could enable GenAI, it became irrelevant to think of it as “smart.” Well, it never really was a great idea to anthropomorphize Ai. But something like ChatGPT coming with an answer to anything is equivalent to someone that is asked a question about a chart in big letters in front of them and they say the answer by reading the answer straight from the chart. The speed of computing has reached the point that Ai can do that with everything online. Would people think you were that smart if you basically answered everything by using an advanced pattern-recognition based search engine website? Not really. That is what Ai basically is.

But that is probably what we get for calling it “intelligence” in the first place – we evaluate Ai based on how close it gets to mimicking human intelligence, instead of looking at how Ai would be classified if humans did the same thing. In the case of passing a course, even at the Master’s level, its the equivalent of being impressed that a student passed a course when someone was holding up flash cards in front of them with the answers for everything. We need to be concerned with how people are lionizing Ai and undervaluing what learning really is.

Obligatory Post on Xitter Alternatives Like Bluesky and Mastodon

With the rapidly accelerating decline of the once mighty Xitter (yes, the “X” is pronounced as “sh”), many people have been looking to alternatives. Many have also written some great guides about these alternatives. I’m not really even sure that I need to write one myself, as everyone reading this has probably already read a half dozen better guides. But I typed this out for some friends in a Facebook Group, so I wanted to save it on my own site before it got sucked into the MetaBlackHole.

First I have to admit I’m actually not on Signal, but I probably need to? I thought giving your phone number to a social media site was a bad thing… is it not anymore? I am also on Instagram, TikTok, LinkedIn, etc. I known they are also sometimes mentioned as Xitter alternatives, but they function differently. I’m going to stick with the three major Xitter alternatives:

Mastodon: Mastodon has been around the longest, and readers of this blog are probably the most familiar with it. People say it is more complicated and that is true. It is decentralized, so you will want to think of joining a server that others you follow are on. That is not a requirement, but it can be sluggish pulling posts from different servers than the one you are on. Longer post lengths also makes it feel blog-like compared to Xitter. Some like that, others think it feels like a slog. The fact that replies are not shown by default is apparently leading to a lot more targeted harassment in the comments according to some, so beware of that. Overall the interface is close(ish) to Xitter. Mastodon may very well come to be considered as something different than Xitter (mainly because it is).

Threads: Threads seems to work mostly like Xitter, but the layout is the most different. Since it is connected to Facebook, you get a lot of people that post the same on both, so it gets a little redundant if you are active on both. It also forces you to see a “For You” recommendation feed by default. You can switch to the feed of just those you follow, but it doesn’t stay. Threads feels more like a comment section of a blog than Xitter, but overall not bad. Just OK to me. Some people use it a lot, but it seems to be the second most popular alternative.

Bluesky: The guy from Xitter sold it to Elon and created Bluesky, so it is the one most like Xitter. A lot easier to to get started. There are even third party tools that will find your Xitter friends on there and even copy your tweets over. Since there are no forced algorithms, it takes some getting used to the differences in what you see in your feed. For instance, if you like seeing the replies that your friends make to accounts they follow but you don’t, you have to change a setting to show that (this view is the default on Xitter). There are guides to do all of this, plus there are some apps and add-ons to tweak your experience in different ways if you feel like geeking out. Bluesky also has “Starter Packs” to help you find lists of people to follow on various topics, but be aware that not all of these are legit and often have people that didn’t ask to be on the list, so make sure to investigate before blindly following or blocking all. Oh, and blocking works a lot better on Bluesky that other apps.

Bluesky seems to be the most popular alternative so far. It’s also where a lot of the critical Ed-Tech people went, so a good amount of the AI skepticism can be found there as well. Some have written about it becoming a progressive echo chamber, but that is mostly just centrist gate-keeping. Many different ideas are discussed and disagreed with on Bluesky every day.

For those that stayed on Xitter, no judgement here. I am still there as well, just much less active. Yes, Elmo is making it horrible. I also don’t blame anyone for leaving. For many, you needed to leave to protect yourself. But some decided to stay and fight, and that is not wrong, either. People have good reasons for doing both. A few people have even created accounts on all the new alternatives while keeping their Xitter account to cross post to all of them. Some have even chosen to use none of them – that is great as well. It has become a bit more like the Wild Wild West Web2.0 days in some ways.

 

ChatGPT is Generating Nonsense and No One Knows Why

Every time I start a new post over the past couple of months, it just devolves into a “I told you so” take on some aspect of AI. Which, of course, just gets a little old after trying to tell people that learning analytics, virtual reality, blockchain, MOOCs, etc, etc, etc are all just like any other Ed-Tech trend. It’s never really different this time. I read the recent “news” that claims “no one” can say why ChatGPT is producing unhinged answers (or bizarre nonsense as others called it). Except for, of course, many people (even some working in AI) said this would happen and gave reasons why a while back. So, as usual, they mean “no one that we listen to knows why.” Can’t give the naysayers any credibility for knowing anything. Just look at any AI in education conference panels that never bring in any true skeptics. It’s always the same this time.

Imagine working a job completely dependent on ChatGPT “prompt engineering” and hearing about this, or spending big money to start a degree in AI, or investing in an AI technology for your school, or any other way people are going big with unproven technology like this?  Especially when OpenAI just shrugs and says “Model behavior can be unpredictable.” We found out last week just how many new “AI solutions” are just feeding prompts secretly to ChatGPT in the background.

Buried at the end of that Popular Science article is probably what should be called out more: “While we can’t say exactly what caused ChatGPT’s most recent hiccups, we can say with confidence what it almost certainly wasn’t: AI suddenly exhibiting human-like tendencies.” Anyone that tries to compare AI to human learning or creativity is just using bad science.

To be honest, I haven’t paid much attention to the responses (for or against) my recent blog posts, just because too many people have bought the “AI is inevitable” Kool-Aid. I am the weirdo that believes education can choose it’s own future if we ever just would choose to ignore the thought leaders and big money interests. Recently Ben Williamson outlined 21 issues that show why AI in Education is a Public Problem with the ultimate goal of demonstrating how AI in education “cannot be considered inevitable, beneficial or transformative in any straightforward way.” I suggest reading the whole article if you haven’t already.

Some of the responses to Williamson’s article are saying that “nobody is actually proposing” what he wrote about. This seems to ignore all of the people all over the Internet that are, not to mention that there have been entire conferences dedicated to saying that AI is inevitable, beneficial, and transformative. I know that many people have written responses to Williamson’s 21 issues, and most of it boils down to saying “it happened elsewhere so you can’t blame AI” or “I haven’t heard of it, so it can’t be true.”

Yes, I know – Williamson’s whole point was to show how AI is continuing troubling trends in education. We can (an should) focus on AI or anything else that continues those trends. And he linked to articles that talked about each issue he was highlighting, so claiming no one is saying what he cited them as saying is odd. AI enthusiasts are some of the last holdouts on Xitter, so I can’t blame people that are no longer active there for not knowing what is being spread all over Elon Musk’s billion dollar personal toilet. Williamson is there, and he is paying attention.

I am tempted to go through the various “21 counterpoints / 21 refutations / 21 answers / etc” threads, but I don’t really see the point. Williamson was clear that he takes an extreme position against using AI in schools. Anyone that refutes every point, even with nuance, is just taking an extreme position in a different direction. To do the same would just circle back to Williamson’s points. Williamson is just trying to bring attention to the harms of AI. These harms are rarely addressed. Some conferences will maybe have a session or two (out of dozens and dozens of session) that talk about harms and concerns. Usually “balanced out” with points about benefits and inevitability. Articles might dedicated a paragraph or two. Keynotes might make mention of how “harms need to be addressed.” But how can we ever address those harms if we rarely talk about them on their own (outside of “pros and cons” arguments), or just refute every point anyone makes about their real impact?

Of course, the biggest (but certainly not best) institutional argument against AI in schools comes from OpenAI saying that it would be “impossible to train today’s leading AI models without using copyrighted materials” (materials that they are not compensating the copyright holders for their intellectual property FYI). Using ChatGPT (and any AI solution that followed a similar model) is a direct violation of most school’s academic integrity statements – if anyone actually really meant what they wrote about respecting copyright.

I could also go into “I told you so”s about other things as well. Like how a Google study found that there is little evidence that AI transformer model’s “in-context learning behavior is capable of generalizing beyond their pretraining data” (in other words, AI still doesn’t have the ability to be creative). Or how the racial problems with AI aren’t going away or getting better (Google said that they can’t promise that their AI “won’t occasionally generate embarrassing, inaccurate, or offensive results”). Or how AI is just a fancy form of pattern recognition that is nowhere near equatable to human intelligence. Or how AI takes more time and resources to fix than just doing it yourself first. Or so on and so forth.

(Of course, very little of what I say here is really my original thought – it comes from others that I see as experts. But some people like to make it seem like I making up a bunch of problems out of thin air.)

For those of us that actually have to respond to AI and use AI tools in actual classrooms, AI (especially ChatGPT) has been mostly a headache. It increases administration time due to dealing with all the bad output it generates that needs to be fixed. Promises of “personalized learning for all” are almost meh at best (on a good day). The ever present existence of the uncanny valley (that no can really seem to fix) makes application to real world scenarios pointless.

Many are saying that it is time to rethink the role of the humans in education. The role of humans has always been to learn, but there never really was one defined way to do that role. In fact, the practive of learning has always been evolving and changing since the dawn of time. Only people stuck in the “classrooms haven’t changed in 100 years” myth would think we need to rethink the role of humans – and I know the people saying this don’t believe in that myth. I wish we had more bold leaders that would take the opposite stance against AI, so that we can avoid an educational future that “could also lead to a further devaluation of the intrinsic value of studying and learning” as Williamson puts it.

Speaking of leadership, there are many that say that universities have a “lack of strong institutional leadership.” That is kind of a weird thing to say, as very few people make it to the top of institutions without a strong ability to lead. They often don’t lead in the way people want, but that doesn’t mean they aren’t strong. In talking with some of these leaders, they just don’t see a good case that AI has value now or even in the future. So they are strongly leading institutions in a direction they do see value. I wish it would be towards a future that sees the intrinsic value of studying and learning. But I doubt that will be the case, either.

Is AI Generated Art Really Coming for Your Job?

You might have noticed this Twitter thread about improvements in AI-generated art work. Well, if you are still on Twitter that is. Here is the thread – well, at least, until You-Know-Who “MySpaces” Twitter out of service:

So let’s take a look at this claim that AI-generated artwork is coming to disrupt people’s jobs in the very near future. First of all, yes it is really cool to be able to enter a prompt like that and get results like this. There is obviously a lot of improvement in the AI. It actually looks useful now. But saying “a less capable technology is developing faster than a stable dominant technology (human illustration)”…?

[tweet 1596581772644569090 hide_thread=’true’]

Whoa, now. Time for a reality check. AI art is just now somewhat catching up with where human art has been for hundreds of years. AI was programmed by people that had thousands of years of artistic development easily available in a textbook. So saying that it is “developing faster”? With humans being able to create photo-realistic drawings as well as illustrate any idea that comes to their mind – where is there left to “develop” in art?

That is like a new car company saying they are “developing new cars faster than the stable industry.” Or someone saying that they have blazed new technology in travel because they can cross the country faster in a car than a horse and wagon did in the past. The art field had to blaze trails for thousands of years to get where it is, and the AI versions are just basically cheating to play catch up (and it is still not there yet).

The big question is: can this technology come up with a unique, truly creative piece of artwork on its own? The answer is still “no.” And beating the Lovelace Test is not proof that the answer is “yes,” because the Lovelace test is not really a true test of creativity.

[tweet 1596680005638975489 hide_thread=’true’]

Yes, all artists stand on the shoulders of others, but there is still an element to creativity that involves blending those influences into something else that transcends being strictly derivative of existing styles. Every single example of AI artwork so far has been very derivative of specific styles of art, usually on purpose because you have to pick an existing artistic style just to get your images in the first place.

But even the example above of an “otter making pizza in Ancient Rome” is NOT a “novel, interesting way” by the standards that true artists use. I am guessing that Mollick is referring to the Lovelace 2.0 Test, which the creator of said test stated that “I didn’t want to conflate intelligence with skill: The average human can play Pictionary but can’t produce a Picasso.”

Of course, the average artist can’t produce an original painting on the level of Picasso either (unless they are just literally re-painting a Picasso, which many artists do to learn their craft). The people working on this particular AI Art Generator have basically advanced the skill of their AI to where it can pass the Lovelace 2.0 Test without really becoming truly creative. And honestly, “Draw me a picture of a man holding a penguin” is a sad measure of artistic creativity – no matter how complex you make that prompt as the test goes along.

But Mollick’s claims in this thread is just an example of people not understanding the field that they say is going to be disrupted. For example, marveling over correct lighting and composition? We have had illustration software that could do this correctly for decades.

[tweet 1596621251627335680 hide_thread=’true’]

Artists will tell you that in a real world situations, the time consuming part of creating illustrations is figuring out what the human that wants to art… actually wants. “The otter just looks wrong – make it look right!” is VERY common feedback. The client probably also created several specific details about the otter, plane, positions of things, etc that has to be present in any artwork they want. Then there are all of the things they had in their head that they didn’t write down. Pulling those details that out of clients is what professional artists are trained to do.

This is where AI in art, education, etc always falls apart: programmers always have to start with the assumption that people actually know what they want out of the AI generator in the first place. The clients that professionals work with rarely ever want something as simple as “otter on a plane using wifi.” The reality is that they rarely even have that specific or defined idea of what they want in the beginning. There is a difficult skill of learning to figure out what people actually want that the experts in AGI/strong AI/etc tell us is probably never going to be possible from AI.

So, is this a cool development that will become a fun tool for many of us to play around with in the future? Sure. Will people use this in their work? Possibly. Will it disrupt artists across the board? Unlikely. There might be a few places where really generic artwork is the norm and the people that were paid very little to crank them out will be paid very little to input prompts. Look, PhotoShop and asset libraries made creating company logos very, very easy a long time ago. But people still don’t want to take the 30 minutes it takes to put one together, because thinking through all the options is not their thing. You still have to think through those options to enter an AI prompt. And people just want to leave that part to the artists. The same thing was true about the printing press. Hundreds of years of innovation has taught us that the hard part of the creation of art is the human coming up with the ideas, not the tools that create the art.

Are We in the Upside Down? Course Hero, Lumen Learning, and All Kinds of Strange Things are Afoot in Ed-Tech

In my last post on The Quick(ish) Guide to Why Some People Don’t Like Course Hero, I stated that I really didn’t want to get into the controversy surrounding recent hirings at Course Hero. That was easy to say when it was just one head-scratching hire, but other things have happened since then that make it hard not to dive in somewhat. But just somewhat!

Part of the problem is that I have been waiting to see what big announcements Course Hero might make about upcoming changes. That hasn’t happened – but surely they have something up their sleeve? They make the claim that empowering students is in their DNA – but that isn’t true in it’s current form. Students can upload content – but its almost always content that others (usually instructors or content companies) have created. There is no real power in that – the students really have no say what is contained in that content. The instructor or company does. When students can upload something of their own creation that then becomes part of a class – that would be empowerment.

But why would they need Course Hero for that? They can already upload content to blogs, Google Drive, Dropbox, Discord, you name any one of hundreds of services. There is nothing special about yet another file hosting service – so either Course Hero has no idea what they are talking about, or there is a big change planned to their core model in the near future.

Even their core product – offering answers to assessments/assignments/etc – is not truly empowering for students. Students just take the answers and turn their course work in without learning the content… so at best Course Hero is extractive for students, not empowering.

Sadly, it has been difficult to get Course Hero to address this issue without deflecting to systemic issues. Of course the systemic issues are real and need to be dealt with – but Ed-Tech critics should know better than to deflect to systemic issues when they work for a company that quite literally drives the adversarial relationship between students and teachers. Giving away answers to anything and everything just adds more pressure to students to cheat. Not to mention that Course Hero’s access model creates even more pressure for students to steal content. Well… or to at least do something like that….

[tweet 1539129886278926336 hide_thread=’true’]

Additionally, Course Hero is also looking to work with instructors. But to do what? Host files? Instructors already have an LMS for that, along with all of the above-mentioned file hosting services. As it stands right now, instructors don’t really need anything new for hosting files or content. So surely they have something else planned, right?

As many have pointed out, there is a need for “reputation-washing” for a company like Course Hero. They are seen as a cheating site that pays little attention to following the law. There probably is something to this – many people have privately expressed irritation at several well-known educators that have defended Course Hero. But they are friends with these people, and don’t want to harm the friendship. I get that. It makes the world feel upside down, and its hard to know how to navigate this weird new situation.

So this leads to the reputation-washing whether Course Hero is wanting it or not. If you look at something like UngradingCon, many of the Session Leaders would probably boycott the conference if they were listed along with the CEO of Course Hero. But when listed with a friend or personal hero that has also aggressively defended Course Hero? Its a lot harder to know what to do when that person is a friend/mentor. Most would still question whether Course Hero and ungrading are a good mix. Like I said in my other post: “I really don’t see a way that Course Hero could co-exist with ungrading, or if students would even bother to use it if grades were low-stakes in any way.”

Certainly there is a lot of value into getting your company into spaces that they wouldn’t be welcome other wise. Even though Course Hero isn’t a strong match for the ungrading world, they are now a part of that community (like it or not).

But surely that is not the whole game plan, right?

Several people recently started to notice that some links to OER resources hosted on the Lumen Learning website started to redirect to the same content on Course Hero. It doesn’t seem like there was an announcement – it just happened. And yes, it was a weird change – Course Hero doesn’t always follow open licenses very well (according to some – the company disagrees). Steel Wagstaff of PressBooks seems to have uncovered the most information about what is going on here.

As several people have noted – Course Hero is blocked on many campuses. It doesn’t take a huge leap of faith to then realize what could happen here:

https://twitter.com/billymeinke/status/1537119423542153217

Another factor to consider – some of central figures at Lumen Learning have been laying the ground work for a partnership with for-profit Ed-Tech companies:

And yes, it is problematic to imply that people who might disagree with partnering with for-profits are “close-minded.” I risk the wrath of some of DW’s fervent defenders by pointing this out, but its not an attack to point out that less problematic language could have been chosen here.

Of course, since Course Hero does not have a reputation in some circles for protecting students very well, some instructors are not happy about having their content moved to Course Hero (and especially with out ever being told that the content was being moved). Other services are stepping up to offer alternatives, again proving that Course Hero is not offering a service that is unique in itself.

And then there are other wild, strange things going on as well that I am not sure exactly where they fit in the post, but they still do somehow:

https://twitter.com/karenraycosta/status/1534969104364339201

So far, it seems that Course Hero is working on getting access to conferences, school servers, and circles of respect that they didn’t have in the past. But the questions still remains… for what purpose? Course Hero does not empower students, they don’t offer a service that instructors can’t get elsewhere, and they aren’t the best place to host OER content. But this is how they are starting to promote their services – even though we know their employees and spokespeople know better. But they also know what is discussed behind closed doors, so you have to wonder what is coming down the pipe to make them start saying these things? Until we find out what that is, the current situation is one of the stranger things happening in the already odd Ed-Tech space.

Heartbreaking Conversations About the Realities of Gun Violence in Schools

I don’t know why news outlets do it this way, but every time there are gunshots at a school in Texas, the only thing you see for a while is “Gunshots reported at Texas School.” If they know someone was killed, then the headline reads “Fatalities reported at Texas School.” Of course, as a parent of a kid in Texas, either one sends me flying to the most recent reports, praying its not my kid’s school.

Which is selfish, of course, but its the new reality living in the wild, wild wes… errr… Texas.

When it doesn’t end up being your kid’s school, but there are fatalities, you are both relieved and sickened (when you realize what too many families are going to go through today).

Then, of course, your kid starts asking all kinds of hard questions:

“Where should I sit in the classroom to avoid getting shot if someone breaks in?”

“Where should I hide if a shooter breaks in?”

“Where should I spread blood on me to look like I have been killed?”

Wow. He should ask those questions. And I did my best to answer (which wasn’t very good).

But it is heartbreaking that he has to ask them in the first place.

He also wanted a police officer permanently assigned to every classroom. I pointed out there aren’t near enough officers to do that. I didn’t have the heart to also talk about how they might not even try to stop a shooter, and the shooter could possibly out gun them if they did. But he has made comments about that as well… so, he knows.

Our kids live in a world where we won’t do what it takes to protect them in schools.

Despite the fact that we have decades of evidence from around the world that gun laws do work, our leaders don’t want to go the proven route. As usual, they have to turn to Ed-Tech to solve problems.

Yes, arming teachers is an Ed-Tech solution. And as a former teacher, I can tell you it is beyond pathetic to think it will work. They expect teachers to fight off some dude brandishing a modified AR-15 and body armor… with a hand pistol? When the police can’t always do the same? If you arm teachers, you will have dozens of students accidentally shot (and probably killed) by those teacher guns before one single teacher gets their first shot off at a shooter. And I would bet they never stop a shooter that way.

Or how about the whole “add more surveillance” options? Really? You want more video footage of carnage that still doesn’t stop anything? How sick is that?

Then there is the whole “door” argument. Also Ed-Tech, and I get why people turn to this option. If you want to do that in addition to gun law reform, that is great. But you also have to realize that school buildings are living, breathing entities that love to find ways to open doors. Teachers know this: they get left open all the time… or blocked when some object falls out of a backpack. Plus it would be literally impossible at many schools to get all students in through a single entrance at the front before noon. And those multiple entry points just love to find ways to stay open.

But wait! The solution is simply to have random inspections to make sure schools are keeping security up… right? Oh, sure. Anyone that has worked at a school knows that 5 seconds after the surprise inspector shows up, every teacher or employee near a security point will be alerted and fixing any issues they see.

Hundreds or thousands of humans inhabit a campus every second it is open – all with a vested interest in seeing their school pass an inspection – and you think you are going to be able to have a “surprise” inspection? Good luck with that.

I used to work as a health inspector. My surprise inspections – where I would show up in between classes and few people saw me come in – were always reported to the cafeteria well before I set foot in there. This was before cellphones and I would literally talk to the secretary in the front office and see hardly anyone else. Yet they still found a student somewhere to run the back way and warn the cafeteria I was coming.

Oh, and shooters can often get in these new secure doors anyways… even when locked and reinforced and made bullet proof…

My son mentioned that he won’t go back to school until he feels safe going. Maybe that is what we need on a national level: give us meaningful, strong gun reform or we don’t take our kids back to school until it happens.

Of course, we would need some massive donations to help support parents that can’t keep their kids home if we did this – but maybe someone should get on that.

And before you ask – yes, I have specific laws in mind to pass. Yes, I know the differences between assault weapons and assault rifles and AR-15 and so on. Quibbling over those things is just wasting time. We all know what we are talking about even if we don’t use the specific correct terms every. single. time. Even if you mean well bringing that up, it is generally a bad faith argument by many others, and it is well past time to #$&?@ stop that.

Also – anyone that wants to say that the Second Amendment is “sacred” – that is blasphemy and you should stop that as well. There is not a single religious text out there that refers to 2A as “sacred” (most are against weapons in many cases – especially the Bible). Go look up what the word “Amendment” means. There is nothing saying that we couldn’t abolish the Second Amendment if we wanted. I used to be against that option, but after decades of waiting for our leaders to enact “common sense” gun laws… I am tired of seeing nothing happen. So I have a lot more sympathy for the “Abolish 2A” crowd.

Does that upset you? Too bad. I have been waiting for decades for real change to happen while supporting those that want to keep 2A… and it has led to nothing but more death and carnage. So I don’t care about 2A anymore. Sorry if that offends you – you should have been pushing to get better gun laws than getting people like me to protect 2A itself. I’m tired of empty thoughts and prayers. Tired of people arguing moot points about definitions and changeable amendments being somehow sacred and Ed-Tech options that are garbage as actual solutions and all the pathetic nothingness that is putting my son’s life at risk. Do something meaningful in this area or just get out of the way.

Metaversity, Tri-Brid Models, and the Same-Ol, Same-Ol “Innovation”

“Is education moving to a “tri-brid” model that flows between in-person, online and simulated environments?” asks the article titled With Money From Facebook, 10 Colleges Turn Their Campuses into ‘Metaversities’ on EdSurge. Of course, the answer is “no” – but you have to wonder how much time and attention will be given to this idea before it fades into the GoogleWave is the future of Education graveyard?

To be honest, I actually enjoy VR – I have fun playing the occasional game in there, or “hanging out” with family in other cities. I think the ability to create simulations makes VR an interesting educational tool. Interactive tools? Meh – don’t hate them, don’t love them. But to move towards making an entire educational experience or course or campus inside of VR sounds like we are going the wrong direction away from the “Sage on the Stage” model. Sitting and watching an instructor in a class room is replaced with sitting there with the classroom strapped to your face (and collecting all kinds of data on decisions you make without even leaving the room). “Only better” as I guess the people involved with this Enagage-supported project would claim.

It was the whole claim of “Tri-Brid” that first caught my attention:

“Arés argues that the rise of VR technologies will shift the current hybrid model of education—which draws on separate in-person and online environments—into a ‘tri-brid’ model, one that moves ‘seamlessly between online, in-person and simulated, without the limits of time, travel and scale.'”

First of – how do you move “seamlessly” between in-person and simulated when in-person is limited by time, travel, and scale? They point out that learners without VR headsets can use a monitor – so let’s be real here. The “simulated” portion is really just another version of Zoom. This is not really a Tri-Brid but an “Extended Hybrid” model. Look at the picture at the top of the article – it is still a Sage-on-the-Stage, still a white professor lecturing an all-white class, the same ol’, same ol’… with a cool 3-D model added to the PowerPoint.

Sounds a lot like Second Life, right? “Not so fast!” the creators say:

“This may bring to mind the now-defunct digital campuses that universities set up 15 years ago using Second Life—but leaders of the new project are quick to claim that this will be way better.”

Well, okay – I have used Occulus, and in many ways it does work better than Second Life. But when people talk about Second Life in this context, they aren’t talking about graphic quality, or interface design, or any of that. Of course we expect those aspects to improve over time. The problem with Second Life classrooms was that for every one of them that were actually interesting, there were dozens more that we just sub-par equivalents to a video conference. It took lots of time and money to create decent scenarios, and half the time the novelty of those wore off and people went back to more traditional online modes.

In other words – for every “look at how the heart functions by going inside of one” simulation that was out there, there were so so so many “here is my classroom camera feed streaming to a screen in a vague virtual re-creation of my classroom.”

There are other foundational problems with the idea as well. Monica Arés, head of Immersive Learning at Meta, had this to say about VR:

Arés, the leader of Immersive Learning at Meta, is a former teacher. She recalls the nagging worry that her lessons might not hold her students’ attention. “I would spend countless hours trying to create lessons that are visually rich,” Arés says. “I knew the second I put that headset on it was the medium I had been looking for.”

Any Instructional Designer and most teachers will tell you that there are all kinds of ways to get student attention beyond just finding “visually rich” lessons. When your foundational idea of what makes learning effective is so skewed like this… I have to worry about the overall project.

And the problems don’t just stop there. David Whelan, founder and CEO of Engage, had this to say:

“Computers started in homes as entertainment, then creeped into school, then into everyday use items and at jobs,” Whelan says. “VR could take the same route.”

This is not true at all. Most people that I know first interacted with computers at schools and Universities long before they had one at home. Read any book on the 80s. Many people were even working on computers at a job before getting one at home. They were really, really expensive at the beginning. Many of us remember arranging our schedules around light traffic time at school/university computer labs so that we could find a free machine.

Computers started at businesses and universities. Its a pretty easy historical fact to look up.

Maybe Whelan grew up in a wealthy home that could afford a home computer early. Maybe he is not old enough to even know where computers started. I honestly don’t know anything about him. Either way – it makes you question the ability of his company to really know what is going on and where the tech world is going.

Then there is the question of whether or not students will actually like being in VR in the first place. The article makes this claim:

“Trying to pay attention in a college course while manipulating an avatar around a virtual classroom can feel a little odd. But the new Stanford study suggests that this kind of setting gets more comfortable for students over time.”

However, I noticed that the article does not go into how comfortable students were overall, or how much their comfort actually increased over time.

Turns out, the answer to both is more like “not very much.”

If you look at the original article in question, the results aren’t that impressive. When students were asked to rate “enjoyment” on a Likert scale of 1 to 5, the results hovered between 3.1 and 3.7. In some ways of crunching the data, it did slightly increase over time – but not by much:

Self-presence and spatial presence (feeling like you are really in the environment) both hovered around 4 on a Likert scale of 1 to 7. Social presence and entitativity (“the degree to which a collection of people is perceived as a single, unified entity”) fared a little better: between 5 and 6 on a 1 to 7 Likert scale. But how much of that is attributed to all of the work online courses have put into increasing those aspects online for decades?

Overall, a more honest reading of the outcomes of the study is that, on average, the reaction to comfort in VR was “meh.” Sure – all factors increased over time, but how much of that increase came from the learners being more aware of those factors because they were asked about them ever week? And is it really that surprising to say that people get more used to using something the more they use it? That doesn’t mean they really like it in the first place.

But you might also say “if they want to waste money on a rabbit hole, what’s the big deal? Its not like they have their sights set on anything bigger.” Well….

“I do hope that things like the socioeconomic divide and geography divide can potentially be bridged in education because of some of these new technologies like VR,” (Greg Heiberger, associate dean of academics and student success at South Dakota State University) says. “Those would be the two tenets I would guess are near the top of their (Meta Immersive Learning) list: making money and giving some of those resources back to make the world a better place.”

Not sure how they plan to eliminate redlining, food distribution, prejudices, and all kinds of other societal problems that drive these divides… through VR (and other tech)? A statement like this kind of feels like the “tossing a roll of paper towels” moment of this whole idea. If there is one thing we have learned about the world, is that you can always count on the rich to give their wealth to the poor and not some huge vanity project purchase.

But obviously, Heiberger needs to talk to Arés about all of this “trickle-down” wealth:

“Arés said that Meta is not focused on earning revenue from the partnership; instead, the ‘main goal is to increase access to education and transform the way we learn.'”

Transform the way we learn – by sticking a white dude avatar in front of a 3-D PowerPoint Screen and then strapping this transformed classroom to students face so they can virtually sit in desks from the comfort of their own homes. Even though those homes might not be “comfortable” for all, and you can only wear an Occulus so long before your face starts hurting. Funny how research studies never examine how deep the red marks in the shape of a headset are on users’ face at the end of these immersive learning sessions.

The Quick(ish) Guide to Why Some People Don’t Like Course Hero

If you are part of certain circles in the education world (especially on Twitter), you probably saw the controversy yesterday about a well-known education critic being hired by the Ed-Tech company Course Hero. I really don’t want to wade into that controversy too much – I don’t know the people involved well enough to comment on their motives. I have never witnessed the whole “change a company from within” strategy ever work, but I know there is no shortage of people who will try. However, Course Hero has run under many people’s radars for a while, and I thought I would go into why some people don’t like the company’s product or business model.

So what exactly is Course Hero? Well, if you read the company hype, you will find things like “partnering with, connecting, learning from and teaching educators in support of them in empowering learners.” Which doesn’t really mean anything specific to be honest. The reality is that they are a resource sharing website, primarily driven by student labor. Students can find answers to test questions, past papers, course documents, and all kinds of materials related to courses they are taking (including entire chapters and courses). After free trials of various kinds, they also have to pay for this access. In turn, they are encouraged to upload documents for other students.

Now, I will say that I am typically sympathetic of students that use websites like this – even though I will still warn them not to.

So before I dive into that problematic system, I will point out to students that using Course Hero can be dangerous. Your institution probably has strongly-worded “Academic Honesty” statements that spell out harsh possibilities for being caught sharing your work with other students or uploading your instructor’s copyrighted content to any website without their permission (and institutions also often claim copyright on course content as well). Even if your intent is to share examples to help other students (something many instructors even encourage), your institution might not see it that way. Plus, I did a quick search through Course Hero yesterday and found a large number of papers that still had the students’ name on them. That means that a random school official could be surfing through their website, see your name, and get you in trouble for a course from a couple of years ago. Course Hero does not appear to be doing much to protect the students that it uses for free labor, so “user beware.”

However, like I said, it is important to understand why students use Course Hero. So many of our institutions still promote high stakes assessment (tests, essays, etc) as the main mode for “weeding out” students (side note: never refer to your students as “weeds”). Sometimes this even comes wrapped in poorly designed courses that don’t do enough to prepare students for these assessments. Students are then given the impression that cheating is the only way other students survive the gauntlet (and in many cases, this is probably true). Focusing on the students that use Course Hero misses the real problem of an institutional system that created the pressure to cheat in the first place.

But remember students – if you are caught using Course Hero, your institution will most likely not do any soul searching on the way they created the pressure to move you in that direction. They will just punish you and move on. Again – user beware.

I see nothing in Course Hero that pushes back against this problematic pedagogy. In fact, it only seems geared to empower that system. I really don’t see a way that Course Hero could co-exist with ungrading, or if students would even bother to use it if grades were low-stakes in any way.

What you have is a company that utilizes free labor (yes, just like other companies like Facebook) and a “freemium” model to get users to start paying. It also has an internal tokening system that creates rewards for uploading content (search Twitter for Course Hero, and most of what you get is users claiming to sell these tokens for cheap). Because most of the users at some point or another are desperate to survive a harsh academic system somewhere, many feel Course Hero is a predatory service relying on student fear. Yes, they do position themselves as a pro-student company, but honestly I don’t see how they are more pro-student than anyone else.

Also of note is the general legality of Course Hero – it’s pretty easy to find many, many examples of how they are in violation of NC licenses. But on top of that, since all material (in the U.S. at least) is automatically copyrighted once it is created – I don’t see how much on their website is technically legal at all (outside of the occasional rare public domain license). You don’t have to agree with copyright laws – I am just pointing out the statues here as they currently stand. In addition, most institutions have added copyright rules that require you to at least get the permission of the instructor, if not the entire institution, before uploading to any external website. Since it would take a massive legal fund to challenge any one of these points, Course Hero probably enjoys a relative “freedom” from legal prosecution. From many accounts I can find online, it is very difficult to get copyrighted material taken down with a simple take down notice. Course Hero does not have a great record of responding to critics of any kind (despite what some might say), including direct legal challenges.

Plus, many institutions will directly name Course Hero as a reason why they have to get proctoring surveillance solutions. Course Hero may not like it (or maybe they do – who knows?), but they are a major player in the course surveillance system. You will hear Course Hero directly named by institutions as one reason they need to increase surveillance. As many people have put it, dealing with a nuclear arms race by adding more nuclear missiles is a step in the wrong direction.

You may disagree with all of these assertions about Course Hero (I am sure the company does). I would refer you back to the title of the post – these are reasons why people don’t like Course Hero. There are many other reasons as well. I’m not here to weigh the praise alongside the criticism.

One of the oldest, cliche moves in the book for tech companies in general is to hire a critic into a high level position at their company. They hope to borrow that critic’s reputation to clean up their image. It never works that way, but still companies try all the time Is that what Course Hero is doing now? Only time will tell. Every single critic that has ever been duped by a tech company in the past all claimed before hand that they were too smart to get duped. Sometimes, they were even hired by someone that really meant it… until that person got forced out by larger forces in the company.

Ultimately, companies don’t really care that much about any of that drama. Drama creates attention, and attention is what they need. They know that when they hire a critic, they also get the loyalty of some of that critic’s friends and colleagues along the way. They know they are getting multiple defenses of their company from many other respected voices… for free. And with Course Hero, you are already seeing that. These defenses range from the normal “I won’t attack someone just for taking a job” (agreed) to the questionable justifications of the company actions to the downright passive aggressive denigrations. One person even made me think “well, Headmaster Killjoy is here to swat down the plebes that dare have a different opinion!” Then there are the attacks and fights. I sincerely hope the people that become that aggressive will realize that they only make people hate Course Hero more when they do that.

Anyways, my only real message here is to please understand why there is so much distrust of Course Hero out there. Most of the disagreement with the recent announcement has been serious and respectful, despite what the defenders of the announcement will claim. Not all disagreements have been cordial, obviously… but the announcement came with the direct statement that “this will upset people.” Why tear into people when they are responding exactly as noted?

Or the bigger question: if Course Hero is a good company that truly engages with it’s critics… they why does it need to be subverted from within? Some people are saying both, and it really doesn’t match if you think about it.

We Know Why You Hate Online Learning – and It Has Nothing to Do With Quality

In some ways, I get why some people are saying they hate online learning. Almost everyone was forced into it – even those that didn’t choose it originally. We live in a time where most people that enter school (or teach at school) are aware that there is an online option. There are a few cases where people want to take or teach online courses when there aren’t any options to do so for the most part. But for everyone else, if you wanted to learn or teach online, you probably were able to choose that. The millions that were forced to switch suddenly last year did so against their first preference, and I get how that frustrates many of them.

Let’s face it – we all know that what has been happening the past two years is often not fully implemented, funded, and institutionally-supported online learning. Most tried hard to make it work, but due to shortages in training, prep time, or funding/support, a lot of it fell short of the true potential of online learning.

Of course, this was also true about face to face learning before the pandemic – even dedicated teachers are held back because of systems that don’t give them enough time, or train them well enough, or give them the money and resources they need. We just act like this is the “Facts of Life” for on campus learning… you take the good, you take the bad, you take them both, and there you have…. a gold standard….?

Nope. Any institutional leader or edu-celebrity that proclaims that on-campus learning is inherently superior to online learning is being disingenuous. They know that reality doesn’t support their claims. They just hate online learning… but not for quality reasons.

The real reason? It’s all about the power and control. Leaders can’t control their students, faculty, and staff remotely like they can on campus. And that control not only brings them a power trip – it also brings in big $$$ for schools when they can manipulate students into spending more money on campus.

And that’s it really: the real reason you have leaders (institutional, thought, and otherwise) claiming that online learning is inferior, and that on campus learning is the “gold standard,” is because they lose power (and the money that comes with that power).

Now – if a student or faculty or even University president proclaims that they hate online learning in and of itself – I get it. We all have personal preferences – I love online learning, but I get why it isn’t for everyone.

But there is a difference between saying one personally doesn’t like it, and saying online learning is inferior, failed, snake oil, etc.

The difference, of course, is research. There really is research showing that there is no significant difference between various outcomes of online learning and on campus learning. Probably one of the best sources to look at for research is the National Research Center for Distance Education and Technological Advancements (DETA) at the University of Wisconsin – Milwaukee’s “No Significant Different” database:

“This site is intended to function as an ever-growing repository of comparative media studies in education research. Both no significant differences (NSD) and significant differences (SD) studies are constantly being solicited for inclusion in the website. In addition to studies that document no significant difference (NSD), the website includes studies which do document significant differences (SD) in student outcomes based on the mode of education delivery.”

Current, the numbers in that database are categorized as:

  • 141 studies that show no significant difference
  • 51 studies that show “Significant Difference – Better Results with Technology” (online usually being said technology)
  • 2 studies that haven’t been indexed yet
  • 0 studies showing “Significant Difference – Better Results in the Classroom”
  • 0 entries showing mixed results

Maybe it is just my bias… but it seems that the results are starting to trend towards online maybe being… better?

Recently I was in a huge Twitter argument with a group of K-12 educational leaders from the UK that were demanding that I provide an article that proves that online learning could even work at all. They had already ignored two responses to these demands from a female colleague of mine – and still demanded that I provide a link of my own even though I had pointed them to those tweets and the DETA database already. So I just refused to give out any more links to people that weren’t going to look at them anyway – and got attacked in all kinds of horrible ways. But it seems like they were under the impression that I had website addresses to killer pro-online education studies memorized and I was just being a jerk in not spitting them out in a few seconds. Look – asking an online educator to provide one article proving that online learning is okay is like asking a Geologist to provide one study that proves that rocks exist within the Earth. A few might have something in mind, but most of us don’t spend a lot of time memorizing what we see as “proof of the obvious.” Others seemed to think that academics have all the time in the world to respond to tweet #54 demanding that one all-proving link. Look – no one owes you free labor. If you ask for something and they don’t give it, learn to respect people’s time enough to accept that maybe they are as busy as you. Especially if you were the one that came in swinging with the “online learning is a dying evil” rhetoric.

It’s all complicated. I will be the first person to tell you it comes down to personal preferences on whether you should do online learning or not, and for most people its not even an either/or. Different contexts call for different modalities for each person at any given moment. We just need to kill the dated and problematic “in-person learning is the gold standard” BS.

See also:

(Cover photo by Clem Onojeghuo on Unsplash)

Using Learning Analytics to Predict Cheating Has Been Going on for Longer Than You Think

Hopefully by now you have heard about the Dartmouth Medical School Cheating Scandal, where Dartmouth College officials used questionable methods to “detect” cheating in remote exams. At the heart of the matter is how College officials used click-stream data to “catch” so-called “cheaters.” Invasive surveillance was used to track student’s activity during the exams, officials used the data without really understanding it to make accusations, and then students were pressured to quickly react to the accusations without much access to the “proof.” Almost half of those accused (7 of 17 or 41%) have already had their cases dismissed (aka – they were falsely accused. Why is this not a criminal act?). Out of the remaining 10, 9 plead guilty, but 6 of those have now tried to appeal that decision because they feel they were forced to plead guilty. FYI – that is 76%(!) that are claiming they are falsely accused. Only one of those six wanted to be named – the other 5 are afraid of reprisals from the College if they speak up.

That is intense. Something is deeply wrong with all of that.

The frustrating thing about all of this is that plenty of people have been trying to warn that this is a very likely inevitable outcome of Learning Analytics research studies that look to detect cheating from the data. Of course, this particular area of research focus is not a major aim of Learning Analytics in general, but several studies have been published through the years. I wanted to take a look at a few that represent the common themes..

The first study is a kind of pre-Learning Analytics paper from 2006 called “Detecting cheats in online student assessments using Data Mining.” Learning Analytics as a field is usually traced back to about 2011, but various aspects of it existed before that. You can even go back to the 1990s – Richard A. Schwier describes the concept of “tracking navigation in multimedia” (in the 1995 2nd edition of his textbook Instructional Technology: Past, Present, and Future – p. 124, Gary J. Anglin editor). Schwier really goes beyond tracking navigation into foreseeing what we now call Learning Analytics. So all of that to say: tracking students’ digital activity has a loooong history.

But I start with this paper because it contains some of the earliest ways of looking at modern data. The concerning thing with this study is that the overall goal is to predict which students are most likely to be cheating based on demographics and student perceptions. Yes – not only do they look at age, gender, and employment, but also a learner’s personality, social activities, and perceptions (did they think the professor was involved or indifferent? Did they find the test “fair” or not? etc).

You can see by the chart on p.207 that males with lower GPAs are mostly marked as cheating, while females with higher GPAs are mostly marked as not cheating. Since race is not considered in the analysis, systemic discrimination could create incredibly racist oppression from this method.

Even more problematic is the “next five steps to data mining databases,” with one step recommending the collection of “responses of online assessments, surveys and historical information to detect cheats in online exams.” This includes the clarification that:

  • “information from students must be collected from the historical data files and surveys” (hope you didn’t have a bad day in the past)
  • “at the end of each exam the student will be is asked for feedback about exam, and also about the professor and examination conditions” (hope you have a wonderful attitude about the test and professor)
  • “professor will fill respective online form” (hope the professor likes you and isn’t racist, sexist, transphobic, etc if any of that would hurt you).

Of course, one might say this is pre-Learning Analytics and the current field is only interested in predicting failure, retention, and other aspects like that. Not quite. Lets look at the 2019 article “Detecting Academic Misconduct Using Learning Analytics.” The focus in this study is bit more specific: they seek to use keystroke logging and clickstream data to tell if a student is writing an authentic response or transcribing a pre-written one (which is assumed to only be from contract cheating).

The lit review of this study also shows that this study is not the only one digging into this idea. The idea goes back several years through multiple studies.

While this study does not get to the same Minority Report-level concerns that the last one did, there are still some problematic issues here. First of all is this:

“Keystroke logging allows analysis of the fluency and flow of writing, the length and frequency of pauses, and patterns of revision behaviour. Using these data, it is possible to draw conclusions about students’ underlying cognitive processes.”

I really need to carve out some time to write about how you can’t use clickstream data of any kind to detect cognitive processes in any way, shape or form. Most people that read this blog know why this is true, so I won’t take the time now. But the Learning Analytics literature is full of people that think they can detect cognitive activities, processes, or presence through clickstream data… and that is just not possible.

The paper does address the difficulties in using keystroke data to analyze writing, but proposes analysis of clickstream data as a much better alternative. I’m not really convinced by the arguments they present – but the gist is they are looking to detect revision behaviors, because authentic writing involved pauses and deletions.

Except that is not really true for everyone. People that write a lot (like, say, by blogging) can get to a place where they can write a lot without taking many pauses. Or, if they really do know the material, they might not need to pause as much. On the other hand, the paper assumes that transcription of an existing document is a mostly smooth process. I know it is for some, but it is something that takes me a while.

In other words, this study relies on averages and clusters of writing activities (words added/deleted, bursts of writing activity, etc) to classify your writing as original or copied. Which may work for the average, but what about students with disabilities that affect how they write? What about people that just work differently than the average? What about people from various cultures that approach writing in a different method, or even those that have to translate what they want to write into English first and then write it down?

Not everyone fits so neatly into the clusters.

Of course, this study had a small sample size. Additionally, while they did collect demographic data and had students take self-regulated learning surveys, they didn’t use any of that in the study. The SRL data would seem to be a significant aspect to analyze here. Not to mention at least mentioning some details on the students who didn’t speak English as a primary language.

Now, of course, writing out essay exam answers is not common in all disciplines, and even when it is, many instructors will encourage learners to write out answers first and then copy them into the test. So these results may not concern many people. What about more common test types?

The last article to look at is “Identifying and characterizing students suspected of academic dishonesty in SPOCs for credit through learning analytics” from 2020. There are plenty of other studies to look at, but this post is already getting long. SPOC here means “Small Private Online Course”… a.k.a. “a regular online course.” The basic gist is that they are clustering students by how close their answers are to each other and how close their submission times are. If they get the exact same answers (including choosing the same wrong choice) and turn in their test at about the same time, they are considered “suspect of academic dishonesty.” It should also be pointed out that the Lit Rreview here also shows they are the first or only people to be looking into this in the Learning Analytics realm.

The researchers are basically looking for students that meet together and give each other answers to the test. Which, yes – it is suspicious if you see students turn in all the same answers at about the same time and get the same grade. Which is why most students make sure to change up a few answers, as well as space out submissions. I don’t know if the authors of this study realized they probably missed most cheaters and just caught the ones not trying that hard.

Or… let me propose something else here. All students are trying to get the right answers. So there are going to be similarities. Sometimes a lot of students getting the same wrong answer on a question is seen as a problem to fix on the teaching side (it could have been taught wrong). Plus, students can have similar schedules – working the same jobs, taking the same other classes that meet in the morning, etc. It is possible that out of the 15 or so they flagged as “suspect,” 1 or 2 or even 3 just happened to get the same questions wrong and submit at about the same time as the others. They just had bad luck.

I’m not saying that happened to all, but look: you do have this funnel effect with tests like these. All of your students are trying to get the same correct answer and finish before the same deadline. So its quite possible there will be overlap that is very coincidental. Not for all, but isn’t it at least worth a critical examination if even a small number of students could get hurt by coincidentally turning in their test at the same time others are?

(This also makes a good case for ungrading, authentic assessment, etc.)

Of course, the “suspected” part gets dropped by the end of the paper: “We have applied this method in a for credit course taught in Selene Unicauca platform and found that 17% of the students have performed academic dishonest actions, based on current conservative thresholds.” How did they get from “suspected” to “have performed?” Did they talk to the students? Not really. They looked at five students and felt that there was no way their numbers could be anything but academic dishonesty. Then they talked to the instructor and found that three students had complained about low grades. The instructor looked at their tests, found they had the exact same wrong answers, and… case closed.

This is why I keep saying that Learning Analytics research projects should be required to have an instructional designer or learning research expert on the team. I can say after reviewing course results for decades that it is actually common for students to get the same wrong answers and be upset about it because they were taught wrong. Instructors and Instructional Designers do make mistakes, so always find out what is going on. Its also possible that there was a conversation weeks ago where one student with the wrong information spread that information to several students when discussing the class. It happens.

But this is what happens when you don’t investigate fully and assume the data is all you need. Throwing in a side of assuming that cheaters act a certain way certainly goes a long way as well. So you can see a direct line from assumptions made about personality and demographics of who cheaters are, to using clickstream data to know what is going on in the brain, to assuming the data is all you need…. all the way to the Dartmouth Medical School scandal. Where there is at least a 41%-76% false accusation rate currently.