we should all fear the false negative.
Below is a copy of a transcription that Robin DeRosa created of a VConnecting session at OpenEd ’18. Robin transcribed part of that conversation and published it on her personal blog under the title: Open Assessment: The Parable of the Scorpion, with Jess Mitchell. An “essay” follows and fleshes out a number of ideas.
Assessment is one of the lines where we determine the Haves and Have Nots, where we can change the trajectory of somebody’s life because of the way we are assessing. I want us to have a pit in our stomach about the possibility that our assessments are getting it wrong. That’s what I’m talking about. The Line. Our tolerance for false positives [should be more] than our tolerance for false negatives…
I want us to have those conversations and I want us to talk about what does it actually mean to assess? What are learning outcomes? And how do we do this in a way that, yes, it can be measured….
I worry that people have an over fondness for data and they treat it as though it’s a pure mountain stream. But what I often say is mountain streams have a lot of crap in them; don’t drink from the mountain stream! And data can have a lot of crap in it, too: how it’s collected, who is interpreting it, what they’re doing with the interpretation of it, what we’ve come to rely on as a “bang for your buck most people.” All of these are very dangerous. We see this manifesting in areas like artificial intelligence and machine learning. We see that when you are part of a marginalized group, you’re not going to be the bang for the buck, you’re never going to be the big numbers.
It points to our failures in assessment, because our failures in assessment are us absolving ourselves of what is very hard. We don’t measure the hard things. So we measure the easy things, and what you measure is what you value, we know that, so what are we measuring? We’re measuring admissions; we’re measuring retention; we’re measuring graduation; we’re measuring in a micro form the regurgitation of information…
Again Jesse and Sean get at some of this in their book on critical digital pedagogy. How are we measuring love of learning? Somebody comes along and says, “I love these two different topics and I want to figure out a way to combine them.” Like, “I love music and I love chemistry.” Where can these two meet? We have been so silly, thinking of the world in these boxes. And the absence of boxes doesn’t mean nothingness, and it doesn’t mean that we have no idea what we’re going to do. The absence of the boxes means we’re free and we can learn so much more and do so much more.
I look at something like in healthcare. There’s a scorpion– I think it’s from Israel, somewhere in the Middle East– its venom can illuminate cancer cells so that when you’re resecting tumors, you can know whether or not you got all the cells. Who on earth went out there into the desert and said, “You know what I’m going to do? I’m gonna to try to get this venom out of this scorpion, and I’m gonna sprinkle it on some cancer cells!” It is magic! But it’s science, too.
I think that our fondness for data and are our lack of criticality about data: there’s a fondness for science but a lack of understanding of what science is. Science is trying to prove the [sic] null, trying to prove it doesn’t work. So what I love about science is it asks the other question, not the “this is my assumption and I’m going to try to prove that,” but how do I disprove my assumption.
You Will Play the Game
I want to start here. Right here. This is the place. This is where we had the “gravel stop.” That’s what my mom called it. Back then there was a train track just ahead. This is a Google Maps Streetview — clearly the railroad track is gone. But the gravel is still there. My mother skid her tires in the gravel and stopped shortly. She pointed her index finger at me and executed a parent-level tirade. I would go to college. I would stop with this resistance to the structure of education — even if it made no sense — I would play along and that was final. That was the best deal she had to offer me. I felt fairly cheated and severely put-upon.
I can’t remember which time this happened: was it when I refused to memorize the countries and capitals of Africa (I wanted to dive in and explore each place and I thought memorizing was a waste of time); was it the time I refused to memorize the periodic table of elements (show me, let me experience them — memorizing made me feel patronized and bored, and besides, any lab I’d been to had them in a poster on the wall); was it the time I had to memorize all the prepositions in alphabetical order — about, above, across, after, against, along, among, around, at, before, by, down, during, for, from, in, into… nope… it wasn’t that time. I was in Grade 5 and I did that one and I’m still bitter about the uselessness of that exercise and how much space it still takes up in my brain.
It was early that I learned that assessments in education were usually a series of memorization responsibilities. There was no thinking, no learning, no exploring. Surely success wasn’t just measured as retention of dates and details! I got more from wandering around in the woods than I did from the classroom. I got more from skipping school to watch my peers build a half-pipe, to watch them work with local authorities to make skating a sport, not a crime. I got more from memorizing song lyrics that expanded my world and my imagination (and they indulged my moody teenage self).
Two Trains
I always had this image when I was in my teens and twenties. There were two trains traveling parallel to each other (in the same direction). I was riding one train and when I looked out the window, through the passing trees between the tracks, I could see to the other train.
That train was where the party was. It was where people were free, happy, curious, full of wonder, discovery, experimentation. They were tromping around in Gombe National Park in Tanzania interacting with gorillas. They were in the Antarctic studying penguins. They were building a machine that would convert mud puddles to potable water. They were mixing culture and music and history and making sense of the world. They were writing brilliant plays. They were the most gorgeous dancers you could imagine. They were sexy, they had great hair, they were cool.
And I was stuck on boring train memorizing the prepositions in alphabetical order. I was struggling through advanced Algebra. I was asked to memorize the periodic table and the countries and capitals of Africa. I wanted to BE in Africa. I wanted to be on that train so badly. And I had no idea how to jump the tracks. I had no idea if/when/ how I’d get to that train. Somewhere (not a specific moment in time) my life jumped the tracks and I found myself more and more on that train. What happened? When?? I mean, I don’t hang out with penguins, but I get to talk to you!
Opportunity Begets More Opportunity
I had opportunities that were (in large part) given to me — and opportunities beget more opportunities. Someone took a chance, then someone else did, then someone else noticed and they too took a chance. And the more people who took a chance, the more the others considering taking a chance felt reassured in their choice.
That was my career path, a kind of experiential act of showing I was capable, then showing I was capable consistently over time, and helping others feel assured I would be capable tomorrow. The very notion still makes me want to act out. Who wants to be predictable? Is predictable success? Is it innovative? Is it dynamic? Is it the sexy train? I don’t think so…
And yet, that is what education expects of us: achieve by showing through assessments that you are capable, that you are capable consistently, and that you are capable over time. But capable of what? Of memorization? Of regurgitation? Of meeting the instructors expectations and never figuring out what success looks like and feels like to me? I called bullshit. And my mom called time-out.
She eked out a compromise where I would play enough of the game that hopefully someone in a position of power would give me an opportunity (that I wouldn’t squander), and that would put me onto the ‘opportunity begets opportunity’ roller coaster of success. She agreed with me, adults WERE PHONIES and the system sucked. And drunk with the power of compromise, she started in on me about the importance of dressing appropriately; yeah, no, our conversation ended abruptly there. This might have been the moment my mother gave me a copy of J.D. Salinger’s The Catcher in the Rye. Indeed Holden…
It’s funny. All you have to do is say something nobody understands and they’ll do practically anything you want them to do. — Holden Caulfield
So, how do you get new experiences? You either take risks yourself (what a luxury!) or someone gives you space/permission to take risks (another luxury for many). Experiences are often given as rewards for achievement, and achievement is measured as following the rules, culminating with assessment. To get opportunities, and this was my Mom’s point, you have to play the game and ACHIEVE.
You got to know when to hold ‘em
know when to fold ‘em
know when to walk away
know when to run…— Kenny Rogers, The Gambler
To get onto the cool train you either work really hard or you step into some wicked luck, or both. It’s all just a gamble! Which brings me to assessment and harm.
Assessment and Harm
You have to achieve to have opportunities and we know opportunities are how you gain skills, experience, insight — the sorts of things you build a livelihood/career/upon. Opportunities are the amino acids (building blocks) of advancement and success. And those building blocks are granted based on performance in assessment.
We design success by showing achievement over time through ASSESSMENT. And in most cases this means scores, grades — we quantify (ASSESSMENT), track it over time (ACHIEVEMENT), and reward it (OPPORTUNITIES).
Are you telling me that the building blocks of the pathway to success really are made up of the details on the periodic table?? I don’t know about you, but this doesn’t look like the sexy train with the people with good hair on it… This looks like a lot of drudgery. And I’m not sure how much learning it represents. Isn’t it just knowing? Learning to me smells more like thinking, exploring, epiphany, critique, questioning…
Where does this drudgery come from? The University determines the minimum requirements (often GPA and SAT scores). The department determines the requirements to graduate with a degree (core and subject-specific). The syllabus lays out the rules. The Grammar Book lays out the rules. RULES RULES
Sign, sign, everywhere a sign
Blockin’ out the scenery, breakin’ my mind
Do this, don’t do that, can’t you read the sign?)— Five Man Electrical Band, Signs
Equitable?
I met a man at the OpenEd 18 conference who came up to me and told me he’d been rethinking his visceral reaction to grammar errors in his students work. He teaches in a poor community in Southern Florida — the kids are often non-native English speakers). He and I both grew up in households full of English speakers with a firm respect for grammar. My mother was a grammarian. In addition to teaching me to lose the stream-of-consciousness chatter, they broke me of using the work ‘like’ as a sentence filler. They made sure no up-speak entered my parlance. This colleague and I achieved in English — through no work of our own — through osmosis.
If you compare us with his students, using the same measures, then we will always win. And not only will we win the assessment game, but we will reap the rewards (the opportunities) that come from that win. SHOULD WE?! I don’t think so. So, how do we do this? What is “fair”? And how do we preserve good grammar?! Breaking this down, we see that those who speak the language (both literally and figuratively), those who play the game well and know how to play the game and whose birth circumstances give them an advantage in the game do well. How is everyone else doing?
Following this up the chain:
- The syllabus — who does it give an advantage to?
- The teaching method — who does it work for?
- The assessment — what methods does it advantage? whose notion of success?
- The department requirements — who do they disadvantage? exclude?
- The University minimum requirements — ever known anyone who is really good at testing but miserable at learning? vice versa? Who gets the advantage because of the way we’ve designed the game??
So, see, we took two steps into assessment and we’re squarely in the space of social justice: equity, diversity, and inclusion. Who is making the rules? Are the rules revisited as culture, humans, the economy, health, the environment change? How thoughtful are the rules that they are careful to not create unintended consequences or perpetuate cycles of exclusion and failure? Do you know the historical answers to these questions? The answer has almost always been those in power, those with influence/money. The dominant culture. Who do the rules disadvantage? Who gets marginalized? Are we surprised by the answer to that??
The fact of the matter is… the game ain’t fair —
There must be some kind of way outta here
Said the joker to the thief
There’s too much confusion
I can’t get no relief— Bob Dylan — All Along the Watchtower
Wait a minute, how did we get stuck here?!
We Design the Game
You see, we are the ones who design the game. And we know the deep relationship between form and function. And education has carved out some well-worn paths of ‘how this is done.’
- Pedagogy: Put them in chairs all lined up and staring at you and you’ll get silence and focus entirely on you. And very little engagement.
- Syllabus: Give them a strict rubric, they will function as though that strictness is a barrier between them and success.
- Assessment: Give them a strict path to success, they will focus on the path, not on the flowers all around them — not the things just off the path.
- High-Stakes Assessment: Give them a go/no-go, some will simply crumble under the pressure.
- Try-to-Think: Lead them through something exploratory — they might sink right back into the rut of what they’ve always done, not knowing how to do it differently. Doing it differently doesn’t come in a recipe. It isn’t some one way — it should be the outcome of creativity, context, thought, and engagement. One Size does not Fit All.
We put education in a straight coat and then we’re surprised when it isn’t whimsical, innovative, creative, and free.
And we do all of this in an effort to be fair. No exceptions. If I made an exception for you, I’d have to make it for everyone. I measure you all on the same scale, standardized! — that is how I know I’m being fair. I codify the structure into the syllabus in as much granular detail as I can to eliminate doubt, and so you all can be assured that I will treat you all the same. I will be fair. But are we all the same?!
An interesting thing happened during a co-design workshop I facilitated: we created an information architecture for a website and its content in an open, exploratory, intersectional, messy, collaborative, creative way. It had arrows and blobs and colours and it stretched across paper and white board and little post-it cards. When we had to present it to a group of stakeholders, an interesting thing happened. All our creativity grew up…
We flatten it, put it into outline form (I. followed by II.; A. followed by B.) and formalized it in our effort to validate it! We made it PHONY And the stakeholders were completely confused. The outline form was neat and simple, but it fell into a taxonomy that was fundamentally flat, not-inter-related, not creative.
We dressed up that creativity, combed its hair, gave it a starched button up, and some serious shoes. And in doing so we lost all the great work we had done. The stakeholders who weren’t at the design session began questioning the taxonomy of what we’d done and got bogged down in the form and function we’d created. We weren’t even talking about the content anymore. We were just talking about form.
We all do this…
When we are confronted with complexity and uncertainty, we lean back on simplicity and completeness and sameness. We approach the world with a transactional check-book accounting expectation — if we document our inputs and outputs, it should all reconcile neatly in the end. Tell me how I’ll be measured and only then will I know how to perform. Tell me the desired outputs and I’ll manhandle the inputs to make them conform.
So we codify the transaction in structure!
This is the syllabus — this is what you’ll be measured on. Strict adherence with little to no spark will get you a B, full-borne deviation will get you a D — because an F is really brutal.
Forget about glitter, forget about good hair — these are marching orders.
Nothing is Neutral
And that will be how we measure success. We will measure you all by exactly the same criteria — because that makes us feel fair; a rule of ‘no exceptions’ makes us feel equitable. But we know it isn’t. And…
we know this does damage… the flattening out and validating of our assessments is NOT A NEUTRAL ACT.
Listen,
Everybody knows
the fight is fixed;
the poor stay poor,
the rich get rich.
That’s how it goes.
Everybody knows the deal is rotten;
old black joe’s still picking cotton for your ribbons and bows
and everybody knows– Leonard Cohen, Everybody Knows
In an act of fairness, achievement is defined by adherence to the syllabus — because we need to measure to feel fair, validated, and objective. No bias here folks! But we all know well that if you’re excluded for any reason at the beginning of this game, you never catch up.
A high school student in Detroit and a high school student at Phillips Exeter will not have the same chances at any part of this equation. The playing field is not level.
And we’re all different. Adherence in many cases is merely conformance with the form of the game to give the intended output or (function). Draw a person — how would you evaluate Picasso, or Miro, or Kandinsky? Have you done so equitably?
What if Kandinsky took longer? What if Miro wanted to do it alone, rather than in a group? What if Picasso wanted to do it somewhere else all together — from another location?
- Pace, Path, Content, Delivery Method
- text, visual, sonification, video…
- individual, group, didactic, participatory
- Motivation — external, internal, positive, negative
- Social support — peer, instructor, other
- Degree of structure
These are just some of the diverse needs or preferences for learning. How do these fit into your rubric? Does your syllabus flex to these? Advantage some? Disadvantage others? Is the preferred mode even available?
And why do we create and tolerate these rules? How else can we get good numbers with which to make our data-driven decisions?
How Worried are We about the False Negative?
And how worried are we that we’re getting it wrong? That we’ll lose someone through a false negative in assessments?!
False positives I worry less about. Perhaps you’ve been innovative in gaming the system — perhaps it will all catch up with you. False negatives I worry A LOT about. Not just the diamond in the rough, but the pearls, the sandstone, the sparkles that we squash out.
Change the Rules, Change the Game, Change the Outcomes
Ask of your students that they understand the rule and then go beyond it. That they undo it from the inside. That they express something in the following of the rule. This is a place for a critical mind. For discovery, wonder, curiosity, creativity, love of learning.
Coming next: Structured Structurelessness…