Feedback: Human and Not

Acts of community.

I’ve been thinking about feedback.

There is a study out of Stanford Law School that says in a blind evaluation, law professors preferred the feedback on student questions about contract law generated by a large language model over those written by other law professors.

The purpose of this feedback is to enhance students’ understanding of legal concepts, based on questions students were asking about those concepts. The study’s authors suggest that this shows that LLMs can possibly “reason,” but given the architecture of LLMs what it shows is that they can simulate legal reasoning. Reasoning, as we should continue to think about it, is a human process of weighing evidence and rendering a judgment, not the product of the manipulation of language tokens on a predictive basis.

I had someone ask me if this changes my opinion about using LLMs to generate feedback on student writing and the answer is no. For one, feedback on student writing is a different kind of process than what the study attempts. For two, LLMs can’t read, and habituating students to valuing feedback from something that cannot read continues to strike me as a bad idea for reasons I go into extensively in More Than Words, but essentially boils down to my belief that we should treat writing as a communicative act between unique intelligences and LLMs simply have no relevance to this experience.

But what about LLM feedback as a supplement to human feedback in school contexts? There are a number of thoughtful people who are exploring this potential and I encourage these explorations because - similar to the Stanford study - each exploration not only reveals more about the capacities and limitations of generative AI technology, they help reveal the structures and values that inform the work teachers and students do inside of our institutions of schooling.

It is this area of “schooling” where LLMs show the most promise. They can be trained against rubrics to provide feedback that coaches students to improve outputs that will be graded against those rubrics. When it comes to the focus of my work - teaching writing - my objection to this feedback is that it is not particularly helpful feedback measured against the goal of helping students develop as writers.

I know this because I spent years giving the kind of feedback that LLMs are getting better at faithfully reproducing, evaluative responses that note features of the text measured against the rubric criteria that describes these features.

Here is a rubric I formerly used in my first-year-writing courses:

I would supplement these ratings with copious marginal comments on the document and summary comment at the bottom of the rubric. This method leans hard into what’s known as “summative” assessment, a description of what is happening in the artifact itself. My end comment attempted some additional “formative” assessment, e.g. “Next time do X, Y, Z…” but as if you count the sheer amount of different data points in the rubric, you’d quickly notice that it would be impossible to offer specific examples for each point. In theory, the students could triangulate the rubric scores, the end comment and the marginal comments in a way that added up to formative assessment, but very few if any did this, given that I had also engaged in the reflection-terminating practice of a numerical grade on the writing itself. (We have scores of studies that show students often do not read the comments when you provide a grade.)

Over time, it became clear to me that this method of feedback, no matter how detailed or thorough, was not going to work to help students develop as writers. Mostly what I was engaged in was a kind of assessment theater meant to provide a justification for whatever grade I put on the assignment. I knew it wasn’t working because my students would show vanishingly little transfer of the deep concepts I thought I was trying to teach. On each assignment, I could coach them to a decent grade, but it was clear that a broader capacity to tackle writing problems independent of that coaching was not happening.

My frustration with this process is really the origin story of what would become my framework of the “writer’s practice,” where over years of iterative experiments I tried to get away from feedback that categorized “what” happened based on the written artifact, to helping the student better understand “why” a piece of writing turned out a certain way. (For both good and ill.)

The feedback became more oriented around an extended conversation about the work writers do, the skills, knowledge, attitudes, and habits-of-mind writers deploy that result in the piece of writing. This required students themselves to be much more mindful and more rigorous when it came to considering their own writing. The grade didn't mean much anymore and no rubric could capture their experience.

My role evolved from judge/rubric applier, to that of reader and editor where I would offer a perspective on what a piece of writing was or was not achieving and requiring students to engage their practices to address these concerns.

Why I did not have this epiphany sooner I can’t tell you, or rather I can, but it’s dumb and mundane. I was simply repeating the kinds of evaluation that I’d been shown was appropriate for “school,” and modeling what had been done to me. Rather than fostering the development of a unique intelligence that could be expressed to the world, the true work of writing, I’d settled for coaching students to get a better grade.

I’d say I regret those years, except they were also the necessary experiences that helped form the views I believe in so strongly today, and why I think human feedback is essential for student writers - or any writer, really - that there is literally no substitute or even supplement to even non-expert response. I well-understand the constraints teachers/instructors work under that makes turning to this LLM aid attractive - I lived them for my entire career - but to give in to this temptation would be a betrayal of my deepest beliefs.

There is another aspect of human feedback that has been bouncing around my head.

Through sheer coincidence, this past I was in Chicago for a conference while my friend, Dave Eggers, was doing a live Q&A at the Old Town School of Folk Music to talk about his new novel, Contrapposto. Dave was in conversation with Kevin Boehm, the co-founder of the Boka Restaurant Group, one of the most accomplished restauranteurs in America (who just opened his 50th restaurant) and, like me and Dave, an alumnus of the University of Illinois who first matriculated to the school in the late 1980s.

More than one of Dave’s teachers from where he grew up (Lake Forest, Illinois) was in attendance at the event (“looking exactly the same,” in Dave’s words), and one theme that came out of the conversation was the importance of feedback as encouragement. Dave mentioned a high school teacher who had written a comment on a paper he’d written about Macbeth that said, essentially, You’re good at this. I hope you become a writer.

As Dave said, the notion had never occurred to him previously, because why would it?It literally put him on the trajectory for what has become his adult life.

There are number of scenes in the book exploring different varieties of feedback, including one set at an art school that’s fairly harrowing.

Boehm has his own version of teacher feedback which landed with particular force, an incident he describes in his memoir, The Bottomless Cup: A Memoir of Secrets, Restaurants, and Forgiveness, that I’m now about halfway through (aside: Holy smokes, what a story!) where he described himself as “mostly an A student, but a lazy one.”

“Only one educator cared enough to call me on it: English teacher Sandy Wands. A towering six feet tall, with a baritone voice, she grabbed me by the arm after I’d turned in a paper that outclassed my earlier work, revealing my untapped potential.

‘Quit selectively using your intelligence,’ she admonished me. ‘You were blessed to be born smart. Don’t waste it.’”

Boehm’s story reminded me of my own encounters with my 8th grade language arts teacher Mrs. Thompson, whom I wrote about previously when I explored the downside of “infinitely patient” AI tutors, and when it’s better for someone to just tell you to “cut the shit.”

Infinite Patience Is Not Good for Education

Sandy Wands and Dorothea Thompson weren’t just teachers, they were humans. Notice how Boehm recalls not just what Sandy Wands said, but how she presented herself. I can still picture Mrs. Thompson, her short steel-grey hair, not quite, but almost a crew cut, and her gravely voice that along with the rumor that she’d served in the military resulted in a nickname rooted in some measure of respect and fear, “Sarge.”

Mrs. Thompson was not necessarily warm in her presentation, but there was little doubt that she cared both about what we were doing and what she was doing. To the extent my own students seemed to think I was a good teacher even when I was torturing them with that rubric I think it was because it was clear to them that I too cared, maybe more than some of them did, and sometimes that enthusiasm might be contagious.

This obvious caring was its own form of feedback, right? Having an adult authority figure take time to read what you’ve written and not respond by correcting, but to say, “this is interesting” can have life-changing consequences, maybe now more than ever when young people are increasingly doubting if there is a place for them in this world.

Maybe showing we care is the high calling of any feedback we can give to students or writers or anyone else who is attempting personal expression. At the least it should be present in any feedback we give, positive, negative, or otherwise.

What do we signal to students when - either in part or whole - we send them to a large language model for their feedback?

That’s a sincere question.

Links

This week at the Chicago Tribune I write about what should be your choice for gift book of the year, The Dog’s Gaze.

At Inside Higher Ed I shared a guest essay by some fine folks arguing what I think should be clear to us all, the mission of higher education is learning.

At Academic Freedom on the Line I have some information on two significant threats to not just academic freedom, but freedom freedom that you can help repel by offering comment.

I haven’t watched/listened to this yet, but here’s Dave on Wild Card with Rachel Martin talking Contrapposto, and apparently, about the same stuff I’m pondering in today’s newsletter:

Want to see how a copyeditor thinks? Take Benjamin Dreyer’s copyediting test, and only then should you check out the answer key.

Fantastically interesting reflection from Ann Kjellberg on what it means to her and to others to review books.

With Space X’s IPO in the news this week I offer this guidance via my friends McSweeney's, “AI Economics for Dummies” by Andrew Singleton.

Recommendations

1. I Who Have Never Known Men by Jacqueline Harpman
2. The Correspondent by Virginia Evans
3. A Manual for Cleaning Women by Lucia Berlin
4. Last Night at the Telegraph Club by Malinda Lo
5. We Were the Lucky Ones by Georgia Hunter

Genie S. - Tulsa, OK

Closing my eyes, searching my gut, waiting for a cover to appear in my mind’s eye: Dinosaurs by Lydia Millet.

1. American Prometheus by Kai Bird and Martin Sherwin
2. Already Free by Bruce Tift
3. Small Things Like These by Claire Keegan.
4. Born to Run by Bruce Springsteen
5. 1929: Inside the Greatest Crash in Wall Street History - and How it Shattered a Nation by Andrew Ross Sorkin.

Keith S. - Charleston, SC

Mostly nonfiction, so I’ll stick with that and lean into music: Sticky Fingers: The Life and Times of Jann Wenner and Rolling Stone by Joe Hagan.

This brings us to the end of another installment of The Biblioracle Recommends. I am grateful for each and every one of you who have read to this, the end.

JW
The Biblioracle