Avoiding dehumanizing writers in our critiques of large language models

This past week, I ended up being quoted in an Inside Higher Ed article and also getting into a conversation on Mastodon with some scholars whose work I deeply admire: Iris van Rooij and Emily Bender. I came away from the latter conversation feeling like I had been a bit misunderstood, in a way that I often am: I’m in a weird position where folks tend to assume I’m super gung-ho about LLMs and I’m just…really, really not?

As I’ve said here before, I have zero interest in using LLMs myself. I’m going to be teaching a class called “Writing about (and with?) AI” in the Spring where my not-so-secret goal is to get students critiquing LLMs and articulating what matters about human writing now that there are tools that can generate writing that at least sometimes passes as human.

(Seriously, I’m constantly reminding colleagues that LLM output is “bullshit”, in the philosophical sense, even though I’m pretty sure my program director hates it when I say that. So it’s just funny that in some eyes, I’m too critical of LLMs and in others, I’m not critical enough. So be it. I’m just doing the best I can to be true to what I know, while staying open to listening and changing my thinking where called for.)

From my place as a linguist/cognitive scientist turned writing professor, I just feel very strongly that *some* of the dismissiveness about LLM-generated writing feeds directly into existing biases about student writers, and especially L2 writers and neurodivergent writers, which leads me to push back against certain framings that I feel are dehumanizing to those groups of writers. On the one hand, I understand how important it is to push back against the corporate hype that says these LLMs are on their way to “AGI”, or that overly anthropomorphizes LLMs. Because no, LLMs do not “understand” the way we do, and they do not have thoughts, experiences, goals, or intentions…and these are all things that are quite central to how (and why) we humans generate language. Those, like Emily Bender, who point to the anthropomorphizing rhetoric from AI-boosters as “dehumanizing” to actual human language users are not wrong to make that point.

But my concern is that the more we emphasize the non-humanness of LLMs, the more we risk dehumanizing actual human writers in another way. Here’s why: we know at this point that what I said about “AI detection” back in February has turned out to be true: false positives are inevitable, and they do seem to disproportionately hit two main populations: those whose native language is not English and those who are neurodivergent (the latter of these, I only have very preliminary data for, but it’s *quite* suggestive). And when I hear people talking dismissively about the output of LLMs, ostensibly with the goal of pointing out just how inhuman it is, I sadly also hear them using a lot of the same language (“predictable”, “no voice”, “generic”) that has been used to criticize writing by inexperienced student writers, and especially by inexperienced student writers whose first language is not English. And then much of the rhetoric about “sounding robotic” or “inhuman” is just straight-up the kind of othering language used against the neurodivergent, especially the autistic. This has real implications for how actual human writers are treated: see the case of Rua Williams, who I talked about in my last post.

Basically, when I push back against certain kinds of critiques of LLM-generated text, and point to important ways that LLM output and human writing ARE similar, it’s not coming from a desire to reinforce the corporate hype that says we should think of these LLMs as our intellectual equals (or even superiors). I don’t think that!! Nope, nope, nope.

Instead, it’s coming from the recognition that false positives are inevitable in “AI-detection”, and thus whatever we say about LLM-generated text, we’re also saying about (some) human writing. My point is that we have to be able to critique “AI-hype” *without* smearing LLM output with the same criticisms that have long been levied against L2 writing, student writing, neurodivergent folks, etc.

All along, the point I’ve been trying to make is that even if you think that LLMs are nothing more than “stochastic parrots”, even if (like me) you’re extremely aware that there’s no-one “there” in an LLM to have an intention, and thus LLMs cannot possibly “mean” anything by the text they produce in the sense we typically mean (oh, how overloaded with related but meaningfully different meanings the verb “to mean” and noun “meaning” are!)…even if we grant all of the points about how very *different* LLMs are from living, breathing, humans, we still must contend with the fact that the *text* that LLMs generate is, in some cases, indistinguishable from the *text* that human writers produce.

We are so accustomed to using writing as a way to test the underlying thought process of the writer, as “evidence” for the mind that produced it, and I know it’s disorienting to realize that the same words, in the same order, could have come from a living, breathing, thinking human, laying their thoughts out on the page, or from an LLM that is just producing a plausible continuation of the prompt, using the model it has built of language through playing the “guess the next word” game billions of times over on an enormous amount of human-generated writing. But that doesn’t make it untrue. (For a great take on why language is not the measure of intelligence we treat it as, see this piece from Karawynn Long: Language is a Poor Heuristic for Intelligence)

And yes, I do have a vested interest here: as part of the research I’m doing into the disproportionate flagging of neurodivergent writing as “AI-generated”, I’ve found that across a variety of “AI text detection” apps, between 10%–40% of a corpus of my own writing is flagged as at least partially “AI-generated”. I am, as I hope anyone would understand, invested in being recognized as fully human, and it’s scary to know that in terms of my writing, the very medium in which I feel most like myself, most able to think and communicate…I am not always seen that way. (It really just confirms the nagging feeling I’ve always had that I’m somehow alien, a feeling that so many neurodivergent folks, especially those diagnosed late in life, have lived with.)

Ever since I first discovered that my own writing gets flagged as “AI-generated”, it has been especially difficult for me to listen to people talk about how “terrible” LLM output is. Even though I know I’m not a terrible writer — I mean, hell, I’ve won awards for my writing! — it’s hard for me not to hear the constant drumbeat of commentary about the awfulness of LLM-generated writing as criticism of my own writing. And so much of it feels familiar to me as a person who writes from the social sciences rather than the humanities — think of the humanists’ common gripe about how “lifeless” scientific prose is. In a recent CNN piece in which academics and experts on academic integrity were interviewed, the output of ChatGPT was simultaneously called “robotic”, “inhuman”, and “voiceless”, and also compared directly to “the writing of a 50-year-old compliance lawyer”. Are compliance lawyers not human? I’m not saying I love reading compliance law text…I really, really don’t…but I’m deeply uncomfortable with identifying it as “inhuman” when it is produced by writers who are just as human as I am, who are simply writing within a genre that is (for many reasons) far more constrained than the ones in which I write.

This Reverse Turing Test that we are now all subject to, where every piece of writing we share is subjected to suspicions about its possible non-human provenance, is itself dehumanizing, and my point is that it’s made even more so when our critical rhetoric about LLMs focuses almost exclusively on how “inhuman” their output is.

I think that part of the reason we find such similarity in the output of an LLM and in (some) human writing is because prediction, the mechanism at the heart of how an LLM works, is also pretty central to human language processing. And of course, the patterns that inform the LLM’s predictions are all based on actual human writing. While I agree with Bender that there are fundamental limits to how much “meaning” can possibly be “learned” by a system that only has access to linguistic form (e.g. Bender & Koller 2020, Bender’s “Thai Library”), I also think that the only reason these systems are able to generate such fluent text, text that can be understood as meaningful by human readers, is because the embedding representations that the statistical model “learns” actually do encode information that would be hard to describe as anything other than “semantic” in nature. (See this piece from my friend Steve Piantadosi, along with Felix Hill, for more on this: Meaning without Reference in Large Language Models.) An awful lot can be inferred from co-occurrence. So while it’s missing the link to the actual embodied world, and thus LLMs can’t be said to “understand” what any of these co-occurrence statistics “mean” in anything like the way we do, *we* are able to understand the output produced by an LLM because it has used the statistical patterns in our linguistic output to develop representations that allow it to respond to prompts in ways that appear “meaningful” and “make sense” (to us!).

(Side note: this is another place where I think my position as a writing professor informs my perspective. We’re always reminding our students that no matter how clear the meaning in their own mind is when they write, the only thing readers have access to is the text on the page — they don’t have access to the writer’s inner thought process. And so readers can’t really be “wrong” about how they interpreted a piece of writing, even if their interpretation doesn’t align with what the writer intended, because once it’s just text on the page, the writer’s intent has to be inferred by the reader based on the text and nothing more. We have a lot of practice doing this, so it’s hardly surprising that we infer intent in text where no intentional actor was involved.)

Sadly, an awful lot of racist, sexist garbage is encapsulated in the statistics of our language, so inasmuch as LLMs are representing the concepts embedded in our language in some fashion, those concepts aren’t necessarily ones we want to amplify. Also, this way of building representations through prediction will inevitably leave LLMs without much representation of language (and the concepts embedded in it via co-occurrence) from those whose writing is less represented in the Common Crawl and other large internet-based datasets that are used to train LLMs. This is a huge inherent problem with this method of “learning” language! And it’s likely to end up accelerating the loss of less widely spoken languages and contributing to further erasure of viewpoints from marginalized voices. (For a great piece about this issue, see this Washington Post article from Viorica Marian: AI could cause a mass-extinction of languages — and ways of thinking.)

This brings me to the topic that kicked off some of the conversations that spurred me to write this piece: whether LLMs are inherently plagiarizing when they generate text. I was interviewed a couple of weeks ago by Susan D’Agostino, a journalist whose work on AI as it relates to higher education I’ve really admired; she was collecting insights from various academics about what “AI” (not just LLMs, but image-generators as well) brings to questions about the murkiness of authorship, and how we do (or don’t) give credit to influences. What I ended up talking about was the fact that, at least as far as linguistic knowledge goes, we are nothing BUT influences, since we learn language(s) from the speech community/communities in which we land as babies, and we further develop our idiolect(s) through media consumption of various kinds, as well as from whatever discourse communities we end up in as we begin writing within specific genres or disciplines. And of course, as we learn language from these communities, we also internalize particular ways of thinking about the world. But even though every word we’ve learned came from someone other than us, it doesn’t make sense to say that we’re “stealing” their words when we use them to express our own thoughts, which are themselves heavily influenced by the cultures and communities we are part of.

So, as I pointed out, I am certain that I have an odd amount of Britishisms in my idiolect for someone raised in the US, because I’ve consumed (with delight!) so much British media — Douglas Adams and Terry Pratchett and Neil Gaiman and Doctor Who and Monty Python and so much more — but could I tell you, when I use a turn of phrase that is more British English than American, whether it came from a Doctor Who episode, or from Pratchett or Adams, and if so, which book? Almost certainly not, unless I’m very intentionally quoting an episode or playing on a very well-known Pratchett turn-of-phrase. To my mind, an “idiolect” is also a model of language learned from input — one that is much, much richer than an LLM, and, crucially, connected to a thinking mind — but both are products of their input. Given that LLMs “learn” a model of English from training on an enormous corpus of English-language texts, I don’t think it makes sense to say that any given word that is used is “stolen” from particular texts in that corpus any more than my words are “stolen” from whatever speakers or writers helped form my idiolect. (Edit 8/29/23: Here’s a really interesting take on these issues as they relate to intellectual property.)

And yet, I do think that Iris van Rooij has a point when she describes LLMs as “automated plagiarism”. While some folks who make this argument point to a misguided idea of LLMs as somehow “copying” from a particular item in a searchable database of existing texts and then “pasting” it into the output, that’s not the basis for van Rooij’s point. What she is saying is that because LLMs are a product of their input but lack any way of pointing to an underlying source, the “ideas” they express, as we make sense of them, don’t have a known provenance. They may actually reflect ideas that would best be attributed to a particular scholar or line of work, or they may not, and as academics, this runs counter to the value we place on scientific integrity. It means that any time we use LLM output, we may be (inadvertently) plagiarizing some other scholar’s ideas, with no way of actually pointing to the source of those ideas. The plagiarism here, in my view, is really on the part of the human writer who chooses to use the output of an LLM without doing any kind of work to fact-check it and find the original sources (the process for which would almost certainly involve rewriting the output), or who tries to pass off the output of an LLM as their own.

(This, by the way, is why the biggest change I’ve made to my writing classes in light of LLMs is to double down on how much I talk about the importance of keeping good notes about sources, to require more explicit reflection on the influences that shape the arguments students make, and to ramp up how much coaching I do around using reference managers, building activities around this into basically every assignment…it’s not that I wasn’t talking about these things before, but it feels so much more important to emphasize now.)

One thing I spoke about with D’Agostino that didn’t make it into the article is that human writers have always had to grapple with knowing that they’ve been influenced by all kinds of things, not all of which are really straightforwardly cite-able in the way that scholarly work that we directly engage with in our writing would be, and some of which are only influences in a diffuse, hard to pinpoint way (like, on some level I know that the fact that I moved from NC to MN as a kid influenced a lot of my thinking about language, but how do I cite that, and for which ideas, exactly?). And there are various strategies that we use to deal with this. In the sciences in particular, Acknowledgements sections do a lot to recognize people whose work doesn’t quite rise to the level of authorship but whose work influenced and improved the final version of the paper: this might include lab employees, funding agencies, commenters at conference presentations, and peer reviewers. In fields influenced by feminist and/or critical theory, we also find Positionality Statements, which try to account for the ways in which the environments in which we were raised, the communities we are part of, the lived experiences we’ve had, and the scholarly positions we hold have influenced the perspectives we offer in our work. In both cases, authors attempt, however imperfectly, to point to those who have influenced the work beyond the individuals whose work is directly cited.

For LLMs, what would be the equivalent of an “Acknowledgements Section” or “Positionality Statement”? Here, I think Data Cards and Model Cards both aim to do something similar: to more carefully document the sources of the input to the model (Data Cards), and the ways in which that input shapes the model’s performance, particularly on issues relating to potential bias (Model Cards). The analogy here to Acknowledgements Sections and Positionality Statements is obviously an imperfect one, and sadly, neither Data Cards nor Model Cards seem to be priorities of the large companies developing LLMs, but I do think that both approaches (especially if taken together) would give us a way of more productively grappling with these questions of influence.

Anyway, I’m supposed to be writing a book chapter right now (with my friend Shawna Shapiro) about why we need both Critical Language Awareness (CLA) *and* Critical AI Literacy (CAIL) for just, equitable writing pedagogy that empowers students as rhetorical agents now that LLMs have hit the scene. We’re using “AI-Detection” as a kind of case study for what goes wrong when we approach our students with a policing mindset and misguided ideas, both about LLMs and about how human language works. It’s still more of a messy outline than a draft at this point, and I’m still thinking about how everything I’ve been talking about in this piece might fit in the chapter, but what has really been driven home to me over the past week is that CAIL really needs to integrate insights from CLA if it wants to avoid dehumanizing writers in the way it critiques LLMs.