If you’ve looked for pedagogical resources on using AI writing in the classroom (“best practices,” info on assignment design, suggestions on how to catch plagiarism) you’ve probably seen notes like this:

“Models like ChatGPT have been recorded in multiple cases to cite sources for information that are either not correct or do not exist” (Wikipedia, citing “Exploring the Boundaries of Reality: Investigating the Phenomenon of Artificial Intelligence Hallucination in Scientific Writing Through Chat GPT References” (Athaluri et al., 2023).)

“GPT-3, the current version which will soon be replaced and upgraded, can include citations and quotations, but they will sometimes be fictional” (“ChatGPT: The State of the Technology and Recommendations for Annenberg Faculty” (January 2023)).

Broadly speaking, this is true. But it’s maybe a bit more cut and dry than that:

ChatGPT is incapable of citing *anything* correctly, at least in the normal, colloquial way that we think about citation and attribution. On the surface, this might seem kind of trivial: the language model doesn’t handle facts and sources like a person. Sure, it’s a bit of pearl-clutching. But, this probably matters a lot when it comes to thinking about what place (if any) AI writing should have in a pedagogical setting.

I’ll break this down into three parts. First, I’ll outline a set of general goals/principles that I think might describe the goal/core of “traditional” (i.e. human) citation practices in an academic setting. I’m not a scholar in this field, so this is less to make a compelling argument and more to define some kind of consensus. Next, I’ll outline how ChatGPT works with respect to drawing from sources–how it cites things. Lastly, I’ll explain why I think this might be a challenge for integrating it into the classroom (at least not without some serious evaluation).

Citations and attribution: Why?

As a writing coach, most of the students I meet with come at the question of “why cite” from a pretty instrumental position: Because I’ve been told to. Being pushed to do something by an appeal to authority isn’t necessarily ineffective, but it can also cause students to misunderstand certain aspects of what they are being told to do. For instance, many students I’ve talked to understand that there is a difference between “academic” articles and “popular” articles, but will only describe the former as a “source.” If we care about students understanding why they should cite, that isn’t a great foothold. But, it does demonstrate something about their training and understanding: They’ve been told that they need to use academic articles as sources in their writing; popular online articles are not academic articles, therefore they don’t count as sources. The logic is wrong in a broad sense, but it serves a function. It really only ends up mattering in their coursework when they are asked to engage with sources that they don’t consider sources–it matters for how they think of “citation” as a practice more than how they perform the kinds of common writing tasks that they are evaluated on.

So, when I want to help students think about citation in a broader sense (i.e. why they are citing), I think it helps to outline a set of things that citations do:

• Citations and references have a rhetorical function
  • • Citations of outside material allow an author to appeal to facts/arguments that are not their own. They can use it to support or contextualize some claim they want to make.
    • Citations of outside material allow a reader to justify to themselves that the author’s argument is persuasive.
• Citations and references have an ethical function
  • • Citations of outside material allow an author to “give credit” to the work of another writer.
• Citations and references have a professional function
  • • Citations of outside material allow an author to demonstrate the scholarly body of work that they wish to contribute to.
    • Citations of outside material allow a reader to find materials that are relevant to a topic of their own interest.
• Citations and references have an instrumental function (especially for students)
  • • Citations of outside material allow a student to demonstrate that they have done the coursework that professors want them to.

This list isn’t exhaustive, but (again, anecdotally), it’s usually more than the students I work with have gotten before.

For the purpose of thinking about AI attempts to cite or attribute claims to specific sources, though, here’s what matters:

For each function, the effectiveness of the citation relies on three characteristics. First, the citation should usually reference a specific claim (argument, fact, statement of value). Second, the claim must tie back to a discrete reference (or set of references). From the reader’s point of view, it also matters that the reference can be found using the citation/reference entry. That’s why we write in APA/Chicago/IEEE/whatever. So that my making up an arbitrary reference structure doesn’t hamper somebody’s ability to find the thing I want them to find.

How do AI writing models like ChatGPT work?

I am not a very good humanities or social sciences scholar, and I was an even worse astrophysicist. But, being bad at both of those things might be helpful here. Certainly, there are more technical explanations of how AI writing works (I’ll cite more throughout), but I think that you can get pretty far by developing a more intuitive sense of the process. One way to do this is just to “ask” ChatGPT how it does what it does. This is dumb. But it’s dumb in a good way–dumb in the same way that testing a value of zero is a dumb way to figure out if an equation doesn’t work for the easiest possible case. It is dumb and it often works. If what ChatGPT explains about ChatGPT’s process doesn’t make any sense, that’s valuable to know. I’ve referenced a chatlog with ChatGPT throughout this post. Feel free to look through the full (long and meandering)  chatlog that I’ve linked to at the bottom of the page.

In broad strokes, here is how ChatGPT does what it does:

1. OpenAI has assembled a large training dataset, drawn from public and private sources. The individual objects that are used in this dataset begin as webpages, books, articles, things like that.
2. The materials in that dataset are preprocessed, cleaned and normalized. You can imagine that non-textual elements (html formatting code, images–at least for GPT 3.5, metadata, etc.) would make it difficult to model language, so that stuff is taken out.
3. Part of this preprocessing involves tokenizing the text into smaller bits–individual phrases and words. The process of segmenting the data like this is automated, but there was probably some kind of manual curation initially (and I imagine there still is to some extent). There are also different ways that the data can be tokenized (e.g. you can split the text according to a set of assigned rules, according to how often different tokens appear, by considering punctuation or not, etc.)
4. ChatGPT (or whatever AI) trains on this processed, segmented data. The “P” in “GPT” stands for “pre-processed.” This means that OpenAI puts the dataset together, sets out some general rules for how to use it, and then lets ChatGPT loose. This is different than how CAPTCHA’s work, where neural networks are trained by lots and lots of people teaching a program over and over again what a fire hydrant is by clicking on pictures of a fire hydrant when they want to access their Facebook accounts.
5. When you prompt ChatGPT with some kind of query, it analyzes the text of your prompt like it did with its training dataset, and then constructs a response according to the probabilities that it has assigned to the various things you are asking for and the various things you might accept as a plausible response. Think of twins who talk over each other. When one starts saying “we finish each other’s…” it makes more sense for the other to interject “sentences” rather than “salamis.” One response is more probable than the other.
6. Apart from it’s pre-processed training, ChatGPT does receive ongoing feedback that shapes how it responds. Some of this has to do with expert intervention (e.g. people at OpenAI trying to dissuade ChatGPT from writing really bad things) and some of it comes from prolonged public usage–you can let ChatGPT know whether a response is or is not what you were looking for.

So how does ChatGPT cite things?

That’s the thing–it doesn’t.

Here’s an example of a short response I asked ChatGPT to write:

Me: Can you write a brief (1000 word) essay about carbon taxes? Can you include references to expert and scholarly opinion?

ChatGPT: Absolutely, I’d be happy to help. Here’s an essay exploring carbon taxes with references to expert and scholarly opinions.

The Role of Carbon Taxes in Mitigating Climate Change

Climate change poses an unprecedented threat to the planet, and addressing it requires multifaceted approaches. Among these, carbon taxes have emerged as a pivotal policy tool, aiming to reduce greenhouse gas emissions by putting a price on carbon. This essay examines the effectiveness, benefits, and criticisms of carbon taxes, drawing from expert and scholarly opinions to assess their role in mitigating climate change.

Effectiveness of Carbon Taxes:

Numerous experts emphasize the efficacy of carbon taxes as a market-based mechanism to combat climate change. William Nordhaus, a Nobel laureate in Economics, highlights the significance of pricing carbon emissions to align economic incentives with environmental goals (Nordhaus, 2007). By internalizing the cost of environmental damage caused by carbon emissions, carbon taxes encourage businesses and individuals to adopt cleaner technologies and behaviors.

Scholars like Gilbert Metcalf suggest that carbon taxes can efficiently reduce emissions by providing a clear economic signal to polluters (Metcalf, 2009). This pricing mechanism incentivizes innovation in low-carbon technologies and encourages investments in renewable energy sources.

Benefits of Carbon Taxes:

Proponents argue that carbon taxes offer multiple advantages. Joseph Stiglitz, a Nobel laureate in Economics, emphasizes that revenue generated from carbon taxes can be used to fund renewable energy projects or offset other taxes, thus promoting sustainable investments (Stiglitz, 2019). Additionally, the predictability of a carbon price allows businesses to plan long-term strategies for emission reductions.

Scholarly research, such as that by Lawrence Goulder, indicates that carbon taxes can lead to substantial environmental benefits, including significant reductions in carbon dioxide emissions over time (Goulder, 1995). Moreover, a well-designed carbon tax can drive technological innovation and create new job opportunities in clean energy sectors.

Criticism and Challenges:

Despite the potential benefits, carbon taxes face criticisms and challenges. One key concern raised by experts like Robert Stavins is the potential regressive impact of carbon taxes on low-income households (Stavins, 2019). The burden of increased energy costs might disproportionately affect those with limited financial resources.

Scholars also argue that the effectiveness of carbon taxes depends on their design and implementation. A poorly structured tax may not achieve emission reduction targets and could lead to market distortions without adequately addressing broader environmental issues.

Conclusion:

Carbon taxes stand as a significant policy instrument in combating climate change, as supported by expert opinions and scholarly research. While challenges and criticisms exist, the consensus among many economists and environmentalists is that well-designed carbon pricing mechanisms, if implemented thoughtfully, can significantly reduce emissions while fostering innovation in clean technologies.

In conclusion, the effectiveness of carbon taxes in mitigating climate change hinges on their design, implementation, and complementary policies. Integrating expert insights and scholarly opinions into policy formulation and execution will be crucial to harnessing the full potential of carbon taxes in the global fight against climate change.

---

References:

Goulder, L. H. (1995). Environmental taxation and the double dividend: A reader’s guide. International Tax and Public Finance, 2(2), 157-183.

Metcalf, G. E. (2009). Designing a carbon tax to reduce US greenhouse gas emissions. Review of Environmental Economics and Policy, 3(1), 63-83.

Nordhaus, W. D. (2007). A review of the “Stern Review on the Economics of Climate Change”. Journal of Economic Literature, 45(3), 686-702.

Stavins, R. N. (2019). The promise and problems of pricing carbon: Theory and experience. Journal of Environment & Development, 28(1), 1-23.

Stiglitz, J. E. (2019). Addressing climate change through price and non-price interventions. The Manchester School, 87(S1), 48-64.

This isn’t too bad! Even better, all of the references actually correspond to things that someone wrote. They aren’t fake; this is not always the case. Let’s take a look at one citation/reference pair:

One key concern raised by experts like Robert Stavins is the potential regressive impact of carbon taxes on low-income households (Stavins, 2019).

Stavins, R. N. (2019). The promise and problems of pricing carbon: Theory and experience. Journal of Environment & Development, 28(1), 1-23.

Here’s the article. You’ll notice an immediate problem: the article is actually written by two authors, Joseph Aldy and Robert Stavins. And the page range is scuffed– the article actually goes from p. 152-180. I won’t hold that against it, though; ChatGPT did better with its reference entry than most citation generators do. So, not the biggest problems. But let’s look at the claim that is being made: [Aldy &] Stavins (2019) argue that carbon taxes may have a potential regressive impact on low-income households. Except they don’t. Here’s what they actually say: “Other socially valuable uses of revenue [from carbon taxes] include reduction of debt, and funding of desirable public programs, such as research and development of climate-friendly technology. The tax receipts could also be used to compensate low-income households for the burden of higher energy prices as well as compensating others bearing a disproportionate cost of the policy” (p. 156).  The closest they get to the claim that ChatGPT attributes to them is when they summarize expert concerns about another policy solution: “Critics of subsidy reform claim it will harm low-income households…” (p. 162)–again, a claim that they push against. If anything, this article is arguing the opposite of what ChatGPT said it is arguing.

Looking back at steps 1-3 in the section above, you’ll see that the data is pre-processed before ChatGPT ever gets a look at it. This means that it only ever encounters a source (a source that you might want it to cite) as a segmented, decontextualized set of tokens. ChatGPT has never “read” Aldy & Stavins (2019). And not just in the sense that ChatGPT doesn’t “read” anything, but more in sense that ChatGPT never had access to Aldy & Stavins (2019); it encountered words, phrases, and maybe whole sentences from Aldy & Stavins (2019) but never in the context that those words, phrases, and maybe whole sentences belonged to Aldy & Stavins (2019).

The same is true for every time ChatGPT “cites” something. Just like ChatGPT is modeling its prose on probability–choosing tokens that will build probable sentences that probably make sense in the given context–it is modeling references and citations to make them seem like probable expressions of citations (because we’ve directly asked for citations, or because the context in which it is writing seems to use them). To be clear: it isn’t choosing from a list of citations and references that overlap with the content of its response and hoping that it picks something plausible. That would be a reasonable assumption given the above example where the citation is more or less correct in terms of the topic that the source is writing about but unfortunately wrong with respect to what is being argued about that topic. Instead, it is simply recognizing that I have asked for a citation, understanding that an APA  citation more or less looks like “Last Name, First Initial. (DATE). Article title in sentence case. Journal Tile in Title Case and Italicized, Vol#(Iss#), page range” and then doing its best to madlib it. The fact that it got so close–the fact that Aldy & Stevens (2019) could be a plausible citation to include in a response like this (barring that it got the claim backwards)–is a function of the token “Aldy & Stevens (2019)” (or some combination of related tokens) appearing so frequently in responses like this. Just like the twin example, ChatGPT finished my salami.

Here’s me asking ChatGPT about it (emphasis mine):

Me: All of the articles that ChatGPT cited in that essay are real.

ChatGPT: Absolutely, those citations were generated based on real academic articles and authors that I’ve been trained on up until my last update in January 2022. The references provided in the essay were simulated to emulate what a well-researched and referenced essay on carbon taxes might look like.

If you need assistance with anything else or have further inquiries, feel free to ask!

Me: So those articles were in your training dataset?

ChatGPT: I don’t have direct access to my training data or know where specific articles might have come from in the dataset. The information I provide, including citations, is based on a wide range of data sources available on the internet and in various publications up until my last update in January 2022. The citations and references I generate are simulated and based on patterns learned during training rather than specific access to individual articles or databases.

Me: But you just told me ” those citations were generated based on real academic articles and authors that I’ve been trained on up until my last update in January 2022″

ChatGPT: I apologize for any confusion. Let me clarify: while the citations and references provided in the essay were structured to resemble genuine academic citations and authors, I don’t have direct access to specific articles or databases. The information I generate is based on patterns and associations in the data I was trained on up until January 2022. These references are simulated and not sourced directly from real publications or databases. If you need accurate citations or references for academic purposes, it’s best to consult reputable academic databases, journals, or publications to ensure the information’s credibility and accuracy.

Me: So ChatGPT just generated reference entries to mimic real articles? To show what entries of real articles in a reference page would look like?

ChatGPT: Exactly. The references provided were generated to mimic the structure and formatting of real academic references or citations that you might find in a scholarly article or paper. They serve as examples to demonstrate how citations from genuine articles would typically appear in a reference list, showcasing the format and style commonly used in academic writing. These reference entries aim to simulate the appearance and structure of citations from legitimate academic sources, but they are not sourced from actual articles or databases. They were created based on patterns and formatting styles learned during training to illustrate what references to real articles might look like in a scholarly context.

Me: But those are real articles. They aren’t simulated.

ChatGPT: I don’t have direct access to external databases or the ability to retrieve specific articles or citations. While the references provided in the essay resemble real academic citations, they are generated based on patterns and structures from the data I was trained on up until January 2022. The information I provide is not sourced directly from real articles, but rather simulated to emulate the structure and format of genuine references to academic works. If the references resemble real articles, it’s coincidental, as they are not based on direct access to external databases or actual publications.

Me: But what are the chances that your generated reference entries exactly match real reference entries?

ChatGPT: The likelihood of my generated reference entries exactly matching real reference entries from existing academic articles or publications is extremely low. While the references provided were structured to resemble genuine academic citations and authors, they are not based on direct access to external databases or specific articles. Instead, they are simulated based on learned patterns and formatting styles from the data I was trained on up until January 2022. If there are similarities between the generated references and real articles, it’s coincidental. The references are not sourced from actual publications but rather created to demonstrate how citations might appear in an academic context. For accurate and reliable references, it’s always best to refer directly to legitimate academic databases, journals, or publications.

Me: It might be that these articles are widely cited, so you have a lot of examples of these reference entries occurring near the kinds of claims that you’ve attributed to them. Right?

ChatGPT: That’s a possibility. The references and citations I generate are based on patterns and structures learned from a diverse range of data available up until my last update in January 2022. It’s plausible that certain articles or authors might be widely cited or referenced across various texts, leading to patterns in the data that influence the way I generate simulated references. However, it’s important to note that while these references are structured to resemble genuine academic citations, they are not sourced directly from specific articles. The information I provide is based on the patterns and associations within the data, and any similarities to real references are coincidental. For accurate and verified citations, consulting reliable academic sources and databases is recommended.

Me: But even if the reference entries are real, it doesn’t mean that they actually correspond to the claims that they are linked to in the essay, right? Like, just because Claim B is attributed to Article B in the essay, it does not mean that Article B made Claim B, right? You’re just pretending.

ChatGPT: Exactly. While the references in the essay may appear to attribute specific claims or information to certain articles, it’s important to note that the connections between the claims and the referenced articles are simulated and not based on direct sourcing or verification. The associations between claims and referenced articles are generated to emulate how academic citations might be structured in a scholarly context. These simulated references aim to illustrate the style and format of referencing but do not imply direct correlations between the claims made and the content of the referenced articles. Therefore, the references provided in the essay are for illustrative purposes and do not necessarily indicate that the referenced articles made those exact claims.

This kind of makes sense for why ChatGPT will hallucinate whole references, sometimes. In fact, that’s the norm–it always assumes the job of modeling some kind of reference if it is asked for it. The fact that it makes it look plausible sometimes is kind of incidental (I have no proof of this, but I’m guessing this points to the kinds of sources/databases that OpenAI uses to assemble its training dataset).

Why does this matter?

To be honest, I don’t know that it matters for any and all AI writing applications. It does matter in an educational setting, though. Everything about the citation modeling procedure I’ve described runs counter to the way that we want students to understand attribution in college.

Above, I outlined the required conditions for a citation or reference to be successful:

• The citation should usually reference a specific claim (argument, fact, statement of value).
• The claim must tie back to a discrete reference (or set of references).
• The reference can be found using the citation/reference entry.

But, because ChatGPT does not engage with entire sources–either in its training, or its responses–its use of citations cannot satisfy any of these:

• The citation should usually reference a specific claim (argument, fact, statement of value).
  • • But, ChatGPT does not draw claims from sources. It constructs plausible claims.
• The claim must tie back to a discrete reference (or set of references). From the reader’s point of view, it also matters that
  • • But, ChatGPT cannot check claims against discrete references because it never encounters discrete references.
• The reference can be found using the citation/reference entry.
  • • But, ChatGPT’s references are not roadmaps to find a source. They are plausible constructions of what a reference might look like.

Moreover, I outlined a few rhetorical, ethical, professional, and instrumental purposes of citations in university writing.  AI writing can satisfy one of those: it uses sources to add rhetorical weight to its arguments, when prompted. And I might even grant it the instrumental one, even if it doesn’t do either well. But it falters on the other two:

• Citations and references have an ethical function
  • • Citations of outside material allow an author to “give credit” to the work of another writer.
      • • But,  we don’t know what ChatGPT was trained on. Short of filing a claim with OpenAI’s legal department, there is no way to confirm that something you’ve written was included in ChatGPT’s training dataset. 
• Citations and references have a professional function
  • • Citations of outside material allow an author to demonstrate the scholarly body of work that they wish to contribute to.
    • Citations of outside material allow a reader to find materials that are relevant to a topic of their own interest.
      • • But, because ChatGPT does not encounter outside materials as discrete sources, it cannot conceptualize or describe an actual scholarly conversation except by accident.

Extra