Digital Learning Week · UT Knoxville
What’s Haunting Our Assessments:
A Hopeful Case for Life After the Zombie Essay
In the Zoom chat
In a word or two: what’s haunting your assessments?
Chapter 1
AI is weird
A test from Colin Fraser, via Gary Marcus
“Generate an image of people having fun at the beach, and subtly include a single elephant somewhere in the image where it is very hard to see without extensively searching. It should be camouflaged by the other elements of the image.”My prompt, after Colin Fraser’s example
“…a really smart, but (occasionally) drunk, intern.”Punya Mishra, “ChatGPT is a smart, drunk intern,” 2023
That image was so 2025.
What about today?
“Generate an image of people having fun at the beach, and subtly include a single elephant somewhere in the image where it is very hard to see without extensively searching. It should be camouflaged by the other elements of the image.”The same prompt, again
Me, to ChatGPT:
great. circle the elephant
Chapter 2
Ghostwriters
The take-home essay.
Still assigned.
No longer alive.
94%
of AI-written exam answers went unnoticed by graders.
Scarfe et al., 2024 · PLOS ONE
https://doi.org/10.1371/journal.pone.0305354
And on average, they outscored the real students.
Five modules of a UK psychology degree. Fully AI-written answers, submitted in secret.
Scarfe et al., 2024 · PLOS ONE
https://doi.org/10.1371/journal.pone.0305354
15.7%
of students in surveys since 2014 admitted paying someone to do their work.
Before ChatGPT. In an Australian study, a survey that rewarded honest answers got 2.5 times as many admissions as a standard one.
Newton, 2018 · Frontiers in Education · Curtis et al., 2021 · Studies in Higher Education
https://doi.org/10.3389/feduc.2018.00067
AI didn’t break the essay.
It showed us the essay was
already for sale.
The source changed.
The drop in exam scores didn’t.
Plagiarism moved from Chegg to ChatGPT. Students who plagiarized more scored lower on the final. The drop was statistically the same before and after.
Chen et al., 2024 · ACM Learning @ Scale
https://doi.org/10.1145/3657604.3662046
“ChatGPT has not created a problem that wasn’tJohn Warner, “ChatGPT Can’t Kill Anything Worth Preserving,” 2022
already present.”
Does the grade still tell you what the student can do?Honestly, no.
And that is fixable by design.
The question isn’t whether AI can write a good essay.
It’s whether writing a good essay is
actually the point.
Chapter 3
Performing isn’t learning
Performing is what you can do
with the tool.Learning is what you can do
without it.
Researchers put EEG headsets on 54 adults and watched their brains write essays three different ways.
ChatGPT only. Search only. Brain only.
Kosmyna et al., 2025 · MIT Media Lab · 54 students, not yet peer-reviewed
https://arxiv.org/abs/2506.08872
Of 18 people who wrote with ChatGPT, 15 couldn’t quote a sentence from the essay they had just written.
In the first session. Only 2 of 18 who wrote with their own brains had the same trouble.
Kosmyna et al., 2025 · not yet peer-reviewed
The ChatGPT group did worse on every level the study measured: brain, language and essay scores.
Kosmyna et al., 2025 · not yet peer-reviewed
“Less stress, better scores, same learning”
275 students. Randomly assigned to ChatGPT, a hint-only AI tutor, or no AI. The AI groups scored much higher on the exercise. No group learned more.
Bassner et al., 2025 · Computers & Education: Artificial Intelligence
https://doi.org/10.1016/j.caeai.2025.100537
ChatGPT raised essay scores. Not knowledge. Not transfer.
117 university students, randomized.
Fan et al., 2024 · British Journal of Educational Technology
https://doi.org/10.1111/bjet.13544
AI helps students perform better while they’re using it. It’s not the same thing as learning.
“An important distinction between tool-supported performance and durable learning”Fesler et al., 2026, Stanford SCALE Initiative
Chapter 4
What replaces the essay
So, what do we do?
Nothing replaces the essay. We change what counts as evidence of learning.
Four patterns
What the research keeps pointing to
Giving students AI rules is not a redesign.
Change the task.
Corbin, Dawson & Liu, 2025 · Assessment & Evaluation in Higher Education
In the Zoom chat
What could a student hand you that shows
how they got there?
Second grade
“Dad, how did Galileo make his telescope?”
“Well, he took two magnifying glasses and stuck them on both ends of a toilet paper tube.”
Grade the footprints,
not just the paper.
Students who made short explanation videos reported bigger gains in learning and confidence.
Three courses, two departments. Face-to-face and remote.
Cardace et al., 2024 · Active Learning in Higher Education
https://doi.org/10.1177/14697874241287208
Graded on results, students cheated more. Graded on how they solved it, they didn’t.
169 university students. Some test questions could only be solved by cheating. Among students told to look competent, the grading rule changed what they did.
Daumiller & Janke, 2019 · AERA Open
https://doi.org/10.1177/2332858419894276
Math
Have students argue with the AI. Grade their reasoning, not its answer.
ChatGPT got continuity and the derivative wrong, and the student had to catch it. A one-student case study.
Urhan et al., 2024 · Interactive Learning Environments
https://doi.org/10.1080/10494820.2024.2308093
Programming
Given a “See Solution” button, 14% pressed it on every problem.
When the answer is one click away, some students never try. 885 students, three problems.
Kapoor et al., 2026 · Australasian Computing Education Conference
https://doi.org/10.1145/3786228.3786233
Programming
Oral exams for 628 students. Students still rated them fair.
An intro computing course for non-majors. Oral scores and computer-exam scores matched only loosely, so the orals may be measuring something the computer exam misses.
Ellis et al., 2026 · ACM ITiCSE
https://doi.org/10.1145/3803400.3809302
Organic chemistry
A chemistry lab swapped some written reports for oral reports. Students scored higher.
Organic Chemistry II, compared with written reports on similar experiments. Online and in-person orals scored the same.
Mehltretter et al., 2026 · Journal of Chemical Education
https://doi.org/10.1021/acs.jchemed.5c00889
You’re not adding a task.
You’re trading one.
EDL 753
A decision record, beside every piece of work
An audit trail, not an AI rule.
Decision record
- I decided
- Because
- I rejected
- I still don’t know
- Who changed my mind
EDL 571
Four live team presentations carry the course’s assessment.
Show What You Know Days.
At scale
722 students. One-on-one oral finals. Grades went up, with no equity gaps.
Bioscience, before and after the switch.
Davey et al., 2025 · Assessment & Evaluation in Higher Education
https://doi.org/10.1080/02602938.2025.2502577
Authentic is not enough. The thinking has to be visible, or live.
Even on authentic assessments, graders mostly couldn’t tell AI-assisted work from student work.
Kofinas et al., 2025 · British Journal of Educational Technology
https://doi.org/10.1111/bjet.13585
Chapter 5
Durable skills and the unAI-ables
Remember when I said “generative AI is weird”?
“When the going gets weird, the weird turn pro.”Hunter S. Thompson
We are the “pros.”
Let’s keep basic, fundamental literacies
at the forefront.
“First of all,” said Serafim, “there are no advanced skills; there is only skillful movement.”Dan Millman, The Journeys of Socrates
“Durable skills.” Employers ask for them in job ads.
That is demand. It is not proof we can measure them.
America Succeeds and Emsi (now Lightcast), 2021: 82 million job postings
A polished essay can’t show any of them.
A skill is only durable if we can see it.
Dr. Brandeis Marshall
What’s unAI-able is what we should measure.
Data scientist; CEO, DataedX Group
UnAI-able
Contextual awareness
Ask for the student’s reading of the situation, not just their answer.
The reading predicted how peers rated them later, beyond the answer they chose.
Rockstuhl et al., 2015 · Journal of Applied Psychology
https://doi.org/10.1037/a0038098
UnAI-able
Conflict resolution
A chatbot simulation helped measure a skill no chatbot has.
215 students. Scores from a chatbot teamwork simulation predicted how peers rated them in a live team task, beyond a written judgment test and a personality test.
Kotlyar & Krasman, 2021 · International Journal of Selection and Assessment
https://doi.org/10.1111/ijsa.12368
UnAI-able
Critical thinking
Two critical thinking tests agreed fully on 45.5% of students.
For 10%, the results were opposite. The format decides what you measure.
Hyytinen et al., 2015 · Studies in Educational Evaluation
https://doi.org/10.1016/j.stueduc.2014.11.001
“What ChatGPT, and other AI tools, can’t replicate is student voice.”Paul Turnbull, Mid-Pacific Institute, on the New View EDU podcast
Chapter 6
The hopeful close
A carefully designed AI tutor beat an active-learning class. It was built on the same teaching principles.
194 Harvard physics students, each in both conditions, in random order.
Kestin et al., 2025 · Scientific Reports
https://doi.org/10.1038/s41598-025-97652-6
“Everything around you was architected by another person.”Abby Covert, How to Make Sense of Any Mess
“All design is redesign.”Larry Leifer, a founding member of the Stanford d.school (and my former boss)
You got this.
Thank you
What’s Haunting Our Assessments:
A Hopeful Case for Life After the Zombie Essay
I’d love to stay in touch.
Sources · 1 of 6
References
- A.I. chatbots defeated doctors at diagnosing illness. (2024, November 17). The New York Times. https://www.nytimes.com/2024/11/17/health/chatgpt-ai-doctors-diagnosis.html (opens in a new tab)
- Barcaui, A. (2025). ChatGPT as a cognitive crutch: Evidence from a randomized controlled trial on knowledge retention. Social Sciences & Humanities Open, 12, 102287. https://doi.org/10.1016/j.ssaho.2025.102287 (opens in a new tab)
- Bassner, P., Lenk-Ostendorf, B., Beinstingel, R., Wasner, T., & Krusche, S. (2025). Less stress, better scores, same learning: The dissociation of performance and learning in AI-supported programming education. Computers and Education: Artificial Intelligence, 10, 100537. https://doi.org/10.1016/j.caeai.2025.100537 (opens in a new tab)
- Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. https://doi.org/10.1073/pnas.2422633122 (opens in a new tab)
- Cardace, A., Hefferon, K., Levina, A., Angert, E. R., Buckley, D. H., Miller, W., Sanfilippo, L., Silva, T., & Aslan, C. (2024). Versatile video assignment improves undergraduates’ learning and confidence. Active Learning in Higher Education, 27(1), 123–141. https://doi.org/10.1177/14697874241287208 (opens in a new tab)
- Chen, B., Lewis, C. M., West, M., & Zilles, C. (2024). Plagiarism in the age of generative AI: Cheating method change and learning loss in an intro to CS course. In Proceedings of the Eleventh ACM Conference on Learning @ Scale (pp. 75–85). https://doi.org/10.1145/3657604.3662046 (opens in a new tab)
- Clemett, V. J., & Raleigh, M. (2021). The validity and reliability of clinical judgement and decision-making skills assessment in nursing: A systematic literature review. Nurse Education Today, 102, 104885. https://doi.org/10.1016/j.nedt.2021.104885 (opens in a new tab)
Sources · 2 of 6
References
- Corbin, T., Dawson, P., & Liu, D. (2025). Talk is cheap: Why structural assessment changes are needed for a time of GenAI. Assessment & Evaluation in Higher Education, 50(7), 1087–1097. https://doi.org/10.1080/02602938.2025.2503964 (opens in a new tab)
- Covert, A. (2014). How to make sense of any mess: Information architecture for everybody.
- Curtis, G. J., McNeill, M., Slade, C., Tremayne, K., Harper, R., Rundle, K., & Greenaway, R. (2021). Moving beyond self-reports to estimate the prevalence of commercial contract cheating: An Australian study. Studies in Higher Education, 47(9), 1844–1856. https://doi.org/10.1080/03075079.2021.1972093 (opens in a new tab)
- Daumiller, M., & Janke, S. (2019). The impact of performance goals on cheating depends on how performance is evaluated. AERA Open, 5(4). https://doi.org/10.1177/2332858419894276 (opens in a new tab)
- Davey, S. K., Birbeck, D., Nallaya, S., Sallows, G., & Della Vedova, C. B. (2025). Utilising one-on-one interactive oral assessments as the major final assessment within a bioscience course. Assessment & Evaluation in Higher Education, 50(7), 1154–1171. https://doi.org/10.1080/02602938.2025.2502577 (opens in a new tab)
- Ellis, S. E., Chao, A., Vadaparty, A., & Bussannagari, P. R. (2026). Oral exams at scale in introductory computing: Student outcomes and perceptions. In Proceedings of the 31st ACM Conference on Innovation and Technology in Computer Science Education V. 1 (pp. 660–666). https://doi.org/10.1145/3803400.3809302 (opens in a new tab)
- Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gašević, D. (2024). Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2), 489–530. https://doi.org/10.1111/bjet.13544 (opens in a new tab)
Sources · 3 of 6
References
- Fesler, L., Martinez Claeys, J. P., Agnew, C., & Loeb, S. (2026). The evidence base on AI in K-12 [Report]. Stanford SCALE Initiative. https://scale.stanford.edu/sites/default/files/The%20Evidence%20Base%20on%20AI%20in%20K-12%20Report.pdf (opens in a new tab)
- Hyytinen, H., Nissinen, K., Ursin, J., Toom, A., & Lindblom-Ylänne, S. (2015). Problematising the equivalence of the test results of performance-based critical thinking tests for undergraduate students. Studies in Educational Evaluation, 44, 1–8. https://doi.org/10.1016/j.stueduc.2014.11.001 (opens in a new tab)
- Ipeirotis, P., & Rizakos, K. (2026). Scalable and personalized oral assessments using voice AI. Communications of the ACM. https://doi.org/10.1145/3831714 (opens in a new tab)
- Kapoor, A., Denny, P., Porter, L., MacNeil, S., & Diaz, M. (2026). Exploring student behaviors and motivations when using AI teaching assistants with optional guardrails. In Proceedings of the 28th Australasian Computing Education Conference (pp. 22–31). https://doi.org/10.1145/3786228.3786233 (opens in a new tab)
- Kestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. (2025). AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting. Scientific Reports, 15, 17458. https://doi.org/10.1038/s41598-025-97652-6 (opens in a new tab)
- Kofinas, A. K., Tsay, C. H.-H., & Pike, D. (2025). The impact of generative AI on academic integrity of authentic assessments within a higher education context. British Journal of Educational Technology, 56(6), 2522–2549. https://doi.org/10.1111/bjet.13585 (opens in a new tab)
- Kosmyna, N., Hauptmann, E., Yuan, Y. T., Situ, J., Liao, X.-H., Beresnitzky, A. V., Braunstein, I., & Maes, P. (2025). Your brain on ChatGPT: Accumulation of cognitive debt when using an AI assistant for essay writing task [Preprint]. arXiv. https://arxiv.org/abs/2506.08872 (opens in a new tab)
Sources · 4 of 6
References
- Kotlyar, I., & Krasman, J. (2021). Virtual simulation: New method for assessing teamwork skills. International Journal of Selection and Assessment, 30(3), 344–360. https://doi.org/10.1111/ijsa.12368 (opens in a new tab)
- Lewis, D. (2022). Impacts of standards-based grading on students’ mindset and test anxiety. Journal of the Scholarship of Teaching and Learning, 22(2). https://doi.org/10.14434/josotl.v22i2.31308 (opens in a new tab)
- Lightcast (formerly Emsi). (2021, April). New research with America Succeeds reveals the high demand for durable skills. https://lightcast.io/resources/blog/new-research-with-america-succeeds-reveals-the-high-demand-for-durable-skills (opens in a new tab)
- Marcus, G. (2024). Taming Silicon Valley: How we can ensure that AI works for us. MIT Press.
- Marshall, B. (n.d.). What’s UnAI-able. https://www.brandeismarshall.com/ (opens in a new tab)
- Mehltretter, S. A., Schmitzer-Torbert, N., & Wysocki, L. M. (2026). Oral reports in an organic chemistry laboratory curriculum. Journal of Chemical Education, 103(7), 3961–3965. https://doi.org/10.1021/acs.jchemed.5c00889 (opens in a new tab)
- Millman, D. (2005). The journeys of Socrates.
Sources · 5 of 6
References
- Mishra, P. (2023, July 26). ChatGPT is a smart, drunk intern: 3 examples. https://punyamishra.com/2023/07/26/chatgpt-is-a-smart-drunk-intern-3-examples/ (opens in a new tab)
- Newton, P. M. (2018). How common is commercial contract cheating in higher education and is it increasing? A systematic review. Frontiers in Education, 3, 67. https://doi.org/10.3389/feduc.2018.00067 (opens in a new tab)
- Rockstuhl, T., Ang, S., Ng, K.-Y., Lievens, F., & Van Dyne, L. (2015). Putting judging situations into situational judgment tests: Evidence from intercultural multimedia SJTs. Journal of Applied Psychology, 100(2), 464–480. https://doi.org/10.1037/a0038098 (opens in a new tab)
- Scarfe, P., Watcham, K., Clarke, A., & Roesch, E. (2024). A real-world test of artificial intelligence infiltration of a university examinations system: A “Turing Test” case study. PLOS ONE, 19(6), e0305354. https://doi.org/10.1371/journal.pone.0305354 (opens in a new tab)
- Spurlock, S. (2023). Improving student motivation by ungrading. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (pp. 631–637). https://doi.org/10.1145/3545945.3569747 (opens in a new tab)
- Urhan, S., Gençaslan, O., & Dost, Ş. (2024). An argumentation experience regarding concepts of calculus with ChatGPT. Interactive Learning Environments, 32(10), 7186–7211. https://doi.org/10.1080/10494820.2024.2308093 (opens in a new tab)
- Walker, B. J., Kalaydzhieva, N., Navarro Lameda, B., & Reynolds, R. A. (2025). Generative AI performance in core undergraduate mathematics: A curriculum-level case study [Preprint]. arXiv. https://arxiv.org/abs/2509.13359 (opens in a new tab)
Sources · 6 of 6
References
- Warner, J. (2022, December 11). ChatGPT can’t kill anything worth preserving. The Biblioracle Recommends. https://biblioracle.substack.com/p/chatgpt-cant-kill-anything-worth (opens in a new tab)
- Warner, J. (2025, February 2). AI boosters think you’re dumb… and want you to stay that way. The Biblioracle Recommends. https://biblioracle.substack.com/p/ai-boosters-think-youre-dumb (opens in a new tab)
- Yik, B. J., & Dood, A. J. (2024). ChatGPT convincingly explains organic chemistry reaction mechanisms slightly inaccurately with high levels of explanation sophistication. Journal of Chemical Education, 101(5), 1836–1846. https://doi.org/10.1021/acs.jchemed.4c00235 (opens in a new tab)
Years are first-publication (online) years. Images: beach scenes generated with ChatGPT; unAI-able images from Pixabay (John Hain; Hans; Ioana Radu). Hunter S. Thompson photo and “Write Smarter” still as used in the original deck.
Handout
Full notes
The talk asks one question: does the grade still tell you what the student can do? The honest answer is no, and the fix is a matter of design. These notes give the evidence behind each chapter, with a link to each study. Full references are in the References section. Most study summaries here come from the papers’ abstracts.
1. AI is weird
AI is impressive and unreliable at the same time. A chatbot outdid physicians at diagnosis in a small study reported by The New York Times. The same kind of tool, asked to hide an elephant in a beach scene, first put the elephant in plain view, then left it out and circled two children when asked where it was. Asked who said a quote, Gemini named Donald Schön and explained his philosophy behind it; he never said it, and Gemini changed its answer only when pushed. Working with these tools is like having “a really smart, but (occasionally) drunk, intern” (after Punya Mishra, 2023).
2. Ghostwriters
The take-home essay is still assigned, but it no longer shows what a student can do. AI did not create the problem. Essays were bought and outsourced long before ChatGPT, and the learning cost of outsourcing was already there. AI made it cheap, fast and visible.
| Study | What they did and what happened |
|---|---|
| Scarfe et al., 2024 (opens in a new tab) | Researchers secretly submitted fully AI-written answers into the exam system in five modules of a UK psychology degree. Markers did not flag 94% of them. On average the AI answers scored half a grade boundary higher than real students. |
| Newton, 2018 (opens in a new tab) | Pooled 71 student surveys (54,514 students, back to 1978). On average 3.5% admitted paying someone else to do their work; in surveys from 2014 on, 15.7% did. Students under-report, so the real share is probably higher. |
| Curtis et al., 2021 (opens in a new tab) | Surveyed 4,098 Australian students with a method that rewards honest answers. 2.5 times as many admitted buying work as on a normal survey. |
| Bretag et al., 2019 (opens in a new tab) | Surveyed 14,086 students at 8 Australian universities. Students unhappy with the teaching and learning environment, or who saw “lots of opportunities to cheat,” were more likely to outsource their work. |
| Chen et al., 2024 (opens in a new tab) | Tracked plagiarism in a large intro programming course before and after ChatGPT. Plagiarism rose and moved from Chegg and CourseHero to ChatGPT. Students who plagiarized more scored lower on the final exam; statistically, that drop was the same before and after ChatGPT (a correlational study). |
| Fischer et al., 2024 (opens in a new tab) | 118 postgraduate management students reported how they used AI on a 2,500-word essay. Most used it to write, paraphrase and reword; about 1 in 10 used it to develop their thinking. |
John Warner, December 2022: “ChatGPT has not created a problem that wasn’t already present.” Students were “responding rationally to the incentives of the system.” See also Dawson, Bearman, Dollinger & Boud (2024), “Validity matters more than cheating.”
3. Performing isn’t learning
Students do better on the task while they have AI. That is not the same as learning, and it is what a take-home product measures. The MIT EEG study is a hook only: 54 students, not yet peer-reviewed. The randomized studies carry the claim. The phrase “tool-supported performance versus durable learning” comes from the Stanford SCALE Initiative’s 2026 review of AI in K-12 education.
| Study | What they did and what happened |
|---|---|
| Kosmyna et al., 2025 (preprint) (opens in a new tab) | 54 students wrote essays with ChatGPT, with search only, or with no tools, while wearing EEG headsets. In the first session, 15 of 18 ChatGPT users could not quote their own essay minutes later (2 of 18 in the brain-only group). The ChatGPT group did worse on neural, linguistic and scoring measures. Not yet peer-reviewed; the later sessions had only 18 participants. |
| Bassner et al., 2025 (opens in a new tab) | 275 intro programming students were randomly assigned to ChatGPT, a hint-only AI tutor, or no AI for a 90-minute exercise. Both AI groups scored much higher on the exercise. On before-and-after knowledge tests, no group learned more than the others. |
| Fan et al., 2024 (opens in a new tab) | Randomized study of 117 university students. ChatGPT raised essay scores but not knowledge gain or transfer. The authors warn of “metacognitive laziness.” |
| Barcaui, 2025 (opens in a new tab) | Randomized, 120 students. On a surprise test 45 days later, the ChatGPT group scored 57.5% and the comparison group 68.5%. |
| Bastani et al., 2025 (opens in a new tab) | High school math. With AI, practice scores went up 48%; when AI was taken away, exam scores were 17% lower for the plain-chatbot group. |
4. What replaces the essay
No one in the research proposes a single replacement format. The common answer is to change what counts as evidence of learning. Four patterns show up across the sources: verify the written work with a short live check; grade the process; replace some written products with live performance; and assess judgment, with students critiquing and correcting AI output. Posting AI-use rules is not a redesign (Corbin, Dawson & Liu, 2025), and making tasks authentic is not enough on its own (Kofinas et al., 2025).
Who has weighed in
| Source | Position |
|---|---|
| Warner, 2018 (opens in a new tab) | Why They Can’t Write: Killing the Five-Paragraph Essay and Other Necessities. Argued the formulaic school essay should die, four years before ChatGPT. |
| Warner, 2025 (opens in a new tab) | “Retreating to analog forms because they can be policed is a missed opportunity.” |
| Lodge et al., 2023 (TEQSA) (opens in a new tab) | Five propositions, including “Focus on the process of learning” and “Apply security at meaningful points across a program.” |
| Kofinas et al., 2025 (opens in a new tab) | Experiments at two UK universities. Even on authentic assessments, markers mostly could not tell AI-assisted work from student work. Recommends assessments that are “performative rather than output-based and asynchronously written.” |
| Corbin, Dawson & Liu, 2025 (opens in a new tab) | Change the structure of the task, not just the rules students are told. |
| Bearman et al., 2024 (opens in a new tab) | Assess evaluative judgement: students judging the quality of AI output and their own work. |
| Roe et al., 2025 (opens in a new tab) | “Assessment twins”: pair a vulnerable task, such as an essay, with a second task that gives a different kind of evidence on the same outcome. Conceptual. |
| Babaa, 2026 (opens in a new tab) | Chemical engineering students assessed on critiquing and correcting AI output. |
Process-visible work
The student hands in the steps between the task and the final product, and those steps are assessed. The evidence is early and mostly descriptive. No randomized study yet compares process grading with product-only grading on learning.
| Study | What they did and what happened |
|---|---|
| Cardace et al., 2024 (opens in a new tab) | Students made short videos explaining course concepts in 3 courses across 2 departments. They reported larger gains in learning and confidence than with regular coursework alone, face-to-face and remote. The study has an experimental component. |
| Daumiller & Janke, 2019 (opens in a new tab) | 169 university students took a test with some questions solvable only by cheating. Students told to focus on looking competent cheated more, but only when told they would be graded on results, not when graded on how they solved the problems. Two later experiments got inconsistent results (Fritz et al., 2026). |
| Oliveira et al., 2025 (opens in a new tab) | Coded 1,450 AI chat exchanges behind 70 graded essays. Students whose chats showed real back-and-forth on ideas tended to get higher grades; students who handed the task to the AI scored below average. |
| Mahmoud, 2026 (preprint) (opens in a new tab) | Engineering students used ChatGPT on take-home exams and handed in their chat logs. The clearest evidence of understanding came when a student spotted a wrong AI answer and explained why. |
| Fassihi et al., 2026 (opens in a new tab) | Writing students could use AI for up to half of a paper and handed in chat logs. The logs showed the instructor exactly how much was AI. |
| Saltan, 2025 (opens in a new tab) | A large software engineering course replaced written assignments with short on-camera answers. Students reported less AI misuse and fairer grading. |
One story per discipline
| Study | What they did and what happened |
|---|---|
| Walker et al., 2025 (preprint) (opens in a new tab) | Math, how it hits. AI took all 8 first-year math assessments at a university. Blind-marked, it earned the equivalent of a first-class degree, and was more consistent than students in supervised exams. |
| Urhan et al., 2024 (opens in a new tab) | Math, what works. A one-student case study. A student argued with ChatGPT about when a tangent line can be drawn. ChatGPT got continuity and the derivative wrong, and the student had to catch it. Grade the student’s reasoning, not the AI’s answer. |
| Kapoor et al., 2026 (opens in a new tab) | Programming, how it hits. 885 intro students had a hint-only AI tutor with a “See Solution” button. Half pressed it at least once; 14% pressed it on all three problems in the lab. Heavy users were lower performers working close to the deadline. |
| Ellis et al., 2026 (opens in a new tab) | Programming, what works. An intro computing course gave oral exams to 628 students alongside computer-based exams. Students rated them fair. The two formats lined up only loosely. |
| Yik & Dood, 2024 (opens in a new tab) | Organic chemistry, how it hits. ChatGPT gave 255 explanations of 85 reaction mechanisms. About a quarter were fully accurate; most sounded sophisticated. Assignment idea: hand students the AI’s explanation and have them find the error. |
| Mehltretter et al., 2026 (opens in a new tab) | Organic chemistry, what works. An Organic Chemistry II lab used conversational oral reports for some experiments. Students scored higher on them than on written reports for comparable experiments; online and in-person orals scored the same. |
Also worth reading: Bastani et al., 2025 (opens in a new tab) (high school math); Dakshit et al., 2026 (opens in a new tab) (fluent but conceptually wrong Calculus I–III solutions); Ohmann, 2019 (opens in a new tab) (oral exams took about the same instructor time as written); DeGlopper et al., 2022 (opens in a new tab) (what you grade changes what students learn).
Grading
| Study | What they did and what happened |
|---|---|
| Lewis, 2022 (opens in a new tab) | College math courses switched to standards-based grading, with clear standards and chances to reassess. Test anxiety went down, and the gap in test anxiety between men and women disappeared. |
| Spurlock, 2023 (opens in a new tab) | Three upper-level computer science courses dropped numeric grades, allowed resubmissions, and asked students to propose their own final grade. Most proposed within half a letter of the instructor. Motivation and effort went up. |
Also worth reading: Krou et al., 2020 (opens in a new tab) (meta-analysis: less cheating where classrooms emphasize mastery); Drummond et al., 2026 (opens in a new tab) (specifications grading in General Chemistry I); Hofmeister et al., 2022 (opens in a new tab) (counterpoint: workload did not drop); Kjærgaard et al., 2023 (opens in a new tab) (no grades in year one).
“This adds work”
The evidence supports a trade, not an addition: assess fewer things, check some of them live, and stop grading every product in full.
| Study | What they did and what happened |
|---|---|
| Tomás & Jessop, 2019 (opens in a new tab) | Counted assessments across 73 degree programs at 14 UK universities. Heavy loads of graded work push students toward surface learning. |
| Pye et al., 2025 (opens in a new tab) | Cutting assessments in final-year modules did not raise grades, and one single assessment made students anxious. Better as well as fewer. |
| Ohmann, 2019 (opens in a new tab) | Oral exams in intro CS. Instructor time was “manageable and comparable to traditional exams.” |
| James et al., 2026 (opens in a new tab) | Short oral exams, under 15 minutes per student, for students who had not yet mastered material. They then did as well on the final as everyone else. |
| Dobslaw et al., 2023 (opens in a new tab) | Counterpoint: oral exams raised teacher workload even in small classes. The trade only works if something else is dropped. |
At scale
| Study | What they did and what happened |
|---|---|
| Davey et al., 2025 (opens in a new tab) | A bioscience course replaced its written final with one-on-one oral exams. Across 722 students before and after the change, grades went up, with no gaps by gender, international status or language. |
| Ipeirotis & Rizakos, 2026 (opens in a new tab) | At NYU Stern a voice AI questioned each student and three AI models graded each transcript. Under $1 per exam. Two cohorts of 36 and 37: this shows cost, not scale. |
| Harrington et al., 2026 (opens in a new tab) | 1,131 intro psychology students took a practice oral exam run by an AI. Most rated it fair; 37% said knowing it was coming made them take the work seriously. |
| Hu et al., 2026 (opens in a new tab) | A proposal for a five-minute “viva-lite” on selected written tasks, with standard questions, recordings and rater checks. |
| Motallebi et al., 2026 (opens in a new tab) | Counterpoint: a map of 20 studies found instructor time for orals “often exceeds that of written exams.” Scaling is not solved. |
From my own courses
EDL 753, the decision record. Beside each piece of design work, students keep a few short entries: what I decided, what made me decide it, what I rejected and why, what I still don’t know, who I talked to and what they changed. It is framed as an audit trail, the standard in qualitative research, not as an AI rule. A few fifteen-minute conversations across the term let students explain their decisions out loud. Nothing is scored, and “I don’t know yet” is a complete answer. The rule I hold to: publish what you will ask, never what a good answer sounds like.
EDL 753, an ungraded course. No points. Students assemble their work and my comments, propose their own grade with a rationale, and then we meet.
EDL 571, Show What You Know Days. Four live team presentations carry the course’s assessment.
5. Durable skills and the unAI-ables
“Durable skills” is a workforce-advocacy name, promoted by America Succeeds, for skills that stay useful across a career. Its evidence is employer demand: in 82 million job postings from 2021, almost two-thirds asked for at least one durable skill (Lightcast, 2021). That counts words in job ads, not skill on the job. A polished essay shows none of these skills. Dr. Brandeis Marshall’s unAI-ables are the ones AI can’t fake, and other fields already assess them: put the student in a realistic situation, have them respond live, and score it with a rubric used by trained observers.
| Study | What they did and what happened |
|---|---|
| Rockstuhl et al., 2015 (opens in a new tab) | Contextual awareness. Seniors watched short video scenes of situations across cultures and explained how they read each one. How well they read the situation predicted later peer ratings of their performance, beyond the response they chose. |
| Kotlyar & Krasman, 2021 (opens in a new tab) | Conflict resolution. 215 undergraduates worked through a teamwork simulation run by a chatbot. Their scores predicted how peers rated them in a live team exercise, beyond what a written judgment test and a personality test predicted. |
| Hyytinen et al., 2015 (opens in a new tab) | Critical thinking. 330 students took an open-ended task and a multiple-choice test, both said to measure critical thinking. They agreed fully for only 45.5% of students; for 10%, the results were opposite. |
| Clemett & Raleigh, 2021 (opens in a new tab) | Across all three. A review of 38 studies of assessing clinical judgment. The most valid and reliable approach: several short, low-stakes observations, scored with a multi-level rubric by trained assessors who check each other. |
How the lists line up
| Durable skill | How it has been assessed |
|---|---|
| Critical thinking | Closest unAI-able: critical thinking. Assessed with open-ended performance tasks, documents and a rubric (Kleemola; Braun). |
| Communication | Closest unAI-able: student voice. Assessed with oral exams and interactive orals (Davey; Ellis). |
| Collaboration, leadership | Closest unAI-able: conflict resolution. Assessed with live simulations scored by observers and peers (Gunasingha; Hruby; Kotlyar). |
| Metacognition | Closest unAI-able: evaluative judgement. Assessed with process evidence and critiquing AI output (Mahmoud; Babaa). |
| Character, mindfulness, fortitude, growth mindset | No close match. Little evidence on assessing these in courses. |
Also worth reading: Lievens et al., 2026 (opens in a new tab) (meta-analysis of 302 studies of situational judgment tests); Gunasingha et al., 2023 (opens in a new tab) (medical students in a staged conflict); Kleemola et al., 2021 (opens in a new tab) (1,469 first-years, open-ended vs multiple-choice); Braun et al., 2020 (opens in a new tab) (a template for critical thinking tasks).
6. The hopeful close
AI also helps on the faculty side. Landgraf’s work (MidPac Institute) shows what neurodivergent students see when they read an assignment, and how faculty can use AI to write clearer ones. Materials to come. A Harvard AI tutor, built on the same teaching principles as the class, beat in-class active learning (Kestin et al., 2025). Assessment is a design problem, not a compliance problem. As Larry Leifer puts it, “All design is redesign.”