RE: Caplan, Selingo, Kitcher, Robbins, Underwood, & Starr
The plausible, implausible, and uncertain claims about AI made by the first six of the Chronicle's recent panel.
Graham Clay
June 05, 2023 • Estimated Reading Time: 20 minutes
[image created with DALL-E 2]
Welcome to AutomatED: the newsletter on how to teach better with tech.
Each week, I share what I have learned — and am learning — about AI and tech in the university classroom. What works, what doesn't, and why.
Let's reflect on the claims that six professors and university administrators made recently in the Chronicle about the effect of AI on higher education.
On May 25th, the Chronicle of Higher Education published an article that contains the brief thoughts on AI of twelve prominent figures in higher education.
In what follows, I provide an opinionated response to the first half of the panel. I highlight claims that I find plausible, implausible, and less certain than they assume, and I explain why I agree or disagree.
On Friday, I will do the same with the claims of the second half of the panel.
Bryan Caplan
One of their plausible claims:
AI will also change the labor market itself. As a general rule, new technologies take decades to realize their full potential. The first true trans-Atlantic phone call wasn’t placed until 1956, almost 80 years after the invention of the phone. E-commerce was technically feasible in the mid-’90s, but local malls still endure. We should expect AI to slowly disrupt a wide range of industries, including computer science itself. This will, in turn, slowly alter students’ career aspirations.
It is plausible that there will be a delay before AI reaches its "full potential" in changing the labor market and students' career aspirations. One reason for the delay is that many of the most impactful uses of AI require the integration of LLMs like ChatGPT with other software. Google is working on integrations between Bard and Google apps (as well as their partners’ services), and Microsoft is hard at work on its Windows Copilot and Microsoft 365 Copilot.
However, I must note two big caveats.
First, the delay between when a technology is invented and when it disrupts the market is decreasing, partly because many technologies — like the internet — have sped up communication and the transfer of information. Caplan's own examples illustrate this. In the US, Amazon achieved retail dominance much faster than (global) phone calling became ubiquitous and affordable. Local malls are largely gone and disappeared rapidly.
Second, in the case of AI in particular, the rate of change is likely to be very fast. AI has already begun disrupting computer science, along with marketing, journalism, graphic design, and other fields. And this is in the space of less than 3 years since GPT-3 was available to a limited userbase (and barely 6 months since ChatGPT was released publicly).
One of their implausible claims:
What will change for students is workload and evaluation. Professors are used to assigning papers and projects to be done outside of class. Using AI to cheat on such work will soon be child’s play. Harsh punishments for cheating might preserve the status quo, but colleges generally give cheaters a slap on the wrist, and that won’t change. Unmonitored academic work will become optional, or a farce. The only thing that will really matter will be exams. And unless the exams are in-person, they’ll be a farce, too. I’m known for giving difficult tests, yet GPT-4 already gets A’s on them.
It is implausible that unmonitored — including take-home — academic work will become optional or a farce.
There are two reasons for this: first, as I have discussed at length here at AutomatED, it is often feasible to design one's course to incentivize students to do unmonitored work at home (including using AI to brainstorm, research, and write) as a condition on successfully completing in-class activities and assessments; second, as our AI-immunity challenge has shown, unmonitored take-home work is more or less AI-immune, depending on the case.
For parallel reasons, it is also implausible that the only thing that will really matter is in-person exams. Of course, it is likely that in-person exams — including oral exams — will increase in prominence and role.
One of the uncertain claims they present as certain:
How will AI change our system? To start, we should bet against any major disruption. Students will not abandon traditional colleges en masse for virtual AI training centers. And since most majors are already barely job-related, students will generally stick with their existing majors.
It is uncertain that there will be no major disruption for higher education from AI.
First, it is not certain that students will enroll in majors at the same rates that they currently do, even if it is granted to Caplan that majors are "already barely job-related." There are a variety of other reasons that students enroll in majors, like the prevalence of certain technologies in popular usage. As the personal computer became ubiquitous in the US in the mid-to-late-70s, and as the dot-com boom occurred in the late 90s, the rate of computer science majors increased accordingly.
Second, higher education could be disrupted even if students do not switch to AI training centers (distinct from universities) and even if they stick to their existing majors. After all, many fields are looking to incorporate significant AI training in their course offerings — this could be disruptive, especially for those fields most readily affected by AI.
Jeffrey J. Selingo
One of their plausible claims:
Let’s start where turnover rates among staff are the worst: admissions. Most colleges accept most students who apply using a selection process that is routine and predictable. AI could be trained to make decisions about who gets accepted — or at least make the first cut of applicants. Yes, colleges will still need humans for recruiting, but even there, AI is increasingly capable of finding and marketing to prospective students.
It is plausible that AI tools will become central to some admissions and recruiting tasks. As Selingo notes, they can help with field-specific tasks, like vetting undergraduates' applications for specific characteristics, and they can help with generic tasks, like composing emails and brainstorming.
One of their implausible claims:
While much of the discussion has focused on what generative AI means for teaching, learning, and research, its immediate impact will likely be felt on functions outside of the academic core.
It is implausible that the immediate impact of generative AI will likely be felt on functions outside of the academic core, where "the academic core" is defined as teaching, learning, and research.
The reason for this is simple and well-understood by many of us in the trenches of teaching: students are early and effective adopters of new technology like generative AI. Indeed, AI plagiarism was a massive and immediate problem this past spring semester.
One of the uncertain claims they present as certain:
But until campuses use AI in that way — to take over for people in jobs that involve processing information or doing repeatable tasks — then we won’t reverse or slow down the upward-cost trajectory of higher education, where most tuition dollars are now spent on functions outside of the classroom.
It is uncertain that only if universities use AI "to take over for people in jobs that involve processing information or doing repeatable tasks" that we will "reverse or slow down the upward-cost trajectory of higher education."
It is not clear that productivity gains provided by AI will match expectations, as productivity growth is hard to predict and explain (see also discussion by Lee Vinsel in the Chronicle piece). Now, all else equal, if productivity goes up from AI, then costs of the services provided by the more productive people will go down, but it is also not clear if all else will be equal — universities might not lower tuition and fees in proportion to reductions in these costs, for a variety of reasons.
(And perhaps more importantly, there are many other ways that the cost of higher education could be reduced, as has been discussed ad nauseum elsewhere.)
Philip Kitcher
One of their plausible claims:
Most of today’s discussions about the impact of AI on education focus on perceived problems in teaching and evaluating students. Yet there is a far larger effect of AI’s advance: Many types of work will be taken over by machines, and jobs will vanish.
It is plausible that, in the long(er) run, much work will be taken over by AI and jobs that can be completed by AI tools will be completed by them rather than humans.
Yet, this is to assume that Kitcher is not referring to jobs vanishing in general — the standard view of economists is that labor-saving technologies have not, to date, decreased labor force participation rates, and other factors are its primary determinants.
One of their implausible claims:
Education aims to help young people develop as individuals and as citizens, not simply to become cogs in the machinery of national production. If the cogs can be made by ingenious AI systems, so much the better. The previously employed people can be liberated to do something else. Not necessarily to expand productivity, but to advance educational goals that we neglect.
Let’s do that by vastly increasing the number of people who educate the young and raising the pay of those who undertake these important tasks — the “main enterprise of the world,” as Emerson once characterized it. Let’s remove the stigma from service work and appreciate its enormous worth. Let’s train more adults and put more of them in the classroom. Let’s give them the opportunity to recognize each child’s individuality, to help young people understand themselves, overcome the difficulties they encounter, and work with one another.
It is implausible that higher education does not (and should not) also aim to train students to work — in addition to "develop [them] as individuals and as citizens." Work is and ought to be an important and rewarding part of many people’s lives.
And many students rightly see higher education as primarily part of their career development, especially Generation Z. They should take seriously the worry that a college degree might not be worth it anymore, economically speaking.
It is also implausible that AI tools will not affect the number of teachers needed in educational settings, especially in K12 schools. It would be great to remove the stigma from service work and teaching, but in the meantime, AI is likely to provide opportunities to personalize and differentiate learning for millions of students who would otherwise lack it. See the case of Khan Academy's use of ChatGPT. It is more likely that AI has a larger effect on the education of the young via personalization and differentiation than via freeing humans from other tasks to teach.
Hollis Robbins
Two of their plausible claims:
14. Specific and local cultural knowledge will become more valuable.
15. Experiential learning will become the norm. Everyone will need an internship. Employers will want assurances that a new graduate can follow directions, complete tasks, demonstrate judgment.
Both of these claims are plausible.
Regarding #14, our investigation into the AI immunity of take-home assignments has already shown that generative AI tools struggle to generate responses as the assessments' field-specific standards increase. ChatGPT can explain general facts about the human reproductive systems, but it struggles to give sufficiently realistic clinical details about experiments into the viability of immunosuppressive drugs in preventing graft rejection post-allogeneic heart transplantation. This will likely change over time, but it is at least currently an area of weakness of many generative AI tools — they are generalists.
Regarding #15, I maintain that in-class activities will increase in importance, flipped classrooms will become more common, and oral performance (whether in exams or not) will be demanded of more students. Experiential learning aligns with many of these predictions. More professors will be seeking to put students in the contexts where they will be in upon graduation, including those reliant on AI tools themselves.
One of their implausible claims:
Higher education will be less about ensuring students know what they’ve read and more about ensuring they read what is not yet known by AI.
It is implausible that higher education will be about ensuring students read what is not yet known by AI. In fact, this is highly implausible, for two reasons.
First, most of what undergraduates read is "known by AI" already, so they cannot read what it is not yet known by it. This will only get more extreme.
Second, most of what undergraduates should read is "known by AI" already, just as much of the mathematical techniques and methods that students should learn are known by calculators, WolframAlpha, etc. Students need to develop skills and knowledge built on basics that AI tools already possess, even if they will ultimately use AI tools for similar purposes later, as I have argued here.
One of the uncertain claims they present as certain:
The written essay will no longer be the default for student assessment.
It is uncertain that the written essay will no longer be the default for student assessment because there is still value to having students write essays in class and there is still value to having students write essays with AI assistance. Perhaps it will lose "default" status. Perhaps not. Many tasks in other fields that can be completed by computers still have central roles in pedagogical contexts.
Ted Underwood
One of their plausible claims:
Students in every major will need to know how to challenge or defend the appropriateness of a given model for a given question. To teach them how to do that, we don’t need to hastily construct a new field called “critical AI studies.” The intellectual resources students need are already present in the history and philosophy of science courses, along with the disciplines of statistics and machine learning themselves, which are deeply self-conscious about their own epistemic procedures. Readings from all of those fields belong in a 21st-century core curriculum.
It is plausible students will need what Underwood mentions, as well as the skills and knowledge distinctive of their fields, to evaluate the responses of AI tools — and which AI tools to use in the first place — even if they will use AI tools heavily in their jobs. In fact, I have argued for a version of this claim in a prior piece.
One of their implausible claims:
Our understanding of AI is strongly shaped by the recent success of ChatGPT and other chatbots. They mold a language model (which in principle could imitate any number of genres) into the familiar form of a question-answering utility like Siri or Google Search: ask a short question and get a short answer. An all-purpose question answerer is certainly easy to use.
It is implausible ChatGPT and other chatbots should be conceptualized as tools where we "ask a short question and get a short answer." After all, ChatGPT can receive long and complicated questions, and produce similar responses. The prompt sizes and modalities will increase rapidly in the coming year, and their responses will, too.
Furthermore, these tools have already been integrated with other software to create functionality far beyond this description, even though the chatbots themselves are unmodified in the process. For instance, ChatPDF uses .pdf analysis software to cleverly feed ChatGPT with bits of a .pdf based on a prompt — ChatGPT is unmodified but is used to produce outputs that do not fit the "short question in / short answer out" model.
One of the uncertain claims they present as certain:
In an academic context, we should approach language models as engines for provisional reasoning — “calculators for words,” as British programmer Simon Willison calls them. Instead of assuming that the model already has an answer to every question in memory, this approach provides, in the prompt, any special assumptions or background knowledge the model will be expected to use. A student might hand in a model of a scientific article and ask for explication of a difficult passage. A researcher might provide a thousand documents, serially, and ask the model to answer the same questions about each one. A writer might provide a draft essay and ask for advice about structure.
It is uncertain that LLMs will remain "engines for provisional reasoning." I take it that Underwood is referring to the future since he refers to the possibility that a researcher provides a LLM with "a thousand documents" (currently, this is far from possible). We have already seen radical increases in LLM accuracy and effectiveness in a short period of time — for instance, GPT4 significantly outperforms GPT3.5 (ChatGPT) on a range of tests — so it is not clear that they will retain their current status as merely provisional reasoners or sources of information.
G. Gabrielle Starr
One of their plausible claims:
More profoundly, it seems clear that the active part of learning is immensely important. Learning requires more than synthesis of information, for it is in the testing of knowledge that we make the biggest gains in understanding. This means that it is how we put our knowledge to work that matters. It also means that learning is a social undertaking, in which we discuss, dispute, verify, reject, modify, and extend what we (think we) know to other people and the world around us.
It is plausible that active learning, synthesis, and application will increase in importance as AI tools become ubiquitous, though they are already very important. Likewise, it is plausible that the social aspects of learning will increase in importance, too. As discussed above, there will be an increased emphasis placed on getting students to perform and display their skills and knowledge in class. Dialogue and discussion will gain increased relevance, especially as worries proliferate about the social skills of a generation of students raised in front of screens.
One of their implausible claims:
What differentiates humans from AI, in part, comes down to that: The pleasures of learning lead us toward creative possibilities, as well as toward active experimentation. For now, humans do that, and they do it pretty well. And I would advise that humans limit AI action — the ability to directly influence the world around us by altering or manipulating it – because it is in action that what humans value most really lies. It is in not just what we do, but in the effects of what we do in the world. It is in ethics that humanity finds itself and determines its own meaning.
It is implausible that the pleasures of learning differentiate humans from AI with respect to creativity and active experimentation. Even though it is highly unlikely that AI tools are presently conscious (so they do not experience pleasure), they can still be very creative. Indeed, this is one of Bard's distinctive traits, as opposed to ChatGPT. Likewise, some AI tools can be set up to actively experiment in contexts that they are left to their own devices, as in the case of the AutoGPT project.
While there are serious concerns about future AI action — like malicious artificial general intelligence (AGI) — it is also implausible that direct influences on the world around us is what should differentiate humans from AI. AI is already directly influencing innumerable aspects of our world, and has been for years.
Finally, even though ethical reasoning and ethics are very important (I am a philosophy professor, so no argument from me there), AI can help humans with ethical reasoning already, and it is likely that AI tools' impact on philosophy and ethics will only increase.
One of the uncertain claims they present as certain:
Nothing ChatGPT does will take away the pleasures of learning.
It is uncertain that ChatGPT and other generative AI tools will not take away the pleasures of learning. There is some evidence that people prefer completing some tasks with AI tools than without. Nevertheless, it is not at all clear that there will not be some pleasures of learning that will be reduced by these tools.
For instance, if there are any take-home assignments that professors will not tend to assign — because of the risk of AI plagiarism — that were uniquely pleasurable for students to complete, then generative AI tools will take away the pleasures of learning. Are there assignments in this category? Or are there some students for whom there are assignments in this category?
