AI Efficiency, Skill Inequality, and the Educator's Role

Presented by

Welcome to AutomatED: the newsletter on how to teach better with tech.

Each week, I share what I have learned — and am learning — about AI and tech in the university classroom. What works, what doesn't, and why.

In this week’s edition, Caleb covers some recent papers on productivity and AI.

Are AI tools making us more productive now?

It’s clearly a big question. One that we should narrow down. Let’s do that by looking at specific professions.

For academics, it would be especially interesting to know whether technologies like ChatGPT have increased general research productivity. As far as I know, there’s been little work into that question, although there are cases where AI has assisted with specific research projects and tasks, like drug discovery, protein modelling, or weather prediction. There are many articles on the ethics of using AI tools for research, but less on whether and to what degree they affect productivity in general.

However, there are promising signs that productivity has increased among business professionals and programmers.

🔬 Looking At Some Studies

One study looked at 453 business professionals and found that they were able to write reports 11 minutes faster. The evaluators graded the reports and the researchers determined that the AI-assisted ones were generally better.

Shaving around 10 minutes off a single task is nothing to sneeze at. The non-AI-assisted professionals created their document in an average of 27 minutes. In theory, in an hour, they could handle 2 such tasks, while an AI-assisted professional could handle 3.

The same study found that AI reduced skill inequalities. Less skilled workers gained more from AI assistance than the more skilled ones. This is because participants in the treatment group tended to get fewer low scores on the first task. Hence, the professionals using AI had more similar scores (more “skill equality”) than those who didn’t.

One can’t read too much into this finding — they’re looking at a simple graded report after all — but it’s suggestive and it’s corroborated by some other studies (like this one, this one, and this one). Let’s return to this point.

Another study found that programmers using AI are able to complete test tasks 90 minutes faster!

Participants were instructed to “write an HTTP server in JavaScript”; that is, complete a technical programming task:

Again — that’s a huge productivity boost. Note that the control group had access to the internet (apart from the AI tool, GitHub Copilot, of course).

Maybe McKinsey Digital’s mid-summer report was right to estimate that generative AI “could add the equivalent of $2.6 trillion to $4.4 trillion annually” to the global economy…

👨‍🏫 Early Implications for Educators?

To my mind, there are several tentative takeaways.

First, we're in the early stages of research. These numbers are huge. That should raise healthy skepticism. Are we seeing anything close to such a huge productivity boost from programmers? My impression (in my technical role) is that we’re seeing some, but there’s no way it’s 126%. The test task isn’t representative of programmer work. Software engineers do many things and most tasks have not experienced the same productivity boost.

We should have similar skepticism about generalizing from the study on business professionals — reports of these kinds may not be representative of a business professional’s tasks.

Second, it hints at where we’ll see productivity gains. Standard reports will be easier to write. Chore-like computer programming tasks will be easier to complete. There’s a point here about the nature of the tasks — routine knowledge work may take up less and less time.

There’s another point about who the productivity gains are going to go to first. If easier work is more susceptible to automation, then we should expect lower-skilled workers to benefit — while automation will make a marginal impact on high-skilled workers. That was suggested by the study on business professionals and also supported by the impressions of some tech workers:

Apart from the hyperbole, my impression is the same.

For the time being, larger and deeper reports will remain elusive. Complex programming tasks involving large code bases, architectural decisions, and coordination across teams will also be difficult to tackle.

As such, the value of being able to tackle these tasks may increase.

It’s at these edges where educators can help. As a first step, this looks like preventing AI misuse to ensure students are capable of the sorts of tasks that AI does poorly. The next step, perhaps, involves helping students and other researchers use AI well, as Graham has argued in discussing the relevant considerations. For example, by helping students improve their judgment after the routine work — work that will be increasingly automated — is done. But to do that, teachers also need to know what the limits of these tools are and have some degree of mastery in the field. Engaging directly and trying out these tools on your own is essential.

🔗 Links

🔎 Speaking of increasing researcher productivity, Elicit recently pushed out new updates to help researchers explore the scientific literature with their AI Assistant.

🏭️ Reports on the ground of AI and productivity:

The CEO of OpenAI followed up with “​​important point here: these systems are much better at doing tasks than jobs.”

📱 Speaking of OpenAI, ChatGPT is rolling out support for voice conversations soon, and we should expect other LLM tools to follow (Pi, for example, already supports voice conversation):

An Advertisement

Why are all your favorite newsletters switching to beehiiv?

It’s because the founding beehiiv team were all early Morning Brew employees who helped scale that newsletter to over 4 million daily subscribers.

Years of trial and error went into building the precise tools, dashboards, and analytics needed to accomplish that. And now every newsletter on beehiiv has access to the same winning formula.

So what exactly does beehiiv offer?

• World-class growth tools like the referral program and recommendation network

• Monetization via the beehiiv Ad Network and premium subscriptions (i.e. beehiiv helps you get paid)

• Seamless content creation with a sleek collaborative editor

• Best-in-class inbox deliverability of 98.7%

• Oh and it’s the most affordable by a mile…

Take your newsletter to the next level — get started for free.