AI as a Thought Partner For Your Research

In partnership with

Welcome to AutomatED: the newsletter on how to teach better with tech.

In each edition, I share what I have learned — and am learning — about AI and tech in the university classroom. What works, what doesn't, and why.

Today, I share results from an experiment where OpenAI's o1-pro AI model provided remarkably thorough and helpful feedback on an economics research paper, revealing that advanced AI can serve as a valuable thought partner for us and offering promising applications for research development and student feedback.

As I note, if you want to participate in this experiment with your research, respond to this email to let me know!

💡 Idea of the Week:
How Good is AI as a Thought Partner?

Can any current AI serve as a genuinely useful thought partner for academic researchers operating at the frontiers of their fields?

To explore this question and provide evidence to you all about the answer, I am conducting a simple experiment with OpenAI's o1-pro, their most advanced reasoning AI model (which requires a $200/month subscription via ChatGPT Pro; its newer, faster, but slightly dumber sibling, o3-mini, is available to all in a low-compute version).

A few months ago, when o1-pro came out, I invited professors to volunteer drafts of their academic papers, which I would then share with o1-pro, asking it to provide thoughtful feedback as a colleague might. The AI would evaluate the paper and offer suggestions for improvement, after which I'd gather the professors' reactions to assess the value of this AI-assisted thought partnership.

Today, I'm sharing results from one particularly illuminating case: a paper about economics education titled “Relevance, Belonging, and Growth Mindsets in Economics: Differences across Institution Types.” The paper was co-authored by Caroline Krafft, Kristine West, Allen Bellas (my correspondent), Ming Chien Lo, Adriana Cortes Mendosa, Gabrielle Agbenyiga, J Dombroski, Nayomi Her, and Joy Moua. Allen Bellas and Ming Chien Lo are at Metropolitan State University, St. Paul, MN, while the rest of the co-authors are at St. Catherine University in the same city.

Crucially, I did not turn on Deep Research, which would have connected o1-pro with the internet (notably, Deep Research is now available to all paying ChatGPT users; more on it here, here, and here). Instead, I tested o1-pro’s ability to engage with the paper based on its training data alone.

In short, the authors received extensive, structured feedback from o1-pro that impressed them in multiple ways, as researchers and as teachers of research methods.

What's most striking is how little guidance the AI needed, all due to the power of o1-pro — like other reasoning models — to prompt itself.

Let's look at what happened.

The Paper

The paper I shared with o1-pro examines why economics as a field has struggled with diversity and inclusion compared to STEM disciplines, focusing on psychological aspects (relevance, belonging, and growth mindset — collectively "RBG") across different institutional settings, including minority-serving institutions (MSIs) and women's colleges.

This was a substantive, data-driven paper with a robust methodology section, analysis of survey data from 24 colleges and universities, and thoughtful discussion of implications for economics education. At 15 pages, single-spaced (excluding the 4 pages of references and 15 tables), it represented the kind of complex academic work that typically requires careful review from knowledgeable colleagues.

My Prompt

The prompt I used relied on no field-specific expertise, as I have none (I majored in economics a long time ago but let’s be honest here). Lengthier prompts with more background information, criticisms from knowledgeable colleagues of the paper, etc. would likely get even better results (more on this below). My prompt just conveys the broad context of my request, without any special insight into how to proceed.

Here is my prompt:

The AI Feedback

Given this limited and generic prompt, o1-pro developed its own comprehensive feedback plan with eight distinct parts:

1. High-Level Framing and Organization

2. Literature Review and Theoretical Foundations

3. Data and Sampling Strategy

4. Measurement of Key Constructs (RBG) and Empirical Methods

5. Presentation and Interpretation of Results

6. Discussion, Contributions, and Limitations

7. Writing Style and Readability

8. Future Directions and Broader Implications

For each section, the AI provided detailed, thoughtful analysis that balanced acknowledgment of strengths with suggestions for improvement. The feedback totaled over 10,000 words — longer than many published papers themselves!

Here's a sample from the "High-Level Framing and Organization" section to give you a flavor of the feedback:

The feedback wasn't just superficial; it engaged deeply with the paper's methodology. For instance, on sampling strategy:

o1-pro also went beyond critiquing the paper to offering thoughtful extensions and implications for future research:

The AI consistently balanced affirmation with constructive critique, maintaining a collegial tone throughout. It identified genuine strengths while suggesting concrete improvements — much like an ideal peer reviewer.

The Professors' Response

The professors were genuinely impressed by the quality and thoroughness of the AI feedback. As Bellas wrote:

While they noted that "the length of the feedback was a bit daunting," they appreciated that "the excellent structure made it easier to read through and digest the feedback and to identify specific things to work on."

Bellas reported that their colleagues immediately recognized the educational potential:

Takeaways and Implications

What does this experiment tell us about AI as a thought partner for academic work? Here are a few of my thoughts:

1. Minimal prompting of the reasoning models like o1-pro can yield high-quality results

The most surprising aspect of this experiment was how little guidance the AI needed. I didn't provide detailed instructions about what constitutes "good feedback" or specify a particular structure. As noted, I don’t know anything about the subject matter, so this prompt is about as good as it’s going to get! Instead, I simply described the basic task in general terms and let the AI determine how to approach it. This may mean that for certain professional tasks, sometimes the best prompt is the simplest one for reasoning models like o1-pro — allowing the AI's capabilities to shine through without constraining them with overly specific instructions. (This matches with the advice of the AI developers, mind you.) But it also leaves room for the possibility that more information in the prompt, which could be provided by a subject-matter expert, would get even better results. I’ve found that the latter can be true in some cases where I do have expertise.

2. AI feedback offers unique advantages over traditional peer review

AI feedback offers several distinctive benefits over human peer review and collaboration:

• Exceptional thoroughness by default

• Consistent structure and organization (the use of bullet points is borderline compulsive from these models, I’ve found, but it is useful for this sort of case)

• No personal biases or academic politics (though there may be biases in the training data, of course)

• Immediate availability (no waiting for busy colleagues)

• A balanced approach that acknowledges strengths while suggesting improvements (reasoning models do this almost to a fault)

3. It's particularly valuable for structure and organization

The AI excelled at analyzing the paper's overall structure, flow, and organization — aspects that even experienced academics sometimes struggle to evaluate objectively in their own work. (I’ve definitely had issues with this…) By offering a fresh perspective on how arguments are sequenced and connected, AI feedback can help researchers strengthen the fundamental architecture of their papers.

4. There are still important limitations

Despite its impressive performance, AI thought partnership has clear boundaries. The AI can't evaluate the factual accuracy of novel research findings (without source material being provided to compare), can't bring disciplinary expertise beyond what's in its training data or what’s shared in the prompt, can't fully understand the nuanced dynamics of academic publishing in specific fields, etc.

5. The educational applications are particularly promising

The authors immediately recognized the potential for using AI feedback in their teaching. For students, receiving detailed, non-threatening feedback from an AI might feel less personal and more constructive than instructor comments. The AI can provide a first round of comprehensive feedback, edited by the instructor if needed, allowing instructors to focus on higher-level guidance or areas the AI might have missed. Furthermore, providing this level of feedback from one professor for all 20, 40, 60, or more students would be extremely time consuming.

✨Pedagogical Suggestions

Here are some practical approaches for integrating this technology into your teaching, including some $0 AI models that are comparable to o1-pro (and how to access them):

✨ What Premium Members Get

As I improve ✨Premium subscribers’ benefits, it can be hard to keep track, especially if you’re new to AutomatED. Here’s a summary…

Core Benefits

• $0 monthly webinars on timely AI topics (last one: Feb. 28, on “Using AI to Create Course Content”; next one: on using AI for non-pedagogical productivity, at the end of March), while everyone else pays $25

• Access to all our practical Guides, including topics like AI-aware assessment design and student AI policies, and Tutorials on how to use AI tools, from using ChatGPT for analyzing teaching data to using LearnLM for tutoring

• Discounts on our focused courses (like our new custom GPT course)

• Annual 1-hour consultation to discuss your specific challenges

• Premium-only sections in weekly newsletters, like the above

You can see all the benefits, including all the Guides and Tutorials, here.

Nearly 100 professors and learning specialists already use these resources to save time and enhance their teaching. Want to join them?