GPT-4o Democratizes Data Analysis

Welcome to AutomatED: the newsletter on how to teach better with tech.

Each week, I share what I have learned — and am learning — about AI and tech in the university classroom. What works, what doesn't, and why.

In this week’s piece, I explain what ChatGPT’s Advanced Data Analysis is, why it is generally reliable, why you should care about it even if your field isn’t quantitative, and how to use it to improve your pedagogy this summer. I also discuss why I think ChatGPT Edu may be overrated for most of us and what Apple is announcing today. Finally, I ask for more submissions and judges for our AI Assignment Challenge.

🧰 An AI Use Case for Your Toolbox: 
GPT-4o Advanced Data Analysis

What It Is

OpenAI just updated Advanced Data Analysis (ADA), their data analytics tool for ChatGPT, so that it leverages the full power of GPT-4o. Formerly known as “Code Interpreter,” this tool significantly expands the LLM’s data manipulation, code execution, and analytical capabilities. This feature allows users to upload various types of data files directly into the chat interface — either directly or from Google Drive and Microsoft OneDrive — where ChatGPT can then analyze it by writing and executing Python code in a secure, sandboxed environment.

ChatGPT can conduct initial codings or thematic analyses in the course of dataset curation(Turobov, Coyle, & Harding, 2024), with the advantage that it can “provide consistent categorization of responses, reducing potential biases that might arise from human interpretation” (Conte et al., 2024).

Next, ADA can read, clean, and manipulate the datasets. For instance, it can remove missing values, merge datasets, and convert data into different formats for analysis​.

Then, via its access to a range of Python libraries (including pandas and Matplotlib), ADA can create functions, perform mathematical calculations, conduct statistical analyses, interpret and translate code across different programming languages, and produce visualizations.

The latest update allows you to use ADA to interact with tables in real-time (demonstration here), and it enables you to use ADA to customize, adjust, query, and download bar, line, pie, and scatter plot charts (demonstration here).

Now, to access these improvements, you need ChatGPT Plus for $20/month, but you will still be able use ADA for free with GPT-4 (the prior model) and some usage limitations, per OpenAI’s announcement from a few weeks ago.

Whether It Is Reliable

Prior to GPT-4o, the consensus was that ADA with GPT-4 was already quite useful and generally reliable in a range of fields, both for novice and expert data analysts.

For instance, financial services experts Pelster and Val (2024) found that, with access to the internet, GPT-4 was “able to provide valuable investment advice and evaluate financial information in a timely manner” and that there is “a positive correlation between ChatGPT-4 ratings and future earnings announcements and stock returns."

Likewise, dermatologists Jairath et al. (2024) found that ADA helps address the problem that there is “an overflow of data but often a lack of expertise to decipher it.” While not replacing “clinician judgment or the nuanced understanding from clinical experience,” ADA is reliable enough to need validation only in “complex cases.”

Few quantitative evaluations of GPT-4o ADA exist so far, but one, conducted by experts in financial data analysis Feng, Li, and Liu (2024), concludes that its performance is “comparable to traditional statistical software like Stata, though some errors and discrepancies arise due to differences in implementation.”

Why Everyone Should Care

ADA is a big deal for all professors. Let me explain.

Of course, if you’re in economics, biology, engineering, or any other data-intensive discipline, ADA’s ability to manipulate, analyze, and visualize complex datasets can enable you and your students to complete field-specific tasks faster and more effectively without learning the nuances of SAS, Stata, or R. While foundational statistical expertise is still needed — especially at the present moment where ADA may fail in tough-to-detect ways for “complex cases” — the utility of ADA merits a close look for field-specific data analysis.

Yet, beyond quantitative fields that can directly use ADA for their work, it can still have an enormous impact.

The reason is that most professors lack access to useful quantitative data on their students' performance. Unlike K-12 educators, who benefit from standardized tests and comprehensive software suites like i-Ready, many professors’ fields do not have standardized tests, corresponding generic standards, and matching tools to determine students' needs, personalize learning, and monitor progress. This gap exists due to the research-driven nature of higher education, which values cutting-edge inquiry and its concomitant creativity and independence.

However, this lack of regimented assessment can leave professors without clear quantitative insights into their students' academic status and progress. Unlike the K-12 teacher, who often has robust dashboards displaying each student’s progress relative to each standard, the professor generally has only their own qualitative analyses and their own bespoke quantitative data.

ADA can fill this gap, providing robust data analysis tools that simplify complex processes, making it possible for each professor to gather and interpret valuable data on student performance. By leveraging ADA, even professors without a background in data analysis can use its intuitive features to enhance their teaching, ensuring they can effectively track and support student progress in a personalized manner.

Here are some use cases that display what I have in mind…

Pedagogical Use Cases

#1 - Analyzing Grade Correlations

Objective: Determine if early performance is predictive of final outcomes within the semester.

What You Do: Gather grades for all assignments, quizzes, midterms, and the final exam. Anonymize them, removing all personally identifiable student information.

What You Use Advanced Data Analysis To Do: Calculate correlation coefficients (e.g. Pearson or Spearman) between each of the early assignments (e.g. first test, first few quizzes) and the final exam grade.

What You Gain: Identifying strong correlations can help in early intervention strategies. If the data shows that early performance is a reliable predictor of final grades, you can focus on supporting students who perform poorly early on. This approach allows you to provide targeted support and resources to students who need it most, potentially improving their overall performance by the end of the semester. (And if there is no correlation, this likely means there is too little connection between early assessments and summative ones.)

#2 - Assessing Difficulty and Discrimination

Objective: Evaluate the difficulty and discriminative power of assignments and exams to ensure they are effectively assessing student understanding.

What You Do: Compile scores for individual questions on exams and assignments. Anonymize them, removing all personally identifiable student information.

What You Use Advanced Data Analysis To Do: Perform item analysis using metrics such as item difficulty index (percentage of students who answered correctly) and item discrimination index (correlation between item score and total test score). ADA can automate item analysis calculations, generating detailed reports on which items are too easy, too difficult, or not discriminating well between high and low performers.

What You Gain: By identifying poorly performing items, you can revise future assessments to improve their quality and fairness. Likewise, ensuring a mix of difficulty levels and well-discriminating items can lead to more accurate assessments of student abilities and knowledge.

#3 - Identifying Patterns in Student Performance

Objective: Understand patterns in student performance across different types of assessments within the semester.

What You Do: Collect scores from various types of assessments (e.g., multiple-choice exams, essays, presentations). Anonymize them, removing all personally identifiable student information.

What You Use Advanced Data Analysis To Do: Use the tool to perform cluster analysis or factor analysis to identify patterns or groupings in student performance. The tool can apply clustering algorithms (e.g., k-means clustering) and factor analysis to reveal underlying performance trends and visualize these patterns.

What You Gain: Discovering performance trends across different assessment types can help you understand how different student groups perform and tailor your teaching methods accordingly.

#4 - Evaluating the Impact of Curriculum Changes

Objective: Assess the effectiveness of curriculum changes over multiple semesters.

What You Do: Compile gradebook data from multiple semesters before and after implementing specific curriculum changes (e.g., new teaching methods, revised course materials). Anonymize them, removing all personally identifiable student information.

What You Use Advanced Data Analysis To Do: Use the tool to perform comparative statistical analyses, such as paired t-tests or ANOVA, to evaluate differences in student performance before and after the changes. The tool can automate these statistical tests, generate comparison graphs, and calculate effect sizes to quantify the impact of the changes.

What You Gain: Quantifying the impact of curriculum changes helps in making informed decisions about which strategies to retain, modify, or discard. Regularly assessing the effectiveness of curriculum adjustments ensures continuous improvement in teaching quality and student learning outcomes.

#5 - Tracking Cohort Performance Trends

Objective: Monitor the performance trends of different student cohorts over time to identify consistent patterns or changes.

What You Do: Gather comprehensive grade data from multiple student cohorts over several semesters. Anonymize them, removing all personally identifiable student information.

What You Use Advanced Data Analysis To Do: Use ADA to conduct trend analysis and regression analysis to track performance changes over time. The tool can generate time-series plots, calculate trend lines, and provide statistical metrics (e.g., R-squared values) to evaluate the strength and significance of observed trends.

What You Gain: Longitudinal analysis helps identify positive or negative trends in student performance, enabling you to understand the factors driving these trends. Recognizing patterns early allows for proactive adjustments to teaching methods, course content, or support services to address emerging issues or capitalize on successful practices.

#6 - Checking Impact of Attendance / Participation

Objective: Examine the long-term relationship between attendance, participation, and academic performance across multiple semesters.

What You Do: Compile attendance records, participation scores, and overall course grades from several semesters. Anonymize them, removing all personally identifiable student information.

What You Use Advanced Data Analysis To Do: Use ADA to perform multivariate analysis, such as multiple regression, to determine how attendance and participation impact academic performance over time. The tool can automate the multivariate regression analysis, produce detailed reports on the relative importance of each factor, and visualize the results using interaction plots.

What You Gain: Understanding the long-term effects of attendance and participation on academic performance provides a holistic view of student engagement and success. Insights from this analysis can inform the development of policies and practices to encourage better attendance and participation, ultimately enhancing student outcomes.

📢 Quick Hits:
News Tidbits for Higher Educators

OpenAI has now introduced ChatGPT Edu, a specialized version of ChatGPT designed for higher education institutions.

• Why it matters: I’m not sure if it matters for most of our readers, to be frank. There are two reasons for skepticism. First, as the below table shows, there is very little that ChatGPT Edu offers that ChatGPT Plus doesn’t already offer. It comes down to a bit more security, and some group management tools (GPT controls, analytics, sharing). It isn’t clear these features are worth it given that I expect the price will be quite a bit higher than the per user cost of ChatGPT Plus. Second, most institutions will have deals with Microsoft or Google that will prevent them from wanting to commit on an institutional level to an alternative. (Please respond to this email and let me know why if you think I am wrong about this.)

Today, at its annual developer conference (WWDC) at 1PM Eastern Time, Apple is expected to announce a significant partnership with OpenAI to integrate ChatGPT under the name “Apple Intelligence” in iOS 18.

• Why it matters: This partnership aims to significantly enhance Siri’s capabilities by leveraging OpenAI’s advanced generative AI technology, just like Apple already relies on Google for search. The deal will allow Apple to offer a more intelligent and responsive Siri — after initially downplaying the import of generative AI chatbots — positioning Siri to compete more effectively with other AI assistants like Google Assistant and Amazon Alexa. It remains to be seen which specific features and integrations Apple reveals today, but they will likely include some that are of high relevance to educators. (I will provide more information next week on this topic…)

Other Links

• OpenAI has begun training its next model, likely GPT-5 (Cade Metz, New York Times)

• Fields Medalist Terence Tao explains how “AI Will Become Mathematicians’ ‘Co-Pilot’” (Christoph Drösser, Scientific American)

• ICYMI: Microsoft released their 135-page Education AI Toolkit to help education leaders plan their institution’s (or department’s") “AI journey”

🔬 Are Your Assignments AI-Immune?

A few months ago, in honor of the one-year anniversary of our (in)famous “AI-Immunity Challenge” and our 3000th subscriber, I announced a new contest.

We currently need submissions in Education, Political Science, Religious Studies, Information Systems, Computer Science, Communication, or Media. We have judges ready for assignments in these fields.

Please volunteer to provide your assignment (or to be a judge). Without more submissions (and more judges), we cannot run the contest!

We will be doling out the result of the raffle for judge volunteers at the end of June.

Here are the rules (click the above links to old pieces for more context)…

Prizes and Rewards

1. For Submitters: Any professor who simply submits an eligible assignment and rubric will get one free month of ✨Premium.

2. If AutomatED Loses: If we get a C or worse on a professor’s assignment — by the terms of their own rubric, as judged by an independent judge expert in their field — then the professor will get one free year of ✨Premium and I will post on my LinkedIN an admission of AutomatED’s “loss.”

3. If AutomatED Wins: If we get an A or a B on a professor’s assignment, then the professor will post on one of their professional social networking accounts an admission of AutomatED’s “win.” (Alternative options are available if this isn’t an option.)

4. For Professors Who Volunteer to Judge: Any professor who volunteers to judge submissions in their field of expertise enters a raffle for one free year of ✨Premium.

5. For Professors Who Judge: If such a professor is called to judge a given submission and offers their judgment on it (in alignment with the rubric), then they get three free months of ✨Premium.

Constraints on Submissions

1. Each assignment must be standalone — not part of a pair or series.

2. The submissions for the assignment must be capable of being typewritten in their entirety (e.g., no oral exams, hand-written essays, dramatic performances, etc).

3. The professor must provide in advance the rubric by which the submissions will be graded by an independent judge, expert in the relevant field. Grades should be labeled A, B, C, D, F, with clear criteria for each.

Constraints on AutomatED

1. Every sentence that we submit must be AI-generated — no edits allowed, except for formatting if needed.

2. We must use only publicly available AI tools.

3. We cannot spend more than 1 hour on each assignment.

4. Each of our efforts must be documented and described in a future piece. The takeaways will be provided in a free weekly piece, while a deep dive will be found in a ✨Premium one.

Constraints on Judges

1. You must provide a grade and a rationale relative to the originally submitted rubric.

2. You must note strengths and weaknesses of the submission.

💸 Refer Someone, Get Prize

It only takes one reference to get the reward.

In turn, the person you refer gets free access to our most popular ✨Premium Guide, namely my 6000-word big-picture tour — from December — of how to plan and develop assignments and assessments in the age of artificial intelligence.

✉️ What You, Our Subscribers, Are Saying

If you used Gemini in Gmail or directed it to access your Gmail via the general Gemini chatbot portal, it definitely had access to your email, assuming you were on the same account. These features come with both Gemini for Google Workspace and Google One AI Premium.