✨Tutorial: A Beginner's Guide to Running Local LLMs

Welcome to AutomatED: the newsletter on how to teach better with tech.

Each week, I share what I have learned — and am learning — about AI and tech in the university classroom. What works, what doesn't, and why.

In this fortnight’s Premium edition, I present a Tutorial on how to set up and experiment with a local open-source LLM on your own computer, so that you can see if this method is viable for your more sensitive use cases (like tasks involving student data).

❓🤖 Why Would I Use a Local LLM?

As professors and learning specialists increasingly integrate AI technologies like large language models (LLMs) into their pedagogy, the privacy and security of student data emerge as concerns. Deploying LLMs locally, as opposed to relying on cloud services, offers a practical solution, at least if certain conditions are met.

Here are some of the considerations you should reflect on before proceeding with a local LLM…

✅ Benefits

1. Overcomes Data Privacy Issues with ChatGPT, Gemini, and Claude

   Using LLMs like ChatGPT, Gemini, and Claude involves transmitting data to the servers of companies like OpenAI, Google, and Anthropic. This means that every prompt and upload — whether it's a student’s homework submission or your lesson plans — passes through and is processed on external servers. (This is true even if you run these LLMs locally because such use requires an API key; the local LLM interface becomes simply a portal to the internet.) These interactions pose a potential risk of data exposure and misuse, which is a significant concern when handling confidential educational data. Furthermore, in some contexts, these prompts and uploads are used by the LLM developers to train and improve the performance of the LLMs.

   By contrast, local LLMs that utilize open-source models like Meta’s LLaMa 3 or Mistral’s Mistral 7B offer a substantial privacy advantage. Once installed, these models operate entirely offline with no need to communicate with external servers or the internet. This "sandbox" environment ensures that all data processing happens locally on the user's computer. You can even run the LLM locally with your internet disabled or in a virtual machine, if you are worried!

   However, as a matter of ethical practice, you should consider informing your students when their data is being processed by LLMs. This enables students to understand how their information is being used, even if you think they have no right to decline to have their work processed. Some students may have reservations about AI processing their data, even locally. (I discuss various solutions to data privacy issues in our comprehensive ✨Premium Guide to Ethically Using AI with Student Data.)

2. Compliance with Regulations:

   Local LLMs help educators comply with educational data protection laws like the Family Educational Rights and Privacy Act (FERPA) in the U.S., which regulates access to educational information and student records. Educators mitigate the risk of inadvertently breaching privacy laws that can occur more easily in other contexts.

👎 Costs

1. Setup Time and Effort:

   Setting up a local LLM is not especially challenging — and I will explain how to do so below — but it requires some technical knowledge and takes more time than simply using ChatGPT, Gemini, or Claude. Professors need to install software, configure settings, and ensure that the environment is suitable for running the model. In addition, local setups require more active/regular oversight, including updates and troubleshooting, unlike cloud services that manage updates and maintenance automatically.

2. Performance Constraints:

   Local hardware may not be as powerful as the servers used by cloud-based AI services, potentially leading to lower performance or slower response times from the LLM. As I discuss below, local LLMs’ context windows are generally much smaller than those available from close-source proprietary models.

   Likewise, although they have made a lot of progress relative to many of the benchmarks, open-source models may not be as powerful as their commercial counterparts (generally they have significantly fewer parameters), which might affect their effectiveness in specific academic tasks such as grading or personalized student feedback.

Subscribe to Premium to read the rest.