Teach Your AI to Keep a Lab Notebook

Welcome to AutomatED: the newsletter on how to teach better with tech.

In each edition, I share what I have learned — and am learning — about AI and tech in the university classroom. What works, what doesn't, and why.

Today, I follow up on Friday’s post to explain the architecture of the memory solution I’ve developed for agentic harnesses and compare it with some alternatives available to you. I hope this piece helps you navigate your options.

Hi—

Friday’s newsletter argued that an AI model matters enormously, but that the harness around it—the software environment that decides what context it gets, what it’s allowed to touch, and what happens to its work when the session ends—matters too. I closed by naming the sharpest version of that problem: better memory isn’t the same thing as a trustworthy record of a project. If you missed it: “Agentic Harnesses Need a Proper Workspace.”

That shift is already visible at enterprise scale. Visa, for example, has open-sourced a multi-model vulnerability harness that maps its codebase, routes agents through a structured workflow, supports remediation and validation, and keeps human judgment at key gates. At Kognity, when I left, we had been building and testing a range of custom harnesses for content production.

I’ve written before about what a reasoning model can offer as a thought partner for research. The question I left hanging in that piece was how you give a thought partner like that a trustworthy, current brief without re-explaining everything each time you sit back down with it. This post explains my answer: the Work Ledger, which I’ve now used on my own consequential projects for months.

🧵 A Chat has an Ending. A Project Doesn’t.

Return to my example from Friday about a six-month course redesign. It starts with student feedback, moves through a new set of learning objectives, picks up an assessment plan, loses that plan when a curriculum committee objects to it, and settles on something else for reasons that mattered enough to change course. Somewhere along the way, someone finds a source worth keeping, someone else raises a constraint nobody had considered, and one promising idea sits unresolved because the evidence for it isn’t in yet.

Swap in a grant proposal, a research paper, an accreditation cycle, or a departmental policy revision, and the shape repeats. A real project accumulates decisions that currently govern, options that were tried and dropped, sources that carry different weight, and questions that remain open. None of that survives well as governing project state in a chat window.

A full chat transcript that you return to repeatedly keeps everything, which makes it a useful archive and a terrible briefing: the plan you abandoned, the mistake you corrected, and the eleven messages you spent realizing you and the AI meant different things by the same word sit at the same weight as the decision that actually matters. Reading it back to reconstruct what you settled on, and then injecting new context that happened outside of the chat (with relationships to prior states explained), is often more work than starting again from scratch.

Your AI’s own memory solves a related but different problem. It’s good at not making you repeat yourself. It is not designed to serve as the canonical record of whether what it remembers still governs the project. An AI that perfectly recalls a plan your committee killed three weeks ago hasn’t remembered your project. It’s confidently wrong in a new way.

Anthropic’s own documentation for its experimental Dreams system names duplicates, contradictions, and stale entries as predictable consequences of memory accumulated across sessions. Its answer is a separate consolidation pass. That is useful. But it is still a different job from governing what may become current before later work relies on it.

The obvious low-lift fix is one big context document per project you maintain and paste in at the start of every session. For some work, that’s enough, especially if it evolves slowly, is repetitive in some regard, and the parts of the document are easy to assemble. If that sounds good to you or fits some of your tasks, I’d rather leave you with it than talk you into more machinery than you need.

But someone has to decide, every time the document grows, what belongs in it and what’s gone stale, and that job doesn’t get any easier just because you keep doing it by feel. It takes a lot of manual effort, especially as projects gain complexity or you have many workstreams running at once. This is the across-time version of what I once called integration effort: supplying the context once is not the same as keeping it current.

🔬 Borrow a Discipline that Already Exists

I trained and taught as a philosopher before working on AI, and the habit that survived the career change wasn’t loyalty to any particular argument (though I am still working on my Humean system on back burner 6). It was a way of handling a claim: state it precisely, say what it rests on, go looking for the strongest reason it might be wrong, and keep straight what you’ve actually established versus what you’re still assuming.

Philosophers carry out this process in a few specialized ways, but really it’s a general pattern. Researchers in a range of fields use versions of the same discipline, often with more paperwork attached. In preregistered confirmatory work, researchers state hypotheses and methods before analyzing results, so planned tests remain distinguishable from later exploration. Lab notebooks record what was done and observed, in order, and corrections go forward instead of erasing what happened. Review does not make a finding true, but it gives someone who did not produce it a chance to challenge the reasoning, evidence, and record. Together, those practices make a conclusion easier to inspect, test, and correct months or years later (cf. software development).

An AI working across dozens of sessions on a project that matters is the kind of long, error-prone process that benefits from that discipline.

So the Work Ledger adapts five of those habits, and each one earns its place by fixing a specific way things go wrong without it.

A briefing memo, not a transcript

Every substantial session begins by your harness being forced to read a small, current set of files: what’s decided, what’s still open, what the project actually looks like right now. That bounded set comes first; older material is retrieved when the task needs it. If your curriculum committee rejected one assessment plan and adopted another, the current file says so, plainly, in one place. Nobody has to reconstruct the governing decision from fourteen old conversations and the tone of a meeting.

Primary sources, kept separate from what you cite from them

Any historian, lawyer, or journalist already knows why you don’t let your notes about a document start standing in for the document itself. The Ledger first makes the project’s source boundaries explicit, then keeps the original sources inside those boundaries—the actual policy, research paper, feedback, or dataset—separate from the smaller, extracted claims a later task uses. The harness is instructed how to make summaries in systematic ways. And the workflow does not let a summary silently outrank the material it summarized, so a task that needs three sentences does not have to load eighty pages by default while a task that turns on the details in the eighty pages can leverage them. Finally, the provenance remains inspectable: you can see what an extracted claim came from without pretending every source plays the same role.

A prospective task record

Before the AI starts a substantial task, it writes down what it’s about to do: the objective, exactly which sources and decisions it’s relying on, what’s still uncertain, and whether it thinks it has enough to go on. Then it does the work.

Writing that down before the task, not after, is the whole point. A record composed afterward can always make a wandering process look tidier than it was. Writing it down up front means that if the AI later asserts something its own stated sources don’t support, the conflict is visible to the owner or reviewer; the task can’t borrow authority from its own confident prose.

A lab notebook for claims that matter

Most day-to-day assumptions, propositions, or conclusions don’t merit being treated as notable claims in their own right. But a few are load-bearing: governing principles, settled conclusions, or other claims later decisions will be built on. For those, the Ledger keeps a dated, append-only assessment history. Entries do not disappear when they turn out to be wrong; material corrections supersede them, with the old entry and the reason still there to be checked.

Before something enters that record, the AI runs it through a structured sequence I call TRACER:

• define the Target claim;

• Reconstruct what it means and what it rests on;

• Assess the evidence and each source’s role;

• Challenge it with the strongest reason it may be wrong;

• Establish the warranted stance and confidence; then

• Record what should change, what later work depends on it, and when to revisit it.

The rigor scales with the stakes—nobody needs a six-part defense to reschedule office hours—but for anything the project will lean on later, the AI has to show its work before it gets to keep the conclusion.

This is the most Graham-opinionated part of the Ledger’s functionality, I would say, because it basically operationalizes how I as a philosopher think about how one should be in a “justificatory stance” about serious claims. (I taught a similar process in my courses, applied to specific philosophical domains.)

A fresh review, and a paper trail that doesn’t get rewritten

Before a change to one of those load-bearing claims becomes official, it is frozen and checked in a fresh pass against the exact candidate being proposed. Change the wording, evidence, dependencies, or affected files, and the old review no longer counts. Only after acceptance does it become part of the project’s durable history.

The durable history itself lives in ordinary Git version control: each accepted checkpoint is dated, attributable, inspectable, and recoverable. (A local Git commit does not itself publish the repository to the internet; pushing it to a remote is a separate action, and needn’t be involved at all if there are reasons to keep the records for a Ledger entirely local.)

The distinctive part is how those pieces fit together. Under this workflow, sources do not become claims by accident. Confidence alone does not make a claim current. Superseded claims remain traceable, and dependent work can be found when a premise changes. The Ledger does not decide the project for you; it makes the current decision, its authority, its evidence, and its consequences inspectable.

This is also the operational answer to an argument I’ve made twice before (AI Integration is Like Hiring and Will Students Need Management Skills as AI Develops?): if AI is something we increasingly manage, its authority, evidence, handoffs, and review cannot live only in the manager’s head.

⚙️ How It Works, Literally

Those are the disciplines. Mechanically, the Ledger is much less exotic: one local folder of ordinary text files, with Git keeping the history. Inside current/, a handful of Markdown files divide up the briefing work: context-pack.md says what the project looks like now; decisions.md says what governs; open-loops.md preserves what has not been resolved; technical-preferences.md keeps stable working preferences; and context-map.md tells the AI where to look next. Original material lives separately under sources/; substantial tasks get dated Markdown records under dossiers/; deliverables live under artifacts/; and the relatively few claims that merit formal treatment go into a small, append-only current/claims.jsonl record. The default version of the Ledger needs no database, background service, or Ledger-supplied program running behind the scenes, and the AI retrieves older material only when the task calls for it instead of loading the whole history every time.

Initialization is a one-time setup inside that folder, not a training run or an import of every conversation you have ever had. You open it as a project in Claude Code or Codex and answer five plain-language questions: what kinds of files will enter the workspace, how much understanding should become reusable, how detailed the source notes should be, how the AI should communicate, and whether it should save approved changes locally for you. Those answers bind a unique Ledger identity to an operating profile; they do not quietly turn on web access, connectors, external actions, public sharing, or remote synchronization. When you add the first sources, the AI preserves eligible originals and their provenance before proposing what, if anything, should become reusable project understanding for you to correct or approve.

After that, you use the AI normally. At the start of a substantial task, the harness first checks that the Ledger’s identity and claim record still agree and that it is not stepping over a half-finished checkpoint. It reads the bounded current brief, then opens one dated task dossier before doing the work: the objective, the sources and decisions being relied on, the remaining uncertainties, and the intended output. Most conclusions never enter the formal claim record. If the work changes something load-bearing, TRACER exposes the evidence, strongest challenge, confidence, consequences, and conditions for reconsideration; a fresh pass checks the exact proposed change; and only an accepted version becomes a local Git checkpoint. Later corrections supersede the old assessment rather than deleting it.

Claude Code and Codex enter through different project instruction files—CLAUDE.md and AGENTS.md—that point to the same shared protocol and the same project-owned state. The claim is not that their internals are identical: their configuration syntax, permission systems, and review mechanisms differ (this is just something we have to navigate, given how Anthropic and OpenAI have approached their harness development), and the Ledger keeps those differences explicit.

The invariant is narrower and more useful: given the same role and authority, both get the same editable paths, source and action boundaries, review and recovery obligations, and saved outcome. Because the authority lives in the folder rather than either product’s private memory, I can close one harness, open the same project in the other, and keep going from the same record—one active writer per shared worktree, so their edits do not silently collide.

🛠️ What this Looks like in Practice

Come back to that course redesign after six weeks away. You open a fresh session. The AI reads the current state first: the assessment redesign is active, the faculty group adopted a shared rubric, and the oral-assessment question is still open. It doesn’t start by rereading every chat and every draft since January.

You ask it to revise the assessment instructions to match the adopted rubric, without touching the still-open oral-assessment question. It writes down, before doing anything, exactly which rubric, which committee decision, and which feedback it’s using, and flags the open question as something to leave alone rather than something to helpfully resolve for you. Then it does the work and checks the result against what it said it was trying to do.

Maybe that’s the whole story: the instructions got better, and nothing about the project’s settled understanding changed. Maybe the revision also reveals that one current decision was incomplete. If so, that gets written up in proportion to how much it matters, checked, and folded into the record. Three weeks later, the next session starts from the corrected understanding, with the source and the reason still there to check.

What this guards against is a specific, recurring list of failures: an abandoned idea resurfacing as a decision, a summary outranking the thing it summarized, a stray file becoming part of the project’s understanding just because it happened to be lying around, or a later session keeping a conclusion while losing the reason for it.

✅ Has it Earned its Keep?

For my own work: yes, clearly. I’ve used this across at least five major active workstreams for months, moving between Claude Code and Codex as the work demands. It has repeatedly done the thing I built it to do: let me pick a consequential project back up and trust that what it says is current, decided, and backed by something I can check.

In one case, the Ledger caught an analysis that was about to reopen work I had already finished. In another, its reviewer rejected a status update because the evidence showed that a beta installation was working, but not the stronger claim the draft was making. The wording changed before it became project memory. That’s not a hypothetical benefit. It is the failure mode I built the Ledger to stop.

I’m not the only one it’s worked for. One beta tester had been using Claude Cowork more than four hours a day for more than six months. After a day or two with the Ledger, she told me, unprompted, that it was already significantly better for her work, even without having the benefit of as much context as her Cowork. She told me that it repeatedly would one-shot the documents she needed from it (she didn’t need to edit them in any significant way afterwards, and she didn’t need to iterate with Code to get them right).

That is the early signal I wanted: the Ledger can help inside somebody else’s day-to-day AI workflow, not just the system I built around my own work. The third post in this series will put it in front of you, so you can see whether it earns its keep in yours.

🗺️ Where it Fits in the Optionspace

Some of the key building blocks here—Markdown files, Git history, project instructions, and a record of sources—aren’t new, and I’d be overselling the Ledger if I told you nobody else had assembled them. TRACER, on the other hand, is entirely novel as a harness instruction set, as far as I know. But, again, several tools are in the vicinity, trying to solve these problems. Here’s the comparison:

One development worth watching: Google’s Open Knowledge Format v0.2 gives portable Markdown knowledge files optional fields for provenance, verification, freshness, lifecycle, and attestation. That points in the same direction: portable context needs explicit trust signals. It still adds a vocabulary, not the Ledger’s governing workflow for what becomes current, what a task may rely on, or how a load-bearing claim change earns acceptance.

But even if Google’s efforts gain traction and maturity, I’m not convinced we’ll end up in a world where massive harness providers like OpenAI and Anthropic converge on a shared file architecture that’s powerful and harness-neutral. It isn’t in their interests from a lock-in perspective or from a model training perspective. Their current products point toward provider-specific model–harness integration instead: Google already offers Gemini CLI and Jules, while Meta says Muse Spark 1.2 was co-trained with Muse Code around goals, compaction, and subagents.

Regardless, I have work to do. So, for now and probably for quite some time, I’ll be working on the Work Ledger: a harness-neutral system for governed work continuity with an evidence trail, for the specific case where being wrong about the current state costs you something real.

The final post in this mini-series lands Tuesday, September 8. I’ll give you a starter version of the Work Ledger that you can put on some real projects of your own and test against everything I’ve just argued, plus a step-by-step guide to get it running.

P.S. I have also built Schedule Workbench for a departmental problem I saw firsthand: assigning faculty to courses through spreadsheets, email, and a great deal of tacit knowledge. The tool I built imports the agreed information, pauses for review, and then makes proposals that a human scheduler can edit, lock, use, or leave alone. It also preserves checkpoints, history, and exports. You can see a walkthrough at scheduleworkbench.com. If struggles with the same process, I would be interested to hear how it works and if you think I could help.