Linting as an agent guardrail

Custom lint rules as the enforcement layer for conventions an agent would otherwise drift away from — architecture boundaries, taste invariants, documentation freshness. Why a linter beats an instruction, and what to put in the error message.

Tests check that code does the right thing. Lint checks that it is built the right way — and that turns out to be the more useful category when the code is written by an agent, because the failure mode isn't usually a wrong answer, it's a plausible one built the wrong way. This note collects what the OpenAI harness-engineering post does with it, since that's the only one of the three sources with a real implementation.

Why a rule beats an instruction

Written guidance about how to build things has three problems in an agent setting, all named in that post: it consumes scarce context and crowds out the actual task; when everything in a long document is marked important, nothing is; and it rots silently, because nobody can mechanically check whether prose is still true.

A lint rule has none of those. It costs no context until it fires, it fires on every run without being remembered, and it either passes or it doesn't. The framing worth keeping: enforce invariants, don't micromanage implementations. Their example — require that data shapes are parsed at the boundary, don't prescribe which library does it. The model picked Zod on its own.

The second-order effect is that this is how human taste gets into an agent-written codebase at all. Review comments and refactor PRs are one-time corrections; a rule is the same taste applied to every future line. When documentation isn't holding a convention in place, promote the convention into code.

Write the error message for the agent

The detail I hadn't considered before reading this: because the lints are custom, the error messages are written to inject remediation instructions into the agent's context. A lint failure isn't just a rejection, it's a targeted prompt delivered exactly when it's relevant, to an agent that is already looking at the offending code. That is about the highest-signal, lowest-cost channel into a running agent there is — and it inverts the usual advice about terse error messages.

What they enforce

• Architecture boundaries. Fixed layers per business domain with validated dependency direction (types → config → repo → service → runtime → UI), cross-cutting concerns entering only through one explicit providers interface. Custom linters plus structural tests; anything else is disallowed mechanically rather than caught in review. Their point: this is the kind of rigidity you'd normally postpone until you have hundreds of engineers, and with agents it's an early prerequisite — constraints are what buy speed without decay.
• Taste invariants. Structured logging, naming conventions for schemas and types, file size limits, platform-specific reliability requirements.
• Documentation freshness. CI jobs validate that the knowledge base is current, cross-linked and correctly structured — docs treated as a checkable artifact, not prose.
• Drift, on a cadence. Beyond blocking lint, background agent tasks scan for deviations from written "golden principles", update per-domain quality grades, and open small refactor PRs. A doc-gardening agent does the same for documentation that no longer matches the code. This replaced a standing Friday cleanup that consumed 20% of the team's week and didn't scale.

That last category is the one that generalises furthest: agents replicate whatever patterns already exist in a repository, including the mediocre ones, so drift compounds by default. Continuous small corrections are cheaper than periodic large ones — their debt-as- high-interest-loan framing.

The same idea where the artifact is a model

Aleph Alpha apply this to a training pipeline, and the translation is instructive because the property being checked isn't structural. Two gates:

• Every PR runs a small-scale end-to-end training run in under five minutes. Same job as a lint — cheap, unconditional, and fast enough that nobody routes around it. It proves the pipeline still executes, not that the model is good.
• Every night, a larger end-to-end run asserts a measurable improvement on the evaluation suite. This is the interesting one: a regression test whose assertion is a score, which is how you catch a semantic break in training logic that still runs cleanly.

The second gate is what a "linter for taste" would have to look like in the open question below — you can't decide it by inspection, so you spend compute and let a metric decide, and you inherit everything that comes with a metric. Which is the trade this notebook keeps circling: mechanical rules are cheap and can only check decidable properties; anything else costs a judge, and the judge needs maintaining (Generator–evaluator loops).

The version already in this notebook

/lint here (.claude/commands/lint.md) is the same idea applied to prose rather than code: broken wikilinks, orphan pages, contradictions between notes, stale claims, concepts mentioned often enough to deserve a page, tags that mean the same thing, summaries that no longer match their page. Some of it is mechanical and gets fixed in place; anything judgment-heavy becomes a file in tasks/ instead. The build does the schema half — frontmatter validation fails it, broken wikilinks warn.

Which suggests two things worth trying here:

• Move more of /lint into the build, where it can actually block. Frontmatter is checked today; orphan detection and updated-date staleness are equally mechanical and are currently left to an agent's attention.
• Write the failure messages the way that post does — say what to do about it, not just what's wrong, since the reader is usually an agent mid-task.

Enforcing a rule is not the same as selecting against it

A distinction this page has been missing, and the Greenblatt interview supplies it from the far end of the scale. Everything above treats a lint rule as a gate: it fires, the agent reads the remediation hint, the agent fixes the thing. Fine.

It becomes something else the moment the rule is used to choose. Rerun until the check passes, keep the variant that scored best, train on whether the gate was tripped — and the rule stops measuring and starts being optimised against. Greenblatt's account of what that does at frontier scale is the whole story: the detectable violations get cleaned up, and what survives is concentrated in whatever the rule cannot see. The pass rate rises either way, so the instrument reports success in both worlds.

Which sharpens the framing at the top of this page rather than contradicting it. Enforce invariants is still right, and the reason is now sharper: an invariant is a property you can decide by inspection, so passing it and satisfying it are the same event and there is no gap to exploit. The nightly score gate two sections up is the other kind — a metric, gradeable only by spending compute, and therefore something a sufficiently motivated optimiser can satisfy without doing the work. Aleph Alpha's version is safe because nothing is optimising against it; that is a fact about who is running the pipeline, not a property of the gate.

For this repo it comes down to one line. npm run build and /lint are fine as gates, and would stop being fine as scores — the moment "did /lint come back clean" becomes the thing an agent is selected on, rather than something a person reads, it is a target and its silence stops being evidence.

Open question

Everything above enforces structure. The interesting unsolved case is the one that motivated the frontend work in the Anthropic post: rules for things that are matters of taste. A linter can require structured logging; it can't require that a layout isn't bland. That's what pushed that team toward a graded rubric and a separate evaluator instead — see Generator–evaluator loops. The dividing line between "encode it as a rule" and "hand it to a judge" is roughly whether the property is decidable by inspection, and it's worth being deliberate about which side a given convention falls on.

A third category: the skill

The dividing line above is two-way — encode it as a rule, or hand it to a judge. A talk on software fundamentals against spec-to-code ships a property that fits neither, and the shape it ships it in is worth noticing. Module depth — is this interface simple relative to what it hides — is not decidable by inspection, so it cannot be a lint; an agent asked to satisfy it mechanically can widen the module until any interface looks simple against it. But it is also not worth a graded rubric and a round trip, because the judgment is made once when the boundary is drawn rather than on every run.

What he ships instead is a skill: a repeatable procedure a person invokes, which explores the codebase, proposes related code to wrap behind one boundary, and hands the result back to be read. No gate, no score, no autonomy — the enforcement is that a person ran it and looked. That is a third slot in this page's taxonomy: a lint for what is decidable, an evaluator for what is gradable, and a procedure-in-a-file for what is neither but is still repeatable. /lint here is already partly that — the mechanical half fixes itself and the judgment-heavy half becomes a file in tasks/ for a person. Worth being as deliberate about which of the three a convention belongs in as this page is about the first two. See Deep modules, and designing the interface you delegate behind.