Episode 37 · Module 8 · AI Security

Governed Writes and Human-in-the-Loop

19 May 2026 · 8:02 · Security for Legal SaaS

8:02 8:02

In Episode 36, we examined how models can leak training data through inference attacks. This episode addresses a different risk entirely: what happens when AI doesn't just analyse — it acts. The principle is simple: AI systems in legal practice should propose, never dispose. They can draft a contract clause, suggest edits to a brief, classify a document, or flag a compliance risk. But filing a court document, sending a client communication, modifying a matter record, or executing a financial transaction — those actions must require a human professional to review, approve, and take responsibility.

Today’s Lesson

Security for Legal SaaS — Episode 37: Governed Writes and Human-in-the-Loop

AI Proposes, Humans Dispose

In Episode 36, we examined how models can leak training data through inference attacks. This episode addresses a different risk entirely: what happens when AI doesn't just analyse — it acts.

The principle is simple: AI systems in legal practice should propose, never dispose. They can draft a contract clause, suggest edits to a brief, classify a document, or flag a compliance risk. But filing a court document, sending a client communication, modifying a matter record, or executing a financial transaction — those actions must require a human professional to review, approve, and take responsibility.

This isn't just good security practice. It's a professional obligation.

Why Legal AI Must Not Autonomously Write

Professional Responsibility Demands Human Judgment

ABA Formal Opinion 512, issued in July 2024, is the American Bar Association's first comprehensive ethics guidance on generative AI in legal practice. It makes the obligation explicit: "GAI tools lack the ability to understand the meaning of the text they generate or evaluate its context, and therefore are not a substitute for the independent professional judgment a lawyer must exercise."¹

The Opinion addresses several Model Rules:

Model Rule	Requirement	AI Implication
Rule 1.1 (Competence)	Lawyer must provide competent representation	Lawyer must understand AI tool's capabilities and limitations
Rule 1.4 (Communication)	Keep client informed of case status	Client must know when AI is being used in their matter
Rule 1.6 (Confidentiality)	Protect client information	AI tool's data handling must preserve confidentiality
Rule 3.3 (Candor)	Duty of candor toward the tribunal	Lawyer must verify all AI-generated citations and analysis
Rule 5.1/5.3 (Supervision)	Partners must supervise subordinates	AI tool use requires supervisory frameworks

The EU AI Act reinforces this globally. Article 14 requires that high-risk AI systems — which includes AI used in legal decision-making — "be designed and developed in such a way that they can be effectively overseen by natural persons during the period in which they are in use."² The oversight must enable humans to understand the system's capabilities, correctly interpret its output, and decide not to use the system or disregard its output.

The Automation Bias Problem

Automation bias — the tendency to trust automated outputs without critical evaluation — is the practical reason governed writes matter. Research consistently shows that humans over-rely on AI suggestions, especially when those suggestions are presented with apparent confidence.³

In legal AI, this manifests as:

Accepting AI-drafted clauses without verifying they reflect the client's negotiated position
Filing AI-generated briefs without checking cited authorities (the phenomenon that produced the Mata v. Avianca sanctions in 2023)
Approving AI-classified documents without reviewing edge cases

Case study: Mata v. Avianca (S.D.N.Y., 2023). Attorney Steven Schwartz submitted a brief containing six fabricated case citations generated by ChatGPT. When opposing counsel could not locate the cases, Schwartz asked ChatGPT to confirm they were real — and it did. The court sanctioned Schwartz and his firm. The failure was not in using AI; it was in treating AI output as a final product rather than a draft requiring verification.⁴

Technical Enforcement: The Draft State Pattern

The "governed writes" principle must be enforced technically, not just by policy. Telling lawyers "always review AI output" is insufficient — the system architecture should make unreviewed AI writes impossible.

Architecture Pattern: AI → Draft → Review → Production

AI generates output → Stored in DRAFT state
                      ↓
Human reviewer receives notification
                      ↓
Reviewer approves / edits / rejects
                      ↓
If approved → Promoted to PRODUCTION state
If rejected → Returned to AI with feedback

Every AI-generated artifact — a contract clause, a document classification, a research memo, a billing entry — enters the system in a draft state that cannot reach production without human approval. The approval event is logged with the reviewer's identity, timestamp, and the specific version reviewed.

Implementation Patterns

Pattern	Description	Use Case
Approval queue	AI outputs land in a review queue; nothing progresses without explicit approval	Court filings, client communications, regulatory submissions
Confidence threshold	Low-confidence AI outputs require review; high-confidence outputs may proceed with lighter oversight	Document classification, email triage
Escalation rules	Certain output types always require senior review regardless of confidence	Privilege designations, conflict checks, financial transactions
Four-eyes principle	Two independent reviewers must approve before promotion to production	High-stakes filings, M&A document production

Key distinction: A confidence threshold does NOT mean "skip human review for confident outputs." It means "route high-confidence outputs to a faster review track and low-confidence outputs to a more thorough one." Even at 99% confidence, a human must see the output before it becomes official.⁵

What "Review" Actually Means

Effective human oversight requires more than a rubber stamp. The IAPP's analysis of human-in-the-loop requirements notes that oversight fails when reviewers lack the time, training, or technical understanding to meaningfully evaluate AI outputs.⁶ For legal AI, meaningful review means:

The reviewer can see the AI's reasoning — not just the output, but what inputs it relied on and how confident it is
The reviewer has domain expertise — a junior associate reviewing a complex derivatives clause is not meaningful oversight
The reviewer has time — if the approval queue contains 500 items and the reviewer has 30 minutes, oversight is theatrical
The reviewer can reject without friction — if rejecting an AI output requires more effort than approving it, approval becomes the default

Specific Governed Write Scenarios in Legal SaaS

Court Filing Systems

An AI that drafts a motion should produce a reviewable document with tracked changes, citations flagged for verification, and a summary of the legal reasoning. The "File" button must require attorney authentication and a certification that the filing has been reviewed. The system should log: who reviewed it, when, which version, and whether they made edits.

Client Communication

AI-drafted emails to clients should enter an outbox that requires explicit send approval. The system should prevent scheduled auto-send of AI-generated content — every communication must pass through a human checkpoint. As ABA Formal Opinion 512 notes, boilerplate consent in engagement letters is not sufficient to authorise unrestricted AI use in client communications.¹

Document Classification and Privilege Review

AI can accelerate privilege review by pre-classifying documents, but the privilege designation must be confirmed by a qualified attorney. Incorrect privilege designations have discovery consequences — a document wrongly marked "not privileged" and produced to opposing counsel cannot be unproduced.

Contract Review and Redlining

AI-suggested redlines should appear as tracked changes, not direct edits. The reviewing attorney must be able to accept, reject, or modify each suggestion individually. The final document should record which changes originated from AI and which from the attorney.

Audit Trail Requirements

Every governed write must produce an audit trail that answers:

Who generated the output? (Which AI model, which version)
Who reviewed it? (Authenticated identity of the human reviewer)
When was it reviewed? (Timestamp, with time zone)
What did the reviewer see? (The exact version presented for review)
What did the reviewer decide? (Approve, reject, modify — and the specific modifications)
Why was it escalated? (If applicable — confidence threshold, document type, matter sensitivity)

This audit trail is not optional. It is the evidence that human oversight actually occurred, and it must be tamper-evident — a topic we will cover in Episode 42.

The Harvard Standard for AI Oversight Liability

A 2024 Harvard Journal of Law & Technology analysis proposed redefining the standard of human oversight for AI negligence. The argument: if a professional claims to have "overseen" an AI system but the audit trail shows they approved 200 outputs in 15 minutes without opening any of them, the oversight was illusory, and the professional bears the same liability as if no oversight occurred.⁷ Meaningful oversight leaves a forensic trail that can withstand scrutiny.

What's Next

Episode 38 covers LLM API Key Isolation and Inference Gateways — how to manage the API keys that connect your legal AI to cloud providers like OpenAI and Anthropic, and why a single leaked key can cost you more than a data breach.

Sources & Further Reading

Sources & references

ABA, Formal Opinion 512: Generative Artificial Intelligence Tools (July 2024).
EU AI Act, Article 14: Human Oversight.
Strata.io, Human-in-the-Loop: A 2026 Guide to AI Oversight.
Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y. June 22, 2023) — sanctions for AI-fabricated citations.
IBM, What Is Human In The Loop (HITL)?.
IAPP, 'Human in the Loop' in AI Risk Management — Not a Cure-All Approach.
Harvard Journal of Law & Technology, Redefining the Standard of Human Oversight for AI Negligence.
Trilateral Research, Human-in-the-Loop AI Balances Automation and Accountability.
Kiteworks, Human in the Loop: AI Compliance and Oversight Requirements.
NYC Bar, Formal Opinion 2024-5: Generative AI in the Practice of Law.
Small Wars Journal, Human-in-the-Loop or Loophole? Targeting AI and Legal Accountability.

Alice: Welcome back to Security for Legal SaaS. I'm Alice.

Dan: And I'm Dan. Episode 37 — governed writes and human-in-the-loop. Alice, this sounds like it's about keeping humans in charge when AI is doing the work?

Alice: That's exactly it. The core principle is: AI proposes, humans dispose. Your AI can draft a contract clause, suggest edits to a brief, classify a document, even flag a compliance risk. But it should never, on its own, file something with a court, send an email to a client, or modify a legal record. A human professional has to review it, approve it, and take responsibility for it.

Dan: Mm. That sounds like common sense, but I'm guessing there's more to it than just telling people "always review the AI output"?

Alice: There's a lot more. Because telling people isn't enough. The ABA issued Formal Opinion 512 in July 2024 — the first comprehensive ethics guidance on generative AI in legal practice. It's very clear: AI tools "lack the ability to understand the meaning of the text they generate" and are "not a substitute for the independent professional judgment a lawyer must exercise." The lawyer remains responsible. That's not a suggestion — it's an ethical obligation under multiple Model Rules. Competence, confidentiality, candor to the tribunal, supervision of subordinates. They all apply.

Dan: Right. And then there's the EU AI Act as well?

Alice: Article 14 of the EU AI Act requires that high-risk AI systems — and legal decision-making qualifies — be designed so that humans can effectively oversee them. Not theoretically. Effectively. That means the human must be able to understand the system's capabilities, correctly interpret its output, and — this is the key part — decide not to use it or override it. If the system doesn't make it easy to say "no, that's wrong," the oversight is meaningless.

Dan: Hmm. I keep thinking about the Mata v. Avianca case. The lawyer who submitted the ChatGPT brief with the fake citations.

Alice: That's the textbook example. Steven Schwartz used ChatGPT to draft a brief. It generated six case citations that didn't exist. When opposing counsel couldn't find them, Schwartz went back to ChatGPT and asked if the cases were real — and ChatGPT said yes. He submitted them to the court. The judge sanctioned him and his firm. The problem wasn't that he used AI. The problem was that he treated AI output as a finished product instead of a draft that needed verification.

Dan: Yeah. So how do you build a system that prevents that?

Alice: You enforce it in the architecture, not just in policy. Everything the AI generates enters the system in a draft state. Draft means it cannot reach production — it can't be filed, it can't be sent, it can't be recorded as official — until a human reviewer explicitly approves it. The approval is a logged event with the reviewer's identity, a timestamp, and the specific version they reviewed.

Dan: Mm. So it's like tracked changes in a Word document? The AI makes suggestions, the lawyer accepts or rejects them?

Alice: Exactly that model for contract review — AI-suggested redlines appear as tracked changes, and the attorney accepts, rejects, or modifies each one individually. For court filings, the AI produces a draft with citations flagged for verification, and the "File" button requires attorney authentication plus a certification that the filing was reviewed. For client emails, the AI draft goes to an outbox that requires explicit send approval — no scheduled auto-send of AI-generated content. Every path from AI output to the real world passes through a human checkpoint.

Dan: Right. But here's what I'm wondering — what about the situations where the AI is really confident? Like, 99% confidence on a document classification. Do you still need full human review?

Alice: <sigh> This is where people get into trouble. You can use confidence thresholds to route outputs — high-confidence items go to a faster review track, low-confidence items go to more thorough review. But "faster" doesn't mean "skip." Even at 99% confidence, a human has to see the output. Think about privilege review in document production. If your AI is 99% sure a document isn't privileged, and it auto-produces that document to opposing counsel without anyone checking, and it turns out the document was a privileged strategy memo — you cannot un-produce it. The damage is done. One percent of a ten-thousand-document review is a hundred documents. You want a hundred privileged documents slipping through?

Dan: Mm-hmm. Good point. So what about the quality of the review itself? Because I've seen approval workflows where people just click "approve" without reading anything.

Alice: That's the automation bias problem. Humans tend to trust automated outputs, especially when the AI sounds confident. A Harvard study proposed that if a professional claims to have "overseen" AI output but the audit trail shows they approved 200 items in 15 minutes without opening any of them, the oversight was illusory. They should bear the same liability as if no oversight occurred at all. So meaningful review requires four things. One — the reviewer can see the AI's reasoning, not just the output. Two — the reviewer has domain expertise. A first-year associate reviewing a complex derivatives clause is not meaningful oversight. Three — the reviewer has adequate time. And four — rejecting is as easy as approving. If it takes three clicks to approve and ten clicks to reject, approval becomes the default.

Dan: Yeah, that fourth point is really practical. If you make the "no" button harder to reach than the "yes" button, you've biased the outcome.

Alice: Exactly. And the audit trail behind all of this matters enormously. Every governed write should produce a record that answers: who generated the output, which model and version, who reviewed it, when they reviewed it, what specific version they saw, and what they decided — approve, reject, or modify. If the matter ends up in litigation or a regulatory investigation, that trail is how you prove that human oversight actually happened. Without it, you're just claiming you checked.

Dan: Mm. And that ties into the audit log episodes coming up later in the series.

Alice: It does — Episode 41 covers audit log design in detail, and Episode 42 covers making those logs tamper-evident with hash chains. But the key point for today is that governed writes aren't just about preventing bad AI output from reaching the world. They're about creating a defensible record that a qualified human exercised professional judgment at every decision point. That's what the regulators want. That's what the ethics rules require. And it's what protects your firm when something goes wrong — because it will, eventually.

Dan: Next episode — LLM API Key Isolation and Inference Gateways. How to protect the keys that connect your AI to the cloud.

Alice: Until then, I'm Alice.

Dan: And I'm Dan.

Alice: Security for Legal SaaS is a series written with AI assistance. Alice and Dan are AI-generated voices — no professional advice here, just education.

Security for Legal SaaS is a series written with AI assistance. Alice and Dan are AI-generated voices — no professional advice here, just education.