The Tool That Removes Client Data Is the Last Place Anyone Looks for It

2026-09-28 · Philip Choo · AI9OS

A practice builds a tool whose whole job is to keep client identifiers out of shared work. It scans documents before they leave, and flags anything shaped like an identity card number, a passport number, a bank account.

To prove the tool works, it needs test data: strings it should catch.

The fastest correct-looking test data is whatever is sitting on the desk. The identifiers from the very matter that prompted building the tool are right there. They are pasted into the tool's test file. The test passes. The tool demonstrably catches real identifiers.

And from that moment, those real identifiers live inside the tool that exists to remove them.

Why nobody finds them

This is uncomfortable to write about, and that is the reason it is worth writing about. The failure is not carelessness. It is where people look.

1. The hiding place is chosen by the reviewer's own picture of the file. A scanner is mentally filed under "tooling". So a careful sweep for client data reads the case folders closely, and skims the tools. The identifiers are not hidden. They are in the room nobody searches, because nobody expects client data in a tool.

2. The control cannot catch itself. Nobody runs the de-identification scan over the scanner's own code. It would look absurd if proposed. So the one piece of the practice that is guaranteed never to be scanned is the scanner.

3. Some places do not forget. Case files are backed up, encrypted, and eventually destroyed under a retention policy. Software, once it goes into version control, is copied and kept indefinitely, and deleting a file later does not remove it from the history. Moving an identifier from the first kind of place into the second turns a managed risk into a permanent one.

The same instinct shows up in a second form: a general-purpose tool quietly keeping one matter's entire narrative as its default input, so that it can be run during development without typing anything.

What it looked like

In the case this comes from, a practice was preparing several days of tooling work to be saved into version control for the first time. A check before that step found that the scanner's own self-test contained several live identifiers from an open matter. Other test values in the very same function were properly invented, which shows this had never been a policy. It was convenience, repeated a handful of times.

A second tool, one that generated diagrams, held that same matter's full narrative as its default. Both were caught one step before they would have been committed to permanent history.

What to do instead

Test data is invented, never observed. A pattern test needs a string of the right shape, and shape is all it needs. An invented string with the correct structure proves the pattern exactly as well as a real identifier does. If test data is real, it was chosen for convenience, and convenience is not a reason that survives being written down.

A general tool holds no case as its default. Ship an invented worked example that doubles as documentation, and load real matter data from somewhere outside the tool. The example can then be shown to anyone, at any time, with no clearance step.

Check at the point of no return, not at the point of intention. A rule that says "do not paste real identifiers" is a habit. A scan that runs automatically on everything about to be saved to permanent history, tools included, is a control.

Exclude by pattern, not by memory. Case-specific files should be identifiable by a rule about their names or locations, not by a list somebody has to remember to update. A list maintained by memory fails on the first busy day.

The part worth keeping

The same sweep turned up something more interesting than the identifiers. It flagged a written lesson that mentioned a real institution and a real sum. The scanner reported that file clean, and was right to, by its own design: prose with no identifier-shaped strings in it is invisible to a pattern matcher. The scanner says so in its own output. It cannot catch a combination of ordinary details that identifies somebody by being unique.

That sentence had been read many times as a caveat. It is not a caveat. It is a stated limit on what the tool covers, and the thing outside that limit is exactly where written case narratives live. The only control over that class of leak is a human reading the text and asking whether it narrows to one person. That is why published pieces here get a human de-identification read, and it is the same point made about a search that cannot find something: a tool's silence is only evidence within the range it can actually see.

If you are trusting a practice with your data

"Where is my data, including inside your tools?" It is an odd question, and worth asking. The places a practice is careful about are the places it thinks of as holding client data. The risk is in the places it does not.