Some agent context should stay home

This week I switched off the shared memory for two of my coding agents. It came up in review. In my software factory, the agents that coordinate the work share memory, and that's a big part of how the factory holds together. But the review found that the same memory was also loading into the prompts of the agent that writes the tests and the agent that writes the code, before either of them had asked for anything. Nothing about the test flow had run yet, but both sides would have started from the same context, which is what the separation was meant to prevent.

We're all trying to figure out memory right now: shared agent memory, brains that connect agents across sessions, factories that share but decompose at the right level, swarms with group memory. The easy answer is to share more. We want shared memory, and sometimes we need isolation. There's another layer here: the context that shouldn't be shared, explicitly or implicitly. That's the layer where we need memory guardrails.

Test isolation is the clearest example I have. Coding agents game tests. Researchers have caught them editing the test, special-casing the exact inputs the test checks, and overriding a comparison so it always passes (ImpossibleBench; EvilGenie). One of the fixes we're trying these days is to separate the agent that writes the tests from the agent that writes the code, and that's what I did.

My reason goes a step further, though. An author who makes their own tests tends to make ones that serve what they're building. The tests come out facile, duplicated, or confirming the route the code already took. Mutation testing catches some of that. It changes the code on purpose and checks that a test fails, which tells you whether the criteria and bounds in the tests, edge cases included, actually test anything. Mutants at least tell you the tests are stronger-ish. They don't tell you whether the tests were written to fit the answer. Separation goes after that directly, because fitting the answer comes from the shared context of the problem and the solution.

So it's not just about gaming. It's about independence. A hypothetical model that never games at all, writing tests with the implementation's route in its context, can still end up confirming that route. It's the same reason a benchmark means less once a model has seen the test set. It's why machine learning kept holdout sets long before it was sexy. The problem is the shared context, not the agent's character. (Okay, it can be the character too.) That's also why separation of duties shows up even where most people are honest: a check is independent because of what the checker can't see, not because of who the checker is.

In my factory, context sits at three levels. The first is what everyone shares: the problem. That's the project context, the spec, the conformance vectors, and the decisions already made. The second is what some roles share. The agents that coordinate the work share run state and memory. The test author, the runner that executes them, and the reviewers share the tests. The implementer, the runner, and the reviewers share the code. The third is what stays home with one role. The tests in progress stay with the test author, and the implementer never sees them. The implementation stays with the implementer. Each reviewer starts fresh, without the other reviewers' findings.

The walls aren't silence, though. Before the work starts, both halves get the same names for the functions and types, so the pieces fit without either side seeing the other. When a hidden test fails, the implementer gets the failure's name and message, never the test itself. What crosses the wall is designed.

The walls also move. The tests go from the test author to the runner when the two halves come together, then to the reviewers, and then into the open when the change merges. A decision I make starts in one conversation and becomes shared once it's written into the design. So where context sits depends on the role, and maybe on the phase of the software lifecycle.

Building this was harder than writing the rule. Context got shared through the harness, through the codebase and the branch, and through the setup of the shared memory. None of those were designed as ways to share context, and all of them could do it. Every time a reviewer looked, it found another channel. That's the implicit part.

A lot of why I push on this comes from being a queer, trans, disabled, neurodivergent person in the world. That's taught me to notice the ways we let power act implicitly, and the value in bringing it into the open and making it explicit.

There's a tradeoff here between ease and independence. Pooling every reviewer's findings would be easier. Letting the test author read the code would be easier. I've worked in regulated spaces throughout my career, where this tradeoff is part of the work. In finance, the person who makes a change isn't the person who approves it. A law firm walls off one team from another, so what one client told the firm doesn't reach the people on the other side. Those fields decided that for certain kinds of work, independence is worth the friction. Agent factories are balancing the same constraints now.

One answer is to keep a single shared memory and control what each agent can retrieve from it. That's one wall, on one channel. Context also moves through the harness, the branch, and the way memory gets loaded before an agent asks for anything. The memory in my factory wasn't something an agent went and retrieved. It was already in the prompt and context.

Another answer is to use a different model for independence, with one model writing the tests and another writing the code. I think that combines with separation rather than replacing it. A different model brings different habits. But if it shares the context, it still sees the route. A wall keeps the route out.

I'm exploring two directions. One is separation that's dynamic rather than fixed: walls that depend on the role and the phase of the work, and that open or close as the work moves. The other is walls inside a shared-memory substrate, rather than between separate systems. Long-running tasks, context that has to last over time, and keeping agents aligned are all pushing factories and companies toward more shared memory. That makes this layer more interesting, not less.

If you're exploring this too, I'd love to compare notes. That goes for people building agent factories or swarms, people working on agent memory or context graphs, and people in regulated work who've lived with walls for a long time. The question is where there should actually be memory guardrails: where we should bring context together and where we should keep it separate.