Skip to content
Loop
Research

Field report · our own run

We tried to break our own premise

Everyone in this category cites research. We wanted to know whether the thing actually works when we run it, so we built it, pointed it at real products, and then ran the single experiment most likely to prove us wrong.

This is a field report, not a citation. The verdict is honest, including the part that didn't work.

What we built

A persona-driven agent driving a real browser. At each step it looks at the actual screen, decides what a person with that intent would do next, does it, and records what happened, with a screenshot behind every step. Not a survey. Not a prediction. A session.

The experiment that could have killed it

The strongest published criticism of AI-driven user testing isn't that the agents are stupid, it's that they're too competent. In peer-reviewed tree-testing studies, the AI sailed through flows where real people get stuck, and therefore revealed nothing. An agent that never struggles is useless.

Our first agents were told, in character, to give up when frustrated. That's a problem: if the agent only "gets stuck" because we scripted it to, the friction is theatre.

So we ran the ablation. We built a second agent with no give-up instruction: a neutral, persistent operator that just tries to finish the task as competently as it can, and stops only when it genuinely can't proceed. Then we ran both agents against the same real product, on the same tasks.

The result

Both agents failed the same tasks, at the same obstacles.

The neutral, unscripted agent did not plow through. It got stuck too, and its struggle showed up as objective behavior: the same action repeated, dead-ends, backtracking, roughly ten different attempts to get past one broken control. Not scripted emotion. Real spinning.

The cleanest case: a location field that hung indefinitely on "finding location." The agent typed into it correctly and still couldn't proceed, for both the scripted and the unscripted persona. That's an emergent, reproducible finding, not an artifact of how we prompted it.

On our evidence, the "too competent" objection did not hold. With the scripting removed, the method still surfaces friction a real person would hit.

The part that didn't work, and the real lesson

Here's what we won't leave out. One of the obstacles our agent "found" was partly our own fault.

Our observer couldn't see inside an embedded frame on the page, so a cookie-consent wall that a human would simply click through was invisible to the agent. It got stuck at a door it couldn't see, and we nearly logged that as a product bug. It wasn't. It was our harness.

That's the real lesson, and it's more useful than the headline: the risk isn't that the agent is too smart, it's that a finding is only as trustworthy as the tool that produced it. A shallow harness makes you chase bugs that aren't there. It's exactly why every finding has to carry its evidence: so you can check it, and so we can catch ourselves.

Where this leaves us

The mechanism is real. The objection everyone raises did not survive contact with the evidence. The problems that remain (seeing embedded content, geographic realism, verifying success against a real funnel) are engineering, not a wall.

Loop is early. This is one pilot: a small sample, one model, one afternoon. We won't dress it up as more than it is. But we ran the experiment designed to disprove us, and reported what broke, which is the only kind of evidence we think is worth anything.

Sources