Robby 5 Rebuild

What it is

Robby 5 is a manifest-driven prototyping harness. You describe a screen and a working route appears — no registration step, no router to edit, no build to trigger.

The generation is the least interesting part of it. What makes the output usable is everything wrapped around generation: what the assistant is made to read before it writes, and what has to pass before the result counts as done.

It's built as three pillars, deliberately separated by how fast each one changes.

A knowledge layer

Slow and curated. Composition rules, layout heuristics, templates, and cards describing the product itself — all plain YAML and JSON, so they're diffable, reviewable, and editable by someone who doesn't write code.

A tool server

Occasional. Fifteen tools that make the knowledge and the live codebase queryable, so the assistant asks what exists instead of assuming it.

A manifest engine

Constant. A build plugin watches for prototype folders and regenerates the route map on the fly. Folder in, route out.

Keeping them apart is why you can add a hundred prototypes without touching the tooling, and refine the design guidance without touching the app. It's also why the whole thing is product-agnostic: the company-specific parts — brand, navigation, record types, vocabulary — live in a config layer. Point it at a different product and nothing about the engine changes.

A rule, or a suggestion

A design system is a rule. To a language model, it's a suggestion.

It invents components you don't have — a plausible DataGrid that isn't in your library, which renders fine in the chat and dies at the pull request. It reaches for colour that isn't yours, so a prototype drifts off-brand the moment someone screenshots it for a deck. And it speaks generic software: items, records, widgets. Stakeholders spend the review translating the demo into their own domain instead of reacting to it.

I wanted to know how bad this actually was rather than assume it, so I built a test that could prove it either way.

I stood up a disposable branch configured as Zendesk — its brand colour, a seven-section navigation, and five context cards describing record types, roles and workflows, vocabulary, and metrics. Then I planted three canaries no model could know from training: a record type called Playbooks, a status renamed Waiting on vendor, and a mobile triage surface called Frontline. Anything that came back using those terms had to have read the context. Anything that didn't had guessed.

The choice of a well-known product was the part that needed guarding. Zendesk's real vocabulary is already in the model's prior — the control runs produced macro, internal note and requester without being told anything — so none of that counted as evidence either way. Only the invented terms could distinguish a session that had read the context from one doing a good impression of it.

Eleven generation runs, each in a fresh session, each given a single sentence of designer prompt that never mentioned the product, the context, or the test. I removed the methodology notes from the branch so no session could read what was being measured.

The individual failures were worse than the headline. Asked for a problem-and-incident view, one session produced ServiceNow: PRB- and INC- reference numbers, P1–P3 priority levels, and a status the cards explicitly forbid. One was handed deliberately wrong vocabulary in the prompt and adopted it wholesale, with zero correction. One never learned the product had the exact surface it was being asked to design. And one built the wrong product altogether — an overdue-invoice escalation flow wired to a NetSuite connector.

And every one of the eleven typechecked, passed lint, rendered without a console error, and satisfied every composition rule.

That's the part worth sitting with. There was nothing to catch. The shell around each prototype was correct, the components were real, the code was clean — and the screen inside it was about somebody else's product. The failure is invisible unless you already know the domain, which is exactly what makes it expensive: it survives review by anyone who isn't the person it would mislead.

Why it happens

A designer learns a design system tacitly. You sit in reviews. You watch what gets sent back and why. Over a few months you develop a feel for which patterns are load-bearing and which are just habit, which rules bend and which don't. Almost none of that gets written down, and it doesn't need to be — it lives in the people, and the people are in the room.

A model has no ambient. It has exactly two things: whatever is in its context at the moment it generates, and a prior trained on every other company's software. Give it nothing about your product and it doesn't fail loudly or refuse. It fills the gap with the average of the internet, confidently, in your component library's syntax.

That's the actual gap. Not that the model is weak — the code it wrote was fine. It's that the knowledge a designer holds implicitly has no path into the one place the model can use it, and nothing in the loop was making it look.

That conversion is the translation layer, and it has three moves.

The translation layer

Knowledge as data, not prose. The design system and the product description live in YAML and JSON — versioned, diffable, reviewed like code, and editable in the app by someone who doesn't write any. Not a preamble somebody remembered to paste into a chat. Prompts decay; files get reviewed. And the cards outrank the request, so when a stakeholder uses the wrong word for something, the prototype still uses the right one.

Queryable, not summarised. Fifteen tools turn the codebase into something answerable. The assistant asks what exists rather than inferring it: every real component with its props and variants, the actual design tokens read from source, the icon set searched by concept, the composition rules, the current state of the system. Everything returns structured JSON rather than prose, so there's nothing to misread.

A constrained output format. The assistant can emit a spec — structured data, not JSX — which the runtime resolves through a registry of twenty-one block types. Unknown types are skipped rather than crashing the page, and there's an eject path to hand-authored code for when a prototype outgrows the declarative form.

Narrowing the output space is the point. You can't hallucinate a component that isn't in the registry.

When asking nicely fails

The project instructions had asked every session to load the product context before building anything. That instruction was there from the start. The eleven-session test is what it was worth.

A hook now sits in front of every write to a prototype and denies it until the session has actually loaded the context. A second hook records that it did, keyed to the session, so the permission can't be inherited from something that happened last week. Where the gate can't tell what happened, it refuses — a gate that resolves ambiguity by allowing is theatre.

The refusal message names the three ways forward, including what to run if the tooling was never built in the first place. That last line matters more than it looks: without it, someone in a fresh clone hits a wall whose only exit is switching the gate off, and a gate whose first lesson is how to disable it is worse than no gate at all.

The decision I'd defend hardest is the escape hatch. It's an environment variable that has to be set before the process starts, which makes it deliberately unreachable from inside the agent's own session. The simpler version — a file the process could check — was considered and rejected, because any agent can create a file.

There's a human on the other side of that tradeoff, so the hatch is real and documented — someone editing a prototype by hand shouldn't be fighting the harness. It just can't be reached by the thing being constrained.

Rerunning the failed probes in fresh sessions took canary pickup from 0 of 11 to 2 of 2, verified end to end rather than inferred from the code.

Where enforcement stops

Ten composition rules run against every prototype — empty states, loading states, destructive confirmation, never colour as the only signal — alongside custom lint rules for hardcoded colour, raw palette classes, and imports that cross prototype boundaries.

One of those rules is unlike the others. It enforces the product's own vocabulary, and its rules are data: loaded from the same terminology entries a non-engineer edits in the app. It's the only check in the system that enforces the thing the system claims to know, and it scans every string — status maps, aria-labels, toast copy — because forbidden words don't only appear in visible text.

But only high-confidence checks are allowed to block. Two rules I care about a great deal — confirming destructive actions, showing loading states — are permanently advisory, because whether they're really violated depends on context that a source-pattern check cannot see. The confirmation dialog might live in a shared component one file over.

Deciding what I wasn't entitled to enforce mechanically turned out to be as much of the design work as deciding what to enforce. A checker that's wrong ten percent of the time doesn't cost you ten percent — it costs you the whole mechanism, because people learn to ignore it.

Role

Sole designer and engineer. Architecture, interface, knowledge schema, tool surface, enforcement, and the tests behind all of it.

The thing I didn't expect going in was how much of it was design work rather than engineering work. Deciding what a model is allowed to know, what it has to ask for, what it can emit, and what it gets refused for is interface design — the user just happens to be a machine. The knowledge schema is an information architecture problem. The refusal message is copywriting, and a badly written one teaches people to disable the feature it's protecting.

It's also the first system I've built where the honest way to evaluate my own work was to try to make it fail, in conditions I couldn't fudge, and write down what happened either way.

What it produces

The harness, the runtime, the knowledge layer, and the tool server are built and running. It costs nothing to operate — everything authors locally and deploys as a static build.

The evidence that it works isn't a testimonial, it's a scoring method. Each round configures a fresh fixture product, plants canaries that can't be guessed, runs generation sessions blind, and greps the output for terms that could only have come from the context.

0/11 → 2/2

Zendesk round — canary pickup before and after the context gate

8/8

Greenhouse round — both layers firing

15

Tools exposing the system to the assistant

$0

Recurring cost to run

The Greenhouse round is the one worth describing, because it's the one that found something. A contradiction trap — where the prompt deliberately asks for a word the context cards forbid — turned out to be the most diagnostic probe of the set. The correct behaviour isn't just to use the right word. It's to use the right word and say why. That distinction only became visible because the round was designed to catch a specific failure rather than to demonstrate a success.

The other outcome is less measurable and probably matters more: the constraint is mechanical now. It doesn't depend on anyone remembering to be careful, including me.

What I'd do differently

I built the evals last. The scored rounds are the strongest evidence the harness works, and they're simultaneously the most fragile process in the project: manual, needing a clean fixture and a fresh session every time. One round was nearly invalidated because a stale server process was quietly serving old context. The scorer should have been built alongside the gate, not after it — and until it's automated, every number is one procedural mistake away from being unreproducible.

The gate proves less than it appears to. It guarantees the session loaded the context. It cannot guarantee that the thing doing the writing read it — subagents inherit the parent's unlock. That limit is inherent to a marker-based design rather than a bug in this one, and I'd rather state it plainly than let a tidy diagram imply otherwise.

I found a fourth way to write a prototype after I'd finished gating the other three. An in-app authoring path writes files without going through any of the gated routes. Leaving it open is defensible, since a human is driving it directly and the gate exists to stop an agent writing from memory. But it was an oversight before it was a decision, and I'd rather record it that way round.

The pattern across all three is the same, and it's the thing I'd carry into the next system: the parts I reasoned about carefully were fine, and the parts I reasoned about instead of testing were where the holes were.