Agentic built, but who confirms it works?
Coding agents write the code and the tests, and the tests only cover what the agent was told. The basics nobody prompts for go unchecked, and nothing proves the app actually works before it ships. So who confirms it works?
TL;DR: The agent that builds the app shouldn't be the only authority on whether it works. So I built an eval for agent-generated web apps. It's the step after the agent says done: it runs the app, signs in, and holds it to the industry's own rules, meant to run on every commit. Proof comes from running the app, not reading it.
Every failure comes with reproducible proof, and what it can't check it says it can't check.
Try it on four public repos: two small workspace apps with flaws planted on purpose, OWASP NodeGoat and DVNA.
Why is this needed now?
Because GenAI-built sign-out just caught me out. The prompt says add logout functionality. The agent's implementation clears the cookie, and its test checks the cookie is gone. Both pass. But the session is still live on the server. Anyone who captured that cookie, from the network, a log or a browser extension, can keep using the account long after the person signed out. Nobody prompted for the session to be destroyed, because nobody thinks to. An app has thousands of cases like this. The hard part isn't getting an agent to test an app. It's deciding what the test should demand, and if the same agent writes the code and decides what success looks like, the missing requirements stay missing.
Your app on trial
As an industry we already have the ground truth for what an app should do. OWASP ASVS 4.0.3 says sign-out must invalidate the session. RFC 8725 says reject a token signed with alg none. Auth0's docs say check the audience. Almost none of it is ever executed against a running app.
So I turned those lines into judges. A judge is one line of that ground truth made executable: a small question with the rule behind it, answered by using the running app rather than reading the code. The sign-out judge signs out, then sends the old cookie again. If the app still knows who you are, that's a fail against ASVS V3.3.1, and both requests are in the report. Others forge tokens, read the cookie the app really set, or ask every route with no cookies at all. The next step is judges written from the app's own ground truth: a record it marks private, a role table, an OpenAPI document.
The eval is built to run on your laptop, in your app's folder, with the code never leaving it. It starts your app, signs in as two test people, through your own form on a throwaway database or through Restaged's twin of Auth0, and goes at every route from the outside. Where a model helps, it uses a coding agent like Claude that's already on your machine, on your subscription. The local version isn't released yet. The demo is the same eval turned loose on public repos in a cloud sandbox, so you can read a report today.
Do existing tools catch it?
I didn't want to claim they don't, so I tried. I made two small workspace apps with flaws planted on purpose, one on Express and Passport and one on Next.js and Auth0. Then I pointed Semgrep and Claude at both. The README that lists the flaws, and every comment that hinted at them, was removed first, so Claude couldn't cheat.
- Semgrep 1.142.1 with its JavaScript, OWASP Top Ten, Node.js, JWT, Next.js and secrets rulesets. On the Express app, 20 findings, three of them planted flaws: the open redirect, the CORS reflection and one unescaped project name. Nothing on the logout route, nothing on session fixation, nothing on the API that needs no sign-in. On the Next.js app, zero findings. And on Ghost, a mature app, fourteen findings and none a real bug, where the eval failed nothing.
- Claude as reviewer. On the Express app it found the logout bug, session fixation and nine of the eleven planted flaws, with eight other items to go through. On the Next.js app it found three of the four and missed sign-out, both times I ran it.
Neither can show what the app does when it runs. The most either can say is that the code looks wrong, and the Next.js miss shows the limit of that. The sign-out code there is stock Auth0 SDK usage, and the SDK is working as documented: its default keeps the whole session in the cookie, so there is nothing on the server to end. The requirement is the app's, and a session kept whole in a cookie needs extra state before sign-out can revoke it. Twice, Claude read correct-looking code and passed it. A reviewer, human or model, can only judge the code in front of it, and this bug lives in a default it never sees. The eval doesn't read code. It signs out, sends the old cookie, and the app answers as the person. It's built for the bugs that look fine, not the ones that look wrong.
These three do different jobs. Semgrep matches patterns in source, Claude reasons about the code, and the eval observes the running app. The tables show what each saw of the same planted flaws.
| Planted in the Next.js app | Semgrep | Claude review | The eval |
|---|---|---|---|
| /api/me decodes the bearer token without verifying it | missed | found | proven |
| /api/projects gives anyone every project, private ones included | missed | found | proven |
| Sign-out leaves the session live, because the SDK keeps it whole in the cookie | missed | missed | proven |
| /admin/members opens for anyone signed in, not just admins | missed | found | not checked |
| Found, of 4 | 0 | 3 | 3 / 0 |
| Model tokens | 0 | 49k | 0 |
The eleven Express flaws one by one: Semgrep 3, Claude 9, the eval 8 proven and 3 to review
| Planted in the Express app | Semgrep | Claude review | The eval |
|---|---|---|---|
| /api/projects gives anyone every project, private ones included | missed | found | proven |
| Sign-in sends the visitor wherever the hidden next field says | found | found | proven |
| Sign-out clears the cookie and leaves the session in the store | missed | found | proven |
| Sign-in keeps whatever session the browser already had | missed | found | proven |
| A project's name is written into the page as typed | found | found | proven |
| A Stripe key committed, then removed, still in the git history | missed | missed | proven |
| The API answers any Origin with credentials allowed | found | found | proven |
| POST /api/check crashes on an empty body | missed | found | proven |
| /api/status passes on another service's answer unchanged | missed | found | to review |
| The footer links to a page nothing serves | missed | missed | to review |
| The add-a-project form has no check against cross-site posts | missed | found | to review |
| Found, of 11 | 3 | 9 | 8 / 3 |
| Model tokens | 0 | 57k | 0 |
Claude did well on these, and 57k tokens is cheap on a 300-line app. But reading code is a different job from proving the app works, and the eval does things a reviewer can't:
- It judges behaviour, not source. A reviewer can only see what's written. The eval sees what the app did: which function turned a request away, what cookie was actually set, what the API said to a forged token.
- It's repeatable. Every judge is fixed code with a written rule behind it and no model in the verdict, so it gives the same answer on every commit, for nothing. Every judge is also measured against apps planted to pass and to fail, and none gives a wrong answer on its cases. A review is an opinion that costs tokens each time and may read the same code differently next time.
- You can see its coverage. Every route it found is listed, every request it sent is in the transcript, and anything it couldn't reach is marked not checked, with the reason. A review's coverage is whatever the reviewer happened to read.
- It catches what only shows at runtime. Session fixation, a crash on an empty body, a 500 on a bad input, the flags a cookie really carries, a key still in the git history, a footer link that 404s. None of those are in the handler you'd review.
- It opens the pages in a browser. The Express run flagged pages that don't fit a phone, and it follows every link the app shows and checks each one leads somewhere. That isn't security, it's the same idea pointed at the UI.
- It doesn't need asking. Nobody has to think of sign-out. The question is already written down, in ASVS, and it's asked of every commit whether or not anyone prompted for it.
However the code is written, whichever framework or SDK it leans on, the question is the same and so is the proof.
What's next
Want to see it run on your own app? Tell me on LinkedIn and I'll send you the report. And if you have a view on what the judges should look for next, MCP servers, endpoints with no cost protection, data leakage, I'd like to hear it.
Or run it on the public repos first.