Prove it by running it
When GenAI writes the code and the tests, who confirms it works? And who checks the things that never made it into the prompt, because everyone assumes they come as standard?
TL;DR: an eval for the app the agent built, not for the model. It boots the app in a sandbox, signs in, and tries to break it from the outside. Every verdict names the rule it broke and the request that proved it. Try it on four public repos: two small workspace apps with flaws planted on purpose, OWASP NodeGoat and dvna.
Sign-out is the one that just caught me out. The prompt says add logout functionality. The agent's implementation clears the cookie, and its test checks the cookie is gone. Both pass. But the session is still live on the server. Anyone who captured that cookie, from the network, a log or a browser extension, can keep using the account long after the person signed out. Nobody prompted for the session to be destroyed, because nobody thinks to. An app has thousands of cases like this.
Run it and see
As an industry we already have the ground truth for what an app should do. OWASP ASVS says sign-out must invalidate the session. RFC 8725 says reject a token signed with alg none. Auth0's docs say check the audience. Almost none of it is ever executed against a running app.
So I turned those lines into judges. A judge is one line of that ground truth made executable: a small question with the rule behind it, answered by using the running app rather than reading the code. The sign-out judge signs out, then sends the old cookie again. If the app still knows who you are, that's a fail against ASVS V3.3.1, and both requests are in the report. The token judges hand the API a token with the algorithm set to none, one with the subject swapped to the other test person, and one minted for a different app. There are twenty-odd judges so far, mostly around sign-in, sessions and tokens, and more than this page has room for.
This isn't a service you upload your code to. It runs on your laptop, in your app's folder, and the code never leaves it. It starts your app, signs in as two test people, and goes at every route from the outside, the way an attacker would. Sign-in through your own form goes to a throwaway database. Sign-in through Auth0 goes to Restaged's twin of Auth0, so your real tenant is never touched. Where a model helps, it uses the Claude Code already on your machine. Your subscription, your workflow, nothing new to sign up for. Every verdict is one of four: to fix, to review, proven, or not checked, and not checked is never quietly a pass. The demo is the same thing turned loose on public repos in a throwaway cloud sandbox, so you can read a report before you install anything.
Do existing tools catch it?
I didn't want to claim they don't, so I tried. I made two small workspace apps with flaws planted on purpose, one on Express and Passport and one on Next.js and Auth0. Then I pointed Semgrep and Claude at both. The README that lists the flaws was removed first, so Claude couldn't cheat.
- Semgrep 1.142.1 with its JavaScript, OWASP Top Ten, Node.js, JWT, Next.js and secrets rulesets. On the Express app, 20 findings, three of them planted flaws: the open redirect, the CORS reflection and one unescaped project name. Nothing on the logout route, nothing on session fixation, nothing on the API that needs no sign-in. On the Next.js app, zero findings. It didn't see the token being decoded without being verified. The other way round is as bad: on Ghost, a mature app, Semgrep raised fourteen findings and none was a real bug.
- Claude as reviewer. On the Express app it found the logout bug and session fixation, with 15 other items to go through. Fair warning: a comment at the top of that file says flaws are planted, so it went looking. On the Next.js app it found three of the four planted flaws and missed sign-out. There the session lives whole in the cookie, which is the Auth0 SDK's default, and there is nothing in the code to see.
- Neither can prove anything. The most either can say is that it looks wrong.
Last time I had an AI code review and an agentic tester both go green over a data leak. Same lesson. When the code and the tests come from a model, the check has to run the app, not read it.
What I'd like from you
Open one of the runs, then tell me:
- Which of these would you have missed in review?
- What ground truth does your app have that nothing executes?
- What should the next judge be? MCP servers, APIs with only an OpenAPI document, routes that spend money?