Auditing an AI-built app: what we check first

Who can read whose data comes first. Then what the browser has been trusted with, and whether the server checks anything it is sent. After those three: backups somebody has restored, tests on the paths that move money, errors that reach a person, and a cap on what the AI can cost. We work in that order because it is the order in which things start to hurt after launch.

An AI-built app audit is a senior engineer reading an app made with an AI builder, its code and its configuration, to find what fails once real users and money arrive. The platforms expect somebody to do it. Bolt.new's terms, updated in September 2026, say the user is "solely responsible" for checking what the AI produced, vulnerabilities and licence compliance included (StackBlitz terms of service, 2026). The licence side is an inventory of its own; the checks below deal with everything else.

The eight checks, in order

The code itself usually runs. Veracode's 2026 tests found AI-generated code syntactically correct nearly every time, yet passing its security checks only 56% of the time on average across more than 100 models (Veracode, 2026). Those are short benchmark tasks scored by the vendor's own scanner, not production apps, but the gap they show is the one these checks look for. Each needs a second test account and a terminal, and the last column gives the shape of the fix, since size depends on the app.

Can other users read your data?

Row-level security is the database checking, row by row, whether the person asking may see it. In a Supabase back end, the kind many Lovable and Bolt.new apps use, it is the main barrier between the public key in the browser and every row in every table.

In March 2025 a researcher checked 1,645 apps Lovable featured publicly and found 170, about one in ten, with row-level security missing or wrong, exposing names, email addresses, financial details and API keys (Palmer, 2025). It became CVE-2025-48757. Lovable disputes it, saying each customer is responsible for its own application's data, and the researcher worked for a competitor. Lovable's later pre-publish scan, Palmer says, checks that policies exist rather than that they are correct.

That distinction is the check. Request a second user's rows from the API with the first user's token, not through the screens. Then read every policy, looking for ones that admit anyone signed in without asking which user, and for tables with no policy at all. File storage has access rules of its own, so check the buckets holding customers' uploads as well.

Is there a key in the browser?

Supabase's public key is designed to sit in browser code, and it is safe only while row-level security is on and correct. Its service key bypasses those policies entirely, and payment, model provider and email keys are just as dangerous in client code. Any of them in the browser is the first thing we report.

Moltbook, a viral social network for AI agents, is the plainest public case. Its founder said he "didn't write one line of code" for it, and a Supabase key in its client-side JavaScript, with no row-level security behind it, left the production database open to reads and writes, exposing 1.5 million API tokens and 35,000 email addresses before it was fixed (Wiz, 2026). Escape, a security vendor, found more than 400 exposed secrets across over 5,600 public apps built on vibe-coding platforms, a sample it says Lovable apps dominate (Escape, 2025).

Search the scripts the deployed site serves for anything shaped like a key, then the repository's history, because a key deleted last month is still in the commits and needs rotating.

Permissions that exist only on screen

AI builders are good at hiding buttons. A user without the admin role never sees the delete control. Whether the server refuses the delete is another matter, and the screen cannot tell you. Take an ordinary user's token and call what their screens hide: the admin endpoints, another account's settings, the export meant for staff. Anything that succeeds is a finding.

We built a field operations platform in a regulated compliance industry the other way round. Every path the browser may call sits on an explicit list at the gateway, and anything else is refused before the API sees it. The people who administer the platform itself sign in against a separate identity tenant, so their tokens fail the customer API's signature check before any route or role is consulted.

An AI-built app needs less machinery than that: the rules written down per role, enforced in database policies or server functions, and one test per rule that fails if the rule is removed.

Validation the server never does

A feature can work perfectly and still be unsafe. In SusVibes, a benchmark of 186 real feature requests where people had previously written vulnerable code, the strongest agent configuration got the feature working in 57% of tasks, and working without a security flaw in 11.8% (Zhao et al., revised September 2026). The tasks were picked for being easy to get wrong, so read it as the rate on risky changes. Code that compiles and misses the point is one of six quiet ways AI coding agents fail, and nothing in the build reports it.

A form that refuses bad input proves nothing about the server, because anyone with a terminal skips the form. Send a negative quantity, a price with twelve decimal places, text where a number belongs and a payload ten times the usual size, then look at what was stored.

A model's reply is input too. On a coaching platform we build, the parser for model output rejects anything malformed rather than guessing, with one written exception: calories are derived from the macronutrients instead of checked against them, because the models we tested reproduce food-label figures that drift 5 to 14% from the arithmetic, and retrying never converges.

Backups nobody has restored

PocketOS, which sells software to car-rental firms, lost its production database to a coding agent in April 2026, and the newest backup its founder could restore was three months old. Railway, its host, later said it had recovered all of the data, and has since made deletions through its API reversible for 48 hours (Railway, 2026).

Restore last night's backup into a scratch database, time it and compare row counts with production. Confirm that development and production are separate databases with separate credentials, and that whatever can drop the database cannot also delete its backups. A dashboard that says backups are enabled proves nothing until one of them has come back. Write the restore time down. An enterprise customer's security questionnaire will ask for it, and it sits on the security list of the 90-day pre-raise engineering plan for anyone about to raise money.

Tests on the paths that move money

Payment code fails in the awkward cases, and generated tests tend to stop at the happy path. Look first for a test of a refund after the money has already been paid out, and one for the same webhook arriving twice. Then a card failing at renewal, if anything renews. If none of those exist, the money path is untested whatever the coverage figure says. What happens when a refund lands after the payout is its own piece of work, and the one people forget to scope. We build multi-tenant platforms where payments are tested before anything else, and our AI-built app audit follows the same list in the same order.

On a field operations platform we build, payment events arrive at an endpoint that reads one field from the body, the event id, then queues it and returns. A worker then fetches the full event from the payment provider over an authenticated call, so nothing in the request is trusted beyond an identifier the provider has to recognise. An invented request fetches nothing, and a repeated one fetches the same event again.

Errors nobody hears about, and the AI bill

Break something in a staging copy and see who finds out. If the answer is a customer, next week, the app has no error reporting. On a client marketplace we build, browser errors, server exceptions and failed responses share one trace id per request, and they group by a deliberately lossy fingerprint: any part of an address that is an id or all digits is replaced first, so a hundred variants of one broken page arrive as one issue. Capture runs inside its own try and catch, because an error tracker that can fail a customer's request has made things worse. A hosted tracker does most of this, and the rule is the part to copy.

Model costs follow the shape of a feature more than its number of users. Anthropic's own figures put its agents at roughly four times the tokens of a chat exchange, and its multi-agent research system at about fifteen (Anthropic, 2025). On a coaching platform we build, the meter on an expensive AI feature is a list of timestamped uses held on the account. Limits are judged over a rolling window when each request arrives, which leaves no reset job to fail. A use is struck off whenever the platform itself fails the generation, so a customer never pays for a failure that was the platform's. Check for a per-user limit in code and a budget alert on the provider account, and ask how the feature gets switched off on a Saturday without a deploy.

Reading the results

Findings in the first three rows mean stop. Close them before another customer signs up, because each one lets a person see somebody else's data or act as someone they are not. Everything below them can be scheduled: validation and restores this month, the rest this quarter, unless the AI bill is already surprising you.

Most findings get fixed in place, in the code you already have. Whether to keep the code at all is a decision for after the audit, with the findings in hand, and the hunch you had before it is worth less than it feels. Investors will run their own version of this list, and the red flags that stall a funding round overlap with it more than founders expect. If the audit points at the process that produced the code, the questions that separate a careful supplier from a vibe-coding one apply to whoever builds the next version.

Questions founders ask about audits

How long does an audit of an AI-built app take?

About a week for a typical app built with Lovable, Bolt.new or Replit, which is how long our AI-built app audit runs. The first checks go quickly. Restoring a backup, reading the payment code and writing up each finding with the shape of its fix take the rest. Several user types or payment flows take longer.

Do we still need a penetration test?

Yes, later. An audit reads the code and configuration to find what is structurally missing. A penetration test attacks the running system to find what is exploitable. Running the audit first means the testers spend their days on what remains, rather than rediscovering access rules nobody wrote.

Can we run these checks ourselves?

Most of them. The tests in the table need a second account, a terminal and a day of patience. Experience goes into judging what a finding means: whether a policy is wrong or merely loose, and which gap somebody outside would reach first. When you pay for an audit rather than working through the table yourself, that judgement is what you are buying.