Adding AI to a product that already has users

Most AI work is not greenfield. There is a system in production, people depending on it, a schema that has accumulated meaning over years, and someone has asked whether a model could do a job the software currently cannot. The interesting engineering is almost never the prompt. It is everything around the call that keeps the rest of the product as reliable as it was the week before. We recently added two model-backed features to a live Express and MongoDB application using the OpenAI SDK: extracting a printed menu from photographs, and grading free-text answers. Almost none of the work was the API call.

Treat the model as a third-party integration, not a new architecture

Give the model call an ordinary function signature and hide everything else behind it. One module exposes a function that takes images and returns normalised menu sections. Another takes a question and an answer and returns a score with feedback. The controllers calling them look like every other controller in the codebase. There is no AI layer, no orchestration framework, no new service. The vision call sits inside a normal Express handler, between a multer upload and a Mongoose write.

That module also needs to know how to be absent. The client is constructed lazily on first use, and a missing API key throws a typed error rather than a generic one, so the route answers with a 503 and a sentence a human understands instead of a 500 and a stack trace. A separate predicate lets the route check configuration before it accepts an upload at all. Staging environments, local development and a lapsed billing account are the same condition, and the feature should switch itself off rather than take a page of the product down with it.

Nothing a model produces should reach a production table unreviewed

The scanning feature never writes to the live menu. Extraction produces a staged record in its own collection with a status of extracting, ready, committed or failed. The owner reviews and edits that payload, and only an explicit commit endpoint creates real sections and items. A misread price is a thirty-second correction on the review screen; the same misread price written straight through is a pricing dispute at a table. Staged records carry an expiry date and a TTL index, so the background monitor clears them roughly thirty days later. They are working data, not a business record.

Constrain the response with a strict JSON schema rather than parsing prose. Two different features get confused here: JSON mode guarantees only that the output parses, while a strict json_schema guarantees the shape as well. Strict mode has a real cost — every property must appear in required and additionalProperties must be false throughout — so an optional field has to be expressed either as a nullable union or as a sentinel value. We chose sentinels: an unreadable price comes back as 0, a missing description as an empty string, and the normaliser turns those back into meaning on our side rather than spreading nulls through the rest of the code.

A schema constrains shape, not truthfulness. Everything is re-validated before storage: prices coerced to finite numbers, negatives clamped, absurd values discarded, nameless items dropped, strings truncated to what the schema and the UI can hold, and enum values filtered against the real list rather than trusted because the schema said so. The extractor also asks the model to self-report confidence as high or low, and we force low whenever the price failed to parse, so the review screen can draw the eye to the rows most likely to be wrong.

Do not let a model infer data that carries regulatory weight. Our prompt and schema record allergens and dietary tags only where the menu prints them, and the instruction never to infer them from ingredients or dish names appears in the system prompt, in the field descriptions and again in the validator. A model deciding a dish contains milk because it is called a carbonara is not a feature, it is exposure. If the source does not declare something that is regulated, the system should return an empty array and say so on the screen.

Store the exact input the model saw — in our case the normalised images, not the originals — because the first time someone disputes what was read, you will want it. And make the commit path idempotent, because re-scanning after a bad read is the first thing a user will try. Ours skips items already on the menu by name. The trap there is filtering a whole section in one pass: every item is compared against the set as it stood before the section, so a dish printed twice on one page passes twice and is created twice. Claim each name as you accept it.

Latency and failure are product decisions, not infrastructure ones

Set timeouts deliberately and per feature, and know how the SDK applies them. The OpenAI client's timeout is per attempt, not per call, so the worst case is the timeout multiplied by retries plus one. Multi-page vision gets two minutes and a single retry, because it genuinely takes that long: four minutes held open at worst. Grading gets a minute and two retries, so three. Being generous with both numbers at once is how you end up with a request that occupies a connection for a quarter of an hour and then fails anyway.

We run both calls inline in the HTTP request rather than pushing them onto the existing job runner. That is defensible when the user is watching a spinner having just taken a photo, and wrong if the work can wait. The honest cost is that a dropped connection loses the result, and you are trusting every proxy between the browser and the process not to cut a long request. If you take that route, write the attempt to the database before you make the call, so a failure leaves a row explaining itself rather than nothing at all.

Then decide, per feature, what failure means. Grading degrades: if the call fails, it returns a neutral score and tells the user the answer was recorded and will be reviewed manually. Nobody loses a submission to a vendor outage. The trade-off is that a silent default becomes a data quality problem, so those rows have to be visible to someone rather than indistinguishable from a real mid-range score. Extraction cannot degrade, because there is no sensible default menu, so it fails loudly with a message that says what to change: a clearer, flatter photo.

One caution on batching. Our grader fans out across a question set with a plain Promise.all, which is fine when the set is small and bounded by the assessment. Promise.all rejects on the first failure, so unless each call handles its own errors you lose the whole batch to one bad response — and pointed at a user-supplied list, the same pattern finds your rate limit in production, in front of a customer.

Cost control is mostly work you do before the request

By the time you are choosing a model, most of the money is already spent or saved. The image pipeline does the heavy lifting: every uploaded page is decoded from HEIC when the magic bytes say it is one, rotated according to its EXIF orientation, resized so the long edge is at most 2000 pixels, and re-encoded as JPEG at quality 82 before being base64-encoded into the request. The vision API rescales anything larger before it tokenises it, so pixels above that ceiling cost upload time and buy no accuracy. The same step is what makes the feature usable on a phone at all.

Around that sit a few unglamorous limits, all of them dull and all of them load-bearing:

• Six pages per scan and three hundred items per import, enforced in the route, the controller and the extractor rather than in one hopeful place.
• A rate limiter scoped to the scanning route — twenty an hour — separate from the app-wide one, because this endpoint spends money and the rest of the API does not.
• An upload size cap on the multipart handler, so an oversized file is rejected before any decoding work happens.
• Authorisation narrowed to the account owner, for the same reason as the rate limit.
• The model name in an environment variable with a cheap default and a per-feature override, so changing it is a deploy rather than a refactor.

Evaluate with fixtures you control

You do not need an evaluation platform to know whether the feature works. We render a synthetic menu as SVG containing exactly the things that trip extractors: dietary markings, a right-aligned price column with two-decimal prices, descriptions in smaller grey type, and footer noise — opening hours, a wifi password, a social handle — that must not become dishes. It renders to a JPEG, goes through the real extraction path, and the script asserts properties rather than exact strings: both sections found, all four dishes found, prices exact, tags present only where marked, footer ignored.

The same script has an offline half that exercises the normaliser with hand-built payloads, including deliberately hostile ones: negative prices, invented dietary tags, allergens outside the statutory list, nameless items. That half needs no API key and no network, which means the majority of the safety logic is testable in CI for free, on every commit. Split evaluation this way and the part that costs money and time stays small enough that you actually run it.

One lesson from writing that script. We originally built the HEIC test fixture with the same image library the production path uses. It produced an AV1-encoded file, which that library decodes happily, so the test passed while real iPhone photos — encoded with HEVC, which most builds of libheif omit for patent reasons — failed at the first byte. We now generate the fixture with a different tool entirely. Build fixtures with something other than the code under test, or you are testing your own assumptions back at yourself.

Why the integration work matters more than the model

Model choice is the part of this project that is genuinely easy to change; ours is an environment variable with a per-feature override. What does not change with a variable is the review queue, the validator, the rate limiter, the degradation path, the fixtures and the decision about which fields the model is not permitted to guess. Those took the time, and those determine whether the feature is trustworthy on a bad day.

A useful test before shipping: if your provider doubled its latency tomorrow, or went down for an afternoon, what would your users see? If you can answer in one sentence, the integration is finished. If you cannot, the prompt is not the problem.

Lucid Code Labs works on software that is already live, including integrations like this one.