Skip to content

AI engineering · 9 min read

Counting the oil a photo can’t see: hidden fats in an AI calorie tracker

How we built Nomvy’s AI pipeline to list cooking oil, ghee and dressings as separate lines, and to avoid counting the same fat twice.

Published · By Kumar, QuantumPlug Technologies LLP

Curved violet ribbon of light

A photo of dal and rice shows the dal and the rice. It doesn’t show the ghee in the tadka or the sugar in the chai beside the plate. In home-cooked South Asian food that fat is often a large part of the meal’s calories, and a camera can’t see it. In Nomvy, our AI calorie tracker for Android and iPhone, we decided not to hide that uncertainty inside one confident number. Hidden fats are listed as their own lines, which the user can see, switch off and correct. This article explains how the pipeline does it: the output schema, the server-side checks, the grounding step and the reconciliation that stops the same oil being counted twice.

Why we made fat a separate line

The simplest design is to ask a model for one calorie figure per dish, with the cooking fat folded in. It is also the hardest to correct. If a karahi comes back at 520 kcal, the user can’t tell whether the model assumed one spoon of oil or four, so they can’t fix the part most likely to be wrong.

We chose the opposite. Every Nomvy estimate has two kinds of row: the foods you can see, and the hidden contributors you can’t, such as cooking oil, ghee, butter, dressings, and sugar or milk in drinks. A hidden row carries its assumption in plain words, such as “1 tbsp oil used in cooking”, so the person reviewing the meal knows what to challenge. Each hidden row has its own switch and amount stepper, and the total and range move as they change it.

That decision shaped everything after it: the schema, the server-side checks and the grounding step, which has to know when a database value already includes the fat the model listed separately.

A schema that keeps visible and hidden apart

Every model call in Nomvy returns structured output defined once as a zod schema in schemas.js. The scan schema, ScanOutput, has an items array for visible dishes and a separate hidden_calories array. Each hidden row has a name, an assumed sentence, grams, calories, macros and a confidence. All properties are required, and optional ones are nullable, because that is what structured outputs expect.

The same schema does two jobs. It is converted to JSON Schema and sent with the request, so the model is held to that shape. Then the response is checked against the same zod schema again on our server. If the JSON is invalid, fails the schema or was cut off at the token limit, the provider layer retries once before moving on. We treat model output as untrusted input, even when it is well formed.

The system prompt is a fixed string with no dates, ids or user data. Anything about the person, such as locale, cuisines, diet, frequent foods and saved “My Dishes”, goes into the user turn in a short context block. That keeps the start of every request identical, so the provider can cache it, and the prompt tells the model to treat that context as hints, never as facts that override what is on the plate.

The prompt does the domain work. The scan system prompt tells the model to list invisible contributors explicitly and to include only fat that is not already counted in an item’s macros. It carries a portion reference, covering things like a 26 cm dinner plate, a 150 ml katori, a 30–40 g roti and 13.5 g of oil in a tablespoon, plus a table of cuisine-aware defaults for hidden fat. The guidance in it is along these lines:

  • South Asian curries, sabzi and dal tadka usually carry oil or ghee per serving, and restaurant versions carry more
  • For deep-fried food such as pakoras, samosas and puris, the absorbed oil belongs in the dish itself, and extra ghee is added only if it is visible
  • A roti or naan with a glossy surface gets a little ghee or butter
  • Salads get dressing unless they are clearly dry, and chai or coffee gets sugar unless the user says otherwise

Turning model rows into editable food items

Model rows are not saved as they arrive. items.js converts each one into the same FoodItem shape the app uses for database foods, barcodes and manual entries, so a hidden oil line can be edited with the same controls as a plate of rice.

For hidden rows, fromHiddenItem clamps grams to between 0 and 500, marks the item isHidden and reads the assumed sentence for a household unit. “1 tbsp oil used in cooking” becomes a quantity of 1 in tablespoons, and “½ tsp” becomes 0.5 teaspoons, so the user sees a unit they recognise instead of 13.5 g. If no unit is found, it falls back to grams.

Every set of macros, visible or hidden, goes through sanitizeNutrients. Values are clamped to plausible ranges, then checked for Atwater consistency: if the stated calories differ from 4 × protein + 4 × carbs + 9 × fat by more than 30%, we trust the macros and recompute the calories. Models make arithmetic slips, and fat, at 9 kcal a gram, is where a small slip becomes a large error.

Grounding, and the double-counting problem

A model’s per-gram values for a dish are guesses, so where we have better data we use it. groundItems in grounding.js matches each item name against the user’s own custom foods, then our curated Regional Pack, then the generic datasets in USDA FoodData Central, each with its own similarity threshold (0.85, 0.80 and 0.82). On a match, the database’s per-100 g values replace the model’s, but the model’s grams are kept. The model is good at seeing how much is on the plate; the database is better at what a gram of that dish contains.

Similarity combines word-level overlap with character-trigram matching, after mapping common spellings to one form: chapati and phulka to roti, karhai and kadai to karahi, daal to dal. A density check rejects any match whose calories per 100 g are below 0.45× or above 2.2× the model’s own figure, which catches foods that share words but not recipes. Grounding is best-effort, so a slow USDA lookup simply leaves the model’s values in place.

Grounding creates a new problem. The Regional Pack holds typical home-preparation values for 315 dishes and 19 basic ingredients, and a home value for a curry already includes the oil it was cooked in. If the model estimated a curry lean and listed a tablespoon of oil separately, grounding raises the curry’s fat while the oil line still adds its own. The same fat is counted twice.

reconcileHiddenFat in the pipeline deals with this. Before grounding, we record the model’s fat for every item. Afterwards, we add up the extra fat grounding introduced across visible items. If that comes to 2 g or more, we reduce the hidden lines whose names look like cooking fat (oil, ghee, butter, lard, makhan and similar) by that amount, converting grams of fat back into grams of the ingredient. A line left with less than 2 g is removed, and the result gains an assumption the user can read: that the oil is already included in the dish estimate. Sauces and dressings are left alone, because a curry’s database value doesn’t include the raita.

Order matters. Smart Memory, which adjusts portions from a user’s past corrections, runs last and never touches hidden items: a learned “you eat more rice than the model thinks” should scale the rice, not the oil. The full sequence is:

  • Record each item’s fat as the model estimated it
  • Ground visible items in data, keeping the model’s grams
  • Add up the fat grounding added and subtract it from generic hidden-fat lines
  • Remove fat lines that fall below 2 g and say so in the assumptions
  • Apply Smart Memory portion adjustments to visible items only

An honest range and one question

A hidden-fat figure is still an estimate, so Nomvy shows a range beside the total. The model returns kcal_low and kcal_high for the whole meal, but grounding and reconciliation change the total afterwards. rangeFor therefore scales the model’s range by the ratio of the final total to the model’s original total, then widens it where needed so it always brackets the figure shown by at least 5% either side, rounded to the nearest 5 kcal.

When the cooking fat is uncertain enough to change the estimate materially, the model may ask one clarifying question. For cooking fat it has to use the id cooking_fat and four fixed options: no oil, a little oil, restaurant-style, and ghee- or butter-rich. The server trims any question to at most four short options and drops it if fewer than two remain.

The answer is applied on the phone, with no model call and nothing taken from the AI quota. applyCookingFatAnswer in the app’s nutrition.ts maps each option to grams of fat for a standard plate (none, a teaspoon, about 1.5 or about 2.5 tablespoons), scales it by the meal’s visible weight against a 300 g plate, clamped between half and double, and updates or adds the hidden fat line. Once a Pro user has given the same answer three or more times for a cuisine, the server pre-selects it next time, and it can still be changed.

Corrections that touch only the fat

Hidden lines are where corrections are most common, so they get the cheapest path. In “Fix it with a sentence”, a few exact phrases are handled by rules with no model call: “no oil” and “no ghee” remove the hidden fat lines, “no sauce” removes sauce and dressing lines, and “add oil” adds a tablespoon of cooking oil from the Regional Pack. Only exact phrases match, so free text that needs the model never takes the rules path by accident.

Anything else goes to the model, which returns operations rather than a rewritten meal: scale, set, remove, add and replace, each keyed by an existing item’s id. New items carry an is_hidden flag, so “with ghee” becomes a hidden line rather than a dish. The server applies the operations and ignores ids it doesn’t recognise, so items the user didn’t mention can’t change. The app shows the result as a “620 → 540 kcal” banner with Undo.

Keeping it fast and affordable

None of this works if a log is slow or expensive, because the free plan includes three AI logs a day. The phone does what it can first. It resizes photos to a 1,024 px long edge and re-encodes them, which drops location data, and it transcribes voice with its own speech recognition, so the server receives a small image or plain text, never audio. The server strips image metadata again before anything reaches a model.

Each scan starts on a small, fast model, Claude Haiku by default, and escalates to a larger one only when overall confidence is below 0.6 or a photo has four or more items. Escalation is the main cost driver, and a South Asian thali often has four items, so that threshold is tunable. Identical requests can be answered from a response cache, and the fixed system prompt is cached at the provider. If the main provider times out or is overloaded, the request moves along a chain of fallback providers that get the same prompt and schema, so the pipeline sees the same shape of data whichever model answered.

Every call is recorded with its model, tokens, latency and cost, and those records are deleted after 30 days. A photo that turns out not to be food gives the AI log back.

How we check it

Jest suites run the pipeline against mocked providers. A photo scan of chicken karahi with naan, for example, must come back grounded to the Regional Pack with the model’s grams kept, carry a cooking-oil line in tablespoons, and total exactly the sum of its items inside its own range. The repo also has an evaluation harness: npm run eval:scan runs the real pipeline on reference meals and reports calorie error by cuisine, and npm run eval:refine checks correction sentences, including Urdu, Hindi and Arabic phrasing, for operation accuracy and for changes to items the sentence didn’t mention. A prompt or model change is meant to re-run both and bump the prompt version stored on every scan.

We don’t publish an accuracy percentage, and the API never returns one. Confidence decides escalation and which of three levels the review screen shows, never a score.

Hidden fat is what a camera sees worst and what matters most in home cooking. We didn’t handle it by asking the model to be cleverer. We gave fat its own place in the schema, treated model output as untrusted, grounded visible dishes in data, subtracted the fat grounding adds, and made the remaining assumption easy to see and change. The pattern travels well beyond calorie counting: let the model estimate, let deterministic code decide, and show people the assumption they are most likely to want to fix.

Common questions

Why doesn’t Nomvy fold cooking oil into the dish’s calories?

Because it is the part of the estimate most likely to be wrong. As a separate line with its assumption written out, the user can see it, switch it off or change the amount without touching the rest of the meal.

How does Nomvy avoid counting the same oil twice?

When a dish is matched to a database entry whose values already include cooking fat, the pipeline measures the extra fat that match added and subtracts it from the generic oil, ghee or butter lines. A line that falls below 2 g is removed and noted in the assumptions.

Does answering the cooking question use an AI log?

No. The answer is applied on the phone from a fixed lookup table, scaled to the size of the meal, with no model call.

All insights