Running Gemma on an iPhone to identify dinner

July 2026 · ~10 minute read · FitDIY engineering notes

FitDIY is a fitness tracker whose App Store privacy label is Data Not Collected. That label is the product: no account, no backend, no analytics, everything on the phone. So when we wanted a MacroFactor-style "photograph your meal, get foods and portions" feature, the standard implementation was off the table before we wrote a line of code. Sending meal photos to a server is data collection. On-device or nothing.

This is the story of shipping that feature: a benchmark that said no, a takeout tray that overturned it, an internally inconsistent SDK release, a token budget nobody documents, and a GPU that lost to the CPU.

Copy the pipeline, not the hosting

The best server-side implementations of this feature share a crucial design decision: the language model never produces nutrition numbers. It identifies what's on the plate and roughly how much; calories and macros come from a real food database. We copied that split exactly:

1. Stage 1 — VLM, on-device: photo (plus an optional text hint) → structured list of candidate foods with portion estimates.
2. Stage 2 — database: map candidates to a bundled generic-food list, USDA FoodData Central, and a cached Open Food Facts lookup. Macros come from the database, never the model.

The result lands in an editable plate — swap a match, fix a portion, remove a row — and nothing is logged without confirmation. The model proposes; the database prices; the user decides.

Model choice: what actually fits in a phone

The target device was a base iPhone 15 — 6 GB of RAM, A16, and pointedly not eligible for Apple Intelligence, which excludes every non-Pro phone older than the 16. That single fact eliminated Apple's Foundation Models as the primary backend, however attractive a zero-download system model sounds.

So: Gemma 4 E2B via LiteRT-LM, Google's official Swift runtime for edge LLMs. Since Gemma 4, the license is plain Apache 2.0 — attribution text in the About screen and you're done.

The eval said no

Before touching the app we ran a spike: is a ~2B-parameter VLM good enough at looking at food? We sampled 100 plates from mmathys/food-nutrients — a cleaned Nutrition5k subset of overhead cafeteria plate photos with per-ingredient mass and macros as ground truth — and scored E2B (and its bigger sibling E4B) against labels, temperature 0, four prompt variants.

Two lessons in one table. Structured output is solved — with fence-stripping and dedupe, every single response parsed. The failure is perception: sausages identified as flatbread, components missed outright, the occasional repetition spiral (one plate came back as five rows of "pineapple"). Prompt iteration moved main-item recall from 38% to 48% and plateaued. And scale doesn't save you: E4B — which can't run on a 6 GB phone anyway — buys ~7–13 points and still lands at half the bar, on the friendliest possible photos.

Decision: park the feature. Ship nothing rather than a scanner that misses half the plate.

…and then a takeout tray changed the answer

While wrapping up, we ran one handheld photo of an actual dinner — a takeout tray of pasta, sauce, and a meatball, the components visibly separate — and E2B identified it perfectly in 13 seconds.

That exposed a confound in the benchmark: cafeteria rig shots are photographically clean but compositionally brutal — foods merged, layered, sauced together. A typical phone photo of a typical meal is the reverse. The dataset was testing the hard case and we'd read it as the average case.

So we set a second, honest gate: ten photos of the developer's real meals — restaurant plates, breakfast tacos, avocado toast, plus two adversarial cases (a bowl of raw sweet potatoes, a maple syrup bottle) — through three output contracts: components-only, dish-level ("spaghetti with meatballs, 650 g"), and a hybrid emitting a headline meal name plus components. Pass bar: at least 7 of 10 identifications a human would accept.

The hybrid contract scored 8–9 out of 10. Better yet, its failures were graceful: cavatelli read as "spaghetti," turkey read as "chicken" — adjacent-name substitutions that are one tap-to-swap from correct in the editable plate, not fabrications. (It even read the maple syrup label.) One contract quirk survived into production code: the model's total_g routinely disagrees with the sum of its own components, so the parser is explicitly forbidden from assuming consistency.

Verdict: ship it — as an explicitly experimental feature, labeled "works best on a few clearly separated foods," with the editable plate mandatory.

Getting 2.58 GB of model into an iOS app

The model obviously can't ship in the binary. It's a user-initiated, resumable background download (Wi-Fi-sized warning, progress UI) into Application Support, excluded from backup, deletable from settings. Without the model, the scan button degrades to an explainer — never a dead button. Integration was where the war stories accumulated:

• The SPM package had to be vendored. LiteRT-LM's manifest uses unsafeFlags linker settings, which SPM rejects in a remote versioned dependency. We commit only the manifest and Swift wrapper (~168 KB); the ~200 MB xcframework still downloads from the checksum-pinned release URL.
• The v0.14.0 release is internally inconsistent. The prebuilt binary's header exposes a chunk-based streaming callback; the tag's Swift sources expect a newer JSON-based one — the streaming path cannot compile as released. We stubbed it and use the non-streaming call, which matches the shipped header.
• maxNumTokens must be 4096, not 1024. Gemma 4's variable-resolution vision encoder emits ~2,394 patch tokens for a 768-pixel photo — not the ~256 you'd expect from the Gemma 3 era. The runtime specializes its graphs to the token budget, so a too-small budget fails tensor allocation (DYNAMIC_UPDATE_SLICE failed to prepare) before inference ever runs.
• Both increased-memory entitlements are required. Loading transiently needs ~4 GB of address space; without increased-memory-limit and extended-virtual-addressing, Jetsam kills the app mid-load. Once loaded, the 2.58 GB file memory-maps down to a ~600 MB–1.1 GB resident footprint.

The GPU lost

End-to-end on the iPhone 15's CPU, warm kernel cache: about 21 seconds tap-to-plate — 0.6 s engine create (mmap is a wonderful thing), ~8.7 s vision encode and prefill, ~11.9 s decode.

Surely Metal fixes that? Measured: 19.4 seconds. The GPU runs vision + prefill 3.5× faster, but decode — the bulk of the time — is memory-bandwidth-bound on an A16 and doesn't move. Meanwhile the GPU path costs 4.9 s of per-scan engine setup (vs 0.6) and a one-time ~19 s Metal pipeline build after every install. A ~10% net win wasn't worth an experimental numeric path the accuracy gate had never seen, so we shipped CPU. If scans need to get faster, the leverage is a smaller or faster model — not the backend.

Why a ±25% meal estimate is still worth shipping

The honest framing: portions from a monocular photo are estimates, and the spike measured them as roughly-right-not-precise. Three product decisions make that useful instead of misleading:

• Identification is the model's job; amounts are the user's. The editable plate prefills grams for a one-flick correction, and an optional text hint ("that's turkey, not chicken") improves results.
• Everything is labeled an estimate, and nothing logs without confirmation.
• The adaptive-targets loop absorbs systematic error. FitDIY recalibrates calorie targets from your real weight trend over time — a consistent bias in food estimates comes out in the wash, which is more than can be said for a skipped log. Adherence beats precision.

Reopening criteria

The spike harness is a permanent asset. The day a better edge VLM appears — a Gemma successor, Apple's models reaching non-Pro hardware, a Qwen-VL-class model under 3 GB — we re-run the same 100 plates against the same bars (≥80% majority-found, ≤30% median calorie error) and the whole downstream pipeline is ready. The scanner ships experimental because that's what it is; the numbers above are why we'd rather tell you that than pretend otherwise.

FitDIY is free, has no account, no ads, and a privacy label of Data Not Collected — the scanner above runs entirely on your phone. Get it on the App Store, or read why it exists.