Benchmark methodology
The accuracy figures on the Holobiont AI landing page come from a benchmark we run ourselves and publish in full. This page explains what was compared, on what data, how it was scored and what the numbers can and cannot tell you. The aggregate results, including confidence intervals, are in the repository file data/landing-benchmark.json for the run named on the landing page.
What is compared
Three arms see exactly the same inputs:
- Holobiont AI: the hosted production API, called the way a customer calls it (
POST /api/v1/food/analyzefor photos,POST /api/v1/chatfor questions), with a normal company API key. - A plain GPT 6 prompt: a general-purpose model asked directly with a short prompt and no further tooling. For photos: “List the foods in this photo with their estimated weight in grams, their calories, protein, fat and carbohydrates, and their vitamins and minerals with amounts. Reply with JSON only.” For questions: the question itself, with the system prompt “You are a helpful assistant. Answer health questions carefully.” Temperature 0. Each call records which model actually answered; a call served by a different model than requested is excluded.
- A plain Gemini 4 prompt: the same short prompts sent directly to the second comparison model. It runs both meal photos and biomarker questions. Each rival has its own paired results and confidence intervals; the comparison switcher changes both arms together.
The Holobiont API is a pipeline, not a single prompt: a preliminary image assessment selects the recognition path for visually ambiguous meals; structured recognition and selective review are followed by grounding of every recognised item in a nutrition database (nutrients per 100 g scaled to the estimated grams; a database entry is used only when its energy density agrees with the recognition step’s own estimate within a factor of 1.4 and a separate yes-no judgment confirms it is the same food as served, otherwise the item keeps that estimate), per-item confidence, and for chat, retrieval from the company’s curated biomarker library and knowledge base with citations. The benchmark measures the whole product against the plain prompt. It does not isolate individual components or name the internal serving models behind the API. The plain comparison models are named explicitly.
Latest refresh (4 November 2026). Only Holobiont AI meal-photo analysis was rerun, on the same frozen 300 photos. The GPT 6 and Gemini 4 responses and database-derived nutrient references are reused from 3 November; their cached files are verified unchanged. The 300-question biomarker answers and blinded reviews are also retained from that run. The refresh therefore measures the updated food pipeline without regenerating the comparison models or biomarker results. Experimental portion corrections are not part of the measured pipeline.
Meal photos
Dataset. Nutrition5k (Thames et al., Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food, CVPR 2021, CC BY 4.0): cafeteria plates photographed from above with a fixed camera, where every ingredient was weighed and its nutrition recorded. We use the published test split only.
Sampling. Dishes with at least 50 kcal and 20 g (near-empty incremental scans make percentage errors meaningless), sampled with a fixed seed (20261101) so the run can be reproduced; 300 dishes with available overhead photos, taken in seeded order. Missing image files are skipped before inference until the full cohort is available; failed or refused model responses are not replaced. Dishes refused by either arm (for example “no food recognised”) are excluded from the paired metrics and counted separately.
Metrics, per dish and then averaged:
| Metric | Definition |
|---|---|
| Calorie error (MAE, MAPE) | Absolute difference between the predicted and weighed calories, in kcal and as a share of the weighed value |
| Within 20 % | Share of plates whose calorie estimate is within 20 % of the weighed value |
| Protein, fat, carbohydrate error | Absolute difference in grams |
| Portion weight error per recognised item | Each recognised item is paired with its weighed ingredient (same name after normalisation, one to one); the absolute difference between the arm’s grams and the scale, averaged per plate |
| Whole-plate weight error | Absolute difference between the sum of predicted item grams and the weighed plate mass |
| Ingredients recognised (recall) | Share of weighed ingredients of 5 g or more that appear among the predicted items, matched by normalised name (plurals folded, cooking words removed, common synonyms) |
| Listed items that were on the plate (precision) | Share of predicted items that match a weighed ingredient; the rest are items that were not there |
| Backed by the nutrition database | Holobiont only: share of the estimated grams whose nutrients come from a database entry rather than a model estimate |
| Fibre, saturated fat, sodium and sugar | Holobiont only: share of plates whose totals include all four, in addition to calories, protein, fat and carbohydrates. The plain prompt was asked for the four macros only, so this is reported, not compared |
| Micronutrients reported | A reference is built from the weighed ingredients: each is looked up in the nutrition database and its vitamins and minerals per 100 g are scaled to its grams. Of the 21 compared nutrients (vitamins A, C, D, E, K, B1, B2, B3, B6, B9, B12, choline, calcium, iron, magnesium, phosphorus, potassium, zinc, copper, manganese, selenium), those at 5 % or more of the adult daily value in the reference count as present; the metric is the share of present nutrients the arm reports with a positive amount |
| Micronutrient amount error | Over the same nutrients for both arms (present in the reference and reported by both), the median absolute percentage error against the reference, averaged over plates. Scoring each arm on its own set would favour the arm that reports only the easy nutrients |
Scoring revision (3 November 2026). The measure penalising nutrients with low reference amounts has been retired. The previous calculation already ignored completely missing reference fields, but low reference amounts are still database-derived estimates, not an independent assay of the photographed meal. A richer nutrition source is therefore not penalised by this measure. Micronutrient detection and paired amount error remain unchanged. The aggregate now includes 12 food measures and 8 chat measures; previous aggregate scores are not directly comparable. Recalculating stored responses under this definition does not represent an improvement in the underlying model.
The micronutrient reference depends on the database matching the weighed ingredient names; plates whose ingredients could not be matched are left out of the micronutrient metrics, and the reference inherits the database’s own values. It is a database-derived reference, not a laboratory assay.
Where the API is ahead, and the overall score. The landing page lists the measures where the API differs significantly in its favour (the 95 % interval of the paired difference excludes zero) and the measures only the API has; the full lists under each section show every measure, including the ones that are even or where the plain prompt is ahead. The overall score scales every compared measure to 0–100 (percentages as they are, error percentages as 100 minus the error, reviewer scores ×20, open-ended errors in grams or kcal relative to the other arm so equal errors score 50 each) and averages them with equal weight. It is a summary of this benchmark, not a general quality grade.
Tuning. The pipeline’s grounding rules and portion prompt were chosen on a separate development sample of 109 Nutrition5k test-split plates (seed 20261102, disjoint from the published sample) by re-running grounding variants offline on cached recognition output. Later runs have reused the published sample for configuration comparisons and operational checks. This refreshed comparison is not a new sealed holdout.
Statistics. Every comparison is paired: the same dish in both arms. The interval shown under each bar is a 95 % bootstrap interval of the mean paired difference (10 000 resamples, seeded). When the interval includes zero we say so: “no significant difference”.
Limitations. Overhead photos from a fixed distance are not phone photos of home meals; the plates are from two US cafeterias; several photos of the same plate at different stages of serving appear as separate dishes, so the samples are not fully independent; portion weight from a single image is uncertain for any method; models are not fully deterministic even at temperature 0, which is why we publish intervals rather than point claims.
Biomarker questions
Question set. The expanded set has 300 questions: 290 synthetic lab-value questions from 34 biomarker families, plus the same ten hand-written general nutrition-lab questions. For each library entry with a reference range and guidance, candidate values are placed above the upper limit (1.2, 1.5 and 2 times that limit), below the lower limit (0.4, 0.6 and 0.8 times that limit), and at 25 %, 50 % and 75 % of the interval between the limits. Values are rounded to two significant digits; duplicate values and values that cross their intended range boundary are discarded. A fixed seed (20261101) selects 290 candidates. Four question templates are used. References remain hidden from every contestant. Multiple questions about a biomarker are repeated synthetic scenarios, not independent patients.
Repeated evaluation. The 3 November run expanded the earlier question and photo cohorts; the 4 November refresh retains those cohorts and reruns food analysis only. Some inputs overlap previous evaluations, and the chat mix contains more synthetic biomarker scenarios relative to general questions than the original smaller runs. Earlier runs have informed configuration choices, so this is a repeatable product comparison rather than a new sealed holdout.
Research citations. Since 2 November 2026 the Holobiont arm may also search the peer-reviewed literature (Europe PMC, PubMed, OpenAlex, with relevance judged before a paper is cited) and link the papers it cites; the share of answers that cite at least one linked paper is reported as an informational measure.
Scoring.
- By code: does the answer state the correct reference range (both limits within 5 %) and place the value correctly (high, low or within range)? Only for the biomarker questions. These automatic checks use nearby numeric matches and status words; they can count incidental or negated mentions, so they are not a reliable semantic verdict parser. The separate blinded review assesses answer interpretation.
- By a blinded reviewer: GPT 6 Astra receives the question, reference material and both answers labelled only “Answer 1” and “Answer 2”. Forty-eight separate agent instances grade frozen, anonymised packets of at most 25 comparisons directly; no reviewer API calls are made. The cached review originally scored four dimensions from 0 to 5: correct range interpretation, actionable and safe nutrition and lifestyle advice, citing sources, and no invented numbers. The published reviewer mean now uses the first three dimensions; the retired dimension remains only in the original audit records. Referring to a clinician is neither rewarded nor penalised. Each pair is judged in both orders by different instances, and the scores are averaged; verdicts that flip count as ties. The collector verifies packet hashes, unchanged source answers, complete judgments and restored answer orientation. The requested cohort contains 600 pairings (300 questions against each rival). Every successful pair is reviewed in both orders; failed responses or responses served by the wrong model are recorded as exclusions, not substituted with another question. The published section counts show the resulting paired support. It shares a model family with the plain GPT 6 comparator, so these are not reviews by an independent model family. Reviewer scores should not be interpreted as directly comparable with an earlier run that used a different reviewer.
Uncertainty and citation checks. Chat intervals resample whole biomarker families (10 000 seeded paired bootstrap resamples); each general question is a separate group. This accounts for several values sharing the same reference and guidance. The point estimates still average over questions. Preference and citation-rate comparisons also receive intervals, and no difference is labelled significant without an interval excluding zero. The automatic citation measure checks structured source objects or explicit source URLs identically for both arms; the blinded reviewer also assesses named sources without links. Research-publication links are informational and do not independently verify a claim or peer-review status. These reporting corrections and the changed question mix mean differences from the earlier 50-question aggregate are not solely model improvements.
Circularity, stated plainly. The Holobiont chat is grounded in the same curated library the reference answers are composed from. This part of the benchmark therefore measures consistency with curated clinical reference data and reviewed answer quality, not independent medical accuracy.
What we do not claim
We do not claim diagnostic accuracy, and the API’s answers are general wellness information, not medical advice. Single-photo calorie and portion estimates can be substantially wrong. A higher aggregate score does not mean a configuration wins every measure; the tables show individual results, paired intervals and excluded requests. The food micronutrient reference is database-derived, and the chat benchmark measures consistency with the curated reference library. Neither is an independent clinical validation.
Chat scoring revision (3 November 2026). “Review: no invented numbers” is no longer included as a measure or in the averaged reviewer score. The reported review mean uses range interpretation, actionable and safe advice, and source citation scores. All values are recalculated from the existing judgments; no contestant answers or reviewer grades are regenerated. Pairwise preferences remain the original judgments made under the four-dimension rubric. The combined aggregate now contains 20 measures (12 food, 8 chat). This is a scoring-definition change, not a model improvement.