[{"data":1,"prerenderedAt":537},["ShallowReactive",2],{"seite-\u002Fen\u002Fopen-source\u002Frechnung-fotografieren\u002F":3},{"id":4,"title":5,"body":6,"description":528,"extension":529,"meta":530,"navigation":531,"path":532,"robots":533,"seo":534,"stem":535,"__hash__":536},"seiten_en\u002Fopen-source\u002Frechnung-fotografieren.md","Photograph the invoice, check the data",{"type":7,"value":8,"toc":517},"minimark",[9,13,24,27,32,35,43,193,208,218,221,225,228,244,247,251,254,261,269,276,290,296,302,306,309,324,331,335,338,341,344,348,351,354,357,368,379,427,430,434,437,448,452,455,458,462,465,488,513],[10,11,5],"h1",{"id":12},"photograph-the-invoice-check-the-data",[14,15,16,23],"p",{},[17,18,22],"a",{"href":19,"rel":20},"https:\u002F\u002Fwartungsheft.ch",[21],"nofollow","Wartungsheft"," is a web app for a car's service book,\nbuilt with Vue 3 as an installable PWA. The core is a photo: you photograph the\ngarage invoice, and the app reads out the garage, date, odometer reading,\namount, and line items. The line items become maintenance entries, and from the\nmaintenance entries the app calculates what is due next.",[14,25,26],{},"That sounds like one API call. In practice it is a chain in which the language\nmodel is only one link. This article describes which links were added and why.",[28,29,31],"h2",{"id":30},"two-stages-instead-of-one","Two stages instead of one",[14,33,34],{},"With Document Annotation, Mistral offers a single-stage route: image in,\nstructured JSON out. I built that in first and tested it with nine real invoice\nimages. One out of nine came back correct. For the others, the model invented\namounts that appeared on no invoice.",[14,36,37,38,42],{},"The route that stayed has two stages. First, ",[39,40,41],"code",{},"mistral-ocr-latest"," reads the\nimage and returns Markdown, tables included. Then a chat model gets only this\ntext, not the image, and uses it to fill a schema. With the same nine images,\nall nine were correct.",[44,45,50],"pre",{"className":46,"code":47,"language":48,"meta":49,"style":49},"language-ts shiki shiki-themes github-light github-dark","const { object } = await withRetry(() => generateObject({\n  model,\n  maxRetries: 0,\n  temperature: 0,\n  schema,\n  messages: [{\n    role: 'user',\n    content: `${prompt}\\n\\n--- OCR-TEXT DES DOKUMENTS ---\\n${ocrText}`,\n  }],\n}))\n","ts","",[39,51,52,94,100,112,122,128,134,146,181,187],{"__ignoreMap":49},[53,54,57,61,65,69,72,75,78,82,85,88,91],"span",{"class":55,"line":56},"line",1,[53,58,60],{"class":59},"szBVR","const",[53,62,64],{"class":63},"sVt8B"," { ",[53,66,68],{"class":67},"sj4cs","object",[53,70,71],{"class":63}," } ",[53,73,74],{"class":59},"=",[53,76,77],{"class":59}," await",[53,79,81],{"class":80},"sScJk"," withRetry",[53,83,84],{"class":63},"(() ",[53,86,87],{"class":59},"=>",[53,89,90],{"class":80}," generateObject",[53,92,93],{"class":63},"({\n",[53,95,97],{"class":55,"line":96},2,[53,98,99],{"class":63},"  model,\n",[53,101,103,106,109],{"class":55,"line":102},3,[53,104,105],{"class":63},"  maxRetries: ",[53,107,108],{"class":67},"0",[53,110,111],{"class":63},",\n",[53,113,115,118,120],{"class":55,"line":114},4,[53,116,117],{"class":63},"  temperature: ",[53,119,108],{"class":67},[53,121,111],{"class":63},[53,123,125],{"class":55,"line":124},5,[53,126,127],{"class":63},"  schema,\n",[53,129,131],{"class":55,"line":130},6,[53,132,133],{"class":63},"  messages: [{\n",[53,135,137,140,144],{"class":55,"line":136},7,[53,138,139],{"class":63},"    role: ",[53,141,143],{"class":142},"sZZnC","'user'",[53,145,111],{"class":63},[53,147,149,152,155,158,161,164,167,170,173,176,179],{"class":55,"line":148},8,[53,150,151],{"class":63},"    content: ",[53,153,154],{"class":142},"`${",[53,156,157],{"class":63},"prompt",[53,159,160],{"class":142},"}",[53,162,163],{"class":67},"\\n\\n",[53,165,166],{"class":142},"--- OCR-TEXT DES DOKUMENTS ---",[53,168,169],{"class":67},"\\n",[53,171,172],{"class":142},"${",[53,174,175],{"class":63},"ocrText",[53,177,178],{"class":142},"}`",[53,180,111],{"class":63},[53,182,184],{"class":55,"line":183},9,[53,185,186],{"class":63},"  }],\n",[53,188,190],{"class":55,"line":189},10,[53,191,192],{"class":63},"}))\n",[14,194,195,196,199,200,203,204,207],{},"The schema is a Zod object via the Vercel AI SDK. The category of a line item is\na ",[39,197,198],{},"z.enum"," with the app's maintenance types. So the model cannot invent\n\"Motorservice\" where the app knows ",[39,201,202],{},"inspektion",". ",[39,205,206],{},"temperature: 0"," does not make\nthe answers correct, but repeatable, and that is the only way to observe an\nerror twice.",[14,209,210,213,214,217],{},[39,211,212],{},"maxRetries: 0"," is there because otherwise the SDK retries twice on its own, on\ntop of the app's own ",[39,215,216],{},"withRetry",". These silent retries used up the rate limit.",[14,219,220],{},"OCR is the expensive and slow part. Its result is cached under the SHA-256 of\nthe image, in memory and in the database. If the same photo is processed again,\nfor example after an error, OCR does not read it a second time.",[28,222,224],{"id":223},"the-prompt-knows-switzerland","The prompt knows Switzerland",[14,226,227],{},"Most errors in the second stage were not reading errors. The model read\ncorrectly and assigned incorrectly. The prompt is therefore a list of\ndistinctions that a person from Switzerland makes without thinking:",[229,230,231,235,238,241],"ul",{},[232,233,234],"li",{},"\"SG 218574\" is a license plate from St. Gallen, not a\nVIN. A VIN has 17 characters.",[232,236,237],{},"If a repair date is on the invoice, that one counts, not the\ninvoice date. The service book needs the day the work was done.",[232,239,240],{},"\"Fr.\" and an amount without a currency mean CHF. EUR only if it says so.",[232,242,243],{},"\"1 014.80\" is one number with a thousands space, not two numbers.",[14,245,246],{},"The vehicle registration document is the same. The fields are numbered and\nlabeled in four languages. The prompt names the four fields that matter: 15\nlicense plate, 21 make and model, 23 VIN, 36 first registration. And it states\nexplicitly that the master number (field 18) and the type approval (field 24)\nare not a VIN.",[28,248,250],{"id":249},"what-the-code-checks","What the code checks",[14,252,253],{},"A prompt is a request. So whatever can be checked is checked by the code, in\npure functions with unit tests.",[14,255,256,260],{},[257,258,259],"strong",{},"Line items against the total."," Many invoices have several description lines\nabove a single labor line:",[44,262,267],{"className":263,"code":265,"language":266},[264],"language-text","Auspuff reparieren\nAuto auf Ölverlust kontrollieren\nArbeit 1.50 Std.   130.00   195.00\n","text",[39,268,265],{"__ignoreMap":49},[14,270,271,272,275],{},"The model attached the 195 francs to every line. On a real invoice, that turned\ninto four line items of 195 francs each, 844.40 in total against a total of\n280.40. The prompt now explains the case. In addition, ",[39,273,274],{},"repairItems"," does the\nmath: if the sum of the line items is more than one franc above the total, it\nmerges consecutive line items with the same amount. But it keeps the merge only\nif the sum matches afterward. Otherwise the form shows a notice and does not\nguess any further.",[14,277,278,281,282,285,286,289],{},[257,279,280],{},"Categories by keyword."," A list of regular expressions overrides the model's\nassignment when the description is unambiguous. \"Auspuff reparieren\" is\n",[39,283,284],{},"auspuff",", no matter what the model thinks. The first match wins, and the order\nis intentional: \"Service mit Ölwechsel\" becomes ",[39,287,288],{},"oelwechsel",", because the more\nspecific rule comes before the general one for inspections.",[14,291,292,295],{},[257,293,294],{},"Duplicate invoices."," The same invoice often arrives twice, once as a photo\nand later in the garage's combined PDF. An invoice counts as a duplicate if it\nhas the same amount down to the centime and a date at most 14 days apart. I\ndeliberately do not compare the garage name: OCR writes it too inconsistently,\nand the 14 days catch invoices where the repair date was read once and the\ninvoice date the other time.",[14,297,298,301],{},[257,299,300],{},"The license plate."," If it is on the invoice, the app assigns the invoice to\nthe matching vehicle, even if a different one is currently open. Spellings like\n\"SG 218 574\" and \"CH-SG218574\" are normalized to one form first. If the plate\nbelongs to no vehicle, the invoice is not preselected in the review list.",[28,303,305],{"id":304},"a-pdf-page-by-page","A PDF, page by page",[14,307,308],{},"Garages like to send a PDF with all invoices of the year. The first approach\npassed the whole PDF to the model in one call. With nine pages, invoices were\nmissing afterward, and the garage from the first one appeared on all of them.",[14,310,311,312,315,316,319,320,323],{},"Now OCR reads all pages, and the model processes each page separately. It also\ndetermines the page type: ",[39,313,314],{},"rechnung"," with its own header, ",[39,317,318],{},"fortsetzung"," of the\nprevious one, or ",[39,321,322],{},"andere"," for terms and conditions and blank pages. So that it\ncan decide, it gets the first 1200 characters of the previous page, explicitly\nfor classification only. Garage, date, and amount may only come from the page\nitself.",[14,325,326,327,330],{},"Putting it together is code again: ",[39,328,329],{},"mergePdfPages"," appends continuations to the\nprevious invoice and fills only empty fields there. The pages run three at a\ntime in parallel, not all at once. All calls go through a proxy of my own, which\nallows at most 20 requests per minute.",[28,332,334],{"id":333},"nothing-is-saved-before-someone-has-seen-it","Nothing is saved before someone has seen it",[14,336,337],{},"The checks find contradictions, but not plausible errors. A misread odometer\nreading that is higher than the last one looks like a correct one. That is why\nno scan saves directly.",[14,339,340],{},"In the invoice form, a scan fills only empty fields. What the user has already\nentered stays; the currency only as long as the user has not changed it. Several\nphotos or a combined PDF appear as a review list. Each row states its source\n(\"Seite 3–4\"), the detected vehicle, and whether the invoice has already been\nentered. Duplicate and unclear rows are deselected.",[14,342,343],{},"Offline, the same applies. Without a connection, the app saves the photo with a\nflag. When the connection comes back, it runs the scan then and again fills only\nempty fields.",[28,345,347],{"id":346},"the-chat-likes-to-claim-it-saved","The chat likes to claim it saved",[14,349,350],{},"The app also has a chat. It can create vehicles, enter invoices and maintenance,\nand set the maintenance schedule, each through a tool. Before entering an\ninvoice, it shows all fields and waits for a \"Ja\".",[14,352,353],{},"The real problem was the opposite. The model wrote \"Die Wartung wurde\neingetragen\" without having called the tool. In the chat that looks like\nsuccess; in the database there is nothing.",[14,355,356],{},"One sentence in the prompt did not fix it. Worse: the model copied example\nsentences for success messages in the prompt word for word, even without a\ntool. So the prompt now describes success messages only by their form.",[14,358,359,360,363,364,367],{},"What helped was a guard in the code. ",[39,361,362],{},"claimsActionWithoutTool"," searches the text\nof the response for a participle with an auxiliary verb (\"wurde eingetragen\",\n\"habe ich gespeichert\") and checks whether a writing tool ran in the same\nresponse. Reading tools like ",[39,365,366],{},"list_vehicles"," do not count. Negated sentences like\n\"noch keine Wartung eingetragen\" are information and stay out, and \"Ich habe\nfolgende Daten erfasst:\" is the preview before confirmation, not success.",[14,369,370,371,374,375,378],{},"If the guard triggers, the code asks once more, this time with\n",[39,372,373],{},"toolChoice: 'required'",". The first version forced this for all steps. Then,\nafter the tool result, another request with ",[39,376,377],{},"tool_choice: \"any\""," went to\nMistral, and it never came back. The maintenance entry was saved, the chat hung\non the loading indicator, on the vehicle page in four out of six attempts. Now\nthe requirement applies only to the first step:",[44,380,382],{"className":46,"code":381,"language":48,"meta":49,"style":49},"prepareStep: ({ stepNumber }) => (stepNumber === 0 ? { toolChoice: 'required' } : {}),\n",[39,383,384],{"__ignoreMap":49},[53,385,386,389,392,396,399,401,404,407,410,413,416,419,421,424],{"class":55,"line":56},[53,387,388],{"class":80},"prepareStep",[53,390,391],{"class":63},": ({ ",[53,393,395],{"class":394},"s4XuR","stepNumber",[53,397,398],{"class":63}," }) ",[53,400,87],{"class":59},[53,402,403],{"class":63}," (stepNumber ",[53,405,406],{"class":59},"===",[53,408,409],{"class":67}," 0",[53,411,412],{"class":59}," ?",[53,414,415],{"class":63}," { toolChoice: ",[53,417,418],{"class":142},"'required'",[53,420,71],{"class":63},[53,422,423],{"class":59},":",[53,425,426],{"class":63}," {}),\n",[14,428,429],{},"After that, eight out of eight attempts went through, five of them via the\nguard.",[28,431,433],{"id":432},"dictation-through-the-same-second-stage","Dictation through the same second stage",[14,435,436],{},"Anyone who would rather speak than type can dictate the invoice. The dictation\nruns through the same second stage as the photo, except that the text comes\nfrom the transcription instead of OCR.",[14,438,439,440,443,444,447],{},"Which model is suitable for this, I measured rather than took from the\ndocumentation, with six sentences from everyday garage work. The sentences were\nvoiced with Piper, not spoken by me; with technical terms like \"Lambdasonde\",\nthat colors the numbers. ",[39,441,442],{},"voxtral-mini-latest"," transcribed them with a\n16.4 percent word error rate. ",[39,445,446],{},"voxtral-small-latest"," understands audio and\ncan call tools, which sounded like the more elegant route. But it rephrased and\nonce answered in English: \"Zahnriemen mit Wasserpumpe ersetzt\" became \"The\nwater pump replaced the fan belt.\" The route via transcript and schema, on the\nother hand, got all five fields right.",[28,449,451],{"id":450},"how-this-is-tested","How this is tested",[14,453,454],{},"The rules above are pure functions with Vitest tests: line items, categories,\nduplicates, merging pages, the guard. They need neither network nor database.",[14,456,457],{},"The Playwright tests intercept the calls to Mistral and return fixed responses.\nThat way they check the form, the review list, and the chat without a run\ncosting money or depending on the model. A separate test project sends real\nphotos and a nine-page combined PDF to the real model. I start that by hand, not\non every change. During one of these runs, it turned out that number fields\nshowed \"214,583 km\", because they lacked the Swiss formatting.",[28,459,461],{"id":460},"what-carries-over","What carries over",[14,463,464],{},"None of this depends on cars. If you want to extract documents with a language\nmodel, you can take this with you:",[229,466,467,470,473,476,479,482,485],{},[232,468,469],{},"Separate reading from understanding. OCR reads, the chat model gets only\ntext and a schema.",[232,471,472],{},"Schema fields with fixed values as an enum, not as free text.",[232,474,475],{},"Write the prompt for the distinctions the model fails at, not for the ones it\nalready handles.",[232,477,478],{},"Recalculate in code everything that can be recalculated, and on a\ncontradiction show a notice instead of guessing.",[232,480,481],{},"Process long documents page by page and assemble them in code.",[232,483,484],{},"Fill only empty fields, and before saving show what will be saved.",[232,486,487],{},"Do not believe a model that reports success; check whether the tool ran.",[14,489,490,491,496,497,500,501,504,505,508,509,512],{},"The source code is open:\n",[17,492,495],{"href":493,"rel":494},"https:\u002F\u002Fgithub.com\u002Fgstrainovic\u002Fwartungsheft",[21],"github.com\u002Fgstrainovic\u002Fwartungsheft",".\nThe pipeline is in ",[39,498,499],{},"src\u002Fservices\u002Fai.ts",", the checks in\n",[39,502,503],{},"src\u002Fservices\u002Finvoice-scan.ts"," and ",[39,506,507],{},"src\u002Fservices\u002Finvoice-items.ts",", the guard\nin ",[39,510,511],{},"src\u002Fservices\u002Fchat-guard.ts",".",[514,515,516],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}",{"title":49,"searchDepth":96,"depth":96,"links":518},[519,520,521,522,523,524,525,526,527],{"id":30,"depth":96,"text":31},{"id":223,"depth":96,"text":224},{"id":249,"depth":96,"text":250},{"id":304,"depth":96,"text":305},{"id":333,"depth":96,"text":334},{"id":346,"depth":96,"text":347},{"id":432,"depth":96,"text":433},{"id":450,"depth":96,"text":451},{"id":460,"depth":96,"text":461},"How Wartungsheft reads garage invoices with Mistral. Two stages instead of one, checks in code, and why nothing is saved before someone has seen it.","md",{},true,"\u002Fopen-source\u002Frechnung-fotografieren",null,{"title":5,"description":528},"open-source\u002Frechnung-fotografieren","uRmFmRcvYr3AVjQTz-50c3jFd2YgAY2twzOJxCeVs4M",1790517213225]