[{"data":1,"prerenderedAt":1428},["ShallowReactive",2],{"seite-\u002Fen\u002Fopen-source\u002Fki-testet-ohne-fenster\u002F":3},{"id":4,"title":5,"body":6,"description":1419,"extension":1420,"meta":1421,"navigation":1422,"path":1423,"robots":1424,"seo":1425,"stem":1426,"__hash__":1427},"seiten_en\u002Fopen-source\u002Fki-testet-ohne-fenster.md","An AI agent tests my editor without a window",{"type":7,"value":8,"toc":1408},"minimark",[9,13,28,34,37,42,45,80,84,95,98,101,105,145,148,462,465,782,789,793,796,1103,1114,1117,1121,1132,1228,1231,1234,1242,1256,1263,1266,1270,1273,1284,1287,1303,1309,1313,1316,1336,1339,1343,1354,1357,1361,1364,1387,1404],[10,11,5],"h1",{"id":12},"an-ai-agent-tests-my-editor-without-a-window",[14,15,16,17,21,22,27],"p",{},"I write my editor ",[18,19,20],"code",{},"zid"," with Claude Code. How that came about is covered in\nthe ",[23,24,26],"a",{"href":25},"\u002Fen\u002Fopen-source\u002Feditor-suche\u002F","article about my editor search",".\nThis one is about the question that comes next: How does an AI agent check\na user interface it can neither click nor see?",[14,29,30,31,33],{},"On the web, this is solved. Playwright opens a browser, clicks, reads the DOM\nand takes screenshots. ",[18,32,20],{}," has none of that. It is written in Zig,\ndraws via wgpu with its own text renderer and builds the layout with Clay.\nThere is no DOM, no developer tools and no browser that takes\nanything off your hands.",[14,35,36],{},"Without help, an agent writes UI code there blind. The end result is\n\"should work\", and then I was the one checking, by hand, in the window.\nI turned that around: The app gets a remote control and eyes, and\nthe agent uses both itself.",[38,39,41],"h2",{"id":40},"three-levels","Three levels",[14,43,44],{},"Testing happens on three levels, and each one catches something different.",[46,47,48,68,74],"ul",{},[49,50,51,55,56,59,60,63,64,67],"li",{},[52,53,54],"strong",{},"Unit tests in Zig"," check logic that needs no UI. PDF\npage navigation lives in ",[18,57,58],{},"ui\u002Fpdf_nav.zig",", the Git history in\n",[18,61,62],{},"git\u002Fgit_history.zig",", the folder dialog in ",[18,65,66],{},"ui\u002Ffolder_ops.zig",". These modules\nknow nothing about Clay and run in milliseconds.",[49,69,70,73],{},[52,71,72],{},"E2E scripts in Python"," start the real app without a window, operate it\nvia JSON-RPC and check the state the app returns as JSON. There\nare 23 scripts with a combined total of just over 4000 lines.",[49,75,76,79],{},[52,77,78],{},"Screenshots"," are written to a file by the app on request. Claude looks\nat them itself.",[38,81,83],{"id":82},"without-a-window-but-with-the-same-loop","Without a window, but with the same loop",[14,85,86,87,90,91,94],{},"The command is ",[18,88,89],{},"zig build run -- --headless --ai=off",". The app then opens\nno window, renders into a buffer of 1200 × 800 pixels and listens on\nport 9999. ",[18,92,93],{},"--ai=off"," turns off the built-in agent with its local\nlanguage model.",[14,96,97],{},"What matters is that headless mode is not a world of its own. It used to have\nits own loop of 27 lines that only picked up results from background work.\nTab switching, clicks in the explorer, deferred closing of tabs and\nbuffered input never ran there. Bugs in exactly these paths could\nonly be reproduced with a visible window, and that window gets in the way when\nI am working on the same desktop at the same time.",[14,99,100],{},"Today, headless runs the same frame loop as the windowed app. The only things skipped\nare what requires a window: window events, the mouse cursor and the\noutput to the screen. Instead of waiting for events, the\nloop sleeps for 16 milliseconds. What is tested headless is therefore the same code\nI use in the window.",[38,102,104],{"id":103},"json-rpc-as-a-remote-control","JSON-RPC as a remote control",[14,106,107,108,111,112,111,115,111,118,111,121,124,125,128,129,132,133,136,137,140,141,144],{},"The RPC server knows 48 methods. Some of them operate the app like a\nhuman: ",[18,109,110],{},"click",", ",[18,113,114],{},"right_click",[18,116,117],{},"move_mouse",[18,119,120],{},"scroll",[18,122,123],{},"key_press",",\n",[18,126,127],{},"type_text",". Others read state: ",[18,130,131],{},"ui_state"," returns open\ndialogs, menus, tabs and focus, ",[18,134,135],{},"editor_state"," lines, cursor and\nsearch bar, ",[18,138,139],{},"pdf_state"," the current page. With ",[18,142,143],{},"element_bounds",", a\ntest gets the position of any layout element instead of guessing\ncoordinates.",[14,146,147],{},"On the Python side, a single function is enough:",[149,150,155],"pre",{"className":151,"code":152,"language":153,"meta":154,"style":154},"language-python shiki shiki-themes github-light github-dark","def rpc(method, params=None):\n    msg = json.dumps({\"jsonrpc\": \"2.0\", \"method\": method, \"params\": params or [], \"id\": 1}) + \"\\n\"\n    with socket.create_connection((HOST, PORT), timeout=10) as s:\n        s.sendall(msg.encode())\n        data = b\"\"\n        while not data.endswith(b\"\\n\"):\n            chunk = s.recv(65536)\n            if not chunk:\n                break\n            data += chunk\n    res = json.loads(data.decode())\n    if \"error\" in res:\n        raise RuntimeError(f\"{method}: {res['error']}\")\n    return res[\"result\"]\n","python","",[18,156,157,184,248,286,292,306,330,347,358,364,376,387,402,447],{"__ignoreMap":154},[158,159,162,166,170,174,177,181],"span",{"class":160,"line":161},"line",1,[158,163,165],{"class":164},"szBVR","def",[158,167,169],{"class":168},"sScJk"," rpc",[158,171,173],{"class":172},"sVt8B","(method, params",[158,175,176],{"class":164},"=",[158,178,180],{"class":179},"sj4cs","None",[158,182,183],{"class":172},"):\n",[158,185,187,190,192,195,199,202,205,207,210,213,216,219,222,225,228,230,233,236,239,242,245],{"class":160,"line":186},2,[158,188,189],{"class":172},"    msg ",[158,191,176],{"class":164},[158,193,194],{"class":172}," json.dumps({",[158,196,198],{"class":197},"sZZnC","\"jsonrpc\"",[158,200,201],{"class":172},": ",[158,203,204],{"class":197},"\"2.0\"",[158,206,111],{"class":172},[158,208,209],{"class":197},"\"method\"",[158,211,212],{"class":172},": method, ",[158,214,215],{"class":197},"\"params\"",[158,217,218],{"class":172},": params ",[158,220,221],{"class":164},"or",[158,223,224],{"class":172}," [], ",[158,226,227],{"class":197},"\"id\"",[158,229,201],{"class":172},[158,231,232],{"class":179},"1",[158,234,235],{"class":172},"}) ",[158,237,238],{"class":164},"+",[158,240,241],{"class":197}," \"",[158,243,244],{"class":179},"\\n",[158,246,247],{"class":197},"\"\n",[158,249,251,254,257,260,262,265,268,272,274,277,280,283],{"class":160,"line":250},3,[158,252,253],{"class":164},"    with",[158,255,256],{"class":172}," socket.create_connection((",[158,258,259],{"class":179},"HOST",[158,261,111],{"class":172},[158,263,264],{"class":179},"PORT",[158,266,267],{"class":172},"), ",[158,269,271],{"class":270},"s4XuR","timeout",[158,273,176],{"class":164},[158,275,276],{"class":179},"10",[158,278,279],{"class":172},") ",[158,281,282],{"class":164},"as",[158,284,285],{"class":172}," s:\n",[158,287,289],{"class":160,"line":288},4,[158,290,291],{"class":172},"        s.sendall(msg.encode())\n",[158,293,295,298,300,303],{"class":160,"line":294},5,[158,296,297],{"class":172},"        data ",[158,299,176],{"class":164},[158,301,302],{"class":164}," b",[158,304,305],{"class":197},"\"\"\n",[158,307,309,312,315,318,321,324,326,328],{"class":160,"line":308},6,[158,310,311],{"class":164},"        while",[158,313,314],{"class":164}," not",[158,316,317],{"class":172}," data.endswith(",[158,319,320],{"class":164},"b",[158,322,323],{"class":197},"\"",[158,325,244],{"class":179},[158,327,323],{"class":197},[158,329,183],{"class":172},[158,331,333,336,338,341,344],{"class":160,"line":332},7,[158,334,335],{"class":172},"            chunk ",[158,337,176],{"class":164},[158,339,340],{"class":172}," s.recv(",[158,342,343],{"class":179},"65536",[158,345,346],{"class":172},")\n",[158,348,350,353,355],{"class":160,"line":349},8,[158,351,352],{"class":164},"            if",[158,354,314],{"class":164},[158,356,357],{"class":172}," chunk:\n",[158,359,361],{"class":160,"line":360},9,[158,362,363],{"class":164},"                break\n",[158,365,367,370,373],{"class":160,"line":366},10,[158,368,369],{"class":172},"            data ",[158,371,372],{"class":164},"+=",[158,374,375],{"class":172}," chunk\n",[158,377,379,382,384],{"class":160,"line":378},11,[158,380,381],{"class":172},"    res ",[158,383,176],{"class":164},[158,385,386],{"class":172}," json.loads(data.decode())\n",[158,388,390,393,396,399],{"class":160,"line":389},12,[158,391,392],{"class":164},"    if",[158,394,395],{"class":197}," \"error\"",[158,397,398],{"class":164}," in",[158,400,401],{"class":172}," res:\n",[158,403,405,408,411,414,417,419,422,425,428,430,432,435,438,441,443,445],{"class":160,"line":404},13,[158,406,407],{"class":164},"        raise",[158,409,410],{"class":179}," RuntimeError",[158,412,413],{"class":172},"(",[158,415,416],{"class":164},"f",[158,418,323],{"class":197},[158,420,421],{"class":179},"{",[158,423,424],{"class":172},"method",[158,426,427],{"class":179},"}",[158,429,201],{"class":197},[158,431,421],{"class":179},[158,433,434],{"class":172},"res[",[158,436,437],{"class":197},"'error'",[158,439,440],{"class":172},"]",[158,442,427],{"class":179},[158,444,323],{"class":197},[158,446,346],{"class":172},[158,448,450,453,456,459],{"class":160,"line":449},14,[158,451,452],{"class":164},"    return",[158,454,455],{"class":172}," res[",[158,457,458],{"class":197},"\"result\"",[158,460,461],{"class":172},"]\n",[14,463,464],{},"The real work lies in the question of which thread may do what. The\nRPC server runs in its own thread, the UI in the main thread.\nIf an RPC call changes the tab list while the main thread is\ndrawing it, the app crashes. That is why input goes into a queue\nthat the main thread works through once per frame:",[149,466,470],{"className":467,"code":468,"language":469,"meta":154,"style":154},"language-zig shiki shiki-themes github-light github-dark","\u002F\u002F\u002F Vom Main-Thread pro Frame aufrufen: gepufferte Eingaben anwenden.\npub fn drainInputs(ctx: *E2EContext) void {\n    var batch: [64]InputEvent = undefined;\n    while (true) {\n        ctx.input_mutex.lock();\n        const n = @min(ctx.pending_inputs.items.len, batch.len);\n        @memcpy(batch[0..n], ctx.pending_inputs.items[0..n]);\n        ctx.pending_inputs.replaceRangeAssumeCapacity(0, n, &.{});\n        ctx.input_mutex.unlock();\n        if (n == 0) return;\n        for (batch[0..n]) |ev| applyInput(ctx.ui_system, ev);\n    }\n}\n","zig",[18,471,472,478,510,538,552,571,615,660,689,704,726,772,777],{"__ignoreMap":154},[158,473,474],{"class":160,"line":161},[158,475,477],{"class":476},"sJ8bj","\u002F\u002F\u002F Vom Main-Thread pro Frame aufrufen: gepufferte Eingaben anwenden.\n",[158,479,480,483,486,489,491,494,496,499,502,504,507],{"class":160,"line":186},[158,481,482],{"class":164},"pub",[158,484,485],{"class":164}," fn",[158,487,488],{"class":168}," drainInputs",[158,490,413],{"class":172},[158,492,493],{"class":270},"ctx",[158,495,201],{"class":172},[158,497,498],{"class":164},"*",[158,500,501],{"class":270},"E2EContext",[158,503,279],{"class":172},[158,505,506],{"class":164},"void",[158,508,509],{"class":172}," {\n",[158,511,512,515,518,521,524,526,529,532,535],{"class":160,"line":250},[158,513,514],{"class":164},"    var",[158,516,517],{"class":270}," batch",[158,519,520],{"class":172},": [",[158,522,523],{"class":179},"64",[158,525,440],{"class":172},[158,527,528],{"class":270},"InputEvent",[158,530,531],{"class":164}," =",[158,533,534],{"class":164}," undefined",[158,536,537],{"class":172},";\n",[158,539,540,543,546,549],{"class":160,"line":288},[158,541,542],{"class":164},"    while",[158,544,545],{"class":172}," (",[158,547,548],{"class":164},"true",[158,550,551],{"class":172},") {\n",[158,553,554,557,560,563,565,568],{"class":160,"line":294},[158,555,556],{"class":270},"        ctx",[158,558,559],{"class":172},".",[158,561,562],{"class":270},"input_mutex",[158,564,559],{"class":172},[158,566,567],{"class":168},"lock",[158,569,570],{"class":172},"();\n",[158,572,573,576,579,581,584,586,588,590,593,595,598,600,603,605,608,610,612],{"class":160,"line":308},[158,574,575],{"class":164},"        const",[158,577,578],{"class":270}," n",[158,580,531],{"class":164},[158,582,583],{"class":179}," @min",[158,585,413],{"class":172},[158,587,493],{"class":270},[158,589,559],{"class":172},[158,591,592],{"class":270},"pending_inputs",[158,594,559],{"class":172},[158,596,597],{"class":270},"items",[158,599,559],{"class":172},[158,601,602],{"class":270},"len",[158,604,111],{"class":172},[158,606,607],{"class":270},"batch",[158,609,559],{"class":172},[158,611,602],{"class":270},[158,613,614],{"class":172},");\n",[158,616,617,620,622,624,627,630,633,636,639,641,643,645,647,649,651,653,655,657],{"class":160,"line":332},[158,618,619],{"class":179},"        @memcpy",[158,621,413],{"class":172},[158,623,607],{"class":270},[158,625,626],{"class":172},"[",[158,628,629],{"class":179},"0",[158,631,632],{"class":172},"..",[158,634,635],{"class":270},"n",[158,637,638],{"class":172},"], ",[158,640,493],{"class":270},[158,642,559],{"class":172},[158,644,592],{"class":270},[158,646,559],{"class":172},[158,648,597],{"class":270},[158,650,626],{"class":172},[158,652,629],{"class":179},[158,654,632],{"class":172},[158,656,635],{"class":270},[158,658,659],{"class":172},"]);\n",[158,661,662,664,666,668,670,673,675,677,679,681,683,686],{"class":160,"line":349},[158,663,556],{"class":270},[158,665,559],{"class":172},[158,667,592],{"class":270},[158,669,559],{"class":172},[158,671,672],{"class":168},"replaceRangeAssumeCapacity",[158,674,413],{"class":172},[158,676,629],{"class":179},[158,678,111],{"class":172},[158,680,635],{"class":270},[158,682,111],{"class":172},[158,684,685],{"class":164},"&",[158,687,688],{"class":172},".{});\n",[158,690,691,693,695,697,699,702],{"class":160,"line":360},[158,692,556],{"class":270},[158,694,559],{"class":172},[158,696,562],{"class":270},[158,698,559],{"class":172},[158,700,701],{"class":168},"unlock",[158,703,570],{"class":172},[158,705,706,709,711,713,716,719,721,724],{"class":160,"line":366},[158,707,708],{"class":164},"        if",[158,710,545],{"class":172},[158,712,635],{"class":270},[158,714,715],{"class":164}," ==",[158,717,718],{"class":179}," 0",[158,720,279],{"class":172},[158,722,723],{"class":164},"return",[158,725,537],{"class":172},[158,727,728,731,733,735,737,739,741,743,746,749,752,754,757,759,761,763,766,768,770],{"class":160,"line":378},[158,729,730],{"class":164},"        for",[158,732,545],{"class":172},[158,734,607],{"class":270},[158,736,626],{"class":172},[158,738,629],{"class":179},[158,740,632],{"class":172},[158,742,635],{"class":270},[158,744,745],{"class":172},"]) ",[158,747,748],{"class":164},"|",[158,750,751],{"class":270},"ev",[158,753,748],{"class":164},[158,755,756],{"class":168}," applyInput",[158,758,413],{"class":172},[158,760,493],{"class":270},[158,762,559],{"class":172},[158,764,765],{"class":270},"ui_system",[158,767,111],{"class":172},[158,769,751],{"class":270},[158,771,614],{"class":172},[158,773,774],{"class":160,"line":389},[158,775,776],{"class":172},"    }\n",[158,778,779],{"class":160,"line":404},[158,780,781],{"class":172},"}\n",[14,783,784,785,788],{},"Read calls still run in the server thread. Shared data then\nneeds a lock. The Git status in the explorer had none: The main thread\nreplaced the map while a test was reading it, and the app crashed in\n",[18,786,787],{},"isIgnored",". Since then, every access goes through three functions that hold the\nlock.",[38,790,792],{"id":791},"a-test-reads-like-a-user-manual","A test reads like a user manual",[14,794,795],{},"This is what the beginning of the test for PDF page navigation looks like:",[149,797,799],{"className":151,"code":798,"language":153,"meta":154,"style":154},"cfg = os.path.join(ROOT, \"tmp\", \"e2e_pdf_cfg\")\nshutil.rmtree(cfg, ignore_errors=True)\nenv = dict(os.environ, XDG_CONFIG_HOME=cfg)\nwait_port_free()\nproc = subprocess.Popen(\n    [\"zig\", \"build\", \"run\", \"--\", \"--headless\", \"--ai=off\", PDF],\n    cwd=ROOT, stdout=log, stderr=subprocess.STDOUT, env=env,\n    start_new_session=True,  # eigene Prozessgruppe, siehe finally\n)\ntry:\n    wait_port(proc)\n    settle(20)\n    # ...\n    check(st[\"pdf\"], f\"PDF-Tab ist aktiv (nach {time.time() - t0:.1f}s)\")\n    check(st[\"pages\"] >= 3, f\"Dokument hat {st['pages']} Seiten\")\n    to_first_page()\n    expect_page(0, \"Bild auf hält am Anfang bei Seite 1\")\n",[18,800,801,826,841,862,867,877,918,958,973,977,985,990,1000,1005,1041,1082,1088],{"__ignoreMap":154},[158,802,803,806,808,811,814,816,819,821,824],{"class":160,"line":161},[158,804,805],{"class":172},"cfg ",[158,807,176],{"class":164},[158,809,810],{"class":172}," os.path.join(",[158,812,813],{"class":179},"ROOT",[158,815,111],{"class":172},[158,817,818],{"class":197},"\"tmp\"",[158,820,111],{"class":172},[158,822,823],{"class":197},"\"e2e_pdf_cfg\"",[158,825,346],{"class":172},[158,827,828,831,834,836,839],{"class":160,"line":186},[158,829,830],{"class":172},"shutil.rmtree(cfg, ",[158,832,833],{"class":270},"ignore_errors",[158,835,176],{"class":164},[158,837,838],{"class":179},"True",[158,840,346],{"class":172},[158,842,843,846,848,851,854,857,859],{"class":160,"line":250},[158,844,845],{"class":172},"env ",[158,847,176],{"class":164},[158,849,850],{"class":179}," dict",[158,852,853],{"class":172},"(os.environ, ",[158,855,856],{"class":270},"XDG_CONFIG_HOME",[158,858,176],{"class":164},[158,860,861],{"class":172},"cfg)\n",[158,863,864],{"class":160,"line":288},[158,865,866],{"class":172},"wait_port_free()\n",[158,868,869,872,874],{"class":160,"line":294},[158,870,871],{"class":172},"proc ",[158,873,176],{"class":164},[158,875,876],{"class":172}," subprocess.Popen(\n",[158,878,879,882,885,887,890,892,895,897,900,902,905,907,910,912,915],{"class":160,"line":308},[158,880,881],{"class":172},"    [",[158,883,884],{"class":197},"\"zig\"",[158,886,111],{"class":172},[158,888,889],{"class":197},"\"build\"",[158,891,111],{"class":172},[158,893,894],{"class":197},"\"run\"",[158,896,111],{"class":172},[158,898,899],{"class":197},"\"--\"",[158,901,111],{"class":172},[158,903,904],{"class":197},"\"--headless\"",[158,906,111],{"class":172},[158,908,909],{"class":197},"\"--ai=off\"",[158,911,111],{"class":172},[158,913,914],{"class":179},"PDF",[158,916,917],{"class":172},"],\n",[158,919,920,923,925,927,929,932,934,937,940,942,945,948,950,953,955],{"class":160,"line":332},[158,921,922],{"class":270},"    cwd",[158,924,176],{"class":164},[158,926,813],{"class":179},[158,928,111],{"class":172},[158,930,931],{"class":270},"stdout",[158,933,176],{"class":164},[158,935,936],{"class":172},"log, ",[158,938,939],{"class":270},"stderr",[158,941,176],{"class":164},[158,943,944],{"class":172},"subprocess.",[158,946,947],{"class":179},"STDOUT",[158,949,111],{"class":172},[158,951,952],{"class":270},"env",[158,954,176],{"class":164},[158,956,957],{"class":172},"env,\n",[158,959,960,963,965,967,970],{"class":160,"line":349},[158,961,962],{"class":270},"    start_new_session",[158,964,176],{"class":164},[158,966,838],{"class":179},[158,968,969],{"class":172},",  ",[158,971,972],{"class":476},"# eigene Prozessgruppe, siehe finally\n",[158,974,975],{"class":160,"line":360},[158,976,346],{"class":172},[158,978,979,982],{"class":160,"line":366},[158,980,981],{"class":164},"try",[158,983,984],{"class":172},":\n",[158,986,987],{"class":160,"line":378},[158,988,989],{"class":172},"    wait_port(proc)\n",[158,991,992,995,998],{"class":160,"line":389},[158,993,994],{"class":172},"    settle(",[158,996,997],{"class":179},"20",[158,999,346],{"class":172},[158,1001,1002],{"class":160,"line":404},[158,1003,1004],{"class":476},"    # ...\n",[158,1006,1007,1010,1013,1015,1017,1020,1022,1025,1028,1031,1034,1036,1039],{"class":160,"line":449},[158,1008,1009],{"class":172},"    check(st[",[158,1011,1012],{"class":197},"\"pdf\"",[158,1014,638],{"class":172},[158,1016,416],{"class":164},[158,1018,1019],{"class":197},"\"PDF-Tab ist aktiv (nach ",[158,1021,421],{"class":179},[158,1023,1024],{"class":172},"time.time() ",[158,1026,1027],{"class":164},"-",[158,1029,1030],{"class":172}," t0",[158,1032,1033],{"class":164},":.1f",[158,1035,427],{"class":179},[158,1037,1038],{"class":197},"s)\"",[158,1040,346],{"class":172},[158,1042,1044,1046,1049,1052,1055,1058,1060,1062,1065,1067,1070,1073,1075,1077,1080],{"class":160,"line":1043},15,[158,1045,1009],{"class":172},[158,1047,1048],{"class":197},"\"pages\"",[158,1050,1051],{"class":172},"] ",[158,1053,1054],{"class":164},">=",[158,1056,1057],{"class":179}," 3",[158,1059,111],{"class":172},[158,1061,416],{"class":164},[158,1063,1064],{"class":197},"\"Dokument hat ",[158,1066,421],{"class":179},[158,1068,1069],{"class":172},"st[",[158,1071,1072],{"class":197},"'pages'",[158,1074,440],{"class":172},[158,1076,427],{"class":179},[158,1078,1079],{"class":197}," Seiten\"",[158,1081,346],{"class":172},[158,1083,1085],{"class":160,"line":1084},16,[158,1086,1087],{"class":172},"    to_first_page()\n",[158,1089,1091,1094,1096,1098,1101],{"class":160,"line":1090},17,[158,1092,1093],{"class":172},"    expect_page(",[158,1095,629],{"class":179},[158,1097,111],{"class":172},[158,1099,1100],{"class":197},"\"Bild auf hält am Anfang bei Seite 1\"",[158,1102,346],{"class":172},[14,1104,1105,1106,1109,1110,1113],{},"Every check prints a line with ",[18,1107,1108],{},"PASS"," or ",[18,1111,1112],{},"FAIL"," and a sentence that\nsays what is meant. This is written for the agent: It reads the\noutput and knows, without a stack trace, which step failed.",[14,1115,1116],{},"Two details came from painful experience. Every run gets a fresh\nconfiguration directory, otherwise the app restores the tabs from the last session,\nand the test measures someone else's state. And the test starts the app\nin its own process group so that at the end it really shuts everything down.",[38,1118,1120],{"id":1119},"what-the-screenshot-shows-and-the-state-does-not","What the screenshot shows and the state does not",[14,1122,1123,1124,1127,1128,1131],{},"The RPC method ",[18,1125,1126],{},"screenshot"," writes a PPM to ",[18,1129,1130],{},"tmp\u002F",". Claude does not read\nPPM files, so the agent converts them before looking at them:",[149,1133,1135],{"className":151,"code":1134,"language":153,"meta":154,"style":154},"from PIL import Image\nfor n in ['repo', 'diff', 'file', 'empty', 'split']:\n    Image.open(f'tmp\u002Fe2e_git_history_{n}.ppm').save(f'tmp\u002Fe2e_git_history_{n}.png')\n",[18,1136,1137,1151,1191],{"__ignoreMap":154},[158,1138,1139,1142,1145,1148],{"class":160,"line":161},[158,1140,1141],{"class":164},"from",[158,1143,1144],{"class":179}," PIL",[158,1146,1147],{"class":164}," import",[158,1149,1150],{"class":172}," Image\n",[158,1152,1153,1156,1159,1162,1165,1168,1170,1173,1175,1178,1180,1183,1185,1188],{"class":160,"line":186},[158,1154,1155],{"class":164},"for",[158,1157,1158],{"class":172}," n ",[158,1160,1161],{"class":164},"in",[158,1163,1164],{"class":172}," [",[158,1166,1167],{"class":197},"'repo'",[158,1169,111],{"class":172},[158,1171,1172],{"class":197},"'diff'",[158,1174,111],{"class":172},[158,1176,1177],{"class":197},"'file'",[158,1179,111],{"class":172},[158,1181,1182],{"class":197},"'empty'",[158,1184,111],{"class":172},[158,1186,1187],{"class":197},"'split'",[158,1189,1190],{"class":172},"]:\n",[158,1192,1193,1196,1198,1201,1203,1205,1207,1210,1213,1215,1217,1219,1221,1223,1226],{"class":160,"line":250},[158,1194,1195],{"class":172},"    Image.open(",[158,1197,416],{"class":164},[158,1199,1200],{"class":197},"'tmp\u002Fe2e_git_history_",[158,1202,421],{"class":179},[158,1204,635],{"class":172},[158,1206,427],{"class":179},[158,1208,1209],{"class":197},".ppm'",[158,1211,1212],{"class":172},").save(",[158,1214,416],{"class":164},[158,1216,1200],{"class":197},[158,1218,421],{"class":179},[158,1220,635],{"class":172},[158,1222,427],{"class":179},[158,1224,1225],{"class":197},".png'",[158,1227,346],{"class":172},[14,1229,1230],{},"Then it opens the PNGs and describes what it sees. There is no separate\nevaluation layer. The model that wrote the code\nlooks at the result.",[14,1232,1233],{},"Some things only exist in the image. While building the folder dialog, the new icons were missing\nfrom every screenshot after the first one. No state field knows about icons;\nthe bug was in the icon atlas budget, which was never reset\nheadless.",[14,1235,1236,1237,1241],{},"Other things are missing from the state only because nobody has asked for them yet. A\nscreenshot from the explorer showed two bugs at once. A click on\na file went through an old code path that wrote the text into the buffer of the\n",[1238,1239,1240],"em",{},"previous"," tab; the previous file was then considered modified and\nshowed someone else's content. And after closing the active tab, the\neditor kept showing the content of the closed tab, but under the name of its\nneighbor.",[14,1243,1244,1245,1248,1249,1252,1253,559],{},"The JSON state could not reveal either one, because it only reported which\ntab is active, not which buffer the editor is currently showing. So the fix\ncame with an extension: ",[18,1246,1247],{},"get_active_tab"," now also returns\n",[18,1250,1251],{},"editor_file"," and ",[18,1254,1255],{},"editor_modified",[14,1257,1258,1259,1262],{},"This has become a pattern. The screenshot finds the bug, a new\nstate field pins it down. Next time, a ",[18,1260,1261],{},"check"," fails, and\nnobody has to look anymore.",[14,1264,1265],{},"Where the appearance itself matters, the test checks the pixels. The\nbuttons below the PDF page light up on hover. The script\ntakes one screenshot with the mouse next to them and one with the mouse on them\nand compares the color at the same spot. No image library is\nneeded for that: PPM is a short text header followed by raw RGB bytes.",[38,1267,1269],{"id":1268},"when-the-test-itself-lies","When the test itself lies",[14,1271,1272],{},"Not every red test means the app.",[14,1274,1275,1276,1279,1280,1283],{},"In the PDF test, the page numbers jumped between two calls without any\npage turning. The cause was in the RPC server. It bound the port with\n",[18,1277,1278],{},"reuse_address",", and on Linux that also sets ",[18,1281,1282],{},"SO_REUSEPORT",". Orphaned\ninstances from earlier runs kept listening, the kernel distributed\nthe connections, and some of the responses came from an old process with\nold state. Today the app binds the port exclusively, and a second start\nreports that it is in use.",[14,1285,1286],{},"Other traps are smaller, but they are all in the project documentation\nso the agent does not have to rediscover them:",[46,1288,1289,1297,1300],{},[49,1290,1291,1292,1294,1295,559],{},"Clay keeps the bounding box of an element that is no longer\ndrawn. Whether a dialog is open is therefore answered by ",[18,1293,131],{},", not\n",[18,1296,143],{},[49,1298,1299],{},"The icon atlas rasterizes at most four new icons per pass. The\nscreenshot helper function therefore renders twice.",[49,1301,1302],{},"A page change only appears in the next frame. Tests poll the\nstate in a short loop instead of waiting once and hoping.",[14,1304,1305,1306,1308],{},"Sometimes the test also finds a real bug that it triggers itself.\nWriting a single screenshot to ",[18,1307,1130],{}," produced around 2000\nfile events. Each one kicked off its own Git status refresh,\nthe queue filled up, and any other task could be\ndropped, including the chat's responses. Since then, the watcher reports identical\nevents only once, and the app refreshes the Git status at most\nonce every 300 milliseconds. For the same screenshot, out of 2037\nevents, two remain and one refresh.",[38,1310,1312],{"id":1311},"the-workflow","The workflow",[14,1314,1315],{},"My instructions to the agent set test-driven development as the\ndefault: first a test that fails, then the smallest code that makes it\npass, then clean up. On a UI, that means in practice:",[1317,1318,1319,1322,1325,1333],"ol",{},[49,1320,1321],{},"The logic behind the feature moves into a module without Clay, with\nunit tests.",[49,1323,1324],{},"For the path through the UI, an E2E script is created, or a new\nstep in an existing one. If the script lacks a state it needs to\ncheck, the app first gets the RPC method for it.",[49,1326,1327,1328,1330,1331,559],{},"The agent runs the script, reads the ",[18,1329,1112],{}," lines, changes the code\nand restarts until everything reports ",[18,1332,1108],{},[49,1334,1335],{},"It looks at the screenshots. If something stands out there, it goes back to\nstep 2.",[14,1337,1338],{},"The commits contain test and code together, so the order is not visible\nin the history. What is visible, however, is why the second level\nis necessary. PDF page navigation came with a clean, unit-tested\nmodule. In the app it still felt jumpy. The fix brought the\nE2E script along with three causes no unit test could see: The server thread read\nstate it did not own, the mouse wheel counted backwards, and\nClay's hover detection reported nothing at all in the PDF view.",[38,1340,1342],{"id":1341},"what-does-not-run-automatically","What does not run automatically",[14,1344,1345,1346,1349,1350,1353],{},"No hook starts the tests after a change, and there is no CI.\nThe agent decides which scripts to run based on the change. For that,\n",[18,1347,1348],{},"AGENTS.md"," lists for each feature which script covers it. One\nrule is phrased as mandatory: After every change to the agent code,\n",[18,1351,1352],{},"scripts\u002Fe2e_ai_tools.py"," must stay green. That too is an instruction, not\nautomation.",[14,1355,1356],{},"A full run of all 23 scripts is not a fixed step. That is the biggest\nopen gap: A change to the explorer that incidentally breaks the terminal tab\nonly gets noticed when someone runs the matching script.",[38,1358,1360],{"id":1359},"what-you-need-for-your-own-app","What you need for your own app",[14,1362,1363],{},"None of this depends on Zig. Anyone who wants to develop a native desktop app with an agent\nneeds roughly this:",[46,1365,1366,1369,1372,1375,1378,1381,1384],{},[49,1367,1368],{},"A mode without a window that runs through the same loop as with a window.",[49,1370,1371],{},"A local interface for input that arrives in the main thread, not\nin the server's thread.",[49,1373,1374],{},"Read calls that return the UI state as JSON, and\none that reveals the position of an element.",[49,1376,1377],{},"Screenshots to a file, in a format the model can read.",[49,1379,1380],{},"Fresh state per test run and a port that only one instance can\nhold.",[49,1382,1383],{},"A model that understands images.",[49,1385,1386],{},"A file that says which script covers which feature and\nwhich trap is already known.",[14,1388,1389,1390,1396,1397,1400,1401,559],{},"The source code is open: ",[23,1391,1395],{"href":1392,"rel":1393},"https:\u002F\u002Fgithub.com\u002Fgstrainovic\u002Fzid",[1394],"nofollow","github.com\u002Fgstrainovic\u002Fzid",".\nThe RPC methods are in ",[18,1398,1399],{},"src\u002Fe2e_server.zig",", the scripts under\n",[18,1402,1403],{},"scripts\u002Fe2e_*.py",[1405,1406,1407],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}",{"title":154,"searchDepth":186,"depth":186,"links":1409},[1410,1411,1412,1413,1414,1415,1416,1417,1418],{"id":40,"depth":186,"text":41},{"id":82,"depth":186,"text":83},{"id":103,"depth":186,"text":104},{"id":791,"depth":186,"text":792},{"id":1119,"depth":186,"text":1120},{"id":1268,"depth":186,"text":1269},{"id":1311,"depth":186,"text":1312},{"id":1341,"depth":186,"text":1342},{"id":1359,"depth":186,"text":1360},"Test-driven development with an AI agent on a native desktop app. Headless mode, JSON-RPC as a remote control, and screenshots the agent looks at itself.","md",{},true,"\u002Fopen-source\u002Fki-testet-ohne-fenster",null,{"title":5,"description":1419},"open-source\u002Fki-testet-ohne-fenster","8yeCZs4E-trFmkFQ1Uat_6abyDymqdvsusXAuf1A_d0",1790517213220]