[{"data":1,"prerenderedAt":627},["ShallowReactive",2],{"blog-\u002Fblog\u002Fgreen-light-is-not-verification":3},{"id":4,"title":5,"body":6,"date":615,"description":616,"extension":617,"meta":618,"navigation":619,"path":620,"seo":621,"stem":622,"tags":623,"__hash__":626},"blog\u002Fblog\u002Fgreen-light-is-not-verification.md","綠燈不等於驗證：同一個專案裡，同一種失敗形狀重複了四次",{"type":7,"value":8,"toc":598},"minimark",[9,17,25,28,33,36,43,45,49,58,77,83,86,89,94,96,100,103,113,128,138,153,156,159,166,184,187,189,193,200,211,214,221,242,255,258,261,271,275,278,291,301,308,318,325,332,334,338,352,355,374,381,387,390,397,411,415,426,436,439,441,444,447,514,520,523,529,535,541,548,550,554,557,560,567,577,579,583,586,589],[10,11,12],"p",{},[13,14],"img",{"alt":15,"src":16},"四個打勾的 checkbox，後面的路徑都斷在半空中","\u002Fblog\u002Fimages\u002Fgreen-light-is-not-verification-hero.png",[10,18,19,20,24],{},"172 個測試全綠。",[21,22,23],"code",{},"send_message"," 的 checkbox 打了勾。專案筆記上寫著「DC-4 done」。實際上這個功能從來沒有真的送出過一則訊息——三個 bug 疊在一起，全被 mock adapter 遮住。而這只是同一個專案裡的第一次。接下來兩天內，同一種失敗形狀又出現了三次，每次穿不同的偽裝。",[26,27],"hr",{},[29,30,32],"h2",{"id":31},"專案背景一句話","專案背景：一句話",[10,34,35],{},"我把自己在用的 LINE TUI 工具泛化成一套聊天基礎建設，叫 chatmux——一個 daemon 管多個平台 adapter，資料落地成 SQLite + JSONL，對外用 MCP 暴露給 Claude Code 和 Neovim。現在是三個 public repo，1637 則訊息、143 個對話在裡面跑著。",[10,37,38,39],{},"寫程式的部分不是這篇的重點。",[40,41,42],"strong",{},"這篇是關於我四次以為自己驗證完了。",[26,44],{},[29,46,48],{"id":47},"第一次mock-綠燈","第一次：mock 綠燈",[10,50,51,53,54,57],{},[21,52,23],{}," 這條路徑跨三層：MCP tool 層組 request、core 轉發給 adapter、adapter 呼叫 linejs 的 ",[21,55,56],{},"sendCompactMessage","。",[10,59,60,61,64,65,68,69,72,73,76],{},"三個 bug 剛好一層一個。tool 層送出的 payload 少了 ",[21,62,63],{},"content"," wrapper，直接丟 ",[21,66,67],{},"{ chat_id, text }","；adapter 把 ",[21,70,71],{},"sendCompactMessage(to, text)"," 當兩個參數傳，但 linejs 要的是一個物件；chat_id 的 ",[21,74,75],{},"line:"," prefix 沒被剝掉。",[10,78,79,80,82],{},"三個都不會被測到。因為每一層的測試對面都是 mock：mock adapter 不檢查 payload 有沒有 wrapper，mock 的 ",[21,81,56],{}," 不在意你傳幾個參數，測試裡的 chat_id 本來就沒帶 prefix。",[10,84,85],{},"三層都綠。整條鏈是斷的。",[10,87,88],{},"後來是為了另一件事跑端對端 spike，才發現這個功能從沒送出過任何東西。補的方式是一個 env-gated 的 live integration test，走真實 linejs、真的發一則訊息給自己。然後做 mutation check：把三個 bug 其中一個還原回去，確認測試會紅，再還原回來確認回綠。",[10,90,91],{},[40,92,93],{},"測試要證明自己有牙，不能只證明自己會綠。",[26,95],{},[29,97,99],{"id":98},"第二次文件寫得很合理","第二次：文件寫得很合理",[10,101,102],{},"chatmux 的招牌是「MCP-native」，旗艦 demo 是「Claude Code 直接當 client 讀你的聊天記錄」。v0.1 的驗收清單上，這條打了勾。",[10,104,105,106,109,110],{},"問題是 MCP server 只 ",[21,107,108],{},"listen(socketPath)","——綁死在 unix socket 上。而 MCP spec 只定義兩種標準 transport：stdio 和 streamable HTTP。",[40,111,112],{},"Claude Code 根本沒有一個欄位可以填 unix socket。",[10,114,115,116,119,120,123,124,127],{},"那 README 和 ",[21,117,118],{},"docs\u002Fmcp-interface.md"," 裡教人填的那段 ",[21,121,122],{},"\"socketPath\": \"...\""," config 是什麼？是一段讀起來完全合理、但從來沒有任何人跑過的設定。當初驗收是用 Bun 的 ",[21,125,126],{},"fetch({ unix })"," 自己打 unix socket 打通的——那是 Bun 專有能力，Claude Code 沒有。",[10,129,130,131,134,135,57],{},"補的方式是加一個 TCP listener（只綁 ",[21,132,133],{},"127.0.0.1","），unix socket 保留給 Neovim 側的 sidecar。這次先寫失敗測試，而",[40,136,137],{},"紅燈的形狀本身給了我一個沒預期到的資訊",[10,139,140,141,144,145,148,149,152],{},"舊的 ",[21,142,143],{},"startMcpServer"," signature 收一個 string，我把新的 options 物件直接丟進 ",[21,146,147],{},"httpServer.listen()","，Node 竟然接受了——結果綁在 ",[21,150,151],{},"*:47717","，所有網路介面。七個測試裡四個「意外通過」，三個紅。",[10,154,155],{},"想清楚這意味著什麼：如果我的測試只斷言「TCP 連得上」，這個把全部聊天記錄暴露到區網的 bind 會全綠通過。",[10,157,158],{},"所以那次之後我寫斷言會多列一條：這個東西不該綁到哪裡去。",[10,160,161,162,165],{},"完成判定我這次沒有用測試綠燈，而是真的跑 ",[21,163,164],{},"claude mcp add --transport http","，開一個 headless Claude Code session、刻意只給 read-only tool，讓它對真實資料查詢。",[10,167,168,169,172,173,176,177,176,180,183],{},"差點又栽在這裡。第一輪 ",[21,170,171],{},"search_messages"," 查「會議」回 0 hits。0 hits 可能是真陰性——語料裡就是沒這個詞；也可能是 CJK 全文搜尋壞掉，而那個形狀跟前面兩次一模一樣。我用語料中已知存在的詞複驗：",[21,174,175],{},"stale_ratio"," 93 hits、",[21,178,179],{},"過高",[21,181,182],{},"報價"," 139 hits。CJK 正常，「會議」是真陰性。",[10,185,186],{},"CJK 搜尋是好的，「會議」就是真的沒出現過——但我沒查第二次，就不會知道是哪一個。",[26,188],{},[29,190,192],{"id":191},"第三次宣告-optional","第三次：宣告 optional",[10,194,195,196,199],{},"寫第二個 adapter 的時候，我刻意不看第一個 adapter 的原始碼，只照 ",[21,197,198],{},"adapter-protocol.md"," 蓋。換語言、換 repo、換平台語義。",[10,201,202,203,206,207,210],{},"這招很有效——逼出 21 條 LINE 特有假設，其中 5 條是「文件沒寫但行為存在」的隱式假設。協定升到 v0.2，最招牌的一條裁決是把 ",[21,204,205],{},"get_message_boxes","（一個 LINE 專有 API）改成 optional：adapter 不支援就回 ",[21,208,209],{},"-32601","，core 跳過。",[10,212,213],{},"看起來很泛化。三個問題疊在一起。",[10,215,216,217,220],{},"第一個是小的：偵測用字串比對錯誤訊息，沒有讀 JSON-RPC 的 ",[21,218,219],{},"error.code","。第三方 adapter 換個措辭就會被當成硬失敗。",[10,222,223,224,227,228,231,232,234,235,238,239,241],{},"更麻煩的是，",[40,225,226],{},"文件承諾的 fallback 寫不通","。文件說 core 會「直接用 ",[21,229,230],{},"get_chats"," 結果做 backfill 排序」，但 ",[21,233,230],{}," 的 response schema 裡沒有任何時間欄位，而 backfill 就是 ",[21,236,237],{},"ORDER BY last_message_at DESC","。誠實回 ",[21,240,209],{}," 的 adapter 會拿到全部 chat 的排序欄位為 NULL，冷啟動的 500 則預算花在隨機對話上。",[10,243,244,245,247,248,251,252,254],{},"最意外的一條：Telegram adapter 也實作了 ",[21,246,205],{},"。內容是 ",[21,249,250],{},"get_dialogs()"," 的第二份包裝，跟它自己的 ",[21,253,230],{}," 同源。所以那條 fallback 路徑一次都沒被執行過，零測試守衛。",[10,256,257],{},"而表面證據看起來像泛化成功——兩個平台的 adapter 都跑得起來。",[10,259,260],{},"判準其實很簡單：要問的是「沒支援的那一邊會怎樣」，不是「有沒有人支援」。第二個實作如果是第一個的複製，泛化程度是零。",[10,262,263,264,266,267,270],{},"閉環的做法是把 Telegram 那份刪掉，讓 fallback 第一次真的被走過，同時給 ",[21,265,230],{}," 加 optional 的 ",[21,268,269],{},"last_message_at","，協定升 v0.3。",[272,273,274],"h3",{"id":274},"舊資料會替你演出已被刪除的元件",[10,276,277],{},"這條線最險的一步在驗證方法，不在程式碼。",[10,279,280,281,283,284,287,288,290],{},"我的 DB 裡 Telegram 那些 chat 的 ",[21,282,269],{}," ",[40,285,286],{},"已經有舊值","——過去 ",[21,289,205],{}," 寫進去的。刪掉 handler 之後，在現有 data dir 上跑，一切正常。排序完全看不出退化。",[10,292,293,294,297,298,300],{},"要換一個乾淨的 ",[21,295,296],{},"CHATMUX_DATA_DIR"," 冷啟動，才看得到真相。結果是 30 個 chat、30 筆 non-null、0 筆假值，回填順序與 ",[21,299,237],{}," 的前五名逐位相同。",[10,302,303,304,307],{},"推廣出去：",[40,305,306],{},"任何「移除資料來源」的改動，都必須在沒有該來源歷史殘留的環境驗證。"," 刪 cache 層、換 migration、移除 denormalized 欄位——舊資料都會扮演那個已經被你刪掉的元件。",[10,309,310,311,314,315,317],{},"而同一條線上，我還差點自己製造一個一樣的掩護。為了保護既有值不被 null 覆寫，我寫了 ",[21,312,313],{},"MAX(COALESCE(a,0), COALESCE(b,0))","。意圖是對的。副作用是：兩者皆 null 時它會寫入 epoch 0，而我的驗證判準正是「",[21,316,269],{}," 有值」。",[10,319,320,321,324],{},"也就是說，防呆機制差點讓驗證判準自己失效。外層包一個 ",[21,322,323],{},"NULLIF(..., 0)","，讓「沒有訊號」保持 NULL、讓退化看得見。",[10,326,327,328,331],{},"抓到這個不是因為更小心。是因為問了一句：",[40,329,330],{},"這個判準在功能完全壞掉的時候，會不會仍然通過？"," 會通過的判準不是判準。",[26,333],{},[29,335,337],{"id":336},"第四次掃描零命中","第四次：掃描零命中",[10,339,340,341,344,345,344,348,351],{},"三個 repo 要轉 public。開工前的 pre-flight 掃描報告寫得很乾淨：憑證檔案（",[21,342,343],{},".env","、",[21,346,347],{},"*.session",[21,349,350],{},"data\u002F","）全歷史從未被 commit、無真實個資、不需要重寫歷史。",[10,353,354],{},"實際上有三處洩漏。",[10,356,357,358,361,362,365,366,369,370,373],{},"第一處是真實的 Telegram ",[21,359,360],{},"API_ID"," \u002F ",[21,363,364],{},"API_HASH","，硬寫成常數放在一個 integration 測試檔裡。不是 placeholder，是真值。",[21,367,368],{},"git log -S"," 查出來是某個 commit 引入的，",[40,371,372],{},"至今仍在 HEAD","——十三個 commit 裡最後七個都含它。改 HEAD 不夠，public 之後任何人都能從 history 撈出來。",[10,375,376,377,380],{},"第二處是真實的 Telegram user ID，寫在 ",[21,378,379],{},"docs\u002Ftesting.md"," 的範例裡。第三處是真實的 LINE user ID，寫在另一個 repo 的 spike 註解裡。",[10,382,383,384],{},"為什麼掃描報「乾淨」？",[40,385,386],{},"因為它掃的是憑證「檔案」和「姓名／電話／email」，inline 憑證常數和平台帳號 ID 不在字典裡。",[10,388,389],{},"零命中只證明你搜的那些東西不在。",[10,391,392,393,396],{},"這次的運氣是時間差。憑證是在 push 之前十分鐘發現的。Telegram 的 ",[21,394,395],{},"api_hash"," 沒有 rotate 機制，推出去就是永久的；而當時 repo 還沒有 remote、沒有任何下游 clone，所以重寫歷史的成本是零。晚十分鐘發現，就會變成「已公開的憑證 + 無法撤銷的金鑰」。",[10,398,399,400,403,404],{},"順手記一條在 repo 文件裡的告誡：",[21,401,402],{},"describe.skipIf"," 不保護任何東西。",[40,405,406,407,410],{},"被 skip 的測試一樣會被 commit、push，一樣會被 ",[21,408,409],{},"git log -p"," 讀到。",[272,412,414],{"id":413},"驗證乾淨的方法有兩層而且都不看本機","驗證乾淨的方法有兩層，而且都不看本機",[10,416,417,418,421,422,425],{},"第一層：不看本機工作區，從 GitHub ",[40,419,420],{},"重新 clone","，再對四個敏感字串跑 ",[21,423,424],{},"git log --all -S","。三個 repo 全部 0 命中。",[10,427,428,429,432,433,57],{},"但這只證明「現在可達的歷史是乾淨的」。GitHub 會保留曾經 push 過的 unreachable object——舊 SHA 的 URL 照樣打得開。所以第二層：用 ",[21,430,431],{},"gh api repos\u002F\u003Cowner>\u002F\u003Crepo>\u002Fcommits\u002F\u003Csha>"," 去打那四個重寫前的 SHA，全部回 ",[40,434,435],{},"422 No commit found",[10,437,438],{},"這才證明含憑證的舊 commit 從來沒抵達過 GitHub。",[26,440],{},[29,442,443],{"id":443},"四次的共同結構",[10,445,446],{},"機械檢查通過（測試綠／文件合理／掃描零命中）→ 宣稱完成（checkbox 打勾）→ 真實路徑從沒被執行過 → 事後覆核才發現。",[448,449,450,466],"table",{},[451,452,453],"thead",{},[454,455,456,460,463],"tr",{},[457,458,459],"th",{},"次",[457,461,462],{},"偽裝",[457,464,465],{},"實際上沒被驗證的是",[467,468,469,481,492,503],"tbody",{},[454,470,471,475,478],{},[472,473,474],"td",{},"1",[472,476,477],{},"172 test 全綠",[472,479,480],{},"三層都對著 mock，整條鏈沒跑過",[454,482,483,486,489],{},[472,484,485],{},"2",[472,487,488],{},"文件的 config 讀起來合理",[472,490,491],{},"沒有任何 client 用那個欄位連過",[454,493,494,497,500],{},[472,495,496],{},"3",[472,498,499],{},"標了 optional、兩個 adapter 都跑得起來",[472,501,502],{},"fallback 路徑一次都沒走過",[454,504,505,508,511],{},[472,506,507],{},"4",[472,509,510],{},"pre-flight 掃描零命中",[472,512,513],{},"掃描字典裡沒有那三種東西",[10,515,516,517],{},"四次的偽裝完全不同。共同結構只有一句：",[40,518,519],{},"我驗證的是「我想到的東西」，不是「實際會發生的事」。",[10,521,522],{},"三條可以直接拿去用的判準——",[10,524,525,528],{},[40,526,527],{},"一、這個判準在功能完全壞掉時，會不會仍然通過？"," 會通過的就不是判準。epoch 0 冒充「有值」，mock 冒充「送出去了」，都是這個問題沒問。",[10,530,531,534],{},[40,532,533],{},"二、退路拿不拿得到它需要的東西？"," 標 optional、寫 fallback、留 degraded mode——只寫清楚主路徑不算，要問「不做那件事的人，用什麼替代」。答不出來的 optional 是把責任丟給別人，還讓文件看起來很泛化。",[10,536,537,540],{},[40,538,539],{},"三、零命中只證明你搜的東西不在。"," 掃描、grep、空結果，全部適用。要嘛擴充搜尋維度，要嘛用已知存在的東西複驗一次。",[10,542,543,544,547],{},"順帶一個規模參考：這條線從 172 個測試走到 235 個，",[21,545,546],{},"tsc --noEmit"," 的 19 個既有 error 從頭到尾沒有增加，協定從 v0.1 走到 v0.3。四次修正沒有一次是靠「更小心」，全部是靠換一個會露出真相的驗證環境。",[26,549],{},[29,551,553],{"id":552},"誰抓到的這件事得講清楚","誰抓到的，這件事得講清楚",[10,555,556],{},"四次都是在宣稱完成之後才被抓到的，但抓到的人不是同一個。",[10,558,559],{},"前三次是我在督導 session 裡覆核時發現的——讀 AI 寫的完成報告，看到「checkbox 全勾、測試全綠」，然後問下一句：這條路徑真的被執行過嗎。第四次不是我，是執行 session 自己在做別的事（把範例配置裡的絕對路徑去個人化）時順手掃到憑證的。",[10,561,562,563,566],{},"我不想把這寫成「人類監督 AI」的漂亮故事。真實的分工比較尷尬：",[40,564,565],{},"AI 很擅長讓機械檢查通過，也很擅長把「檢查通過」寫成「完成」。而我很擅長在它交報告時偷懶不追問。"," 四次裡有三次，是因為我剛好在覆核而不是剛好在趕進度。",[10,568,569,570],{},"專案在這裡，包括那四次的完整施工紀錄都在 repo 的追蹤文件裡：",[571,572,576],"a",{"href":573,"rel":574},"https:\u002F\u002Fgithub.com\u002Fechoedinvoker\u002Fchatmux",[575],"nofollow","github.com\u002Fechoedinvoker\u002Fchatmux",[26,578],{},[29,580,582],{"id":581},"如果你的團隊也在用-ai-寫程式","如果你的團隊也在用 AI 寫程式",[10,584,585],{},"那你大概已經遇過這個問題了：AI 交回來的東西測試全綠、報告寫得很漂亮，但你不確定它到底有沒有真的跑過那條路徑。而你沒有時間逐條追問。",[10,587,588],{},"如果你正在把 AI coding 導進團隊流程，卡在「怎麼定完成判定」這一關，可以找我聊——沒有簡報，也不賣套裝方案，先確認你的問題長什麼形狀再談解法。加 LINE 直接描述你的狀況就好，十分鐘大概就能知道值不值得繼續談。",[10,590,591,592,597],{},"我的顧問 LINE 本身就是我自己做的：",[571,593,596],{"href":594,"rel":595},"https:\u002F\u002Fmattchang.dev",[575],"mattchang.dev","。你跟它的對話，就是你正在評估的那個東西的 live demo。",{"title":599,"searchDepth":600,"depth":600,"links":601},"",2,[602,603,604,605,609,612,613,614],{"id":31,"depth":600,"text":32},{"id":47,"depth":600,"text":48},{"id":98,"depth":600,"text":99},{"id":191,"depth":600,"text":192,"children":606},[607],{"id":274,"depth":608,"text":274},3,{"id":336,"depth":600,"text":337,"children":610},[611],{"id":413,"depth":608,"text":414},{"id":443,"depth":600,"text":443},{"id":552,"depth":600,"text":553},{"id":581,"depth":600,"text":582},"2026-07-25","測試全綠、checkbox 打勾、文件寫好——然後發現那條路徑從來沒被執行過。我在同一個開源專案裡踩了四次，每次偽裝不同。這是四次的拆解和三條可以直接套用的驗證判準。","md",{},true,"\u002Fblog\u002Fgreen-light-is-not-verification",{"title":5,"description":616},"blog\u002Fgreen-light-is-not-verification",[624,625],"testing","claude-code","AQYTq3yLEjhzx1qrTXoxIPQwj7ex1RmzrEqKrbEDLVk",1784946204529]