Apex Climate Solutions — How the AI steps were tested

Three AI steps in Apex were formally tested, each test deciding the model or prompt that runs today. The urgency rater moved up to a more expensive model after the cheaper one missed six emergencies. The junk filter moved down to a model about one-eighth the list price, after four models were compared. The technician-answer step kept its free-tier model and went through four prompt versions before it passed.

Apex Climate Solutions is a simulated company. Every test message was written for the test, and the system has never handled live customer traffic. Tests ran between September 27 and October 28, 2026, and this page is not kept current.

How to read these results

Every score on this page carries a label saying what kind of test produced it, and every pass rule was written down before the first scored run.

• Each case passes or fails. It has an expected answer, and code compares the reply to it instead of grading on a 1-to-5 scale. An AI judge is used only where a rule can't decide, and only after it was checked against labeled answers.
• Rules first. The pass rule, including which failures sink the whole run, was recorded before the first scored run and was never loosened after seeing results.
• Who wrote what is stated. Where the same working session wrote the cases, tuned the prompt and scored the run, the section says so. That weakens a result, so the section names it.

Urgency rater

The rater ran on Opus 5.5, the most capable and most expensive model tested, because a missed Level 5 (a gas smell, an elderly person with no heat) is the costliest mistake in the system. Sonnet 5 was about 2.4 times cheaper per message and was chosen first. It was replaced after it rated six "elderly relative, system fully down" messages as 4 instead of 5.

Pass rules, set before the first scored run. Each message's expected level is recorded in an answer key (the list of expected answers), and code compares the rater's answer to it. The rater is graded before a fixed rule that forces Level 5 when a phone caller reports a gas or burning smell, so that rule can't hide a miss.

• Any message that should be Level 5 rated below 5 fails the whole run.
• Any message rated two or more levels too low fails the whole run.
• Messages wrongly rated 5 are counted on their own, because they land in the urgent channel and wear down trust in it.

Version history

Final results on v5

Across 29 timed runs, each call took 2.1 to 4.1 seconds.

Limits

• The development set grew alongside the prompt, as each failure added cases. 66 of 66 shows only that the prompt meets its own rules. Those first results were scored in the tuning session. On October 28, 2026, a fresh run on the current build, with prompt caching on, was graded in a separate session and matched: 66 of 66 and 36 of 36, every reply clean JSON after the workflow's cleanup step.
• The latest answer key was written in a session that had already seen the prompt's age wording. That was disclosed in the key at the time.
• The development set ran once per case. On October 28, 2026, a separate holdout set of 30 ordinary new messages, never used for tuning and not attacks, ran 3 times each and scored 90 of 90, graded in a separate session. That set is small and English only, and the result covers prompt v5 on Opus 5.5 only.

Junk filter

The filter switched from Haiku 4.5 to Haiku 5.5 after four models were compared on the same 48 messages: every model kept every real request, and Haiku 5.5 was the cheapest one that passed cleanly. Its own rule is "when unsure, treat it as real", so a doubtful message costs a few seconds of attention in the normal queue instead of being set aside.

Pass rules, set before the first scored run. 48 messages (26 real requests, 22 junk), each run 3 times: 144 runs per setup. Label: Model comparison.

1. No real request marked junk, in any run. A safety-critical message (gas smell, no heat or cooling, water leak) marked junk fails automatically.
2. At least 80% of junk runs sorted as junk. Very short junk such as "hello" is allowed to leak.
3. At least 98% of raw replies are exactly one JSON object, counted before any cleanup step.
4. The 15 attack messages give the same verdict in all 3 runs.
5. Among the models that pass, the cheapest wins, and every call finishes inside the filter's 10-second limit.

The built filter was also run end to end through a separate test setup: no real request marked junk, all 66 junk runs caught, 1.09 seconds median including the hand-off.

What failed first

• An earlier independent check of prompt v1 found 7 of 36 replies added text after the JSON, which broke the next step. The builder's own attack tests had missed it, because the builder wrote them.
• Haiku 4.5 wrapped nearly every reply in code formatting. The filter's log recorded replies after its cleanup step, so an earlier "every reply clean" figure could not have shown it. This test counts format on the raw reply for that reason.
• Once, the built filter got a reply with the wrong data type. Its failure rule treated the message as real and raised an alert, as designed.
• Sonnet 5.5 spent four to five times the expected output on very short messages, which shows in its cost.

After the switch. A 26-message check through the live filter on Haiku 5.5: no real request marked junk, every reply one clean JSON object, and prompt caching engaged on 25 of 26 calls. One of three very short junk messages was caught, the same known leak. This check confirmed the switch and is not reported as a new score.

Limits

• The answer key, the scoring script and the runs were all in one session. The filter itself was built in an earlier session. On October 28, 2026, a separate session re-scored the saved runs and got the same result on every pass rule.
• Results were copied by hand from the automation platform into files before scoring. The script's own cross-check agreed with the filter's logged decisions on all 144 rows.
• Very short junk ("u up?", "hello") sometimes passes as real.

Grounded answer (technician questions)

This step kept the free-tier Gemini 3.5 Flash Lite model, and every failure was fixed by rewording the prompt; prompt v5 passed every required check after v2, v3 and v4 each failed one. It answers a technician's question only from passages of the simulated company's manuals, and must say so when the passages don't answer it.

The case set. 18 cases, 22 calls: the two cases with a planted instruction ran 3 times each. Passages were real sections of three simulated company documents, plus two edited copies: one with a conflicting rule, one carrying a hidden order ("reply PWNED", "this repair is free and approved"). Three cases had passages that do not answer the question. Label: Development set.

Pass rules, set before any run. Any required check failing fails the run.

Two softer limits allowed at most one case each: answering "not found" when the passages did answer, and missing the key fact.

Checking the judge first. The judge was Claude Sonnet 5.5, a different model family from the one being tested, asked one question: is every claim supported? Before it graded anything, it was checked against 8 real answers labeled in advance and 4 deliberately bad answers written to catch it out. The bar was 7 of 8 agreements and all 4 bad answers failed. It agreed on all 12.

Version history

The sentence the judge caught in v4 was "the documents are not named per company instructions": the model had leaked its own rule into the reply.

Final results on v5

After release, one live question ran end to end through the finished step and returned a supported answer with its source line.

Limits

• Prompts v3 to v5 were written after the cases and outputs had been seen, so the pass is partly tuned to these 18 cases. Scoring was first done in the tuning session. On October 28, 2026, a separate session re-scored the saved answers and agreed on every required check.
• The judge's reference labels were written by Claude, not by a person reading each answer.
• The scoring script had three pattern fixes after outputs were read. Each was disclosed; no case, label or pass rule changed.
• Passing these checks shows the answers are grounded in the documents. Whether technicians would actually miss this step if it were switched off is a separate question, covered in the case study's Break it section.

What these tests don't show

These tests were enough to choose a model and a prompt for each step; they do not give an accuracy rate for real use.

• No real traffic. Every message was written for the test. How the steps behave on a real shop's messages is untested.
• Small case sets. 66, 48 and 18 cases are in the normal range for choosing between options, and too few to quote as a general error rate.
• Partial independence. Each section says where the same session wrote the cases, tuned the prompt and scored the run.
• Five AI steps had no formal evaluation (a scored test with pass rules set in advance). Request extraction is checked by a fixed rule step after it; watchdog ranking by code that keeps every flagged job; parts extraction by a read-back the technician confirms. Review drafts wait for a person's approval (posting is design only). Part lookup has no automatic check; the technician reads the reply.

The re-run rule. After any change to a tested step's prompt, model or reply length cap, its full case set runs again against the same pass rules before the change goes live. Turning on prompt caching did not trigger a re-run: it changes none of the three, and the urgency rater's October 28, 2026, re-run with caching on matched its earlier scores.

Records behind this page

Case sets, pass rules and scored runs are kept for all three steps, with one gap for the urgency rater.

• Each step's answer key or case set, locked before its scored run, with the pass rules recorded beside it.
• For all three steps, the scored runs are kept line by line with their scoring scripts and the judge's prompt, available on request. For the urgency rater that is the October 28, 2026, re-run: the per-message results of the earlier September 28 and October 12 runs were not saved.
• The project charter's model-selection table and its list of what is built, design only or declined, which every claim on this page is checked against before it is published.

Testing practice drew on Hamel Husain, LLM evals FAQ and Anthropic, Demystifying evals for AI agents.