Skip to content
Menu
Case study

Apex Climate Solutions: automating intake and operations for a heating and cooling company

A portfolio demonstration, not a product I maintain. Apex is a simulated company, every test message was written for the test, and no live customer traffic has run through it. Figures are as of the publish date.

AI Agent Manager / AI OpsShow all

If you're hiring an AI Agent Manager, these three sections show how the three main AI steps were tested, what happens when one gets it wrong, and what it costs to run.

  1. Tests first
  2. Break it
  3. Cost page
AI Project / Program ManagerShow all

If you're hiring for AI delivery, these three sections show how I planned, tracked and corrected the work as it went.

  1. Record
  2. How I build
  3. Cost page
AI Product ManagerShow all

If you're hiring an AI Product Manager, these three sections show what I chose, what I rejected, and the evidence for each.

  1. Decisions
  2. Tests first
  3. Build
AI Automation SpecialistShow all

If you're hiring for automation, these three sections show how failures come to light, what I built, and how I documented the work.

  1. Break it
  2. Build
  3. As-built

Apex is a system of 27 Make.com workflows for a small heating and cooling (HVAC) shop: it triages the requests the shop receives, answers technicians' questions from the company's own manuals, and flags jobs, stock and permits before they slip. It shows where AI belongs in that work and where a person decides. I built it on Make.com, Neon databases and Slack, with eight AI steps: three of them, the urgency rater, the junk filter and the technician answer step, went through formal evals (scored against fixed test sets), and the other five sit behind a fixed check or a person's review.

Time freed
30 hrs /month
staff hours freed for a five-technician shop (modeled)
Saved
$12,420 /year
saved after paying its own running cost (modeled) · see the cost page
Cost to run
$59 to $121 /month
for a five-technician shop (estimate)

Who owns each step

CUSTOMERFIXED RULESAI CALLSOFFICE STAFFTECHNICIANSends arequestRecords andchecks itSorts andrates itAssigns thetechnicianSays theparts usedReads thepartsConfirms theparts usedDrafts areview replyApproves everypublic reply
Technician question, from field to answer
FIXED RULESAI CALLSTECHNICIANAsks a fieldquestion in SlackFinds matchingpassagesAnswers from them,citing the sourceReads the answerand its source
  • Fixed rule
  • AI call
  • Person
  • Hand-off to a person
Gray boxes are fixed rules, purple boxes are AI calls, amber boxes are people. The AI sorts, rates, reads, drafts and answers. The human in the loop assigns the technician, confirms parts before stock changes, and approves every public reply. Below, technicians' questions are answered only from the company's own documents, with the source shown.
Step 1

Brief and constraint

A small HVAC shop loses time in three places: sorting incoming requests by hand, technicians hunting for answers, and problems noticed only after they cost a job.

  • Intake. Requests arrive by phone, web form and WhatsApp, and a person reads each one, so a no-heat emergency can sit behind a routine maintenance request.
  • Field knowledge. Refrigerant procedures, city permit rules and warranty terms live in people's heads. Technicians phone the office or guess.
  • Oversight. Overdue jobs, empty truck stock and permits stuck in pending too long come to light only after they cause a missed job.

I set three constraints: one simulated client; free tiers wherever they held up; and, once the free plan proved too small, a ceiling of 10,000 Make.com operations (billed workflow steps) a month.

Three things reach a customer before a person sees the request: the voice agent's live conversation, which follows a set script to gather the caller's details; the WhatsApp acknowledgment; and the form confirmation email. After that, each request is recorded and checked by fixed rules, unknown senders also pass the junk filter, and real requests are rated and posted to Slack, where a person reviews and decides.

Step 2

Map

Three components share three Neon databases and one Slack workspace: intake and triage, technician answers, and an operations watchdog.

Three Neon databasesIntake and triageTechnician answersOperations watchdogOne Slack workspace
  • Component
  • Store
  • Slack or input
Three components, three Neon databases, one Slack workspace

Here is one request, step by step:

  1. Saved first. The message is written to the database before anything judges it, which also catches duplicates.
  2. Screened by rules. A rule-based filter checks it, then the sender is looked up by exact match against known customers.
  3. Junk check, for unknown senders only. A message from someone who isn't a customer goes through the junk filter (an AI call).
  4. Read and rated. One AI call pulls out the name and equipment, and another rates the urgency. A fixed format check inspects every AI reply.
  5. Handed to a person. The request is posted to #urgent or #normal-triage in Slack, where a person reviews and decides.
Database record;catches duplicatesRule-based filterExact-match customer lookupJunk filter(unknown senders only)Name and equipment extractionUrgency ratingFormat check on every AI replyPost to #urgent or #normal-triageA person takes it from there
  • Fixed rule
  • AI call
  • Person
  • Hand-off to a person
One request, from first message to the triage channel
Step 3

Decisions

Why this matters to you

Each choice names what it beat and why, including one I reversed when new evidence came in.

Next section: Tests first

I made each choice against a named alternative, for a stated reason. Model choices link to the test behind them.

  • Urgency rater on the most capable model. It started on the cheaper Sonnet 5, which matched on all 43 test cases. Two prompt versions later, it missed six emergencies, so I moved the rater to Opus 5.5.
  • Junk filter on the cheapest model that passed. I compared four models on the same 48 messages. Haiku 5.5 kept every real request at about one-eighth the cost of Haiku 4.5, the model it replaced.
  • Technician answers on a small, free-tier model, Gemini 3.5 Flash Lite. On the free tier, the larger Flash models were capped at 20 requests a day; Flash Lite allowed 500. An answer that cites nothing is blocked, and the technician sees the source it cites, so a bigger model would buy little. I fixed every failure by changing the prompt.
  • AI where judgment is needed, fixed rules everywhere else. I weighed running intake as a single AI agent, with a prompt listing the steps and the agent interpreting them and choosing a tool for each. I rejected it: no intake case needs an agent's loop of searching and retrying, and fixed steps keep each decision testable and auditable, with a plain record of every step and no risk of runaway cost. Intake instead runs on fixed rules wherever the logic is predictable, and reserves three single-purpose AI calls for the steps that need judgment: screening junk, extracting details and rating urgency. Every request still reaches a person before the company acts on it.
  • Make.com over Zapier and n8n. Make bills per operation, has visual branching, and lets one workflow (a Make "scenario") call another directly. Self-hosting n8n adds a server to run and secure.
  • Directus over Baserow for the office screen. When I checked during the build, Baserow couldn't sit on the existing database. Directus reads it as it is and limits each person to the columns they need.
  • Watchdog as one database function. My first design looped through every open job in Make, which would have cost about 57,000 operations a month, more than five times the 10,000 budget. I moved the checking into the database instead. One function applies every rule, from overdue jobs to stalled permits, and returns each neglected job with its reason; Make just calls it every 30 minutes through the working day. The rules live in one place, and I tested their time limits and stock thresholds on a copy of the database with a fixed test clock, without spending Make credits.
  • Prompt caching, adopted after measuring it. Caching lets the AI provider reuse the unchanging part of a prompt at a lower price. Testing it on a copy of the workflow found two silent problems: the original Make template couldn't carry a cache marker, and automatic caching would have marked the customer's message, which changes every time. I built a new template with the marker on the fixed instructions, then compared the 5-minute and 1-hour cache options and adopted the 1-hour one. It cuts the urgency rater's token cost by an estimated 63 to 73% (modeled, not yet measured over a real month).
Step 4

Tests first

Why this matters to you

The three tested AI steps each had a pass rule written before their first run, and one failed test changed the model.

Next section: Break it
Why this matters to you

Three model or prompt choices were each settled by a test that could have failed them.

Next section: Build

I designed the tests before building: 30 test cases, ten for each component, written with their expected results before any component existed. Writing them exposed three gaps in the design, which I fixed before the build. Three AI steps, the urgency rater, the junk filter and the technician answer step, then went through formal evals, with each pass rule set before the first scored run. Each eval refined the step's prompt, settled its model, or both.

Urgency rater

Moved up to Opus 5.5 after the cheaper model missed six emergencies. 66 of 66 on its development set. Then, on messages not used while writing its prompt, it rated all 12 trick messages and all 30 everyday messages correctly, in all 3 runs of the test.

Junk filter

Switched from Haiku 4.5 to Haiku 5.5, at about one-eighth the price. None of 26 genuine service requests was marked junk, in 3 runs each. 22 junk messages, also run 3 times each, were caught in 61 of 66 runs (92%). Very short junk like “hello” can slip through; it lands in the normal queue for a person to dismiss.

Technician answers

Kept the small, free-tier Gemini 3.5 Flash Lite. Prompt version 5 passed every required check after versions 2 to 4 failed.

Read full eval report

Read the full eval report: pass rules, every version, and what the tests don't show.

Eval card

Urgency rater

What it does
Rates each request 1 to 5
Pass rule
  1. Any message that should be Level 5 rated below 5 fails the whole run.
  2. Any message rated two or more levels too low fails the whole run.
  3. Messages wrongly rated 5 are counted on their own, because they land in the urgent channel and wear down trust in it.
Final score
  • 66 of 66 Development set
  • 12 of 12 trick messages rated correctly, in all 3 runs New cases, attack
  • 30 of 30 everyday messages rated correctly, in all 3 runs New cases, ordinary
Limits
The development set (the cases I tuned the prompt against) grew with the prompt, so 66 of 66 shows only that it meets its own rules. The fairer test used a separate holdout set: 30 everyday messages it was never tuned on. The whole set ran 3 times, and every message was rated correctly every time: 90 ratings, 90 correct.
Eval card

Junk filter

What it does
Sets aside messages that aren't service requests; when unsure, treats them as real
Pass rule
  1. No real request marked junk, in any run. A safety-critical message (gas smell, no heat or cooling, water leak) marked junk fails automatically.
  2. At least 80% of junk runs sorted as junk. Very short junk such as "hello" is allowed to leak.
  3. At least 98% of raw replies are exactly one JSON object, counted before any cleanup step.
  4. The 15 attack messages give the same verdict in all 3 runs.
  5. Among the models that pass, the cheapest wins, and every call finishes inside the filter's 10-second limit.
Final score
  • Real requests marked junk (of 78 runs): 0
  • Junk caught (of 66 runs): 61 (92%)
  • Raw reply exactly one JSON object: 144 of 144
Model comparison
Limits
Very short junk such as "hello" sometimes passes as real, by design.
Eval card

Technician answers

What it does
Answers from the company's manuals, citing the source
Pass rule
Any required check failing fails the run.
  1. Reply is one clean JSON object, citing only passages it was given
  2. Every number copied exactly from the passages
  3. The three unanswerable cases return "Nothing in the company documents answers this."
  4. Every claim is supported by the passages: no coverage promises, no rules blended across zones
  5. Instructions hidden in the test material are never obeyed, and those cases give the same answer in all 3 runs
  6. Known wrong values never shown
Final score
  • Prompt v5 passed every required check
  • One minor limit was reached by one case, within its allowance: a cautious "not found" answer
Development set
Limits
I wrote prompt versions 3 to 5 after seeing the test cases, so the pass is partly tuned to them.

Two of the three formal AI tests were first scored in the session that tuned them; all three were re-scored in a separate session before any result was published. The eval report says where a prompt was tuned on its own test cases. Scoring kept separate from building

Step 5

Build

Why this matters to you

What I built, what is design only, and where the AI steps hand off to a person.

Why this matters to you

Three of the 27 Make.com workflows, and a register of every AI step with what happens when it fails.

Next section: As-built

Each AI step has a stated job and a set path when the call fails; all but part lookup have a check after it.

The junk filter's steps (Make modules) laid out in the Make.com editor
The junk filter as a Make.com workflow (a Make "scenario"), in the editor

Junk filter

What it does
Sets aside messages that aren't service requests. When unsure, it treats the message as real.
One decision behind it
If the AI call fails or returns something unusable, the message goes through as real, so nothing is lost.
A Slack thread with a question and an answer quoting a section of the diagnostic guide
A technician's question, answered from the manuals with the source cited

Technician answers

What it does
A technician asks in Slack and gets an answer from the company's manuals, with the source cited.
One decision behind it
Cited sources come from database records, never from the AI's own text, and an answer that cites nothing is blocked.
A Slack thread where a technician asks whether they have run capacitors on their truck and the reply gives the count
A truck-stock lookup in Slack

Truck-stock lookup

What it does
A technician asks what's on a van and gets the count back.
One decision behind it
A fixed rule that matches nine phrases decides which questions are stock questions.

Agent register

Eight AI steps, plus the model that indexes the manuals for search. I matched each model to the cost of a wrong answer, not to how hard the task looks.

AI stepJobModel and whyCost of a wrong answerTested onChecked after byIf the call failsTime per callStatus
Junk filterSets aside messages that aren't service requests; when unsure, treats them as realHaiku 5.5, the cheapest that passedJunk let through: a few seconds in the normal queue. A real request marked junk: held for the next morning's digest (emergency words always pass)48 messages, 3 runs eachThe urgency rater and the dispatcher see everything it lets throughMessage treated as real0.71 s medianBuilt
Request extractionPulls the customer name and equipment out of the messageHaiku 4.5; low stakes, checked right afterA field left unmatched on the triage cardNo formal testA fixed database match; a near-match is shown as a suggestion, never appliedJob continues with those fields unmatchedNot measuredBuilt
Urgency raterRates each request 1 to 5Opus 5.5; the only step where a wrong answer can leave an emergency waitingA missed emergency66 development cases; 12 attack messages, 3 runs each; 30 everyday messages, 3 runs eachOn a voice call, a fixed rule forces Level 5 when the caller reports a gas or burning smell; the dispatcher decides what happensNo job created; raw request posted to #urgent for a person2.1 to 4.1 sBuilt
Technician answersAnswers from the company's manuals, citing the sourceGemini 3.5 Flash Lite, small and free-tier, with a daily limit that fits the demo; the citation check does the reliability workA wrong fix in the field18 cases, 22 calls; AI judge checked first, 12 of 12An answer that cites nothing is blocked; the technician sees the sourceApology to the technician; alert raisedNot measuredBuilt
Watchdog rankingOrders the digest when two or more jobs are flaggedGemini 3.5 Flash LiteJobs listed in the wrong orderNo formal testEvery flagged job reaches the digest whatever the ranking returnsOne retry; if still unusable, the digest posts unranked with a note saying so, and an alert is raisedNot measuredBuilt
Parts extractionReads the parts a technician says they usedGemini 3.5 Flash LiteA wrong stock countNo formal testRead back in the job thread; the technician confirms or discards before stock changes, and can undo afterApology to the technician; alert raisedNot measuredBuilt
Part lookupReads which part a stock question asks aboutGemini 3.5 Flash LiteA stock count for the wrong partNo formal testNo automatic check; the technician reads the replyOne retry; then "try again" to the technician and an alertNot measuredBuilt
Review reply drafterDrafts a reply to a customer reviewGemini 3.5 Flash Lite; the person who approves each draft is the quality checkA poor public reply, if approvedNo formal testA person approves every draft in Slack; Yelp replies are posted by handAlert raisedNot measuredBuilt (approval gate); pulling and posting Google and Instagram reviews is design only

The re-run rule. After any change to a tested step's prompt, model or reply length limit, its full test set runs again against the same pass rules before the change goes live.

Where a person decides: the urgency rating shows on the office screen as advice, and a dispatcher assigns the technician. Every review reply waits in Slack for a person's approval; posting to Google and Instagram is design only.

A drafted review reply in Slack with approve and discard reactions
A review reply waiting for a person's approval
A job record on the office screen showing its urgency rating, status and an empty assigned-technician field
A job on the office screen, with its urgency rating as advice

Built, design only, declined

Built and run on test messages

Intake on WhatsApp, web form and an AI voice agent (web calls only; no phone number is rented); urgency rating, junk filter and extraction; technician answers; the watchdog digest, its summary of flagged jobs (tested on a copy of the database); the loop that updates truck stock from parts used; the review approval gate.

Design only

Pulling in and posting Google and Instagram reviews; sending job records into Jobber; WhatsApp calling; a retention-locked copy of the audit log (one that can't be altered or deleted).

Declined on purpose

A replay queue for failed sends (a person resends by hand); multi-region hosting and failover; detecting a manual edited outside the system.

Step 6

Break it

Why this matters to you

When an AI reply is malformed or the call fails, a fixed rule takes a safe path, so the request still reaches a person.

Next section: Cost page
Why this matters to you

Most failures take a safe default or reach a person in Slack; the few that don't are named as accepted limits in the technical as-built.

Next section: Build

Most failures take a safe default or reach a person in Slack; the few that don't are named as accepted limits in the technical as-built. I designed the tests to find the AI's mistakes.

One failure route: if the urgency rater returns a reply that fails the format check, no job is created; the raw request is posted to #urgent for a person. Operational alerts use a fixed six-line format.

Urgency rater replyFormat check failsNo job created; raw requestposted to #urgent for a personAlert
  • Fixed rule
  • AI call
  • Failure route
When the urgency rater's reply fails the format check
A Slack alert naming the failed step, what happened and what to do next
A health alert in the six-line format

What the tests caught

  • The cheaper urgency model rated six "elderly relative, system fully down" messages as level 4 instead of 5.
  • An independent check of the first junk-filter prompt found 7 of 36 replies added text after the expected reply, which would have broken the next step. My own tests had missed it.
  • The junk filter's original model wrapped 140 of 144 replies in code formatting. A cleanup step had been hiding it.
  • Asked about a Lennox furnace, the answer step replied from a generic furnace entry. Two prompt versions later, it leaked one of its own instructions into a reply. The AI judge, a second model that grades each answer, caught it.
  • A test with a forged request crashed the check that verifies incoming messages, because the message body was empty. I fixed it the same day.

Only intake counts as essential. I asked of each component what would break if it were switched off. Intake is the only step where a delay costs a customer within minutes. For the watchdog the cost builds over days, and technicians can call the office, so I treat those two as important but not critical.

Step 7

Record

Why this matters to you

I dated design decisions with their reasons, noted where a reason was never recorded, and from mid-September kept every open item in a backlog stating what had to be true for it to count as finished.

Next section: How I build

I kept two records. Design decisions went into the charter's dated decision log with the reasoning at the time; where a setting's date or reason was never recorded, the record says so rather than filling it in afterwards. From mid-September, every open item went into a Jira backlog with a definition of done: what had to be true to close it.

The public charter holds the scope, the architecture, the list of what's built, designed only or declined, and the decision log. How I run this kind of work, from charter to map to backlog to build and test, is on the How I build automation page.

Supporting page

Cost page

Why this matters to you

I measured the automation budget, found it 16% over plan, and projected it back under plan after four fixes.

Cost page
Supporting page

How I build

Why this matters to you

How I plan, build and test an automation, with a check before I publish any claim.

How I build
Supporting page

Cost page

Why this matters to you

I estimated the budget, measured it over a month, then corrected it when the measurement came in 32 points higher.

Cost page
Supporting page

As-built

Why this matters to you

The write-up of the system as built: how it's wired, what happens when a step fails, and where it differs from the plan. About a 20-minute read.

As-built
Step 8

Post-project reflection

The four working rules I followed, from not grading my own work to checking every claim before publishing it, are on the How I build automation page.

What changes at real volume

  • Apex's running cost follows message volume, not team size. At 30 to 40 messages a day, every day, it would rise from about $59 to $121 a month to about $173 to $273, including paid Slack (estimate). The cost page breaks this down and compares it with other options.
  • The AI steps behind technician answers and the watchdog run on Google Gemini's free tier. Real customer data would need a paid Gemini plan; intake's AI steps already use Anthropic's paid API.
  • In a heat wave, the two technicians covering one zone can't meet a 15-minute emergency response, whatever the software does. Apex can only make the shortfall visible.

What I'd do differently

  • Before building, I'd talk to three to five shop owners about missed calls and job triage and what software they already use for the office work. Then I'd check which steps an existing tool already automates, so I build only what pays off and don't recreate what is already available.
  • I'd measure the automation budget in the first full week instead of estimating it. My estimate was 84% of plan; the first measured month came in at 116%.
  • I'd score every test in a separate session from the start. I first scored two tests in the session where I tuned them, and had to rescore them later. I'd also log every per-case result from the first run. I had to re-run the urgency rater's test because its per-case results weren't kept.
  • I'd start every step with the simplest tool that does the job: a fixed rule first, a single AI call only where a step needs judgment, and an AI agent (one that plans its own steps) only where nothing simpler works. Intake began as an agent design and, behind the voice agent, turned out to need AI at just three points: spotting junk, pulling out the customer's details and rating urgency. Everything else became fixed rules.