Skip to content
All work

Claire IT AI

A medical-coding AI assistant. I raised its retrieval hit rate from 28% to 75% and cut p90 latency by 40%.

Client
Claire IT AI
My role
Lead engineer
Team
3–4 engineers
Period
2025 – 2026

Results

  • 28%to 75%

    Retrieval hit rate, pasted charts+47 pt

  • 83sto 50s

    p90 answer latency−40%

  • 10 in 20to 0 in 20

    Failed chart searches−10

  • 17 of 30

    Reviewer-flagged failures repairedAuto-repaired

Context

Claire IT AI is an AI assistant for medical coders. A coder pastes a clinical chart or asks a question, and Claire assigns or explains ICD-10, CPT and HCPCS codes, maps them to HCC risk-adjustment categories, and grounds the answer in official guidelines.

It is a paid SaaS with Stripe tiers, with 146 accounts by September 2026. In this domain a plausible answer isn’t enough: the code has to be exactly right.

My role

Lead engineer and repo owner from May 2025 to October 2026, coordinating 3–4 contributors. I built the core platform, the RAG pipeline, answer verification, the eval harness and billing. I built the medical-code data platform solo.

The problem

I measured both problems on an eval set of real, coder-reviewed cases.

  • Retrieval. A client reported that pasting a full chart retrieved nothing. On 20 eval charts, half the searches failed, and confirmed codes appeared in retrieved passages only 28% of the time.
  • Speed and cost. Answers took 83 seconds at p90, and the answer call was most of the cost.

What I built

Each request runs through a pipeline: coding context from a pre-chat form, a retrieval orchestrator, the answer model, then verification before the user sees anything.

Claire IT AI request pipeline
  1. 01Chart or question
    • Pasted clinical chart
    • Coding context from a pre-chat form
  2. 02Retrieval orchestrator
    • Rewrite and decompose
    • One search per condition
    • Route to partitions
  3. 03Knowledge and codes
    • AWS Bedrock Knowledge Bases
    • ~169k codes in Postgres
    • HCC crosswalk, ~413k mappings
  4. 04Answer model
    • Context de-duplicated
    • Prompt caching
  5. 05Verification
    • Fabricated, non-billable, laterality and HCC checks
    • Buffered until it passes
    Loop: repair, then re-check
  • Data platform. Stealth scrapers (Camoufox, Apify) and resumable ingestion built a Postgres database of ~169k ICD-10, CPT and HCPCS codes.
  • Retrieval orchestrator. Rewrites and decomposes the query, decides whether the knowledge base is needed, routes to partitions and pulls matching codes. The knowledge base moved from Milvus to Ragie to AWS Bedrock Knowledge Bases; a teammate ran the Bedrock document migration.
  • Retrieval fixes. I traced the chart failures to five faults: quoted text treated as a required phrase, invisible Unicode spaces, over-long queries, the wrong fiscal-year code edition, and charts searched by their instructions instead of their conditions. A chart now becomes one search per condition.
  • Latency and cost. The knowledge-base context was sent twice per answer, and prompt caching never fired. I fixed both and upgraded the AI SDK.
  • HCC crosswalk. A deterministic ICD-10 → HCC lookup, ~413k mappings across 10 CMS/HHS models, so categories come from tables, not model memory.
  • Answer verification. Answers are buffered, checked for fabricated, non-billable, wrong-laterality and wrong-HCC codes, repaired by the main model, then checked again.
  • Eval harness. Coder-reviewed datasets run through the live pipeline with an LLM judge, deterministic HCC checks and prompt A/B tests.

Results

  • Retrieval hit rate for pasted charts: 28% → 75%.
  • Failed chart searches: 10 in 20 → 0 in 20.
  • p90 latency down 40% (83s → 50s) and cost per answer down ~14%, validated on the eval set before rollout.
  • Verification repaired 17 of 30 reviewer-flagged failures before users saw them.

What I’d do differently

  • Make the eval set a gate much earlier. The harness arrived about 16 months in, and a client found the chart faults before we did. Running it on every retrieval change, especially across two vector-store migrations, would have caught them first.
  • Track cost and cache hits per answer from the start. The duplicated context and dead cache went unnoticed until a cost audit.
  • Put lookups in tables from the first release. HCC categories were first answered by the model, and a reviewer caught mis-mappings. Anything deterministic belongs in data, not a prompt.