Quick Summary
- AI often finds many issues, a portion are highly useful; the rest still take time to review
- Custom prompts let you tell the AI what to focus on and what to ignore (e.g. your documentation style)
- A prompt library applies your preferences across every job so you don’t re-enter them
- Each run gets better: refine the prompt after the first project and improve hit rate on the next
Real feedback from firms running AI plan check: “We got about a hundred useful things. The annoying thing was we had to double-check every one of them.” When the tool finds 250 issues and 100 are actionable, you still spend time reviewing the other 150. Custom prompts and prompt libraries are how you dial up the hit rate and cut that review time.
The Hit Rate Problem
AI plan checkers surface a lot. Some findings are exactly what you need: code conflicts, coordination errors, missing details. Others are technically correct but not useful for your workflow: documentation conventions you don’t follow, local interpretations you’ve already decided, or issue types you’ve agreed to ignore. The more of those you get, the more time your team spends clicking through and verifying “is this real or not?”
One engineering firm ran a large set and saw about 100 useful issues out of 250. They fixed most of the useful ones. The cost was the time to review all 250, including 3–4 hours × 2 people to double-check the ones that weren’t useful. The ask: “How do we get it down to 100 useful out of 150 instead of 100 out of 250?”
Use Custom Prompts to Focus (and Ignore)
A custom prompt is instructions you give the AI before it runs: what to emphasize, what to skip, and how to treat certain situations. For example:
- Ignore documentation style. “We draw 8-inch walls for block; we don’t draw 7⅝". Don’t flag nominal 8-inch vs 7⅝" as an error.”
- Skip certain issue types. “Don’t include ASTM reference checks for this project” or “Don’t flag door tag format, we use a different convention.”
- Focus on high-value checks. “Prioritize fire separation and egress; we’re most concerned about life safety on this job.”
Once you’ve run a job and seen which findings you consistently dismiss, you can add those to the prompt for the next run. The goal isn’t to hide real problems, it’s to stop surfacing things that don’t add value for your team so you spend less time on non-issues and more on the ones that matter.
Prompt Library: Reuse Across Jobs
Typing the same instructions into every project is tedious and easy to forget. A prompt library lets you save a set of instructions (e.g. “South Florida residential + building code mix,” “Ignore 8\" wall documentation,” “Skip ASTM refs”) and apply it to your account so it’s used on every check. You refine it once, and every job benefits. Firms that run 5–10 checks a week see the hit rate improve by the second or third run as they add “don’t include X” and “focus on Y” to their library.
JSO / internal guidelines
Some customers already have internal plan-check or JSO guidelines (tone, formatting, issue types). Those can be adapted into a prompt or prompt library so the AI aligns with how your team already works, another way to improve hit rate and consistency.
Each Run Gets Better
You don’t have to know every exclusion upfront. Many users don’t know what to exclude until they’ve seen one full run. So the workflow is: run the first project, note which issues you consistently mark as “not useful” or “ignore,” then add those to your prompt (or library) for the next project. On the next run you should see fewer non-issues and a higher share of useful findings, e.g. 100 useful out of 150 instead of 100 out of 250. That’s less time reviewing and the same (or more) value.
Summary
Improving AI plan check hit rate isn’t just about the model, it’s about telling the tool what matters for your office. Custom prompts and prompt libraries let you ignore the noise (documentation style, ASTM refs, local conventions you’ve decided against) and focus on the issues you actually fix. Refine after each run and you’ll spend less time double-checking and more time fixing the things that matter.