Keldura engineering
734 pull requests a month with AI agents: the hidden GitHub bill nobody warned us about
Real numbers from building keldura.ai: what shipping ~24 PRs a day with AI agents does to your Actions invoice, your review-bot quotas, and your CI design.
On July 13, our repo merged 62 pull requests in one day.
The engineering team is me.
The implementation is split between Claude Code and Codex — with Grok 4.5 recently joining, and becoming genuinely good at delivering end-to-end: sticking to the goal, following instructions, finishing the PR. Grok also runs part of our local review pass. The allocation rule is less glamorous than any agent-architecture diagram you've seen this year: whoever still has quota gets the work. Mostly that's Claude Code and Codex.
Five subscriptions power this — Claude Max, Codex Pro, SuperGrok Heavy, Google AI Pro, JetBrains AI Pro — and here's the first thing nobody tells you: five is barely enough. One or two days a month everything hits zero at once and work just… stops. We survive on Codex's frequent quota resets and Claude Max's recently doubled limits, not on headroom.
The subscriptions were the bill I expected. This post is about the ones I didn't.
Part 1: The bill arrives
When I audited a week of GitHub Actions runs, one number explained everything: of our last 200 workflow runs, ~83% were orchestration — crons, sweeps, status publishers — not tests. The root cause is a single pricing rule: on private repos, every job bills a minimum of one minute. At agent volume, your bill is driven by how many jobs start, not how long they run.
One story to show the shape of it. We split our frontend test suite across 4 runners and measured an 85-second wall-clock win. What we didn't measure: each shard paid its own full clone of a 176 MB repo, its own apt-get, its own npm ci — and a build-cache bypass we'd added for a fair A/B experiment, then forgot to turn off. Net: ~16–20 extra billed minutes per run, to save 85 seconds. We'd left the lab equipment running in production.
The same audit found a cron billing ~2,900 no-op minutes a month, a "blocked run" cost floor that had silently grown 67% past what our own cost reporter could classify, and a workflow_run trigger double-firing on everything. All ten traps, with fixes:
Part 2: We graded our review bots. The only "Critical" was a false positive.
Four AI review bots comment on every PR. We scored 596 of their findings across 90 merged PRs against the one ground truth that can't be gamed: did the maintainer actually accept it? No keyword matching, no trusting the bot's own severity badge — which was the right call, because the only finding labeled "Critical" in all 596 was a false positive.
| Bot | Findings | Accepted (of answered) | High-value | Noise |
|---|---|---|---|---|
| Codex | 209 | 95% | 78% | 3% |
| CodeRabbit | 111 | 82% | 18% | 20% |
| Qodo | 131 | 67% | 22% | 43% |
| Gemini | 145 | 56% | 12% | 55% |
The spread is wild, but two subtler findings changed more of our behavior. No single bot is enough: 65% of flagged locations were caught by exactly one bot — mid-table Qodo was alone on a migration that would have clobbered admin-owned rows and a real Postgres deadlock. And noise isn't free: our branch protection requires every thread resolved before merge, so a wrong comment still costs an agent a full round to answer — on top of one bot's 172 auto-summaries averaging 9,500 characters.
Result: Codex as anchor, Qodo scoped to backend/migrations with summaries muted, Gemini demoted to on-demand fallback. The transferable 5-step grading method:
(One repo — Scala + Angular — one 90-PR window, our configs. Your ranking may differ. The method won't.)
Part 3: Stop feeding the meter
The loop that connects the two bills: every push re-triggers every bot and a CI cycle. One hard PR took 27 fix rounds. Rounds × bots × CI = exhausted quotas, a fat invoice — and no review coverage left for the next PR.
The fix is one variable. Cloud quota per PR ≈ commits the bots see × number of bots — and only the first factor is yours to control:
Two receipts from running this loop, and one trick: the local review pass caught ~50 defects pre-push on one heavy PR (each would have been a 15–30 minute cloud round); holding CI until reviews converge cut ~68% of billed minutes — after which the bots became the critical path at 18.5 min median vs 6.5 for tests (you don't eliminate the queue, you move it); and flipping a PR to draft whenever the next push isn't a merge candidate makes iteration free, because bots skip drafts.
One scar: a quota-dead bot must never block a merge — which means a PR can merge with zero remote review, silently. It happened: a degraded fallback said "0 findings" twice where the anchor bot, quota restored, found 11 real issues. Degraded coverage must be explicit, never silently green.
Steal this
- Count jobs, not workflows — every job bills ≥1 minute on private repos.
- Re-count your cheapest path's jobs after every workflow refactor. The floor creeps.
- Measure sharding in billed minutes, never wall-clock.
cancel-in-progress: trueon event-triggered orchestration; crons are backstops, not engines.- Skip CI on drafts; use drafts to shield bot quota during iteration.
- Grade review bots by maintainer acceptance — anchor one, scope the specialists, demote the rest.
- Review locally on subscriptions; spend cloud quota on merge candidates only.
- Gate expensive CI on review convergence, with ceilings so nothing wedges.
- Build the cost reporter first — and verify it by running it, not reading it.
AI agents made one developer ship like a twenty-person team. The infrastructure underneath still bills and reviews like it's one human pushing twice a day. Until the platforms catch up, the meter is part of your architecture — design for it the way you design for latency.
We build Keldura — living AI knowledge bases that turn the sources you follow into cited answers, briefings, and cheatsheets.