Methodology & honest metrics
An independent auditor only works if you can check its work.
Spendassay is the neutral, cross-vendor layer between what a company spends on AI and what its engineering teams actually ship. Neutrality is a claim you have to earn on the page, not assert in a logo. This is how every number is built, what tier of evidence stands behind it, and the lines we will not cross to make a chart look better.
How we measure
Six layers, from raw coverage to a dollar-denominated range
No single metric survives contact with a skeptical CFO. Spendassay builds value up in six layers, each one auditable on its own. The higher the layer, the more inference it carries — so the higher the layer, the weaker the evidence tier we are willing to stamp on it.
Seats provisioned, active vs. dormant, license and API-key inventory across every vendor. The denominator for everything above. Straight from billing and admin APIs.
Suggestions surfaced and accepted, tokens consumed, sessions and prompts, feature mix. Read directly from vendor telemetry. A signal of engagement, never yet a claim of value.
PR throughput, cycle time, review latency, deploy frequency — team-level, from Git and CI metadata. What the org would track with or without AI.
Rework and churn, bugs per PR, unreviewed-merge rate, incidents per PR. Rendered beside — never after — the throughput numbers in L3. This layer is mandatory, not optional.
Delivery and quality deltas priced using loaded engineering cost, then set against real AI spend. Wherever possible the delta is measured against a matched no-/low-AI cohort inside the same company rather than an industry constant.
Net value as a band, not a point. Every input is editable, every formula is visible, and the output always carries a sensitivity range. This is the only layer we let be modeled — and we label it as such, every time.
The evidence-tier ladder
Every number Spendassay renders carries one of three tiers. The tier is not decoration — it tells you exactly how much inference sits between the raw log and the figure on screen. We never let a lower tier borrow the authority of a higher one.
Read directly from a system of record — billing, vendor telemetry, Git, CI, the incident tracker. No modeling, no estimation. If two people pulled the same window, they would get the same number. Coverage, usage, delivery, and the quality counter-metrics all live here.
A measured difference between two comparable groups — an AI-heavy team against a matched low-AI cohort inside the same company, same period, same work profile. Stronger than an industry benchmark because the comparison is internal, but it still attributes a gap to AI, so it is a tier below measured.
A projection built on explicit, editable assumptions — dollar values, adoption curves, discount rates. Useful for planning and honest about its own uncertainty. Modeled figures always ship with a sensitivity range and never appear without this chip.
What we never do
The constraints are the product
An auditor is defined by what it refuses to do. These are hard architectural limits, not settings — they survive a works-council review and a security questionnaire because they are enforced below the UI.
No individual developer league tables
Spendassay reports at the team level, always. There is no per-person ranking, no surveillance leaderboard, no way to slice a metric down to a single engineer. Productivity data that can be turned against a person is not something we are willing to build.
No ingesting your code content — metadata only
We read commit and PR statistics: counts, timestamps, churn, review events. We do not read, store, or transmit diffs, source, or file contents. The audit runs entirely on metadata, which is all the measurement actually requires.
Read-only — we never mutate customer systems
Spendassay pulls from billing, telemetry, and Git/CI through read-scoped access. It cannot merge a PR, change a license, revoke a seat, or write to any customer system. An auditor that can change the books is not an auditor.
Team-level with a minimum cohort of 8
No cohort smaller than eight people renders a productivity figure — a hard floor, not a default. Below it we suppress the number rather than risk re-identifying an individual from an aggregate.
Why counter-metrics are mandatory
A throughput number without its quality tax is propaganda
The easiest way to lie with an AI dashboard is to show acceleration and stop there. More PRs, shorter cycle time, more epics closed — all true, all rising, all meaningless until you know what they cost. So Spendassay renders every throughput gain beside its counter-metric, on the same surface, at the same time. You cannot see the speedup without seeing the tax.
Rework / churn ≤ 21 days
Lines rewritten or deleted within three weeks of merge. AI-drafted code that never stabilizes is negative throughput wearing a green badge.
Bugs per PR
Defects traced back to a pull request. Shipping faster only counts if the defect rate holds.
Unreviewed-merge rate
Share of PRs merged without a substantive human review. Velocity bought by skipping review is debt, not delivery.
Incidents per PR
Production incidents attributable to a change. The truest tax on acceleration — and the one that spikes first.
This is not a hypothetical. In Faros AI's 2026 analysis, teams that pushed epic throughput up with AI saw incidents per PR rise roughly 3x and code churn climb sharply over the same window. The velocity chart looked like a win; the instability chart, published beside it, looked like a warning. A single-metric dashboard would have shown only the first. That gap is the entire reason counter-metrics are product law here, not a toggle a customer can turn off.
The ROI model is a range, not a number
Any tool that hands a CFO a single ROI figure is selling something
Layer 6 is the only place Spendassay models, and it does so out loud. The output is a band with a payback window, every input is editable, and the formula is printed on the same page as the result.
The L6 formula
Net value = ( Δ delivery value − Δ quality cost ) − AI spend
Δ delivery value = throughput delta (L3)
× loaded engineering cost per unit
Δ quality cost = counter-metric delta (L4)
× cost per rework / bug / incident
AI spend = seats + tokens + infra (L1)
ROI range = Net value ± 20% (assumption sensitivity)Editable assumptions
Loaded cost per engineer, cost per rework hour, cost per incident, adoption ramp — all inputs, all editable. Change one and the range recomputes in front of you. We ship defaults; we never hide them.
±20% sensitivity
The headline is always a band, not a point. The ±20% envelope makes the model's uncertainty legible instead of laundering it into false precision.
The single-counting rule
A given hour of saved effort is counted once. Time saved cannot also be booked as new features shipped and as headcount avoided. Double-counting is how AI ROI decks reach implausible numbers; we forbid it.
The J-curve — payback in 6–18 months
Adoption gets worse before it gets better: a verification-tax dip as teams learn to review machine-drafted code, then payback. Spendassay models the dip explicitly and places typical payback in a 6–18 month window rather than promising day-one returns.
The research we build on
Standing on the published record, not on our own benchmarks
Spendassay does not ask you to trust a proprietary productivity score. The method is assembled from the public research on AI and software delivery — including the studies that complicate the optimistic story. We cite the work that constrains us, not just the work that flatters us.
METR
2025 / 2026Developers using AI tools estimated a 20% speedup on real tasks; the measured effect was a 19% slowdown. The perception gap — not the tool — is the thing to audit. We report ranges, never single-point speedups, because the primary study on the question found self-report and measurement pointing in opposite directions.
Faros AI — "Acceleration Whiplash"
2026Teams that accelerated epic throughput with AI saw incidents per PR rise roughly 3x and code churn climb sharply. The finding that makes counter-metrics non-negotiable: throughput and quality move on different clocks, and the quality bill arrives later.
GitClear
2025 / 2026Large-corpus analysis showing rising code churn and duplicated ('copy-pasted') blocks coinciding with AI-assistant adoption. Grounds our churn and duplication counter-metrics.
DORA — State of DevOps
2025AI is an amplifier, not a cause: it magnifies the throughput and the instability of the system it lands in. Strong delivery practices get faster; weak ones get faster at breaking. Our normalization reads AI's effect against a team's existing DORA baseline.
CACM / Ziegler et al.
Copilot acceptance rate correlates with perceived productivity but is not itself a productivity measure. We treat acceptance and suggestion-usage as coverage signals (Layer 1–2), never as delivery outcomes.
LinearB — APEX
Documents the AI adoption J-curve and the 'verification tax': an early dip as teams learn to review machine-drafted code, then payback. Source of our 6–18 month payback window and single-counting discipline.
Microsoft — Viva / workplace analytics
Establishes minimum-cohort floors for reporting team analytics without re-identifying individuals. Basis for our hard 8-person cohort minimum.
DX — AI Measurement Framework
A multi-dimensional frame — utilization, impact, cost, and developer experience — that resists collapsing AI value into one number. Shapes the layered stack rather than a single headline metric.
See the method run against a real audit
Every number in the demo carries its evidence tier, every throughput chart carries its counter-metric, and every ROI figure is a range you can re-derive. Check our work.