How to measure agentic AI maturity in your engineering org

Measuring agentic AI maturity in an engineering org means scoring two things separately: how much of each workflow an agent runs without a person, and how well the actions it takes are controlled. Adoption surveys measure neither. A team can be 100 percent on coding assistants and still have no inventory, no policy that decides anything outside the model, and no record of what the agents did last Tuesday.

That gap is common and it is measurable. In the Stack Overflow 2026 survey, which drew more than 30,000 respondents, 66 percent of AI users name coding assistants and agents as their top use case, ahead of chatbots at 63 percent, and 73 percent use them daily. The same survey reports 48 percent trust AI output only when they can easily verify it. Trust is conditional, and the evidence behind it is exactly what a maturity measurement should check.

The quick win: score one workflow this week

Do this before any survey, any steering committee and any tool purchase.

1. Pick one workflow your engineering org runs every day: bug triage, first-pass code review, dependency updates or test authoring.
2. Answer 20 questions about it, 18 on how the work runs today and 2 on the controls around it. Alyria publishes the same tracks free at alyria.ai: the software development assessment covers plan, design, implement, test, deploy and observe.
3. Read the phase with the lowest score, not the total. That is your next piece of work.
4. Write down one sentence per control question: what evidence has to exist before a high-impact action, and whether policy can stop an action based on what the session already did.

An afternoon produces a defensible answer for one workflow, and it usually contradicts what people assumed. Repeat it for a second workflow next month rather than scoring the whole org at once.

What adoption numbers miss

Assistant seat counts, edit-acceptance rates and lines-of-code suggestions all measure activity. None of them measure whether the work is finished by the agent or by a person cleaning up after it, and none measure control.

The reliability literature is a useful corrective here. On the tau2-bench leaderboard, Claude Opus 4.5 solves 79.61 percent of retail tasks on a single attempt, and 51.75 percent of tasks succeed on all four of four attempts. The best entry on the Berkeley Function Calling Leaderboard scores 77.47 percent overall and 68.38 percent on multi-turn tasks. An agent that is right most of the time is not an agent you can leave unattended on every workflow, and maturity includes knowing which workflows those are.

The security literature explains the same point from the other side. In the largest public agent red-teaming competition, 1.8 million attacks produced more than 60,000 successful policy violations across 22 frontier agents, and the authors report that nearly all agents violate policy for most behaviours within 10 to 100 queries (Gray Swan and UK AISI, arXiv 2507.20526). Measurement that ignores this reports a fleet that is productive and does not report a fleet that is exposed.

The two axes, and why they stay apart

Automation is who performs the workflow. Level 0 means a person does the work, or the workflow does not exist yet. Level 1 means a person prompts the AI each time and checks the result. Level 2 means an agent runs the workflow end to end when it is triggered.

Controls are two questions scored on their own scale of 0 to 2. First, what externally verified evidence must exist before a high-impact step can proceed. Second, can runtime policy stop an action because of the state of the task and what the agent already read.

Keeping them apart is not pedantry. A program at level 2 with controls at 0 is the combination that shows up in incident reviews, and averaging the two into one score hides it. The agentic AI maturity model sets out the levels and the controls expected at each one.

What to measure, and what each measurement proves

Measurement	What it actually proves	Where it comes from
Share of workflows an agent completes end to end	Automation level, per workflow	Workflow owner answers, checked against a sample run
Agents, CLIs, versions and MCP servers per endpoint	Whether the inventory is real and current	Endpoint inventory, not a spreadsheet
Policy decisions per week, by rule and disposition	Whether policy is deciding anything	Decision logs with rule ids attached
Rules still in monitor mode after 30 days	Whether enforcement is actually promoted	Console state per rule category
High-impact actions with recorded approval	Whether approvals are designed or accidental	Audit records for deploys, payments, merges
Exported and verified audit records	Whether evidence survives the laptop	Audit export, hash verification

Two numbers deserve special suspicion: the count of "AI tools in use" and the count of "policy violations prevented". The first is usually scraped from a directory and misses local MCP servers, which live in client config files. The second is meaningless without a monitor period, because a rule that has never been run against real traffic has never been tested.

Collect evidence, not opinions

For automation, ask the person who owns the workflow whether the agent finishes it or they do, then verify by watching one run. For controls, four artefacts answer most of it:

• An inventory of AI tools and MCP servers on a sample of endpoints, including their versions. Alyria Beacon produces this read-only in about 30 minutes, and one command removes it, which makes the sample easy to take without committing to a rollout.
• The written policy, in whatever form it exists, plus the rule ids attached to recent decisions.
• The decision log for a week, including decisions in monitor mode that were never enforced. Lyra evaluates policy locally at the endpoint, so those decisions exist even for calls that never reach the cloud.
• The audit export, run once by someone who is not the program owner, and verified.

If the fourth item takes more than an hour to produce, that is the finding. Evidence that cannot be exported under pressure is not evidence.

Score the controls honestly

The two control questions are where most engineering orgs score 0 or 1, and the honest answer is usually 1: something is automated or advisory, and nothing enforces the whole decision at runtime with a record.

Consider the shape of the risk. On the Gray Swan indirect prompt injection benchmark, an attacker with 15 attempts succeeded against Claude Opus 5 two percent of the time, and GPT-5.6 Luna reached 43.9 percent (Anthropic Transparency Hub). For a fleet running several models, a success rate that low within 15 attempts still means attacks succeed at fleet scale, which is why the control question is about what happens after the model is wrong rather than whether the model can be wrong.

Tool poisoning is the same argument from the supply side. Across 20 agents tested against 45 live MCP servers, the highest attack success rate reached 72.8 percent and no model refused more than 3 percent of the attacks (MCPTox, arXiv 2508.14925). A separate study of 1,899 open-source MCP servers found 7.2 percent with general vulnerabilities and 5.5 percent showing MCP-specific tool poisoning (MCP at First Glance, arXiv 2506.13538). An org that cannot list its MCP servers cannot score above 0 on either control question, whatever its agents can do.

Run the measurement on a cadence

Quarterly is too slow for a fleet that changes weekly, and continuous is too noisy to govern by. A workable cadence:

• Monthly: re-run the inventory diff and the policy decision review. Two questions only: what appeared, and which rules decided something unexpected.
• Quarterly: rescore the 20 questions for the two or three workflows that matter, and confirm the audit export still verifies.
• On every agent rollout: ask which of the five controls the new tool changes. If the answer is unknown, the rollout is not ready.

Alyria supports the cadence with the parts a marketing page cannot: Beacon keeps the endpoint inventory current and evaluates Lyra policy on every model and MCP call locally, Spectra routes the decisions to Observatory or your own SIEM over OpenTelemetry, and the audit chain is signed and hash-chained so a verifier can check it later.

Common measurement mistakes

• Counting seats as adoption. Seat counts measure procurement, not automation.
• Scoring workflows you have not watched. The owner's description and the actual run differ most at the edges: retries, partial failures and the hand-off back to a person.
• Averaging controls into the automation score. It produces a comfortable number and hides the gap that matters.
• Skipping monitor mode in the assessment. If a rule has never run in monitor mode, the org does not know what it would block.
• Measuring only engineering. The same 20-question shape exists for security operations, procurement, customer service and financial operations, and a fleet does not stop at the engineering boundary.

Frequently asked questions

How many workflows should we score first?

Two. One that is already automated, so the control questions have something to score, and one that is still manual, so the automation question is honest. More than two turns the exercise into a survey.

Is this the same thing as a DORA or SPACE metric?

No. Those measure delivery performance. This measures who performs a workflow and how agent actions are controlled, which is a different question needing different evidence.

We use four agent vendors. Does that change the score?

It changes the cost of the controls, not the levels. Four vendors means four sets of client-side settings that drift apart, which is why the inventory and the policy layer need to be cross-vendor. Alyria governs Claude Code, Cursor, Codex, GitHub Copilot and whatever ships next quarter from one policy.

What is the single most useful number?

The count of high-impact actions in the last 30 days that have a recorded approval attached. It is small, checkable, and answers both control questions in one go.

Can we measure without installing anything?

Yes for automation: the workflow owner and one observed run cover it. Controls are harder, because a written policy is a claim until something evaluates it against real traffic.

Get a full Alyria maturity assessment

Alyria assesses your agents and controls across functions using Compass (compass.alyria.ai), and delivers a scored maturity report and a roadmap. Talk to us about a full assessment, or start with the free track for your function at alyria.ai/assessments.