AI coding agent governance RFP: the checklist, with tests you can run

A governance RFP for AI coding agents (Claude Code, Cursor, the Codex CLI, GitHub Copilot) gets scored on slides unless the evaluators bring tests. The list below is a scoring sheet: seven requirement groups, each requirement phrased so a vendor either passes a named test or answers honestly that it cannot. Run the tests on one laptop inside the evaluation window. Most take an afternoon and none need the vendor's own environment.

Download the checklist as markdown to paste into your RFP template. It carries the same content without this page's context.

1. Inventory that survives the config files

The first question an RFP should force is about visibility, and the failure mode is a gateway list. MCP servers live in per-client config files on each laptop, and a stdio server never crosses the network, so anything that only sees proxied traffic under-reports by design.

• The vendor inventories every installed AI tool and version on the endpoint: Claude Code, Claude Desktop, Cursor, the Codex CLI, VS Code with GitHub Copilot, the Copilot CLI, and the ones nobody mentioned.
• It reads each client's own configuration scopes, including project scope, user scope, and administrator-deployed managed files.
• It sees stdio servers, which run as child processes and touch no network.
• For each server it records the transport (stdio, HTTP, SSE), the launch command or URL, whether the version is pinned, and the first and last time it was seen.
• It answers "what changed since last week" without being re-run by hand.

Test 1: the config sweep. Before the demo, run this on the evaluator laptop, from the live how-to how to find every MCP server on your developer laptops:

grep -rl --include='*.json' -e '"mcpServers"' -e '"servers"' ~ 2>/dev/null
grep -rn -e '^\[mcp_servers' ~/.codex/config.toml 2>/dev/null

Then ask the vendor to produce its own inventory for the same machine. Expected result: the vendor's list is a superset of yours, and down-level or unapproved entries are the same on both. A list that misses an entry you just added by hand is a demo feature, not an inventory.

2. Policy that runs at the call site

A deny only counts if it happens before the call executes and cannot be talked around. Anything evaluated after the fact is monitoring, and anything evaluated in the cloud fails when the laptop is offline.

• The vendor can deny a tool call (MCP call, shell command, or file read) before it executes, and the deny is final.
• The same policy covers every AI client installed on the machine, not only the one with a hook API.
• Policies are written as code and versioned, so a rule change has an author and a date.
• Enforcement starts in monitor mode: the deployed rules record what they would have blocked before they block anything.
• Sequence rules are expressible: "after this action, refuse that next action" (for example, after reading a secret, no network egress). The semantic policy examples page shows what that looks like.

Test 2: a deny that is final. With the vendor's product in enforcement mode, attempt the same intent in several shapes: cat .env, /bin/cat .env, head .env, and python3 -c "open('.env').read()". Expected result: every shape is denied, and the local record names the rule that fired for each. Vendors pass this on literal string matching only in the demo; ask them to re-run it after you obfuscate the command.

Test 3: offline enforcement. Disconnect the network, then trigger a rule with a stdio MCP server (a filesystem server reading outside its allowed root is enough). Expected result: the block still fires, because the decision was made on the laptop. A vendor that needs a round trip for the verdict will fail this test in the first minute and the explanation will usually be honest.

3. Secrets and credentials

Agents inherit whatever tokens sit in their configs, so an RFP that stops at scanning misses the actual fix, which is that the agent should never hold the credential.

• The vendor flags plaintext credentials in client configs (the env block of an mcpServers entry is the common case) and reports which endpoints have them.
• It offers short-lived leases instead of standing tokens: the agent receives the credential at the moment of use, and it expires.
• Credentials are never written into the model's context.
• Custody is verifiable: you can hold or revoke the key (BYOK or HYOK on enterprise terms) and confirm revocation in your own KMS or cloud audit log.

Test 4: the standing-token sweep. Run a task that needs a cloud credential (one AWS CLI call is enough) through the vendor's lease flow, then run a limited smoke check on the config files and credential prefixes above. Treat it as a prefix and config-file check, not as proof for the broader claim:

grep -rn "AKIA" ~/.aws 2>/dev/null
grep -rn "AKIA\|ghp_" ~/.claude.json ~/.cursor/mcp.json ~/.codex/config.toml 2>/dev/null

Expected result: nothing standing in the file locations the sweep reads, the lease visible in the vendor's audit record. This cannot establish a disk-wide or context absence claim: a github_pat_ token, a credential held by a tool outside these four paths, or a token a vendor leaks into the model's context will not match these greps. Score the disk-wide and context claims only on separate evidence from the vendor, or on your own DLP scan; the background is in keeping coding agents away from secrets.

4. Evidence an auditor accepts

If the vendor claims governance, its audit trail is the deliverable. GitHub's own documentation notes that Copilot's audit records do not include local tool calls, so an RFP should make the claim testable rather than take a screenshot as evidence.

• Every tool call and model decision is recorded with the rule that applied, the identity that made the call, and the client it came from.
• The record is tamper-evident in a way a third party can verify, not just immutable by insider policy.
• The chain covers every client on the machine in one place, not one log file per tool.
• It exports in a standard format (OpenTelemetry fits an existing SIEM) to where your SOC already watches.

Test 5: the tamper check. Have the vendor walk you through modifying one record in place: the expected result is a failed verification, not a memo. Then export one day of records to your SIEM staging environment and confirm the same policy decision is queryable there. The criteria are in what an audit trail for AI coding agents has to prove.

5. Native controls: the vendor's own position

The client vendors ship real controls, and a governance RFP should ask a candidate vendor to place itself honestly relative to them rather than pretend they do not exist.

• The vendor names the native controls per client: Claude Code managed settings and managed MCP configuration, Cursor hooks, Codex sandbox and approval policy, Copilot CLI hooks, and links the documents like they are public knowledge, because they are.
• It states what its layer adds beyond them, which for most implementations is fleet-wide visibility and one policy and audit layer across every client on the machine.
• It keeps the native controls running rather than replacing them; MDM stays in charge of the managed files.
• The rollout plan starts read-only and moves to enforcement per rule, with the false-positive rate seen in monitor mode first. The order is written up in the Claude Code post and generalized in the rollout docs.

Test 6: the monitor-mode week. Deploy in monitor mode on the evaluator cohort for a week. Expected result: a report of what the rules would have blocked, with counts, and zero developer-visible breakage. A vendor that skips this step is afraid of its own rules.

6. Runtime and fit questions

The last group separates platform claims from delivery facts. These do not need a test environment; they are scored on the answer alone.

• Where does inference happen for each agent, and what stays inside the customer boundary?
• What happens to enforcement when the laptop is offline? (Already tested, but the vendor should describe the same behavior.)
• What does the vendor not do: no kernel driver, no EDR replacement, no gateway replacement where a gateway is the right tool? The endpoint vs gateway vs cloud runtime guide is a fair baseline for what each layer is responsible for.
• Time to first inventory: about 30 minutes from install to a fleet report is the number to hold from any vendor offering an endpoint agent; anything longer is worth a why.
• Removal: one command, no residuals.

Try it with Alyria

Score us the same way as everyone else: install Alyria Beacon on the one evaluator laptop in monitor mode, run the config sweep from test 1, and compare its inventory with your own list before you decide anything. The install is read-only, first inventory takes about 30 minutes, and removing it is one command, so it costs the evaluation nothing. Start free and read where Alyria fits for what it will not do for you.

Scoring

Score each requirement pass, partial, or fail. A vendor that fails inventory (group 1) cannot compensate with a strong audit story, because you cannot audit what you never saw. Groups 2, 3, and 4 are the enforceable core: a vendor that passes those three is delivering governance, whatever the slides say. Groups 5 and 6 tell you how it will behave after the first contract month.

The same controls this checklist scores are what Alyria Beacon, Lyra, Umbra, and Spectra are built around: endpoint inventory and policy evaluation with no cloud round trip in the enforcement path, short-lived credential leases, one hash-chained audit chain per fleet, and native OpenTelemetry routing to your SIEM. Where Alyria fits states plainly what it does not do, and the OWASP MCP Top 10 mappings show which control covers which risk from the vendor's own list. For a read of where your program stands before scoring anyone, start with the AI security maturity assessments.