# AI coding agent governance RFP checklist

Alyria, CC BY 4.0. Score each requirement pass, partial, or fail. The tests
assume one evaluator laptop, the AI clients your developers already run, and
one evaluation window. The full context for each test is on
/resources/ai-coding-agent-governance-rfp.

## 1. Inventory that survives the config files
- [ ] Inventories every installed AI tool and version on the endpoint
- [ ] Reads each client's configuration scopes: project, user, managed files
- [ ] Sees stdio servers, which touch no network
- [ ] Records transport, launch command or URL, version pinning, first and last seen
- [ ] Answers "what changed since last week" without a manual re-run

Test 1 (config sweep):

    grep -rl --include='*.json' -e '"mcpServers"' -e '"servers"' ~ 2>/dev/null
    grep -rn -e '^\[mcp_servers' ~/.codex/config.toml 2>/dev/null

Expected: the vendor's inventory is a superset of yours.

## 2. Policy that runs at the call site
- [ ] Denies a tool call before it executes; the deny is final
- [ ] One policy covers every AI client on the machine
- [ ] Policies are code, versioned, with author and date
- [ ] Monitor mode records what would have been blocked before blocking
- [ ] Sequence rules are expressible (after action A, refuse action B)

Test 2 (deny that is final): attempt cat .env, /bin/cat .env, head .env and
python3 -c "open('.env').read()" with enforcement on. Every shape denied; the
record names the rule that fired.

Test 3 (offline enforcement): disconnect the network, trigger a rule through a
stdio MCP server. The block still fires.

## 3. Secrets and credentials
- [ ] Flags plaintext credentials in client configs, per endpoint
- [ ] Offers short-lived leases instead of standing tokens
- [ ] Credentials never enter the model's context
- [ ] Custody is verifiable and revocable on your side (BYOK or HYOK)

Test 4 (standing-token sweep): run one cloud credential task through the lease
flow, then:

    grep -rn "AKIA" ~/.aws 2>/dev/null
    grep -rn "AKIA\|ghp_" ~/.claude.json ~/.cursor/mcp.json ~/.codex/config.toml 2>/dev/null

Expected: nothing standing in the config locations and prefixes the sweep reads;
the lease is in the audit record. This is a prefix and config-file smoke check,
not a disk-wide or model-context proof: a github_pat_ token or a credential
held elsewhere will not match. Score the broader claims on separate evidence.

## 4. Evidence an auditor accepts
- [ ] Every call recorded with the rule applied, the identity, and the client
- [ ] Tamper-evidence verifiable by a third party
- [ ] One chain across every client on the machine
- [ ] Exports in a standard format (OpenTelemetry) to your SIEM

Test 5 (tamper check): modify a record in place; verification fails. Export one
day of records; same policy decision queryable in the SIEM.

## 5. Native controls: the vendor's own position
- [ ] Names the native controls per client (managed settings, managed MCP
      configuration, hooks, sandbox, approvals) and links the vendor docs
- [ ] States what its layer adds: fleet visibility, one policy and audit layer
      across clients
- [ ] Keeps the native controls running; MDM stays in charge of managed files
- [ ] Rollout starts read-only; enforcement moves per rule after monitor mode

Test 6 (monitor-mode week): one week on the evaluator cohort; report of what
would have been blocked, with counts; zero developer-visible breakage.

## 6. Runtime and fit questions
- [ ] Where does inference happen per agent?
- [ ] What does enforcement do offline? (Tested; vendor should describe the same.)
- [ ] What the vendor does not do, stated plainly
- [ ] Time to first inventory stated and then held
- [ ] Removal is one command, with no residuals

## Scoring

Pass, partial, or fail per requirement. Group 1 failing cannot be compensated
by a strong audit story. Groups 2, 3, and 4 are the enforceable core. Groups 5
and 6 predict post-contract behavior.

Reference: /resources/ai-coding-agent-governance-rfp on alyria.ai.