Agentic AI security platforms: how to evaluate them

The evaluation is a set of demonstrations against your own fleet, not a feature comparison.

The category is crowded and the labels overlap. AI security posture management, AI gateways, prompt firewalls, data loss prevention with an AI module, endpoint agents with an AI tab and specialist agent-governance platforms all claim the same problem. The differences that decide the outcome only appear when a product meets a hostile tool and a real config file.

Why does platform evaluation matter now?

Buying decisions are being made this quarter, while the requirements are still moving. Gartner expects over 40% of agentic AI projects to be cancelled by the end of 2027 because of escalating costs, unclear business value or inadequate risk controls (Gartner). A platform that cannot produce risk evidence is part of the cancellation statistic.

The fleet you are buying for is already large. In the 2026 Stack Overflow survey, 73% of respondents used coding assistants and agents daily and 31% of daily users spent four or more hours a day with them (Stack Overflow Developer Survey 2026). Whatever you buy has to work at that scale without a per-agent project.

The threats you are buying against are measurable, which means the evaluation can be too. In one study of 20 agents and 45 live MCP servers, o1-mini was steered by tool poisoning 72.8% of the time, and no model refused more than 3% of attacks (MCPTox, arXiv:2508.14925). A platform that cannot stop a poisoned tool call before execution is not addressing the failure the data describes.

The credential problem is the other half of the brief. In a survey of 5,200 open-source MCP servers, 88% required credentials, 53% relied on long-lived static secrets and 79% passed API keys through environment variables (Astrix). A platform that inventories keys without offering a way to remove them leaves the buyer with a report and a standing risk.

How exposed is your setup? Compass gives you an independent starting read before you talk to any vendor: compass.alyria.ai.

How is an agent security platform different from AI-SPM, a gateway or EDR?

The labels describe where the product sits, and the position decides what it can see and stop.

Category	Primary job	What it sees	What it can stop	Typical gap for agents
EDR with an AI module	Endpoint threats and process behaviour	Processes, files, network	Malicious executables and behaviour	Understands tools, prompts and MCP semantics poorly
AI gateway	Control traffic to models	Requests and responses crossing it	Model calls and some payloads	Misses local models, stdio servers, file access
Prompt firewall	Classify and filter content	Prompts and completions	Content patterns	Probabilistic, and blind to what the agent then does
AI-SPM	Posture and inventory	Configurations, cloud resources	Policy violations after the fact	Observes rather than enforces per call
DLP with an AI module	Detect data movement	Traffic and endpoints	Exfiltration patterns	Unaware of tool authority and agent identity
Agent governance platform	Bound what an agent may do	Endpoint agents, tools, credentials, calls	Tool calls, egress, credential use	Requires endpoint deployment and change management

In practice: The question is not which category is best. It is whether the product can make a decision before an action, on the surface where your agents actually run.

What are the evaluation criteria?

Six criteria decide most outcomes. Score each against a demonstration rather than a claim.

Coverage of the endpoint reality

Can it see local agent CLIs, IDE extensions, MCP clients and servers, including stdio servers that never touch the network? Can it show the config file that enabled each server and the credentials each agent can reach?

The test is simple: run it on a developer laptop and compare the count against your own inspection. A product that reports eight servers on a machine with thirty has a coverage problem it will not solve with configuration.

In practice: Ask for the tool list each server exposes, not just the server names. Tool-level coverage is what makes policy possible.

Enforcement, not observation

Can it refuse a specific tool call before it executes, with the tool, the arguments and the destination in view? Can it hold rather than block, and does the hold stop the agent rather than let it proceed?

A product that only reports what happened after the fact leaves you with an incident timeline and no control. Observation is valuable, and it is the smaller half of the requirement.

In practice: Test with the endpoint offline. Local enforcement survives a network partition; a cloud-only decision does not.

Policy expressiveness

Can policy express capabilities, arguments, data classes and destinations? Can it express an information-flow rule, such as refusing a send that carries data read in the same session? Can it distinguish read from write, and write from irreversible?

A policy language that only allowlists tools cannot describe the incidents you actually face. Ask for the rule that refuses credential-path reads, and for the rule that stops a high-impact action pending approval.

In practice: Ask the vendor to write your three hardest rules during the evaluation. Their ability to express them is the product.

Identity and secrets

Can it attribute an action to an agent rather than a shared account? Can it issue or broker short-lived credentials, or only inventory the keys already in config?

The second question separates a policy tool from a governance platform. An agent on a personal token with a perfect policy layer still carries a durable secret in a file.

In practice: Ask what happens to a leaked config file. If the answer does not involve an expiry, the identity story is incomplete.

Evidence and audit

Can a decision be exported with the agent, the tool, the arguments that mattered, the policy version and the outcome? Is the record tamper-evident? Can it reach your SIEM natively?

An audit answer that requires the vendor's console cannot be used by an auditor who does not have access. Native OpenTelemetry to your own collector is the cheapest way to keep the evidence yours.

In practice: Ask for an export of a specific week and read it. If the record cannot be read outside the console, it is a screenshot.

Operability and blast radius

What does deployment look like, what happens on upgrade, how is the agent stopped, and what does a false positive cost? How long does revocation take?

The operational questions decide whether the platform survives its second quarter. A control that blocks work is disabled by the first engineer who is blocked, usually quietly.

In practice: Measure time from decision to revocation during the evaluation. Under five minutes is a control; under an hour is a process.

How to run the evaluation

The sequence is designed so a bad fit is discovered in week one rather than at contract signature.

1. Write the three rules you must be able to enforce

Pick the rules that map to your worst realistic incident: a credential read, an unattended write to a production system, an external send carrying restricted data. Every vendor gets the same three.

Tip: Derive them from an incident or a red-team finding, not from a framework list. Rules with an owner get enforced.

2. Insist on inventory on day one

If the product needs a two-week scoping exercise before it can list agents, that is a product property. Give the vendor a test laptop and a deadline of one day.

3. Test with a hostile tool

Connect a deliberately malicious server: one tool that asks the model to read a key file, one that returns instructions in its results, one that renames itself mid-session. Confirm what is blocked, what is logged and what the model actually does.

Tip: A vendor who refuses this test has told you the answer. The hostile server is the closest thing to a fair fight in an evaluation.

4. Check the denial path

Confirm the agent stops rather than retrying, that the refusal is recorded, and that a denied action does not reappear through a different tool in the same session.

5. Verify attribution and identity handling

Take one action and trace it end to end: which agent, which identity, which lease, which policy version, which approver. If any of those is missing, the evidence chain has a gap an auditor will find.

6. Measure the false positive cost

Ask what the platform would have blocked during the previous week of normal work. A control that would have stopped 400 legitimate actions needs tuning, not deployment.

Tip: Run monitor mode for a week before enforcement. The list of what would have been blocked is the most useful artefact the evaluation produces.

7. Test the offline and degraded paths

Disconnect the endpoint, restart the service, expire a credential. Confirm enforcement continues and the evidence queues rather than disappears.

8. Score against the criteria, with the demonstrations attached

A score without evidence becomes a preference. Attach the export, the denial, the inventory diff and the rule text to each score so the decision can be defended in a procurement review.

In practice: The winning product is usually the one whose denial you could demonstrate, not the one whose dashboard you liked.

Where the platform fits in the agent lifecycle

Design

The platform informs design by showing what similar agents already reach. That evidence is more useful than a template, because it reflects your own estate.

Note: A design review informed by an inventory of what exists produces narrower scope than one informed by a capability catalogue.

Development

Policy authored as code, reviewed with the change, and shipped with the agent. Anything configured only in a console will drift from the repository.

Pre-deployment testing

Run the hostile tool test as a gate. A build that cannot demonstrate the denial is not ready for a production action.

Note: Automated denial tests catch a policy regression cheaply. They are worth more than a weekly review of a dashboard.

Deployment

Confirm enforcement, attribution and evidence export before the agent's first real action. A control deployed after the first incident is an admission.

Runtime

The platform decides, records and alerts. Operationally this is where the daily work happens, so the alert volume has to be tuned early.

Continuous monitoring

Report denials, drift, new servers and pending approvals to the people who own the agents. A monthly report to a security team nobody in engineering reads changes nothing.

Note: The most valuable ongoing output is a short list of agents whose capability changed this week, sent to their owners.

Decommissioning

Removing an agent should remove its identity, its tool registrations and its policy. Confirm the platform reflects that, or the register decays.

The takeaway: Evaluate on demonstrations: inventory in a day, a denial you watched, an export you read and a revocation you timed.

Worked example: a three-vendor evaluation

A software company with 600 engineers runs a two-week evaluation with three platforms: an extension of its existing endpoint tool, an AI gateway and a specialist agent-governance platform.

The test set. Three rules: no read of SSH keys or cloud credential files, no unattended push to a protected branch, no external send carrying restricted data read in the same session. Plus a hostile MCP server with a poisoned description.

Vendor A, the EDR extension. Inventory takes four days and reports 12 agents and 6 servers on a test laptop that has 22 and 19. It blocks execution of unknown binaries well and has no concept of a tool call. Denial of the credential read is not possible; it can alert after a file access. Scored high on platform consolidation, low on the requirement.

Vendor B, the gateway. Inventory is immediate for traffic, and the local stdio servers are invisible. The credential read never crosses the gateway, so rule one is unenforceable. The poisoned description is caught by a content filter in three of ten attempts. Scored high on model traffic control, low on coverage.

Vendor C, the specialist platform. Inventory on day one reports 22 agents and 19 servers with config paths. Rule one denies before the read and records it. Rule two holds the push with the diff attached. Rule three refuses the send because the information-flow rule sees the data class read earlier in the session. The hostile server's poisoned tool is denied on its second call when it appears mid-session, and the denial is exported to the SIEM the team already runs.

The scoring. Coverage and enforcement decide the outcome; operability and evidence separate the two viable options. Vendor A remains in the stack for endpoint malware, because the specialist platform does not replace it. Vendor B is proposed for a narrower use case around model traffic, which is what it is good at.

The decision. Vendor C is selected with a 12-month term and a success measure: the gap between observed and registered agents under 10%, and denial rate under 2% after tuning. Both figures come from the evaluation rather than from a slide.

How standards and regulations influence the purchase

Framework	Primary focus	What the buyer must be able to evidence
NIST AI RMF	Govern, Map, Measure, Manage	Inventory, risk tiering, measurable monitoring
ISO/IEC 42001	AI management system and supplier control	Documented controls, supplier review, records
ISO/IEC 42005	AI system impact assessment	Assessment records per deployed agent
EU AI Act	Obligations by risk class	Logging, oversight and technical documentation
OWASP Agentic Top 10	ASI01 to ASI10 threats	Coverage of the named agent threats
OWASP MCP Top 10 (beta)	Tool and server level risks	Tool-level policy and server inventory
SOC 2	Controls over a period	Access, change management and monitoring evidence

Further reading: Agentic AI security for the control set the platform must cover, AI TRiSM for agents for the operating model around it, and Policy as code for AI agents for what to ask about policy authorship.

How Alyria handles it

Alyria is the endpoint agent-governance platform for the agents your people run, and the evaluation above describes how it is built. Beacon is a signed daemon on each machine that inventories agent CLIs, IDE extensions, MCP clients and servers, including stdio servers, along with the config that enabled each one and the credentials in reach. Inventory is available on the first machine rather than after a scoping project.

Lyra enforces at the call, locally, with the arguments in view. It expresses capabilities, arguments, data classes and destinations, and it is information-flow aware, which is what allows a rule that refuses a send carrying data read in the same session. Enforcement is local, so it holds when the endpoint is offline.

Umbra issues short-lived credential leases rather than only reporting on keys, sealed to device hardware keys Alyria never holds. Spectra exports signed, chained OpenTelemetry to Alyria Cloud or directly into your SIEM, and Observatory holds policy decisions, detections, CVE posture and fleet health in one plane.

Start free at alyria.ai and run the inventory test on one laptop before you run an evaluation: app.alyria.ai/signup.

The free Agent Exposure Report is the read-only version, and the platform overview documents the architecture you would be evaluating.