Most enterprise agent evaluations still borrow a SaaS-era checklist: feature list, security questionnaire, per-seat quote. That checklist misses the part that actually decides whether a pilot survives contact with production: what happens when the agent is wrong, who pays when usage spikes, and whether you can switch vendors without rebuilding every integration.
Three shifts in 2026 changed what a real evaluation needs to cover. Pricing moved away from flat per-seat licenses toward usage and outcome-based models that change what a vendor is incentivized to sell you. The tool-integration layer most agents depend on, the Model Context Protocol, moved out from under a single vendor's control. And for the first time, a named, peer-reviewed list of what actually goes wrong in agentic systems exists, instead of buyers relying on vendor claims and gut feel. All three change the questions worth asking before you sign anything.
Start with the job the agent replaces, not the demo it runs
A demo shows an agent completing a clean, scripted task. Production shows an agent handling the messy cases the demo skipped: ambiguous input, a downstream system that is down, a customer who changes their mind mid-conversation. Before scoring any vendor, write down the specific job, the current cost of doing it manually, and which failure modes are acceptable versus which are not.
This step sets the baseline every pricing and security question depends on. An agent that drafts internal marketing copy has a different acceptable failure mode than one that issues refunds or pushes code to production. Teams that skip this step end up scoring vendors on raw capability, which rewards whichever agent has the flashiest demo rather than whichever fits the job.
Write the job spec in terms a vendor can be held to later: the trigger that starts the agent's work, the systems it needs to read or write, the maximum acceptable error rate, and who gets paged when it exceeds that rate. Vendors that ask you clarifying questions about this spec are usually easier to run a fair pilot with than ones that jump straight to a demo.
Pricing model reveals vendor incentives
How a vendor prices an agent tells you what they are optimized to sell you, not only what it will cost. Salesforce CEO Marc Benioff described this directly when Agentforce launched, telling investors pricing would move to a consumption basis, roughly $2 per conversation, because per-seat licensing works against a product meant to reduce the need for seats. An agent that genuinely does the work of ten support reps is a bad deal for a vendor charging by seat, and a good deal for one charging by resolved conversation.
Salesforce's own current pricing page shows how far that logic went. Agentforce now sells three ways at once: the original $2-per-conversation model for customer-facing agents, a per-user license, and Flex Credits, a prepaid consumption pool priced at $500 per 100,000 credits where each standard action costs 20 credits and a voice action costs 30. That is a vendor hedging across pricing models because no single one fits every buyer's usage pattern, which is itself a useful signal: ask any agent vendor which of their existing customers pushed them to add a second or third pricing model, and why.
- Per-seat: a flat fee per named user. Easy to budget, but rewards a vendor when the agent under-delivers and you still need the human seats.
- Usage-based: priced on tokens, API calls, or conversations. Costs scale with volume, predictable for steady workloads and risky for spiky ones.
- Credit or action-based: a prepaid pool spent per discrete action, Salesforce's Flex Credits being the clearest current example. Granular and auditable, but it pushes the burden of forecasting usage onto the buyer.
- Outcome-based: priced on a completed result, such as a resolved ticket or booked meeting. Aligns vendor and buyer incentives most closely, but needs a precise, contractual definition of what counts as a completed outcome versus an escalation.
- Hybrid: a base platform fee plus usage or outcome charges. Gives the vendor predictable base revenue and the buyer costs that scale with value actually delivered.
Ask for a worst-case cost model, not just the advertised rate. Have the vendor run your actual volume, including retries, escalations, and conversations that never reach a successful outcome, before you compare price across a shortlist. On a credit or action-based model, ask specifically what counts as a billable action: a single failed tool call and its retry may both draw down your credit pool even though the customer never got an answer.
Tool integration and lock-in risk
Agents are only as useful as the tools and data they can reach: your CRM, ticketing system, codebase, calendar. How a vendor handles that integration layer determines how hard it is to switch later.
The most consequential shift here in 2026 was not a product launch. In December 2025, Anthropic donated the Model Context Protocol to the Agentic AI Foundation, a directed fund under the Linux Foundation. The Linux Foundation's announcement named Anthropic, Block, and OpenAI as co-founders, with Google, Microsoft, AWS, Cloudflare, and Bloomberg backing the foundation as members. MCP joined goose (Block) and AGENTS.md (OpenAI) as founding projects, under the same neutral governance model that already hosts Kubernetes and Node.js. IBM, Cisco, Datadog, Docker, Oracle, SAP, Salesforce, and Snowflake joined as Gold members, which matters because it means the vendors you are evaluating are, in most cases, funding and governing the same protocol rather than competing to replace it.
That governance change matters for a question buyers used to struggle to answer: if this vendor disappears or changes terms, how much of your integration layer survives? A platform built on an open, multi-vendor-governed protocol answers that question far better than one with a proprietary connector format you would have to rebuild from scratch. MCP's own numbers back up how far this spread in a single year: the Anthropic announcement puts monthly SDK downloads above 97 million and active public MCP servers above 10,000, with client support across ChatGPT, Claude, Cursor, Gemini, and Microsoft Copilot.
- Does the platform support MCP or another portable, non-proprietary tool protocol, or only proprietary connectors?
- Can you export agent definitions, prompts, and tool configurations in a format another platform could import?
- Are credentials and data connections scoped per agent, or does switching mean re-authorizing everything at once?
- If the vendor is acquired or discontinues the product, what is the actual migration path, in writing?
- Is the vendor a member of, or actively contributing to, the Agentic AI Foundation or an equivalent open governance body, or are they running a closed fork?
The OWASP Agentic Top 10, translated into buyer questions
OWASP published the Top 10 for Agentic Applications on December 9, 2025, announced at Black Hat Europe alongside the OWASP Agentic Security Summit and built with input from more than 100 practitioners. It exists because the older LLM-focused Top 10 treats the model as something that receives a prompt and returns text. An agentic system plans, holds memory, calls tools, talks to other agents, and acts with delegated credentials across live systems, which creates failure modes the LLM list never had to cover.
The list runs ASI01 through ASI10. Each category translates directly into a question worth putting to a vendor during a pilot, not just a risk to read about.
- ASI01, Agent Goal Hijack: can an untrusted input (a webpage, an email, a document the agent reads) redirect what the agent is trying to accomplish? Ask how goals are isolated from ingested content.
- ASI02, Tool Misuse and Exploitation: can the agent be tricked into calling a legitimate tool in a harmful way? Ask what validation sits between the agent's decision and the tool actually executing.
- ASI03, Agent Identity and Privilege Abuse: does the agent run under a scoped identity, or does it inherit a human's or a service's full permissions? This is the single most common gap in early pilots.
- ASI04, Agentic Supply Chain Compromise: where do the agent's tools, plugins, and MCP servers come from, and who vets them before they are reachable in production?
- ASI05, Unexpected Code Execution: if the agent can write or run code, what sandbox contains it, and what can that sandbox reach on your network?
- ASI06, Memory and Context Poisoning: can bad data written into the agent's memory in one session corrupt its behavior in a later, unrelated session?
- ASI07, Insecure Inter-Agent Communication: when this agent talks to other agents, is that channel authenticated and logged, or implicitly trusted?
- ASI08, Cascading Agent Failures: does one agent's mistake trigger a chain of downstream agent actions before a human can intervene?
- ASI09, Human-Agent Trust Exploitation: can the agent's own explanations be used to talk a human reviewer into approving something it should not?
- ASI10, Rogue Agents: is there a kill switch that works even if the agent's own process is unresponsive or actively resisting shutdown?
Where NIST fits alongside OWASP
OWASP's list tells you what can go wrong inside an agentic system. NIST's Center for AI Standards and Innovation (CAISI) launched an AI Agent Standards Initiative on February 17, 2026, and it addresses a different, complementary question: how agents from different vendors identify themselves and interoperate safely across an ecosystem you do not fully control.
The initiative is organized around three pillars: industry-led standards development, community-led open-source protocol work, and foundational research into agent security and identity. Concretely, that included a public request for information on AI agent security with a March 9 response deadline, and a draft concept paper through NIST's National Cybersecurity Center of Excellence on agent identity and authorization. It extends the existing NIST AI Risk Management Framework, first published in 2023 before agentic architectures were a mainstream deployment pattern, to systems that can take irreversible actions, like sending an email or issuing a refund, before a human notices something went wrong.
Neither framework replaces vendor due diligence. Use them as shared vocabulary: ask a vendor directly whether they have mapped their product against the OWASP ASI categories above and whether they are tracking NIST's agent identity and authorization work, and treat a vague or dismissive answer as a real signal. For the full pilot-security checklist, including least-privilege service accounts and kill switches, see the agent-security checklist and MCP supply-chain risk breakdown.
A 60-day pilot scorecard
Run the pilot before the contract, not after. Score every shortlisted vendor against the same list, using your own volume and your own failure cases, not the vendor's demo environment.
- Define the job, the current manual cost, and which failure modes are acceptable, before any vendor sees the brief.
- Get a worst-case cost model from every vendor, run against your real volume including retries and escalations.
- Confirm the tool-integration layer: MCP or a portable equivalent, exportable configurations, per-agent credential scoping.
- Walk through the ASI01-ASI10 categories directly against this vendor's architecture, and confirm whether they track NIST's agent identity and security work.
- Run the agent on your messiest real cases, not the vendor's cleanest demo script, and log every failure.
- Require exportable audit logs before the pilot ends, not as a promised future feature.
- Assign a named owner and a rollback path for the agent before it touches anything in production.
Categories worth shortlisting from show the range: coding agents like Cursor and Devin, orchestration frameworks like CrewAI and AutoGen, conversational platforms like Botpress, tool and connector layers like Composio, and RPA-heritage vendors extending into agentic automation like Automation Anywhere and Blue Prism. The right comparison set depends entirely on the job defined in step one.