Cithren

Adopt or skip / Decision guide

How to evaluate a new tool for your AI agent

Evaluate a new tool against a specific job your agent already needs to do. Compare it with your current approach on representative cases, check its permissions and failure behavior, then adopt only if the improvement justifies the added setup and attention. A convincing demo is a reason to test, not a decision to adopt.

Name the job and the baseline

Write the missing capability as an observable task: “retrieve the current refund policy and identify the source” is more useful than “improve research.” Identify who uses the output, what makes it acceptable and what the agent must not do. A tool without a named job is another thing to maintain.

Then document the baseline. That might be an existing tool, a manual step or the agent declining work it cannot verify. Include the human correction involved. The relevant question is whether the candidate improves this workflow, not whether it looks impressive on its own.

Build a fair comparison

OpenAI's evaluation best practices recommend defining an objective, assembling a dataset, setting metrics and comparing results. They also address edge cases, variable model behavior and human judgment. Those are evaluation principles, not evidence that any particular tool works.

For a bounded tool decision, collect cases from the work you expect: routine requests, ambiguous requests and cases where the correct response is to stop. Use synthetic or properly sanitized material rather than private logs. Keep the task, model instructions and acceptance rules consistent across the baseline and candidate. Record any unavoidable differences.

Choose criteria before seeing the candidate's output. Assess task correctness, source support, unnecessary actions, time spent checking and operating cost where observable. Repeat cases that depend on model choices; one successful run cannot establish reliability. Have a person inspect outputs when correctness needs judgment. Record failures separately instead of hiding them inside an average.

Check boundaries and failure behavior

Inspect the permissions actually requested, not just the capability name. Does a reading task require write access? Can you limit the account, data and actions available during evaluation? Who approves consequential actions? If you cannot explain the access boundary, do not approve a live trial.

The Model Context Protocol security best practices discuss authorization and consent risks. Using MCP is not, by itself, a trust assessment. Apply the relevant checks to the actual server, credentials and requested scopes.

Test a denied request, unavailable service and incomplete response in a controlled setting. Look for visible failure rather than confident invention or silent action. Decide how a person will notice and recover. Passing a normal task is not permission to bypass these boundaries.

Hypothetical example / Not a product result

A better search result is not the whole job

Suppose an agent drafts internal answers from a policy folder. A proposed retrieval tool returns relevant passages, but an archived policy appears beside the current version. The evaluation includes a straightforward question, conflicting versions and a question the folder cannot answer.

The adoption gate is not “finds relevant text.” It is “identifies the current policy, preserves its source and declines unsupported answers.” If the candidate cannot distinguish versions, narrow its scope or skip it. If it passes those checks with acceptable review effort and read-only access, a limited trial may be justified. These are decision conditions, not measured outcomes.

Tool adoption worksheet

Copy this into your own notes. Complete the first four rows before testing; attach observations afterward. This is Cithren's editorial decision framework, not a vendor certification or an automated assessment.

Adopt, skip or gather more evidence
Decision fieldWhat to record
Job and ownerThe task, intended user and person responsible for the decision.
Current approachBaseline output, manual correction and the gap worth solving.
Acceptance gatesRequired behavior, forbidden actions and evidence that would count as failure.
Test casesRoutine, ambiguous, unsupported and unavailable-service cases; expected behavior for each.
Observed differencesBaseline versus candidate, repeated-run variation and remaining human review.
Access and upkeepScopes, consent, setup burden, costs, maintenance owner and unresolved concerns.
Decision recordAdopt within a named boundary, skip with a reason, or defer pending specific evidence.

Make the decision explicit

Adopt when the candidate meets the gates and its benefit survives the added operating burden. Skip when it solves no current problem or fails a non-negotiable boundary. Defer when evidence is missing, naming what would resolve the uncertainty. Do not turn “interesting” into a permanent integration.

If adoption would displace something already in use, the question changes. Read when to replace an AI agent tool, and when to keep it before committing to the switch.