← all checklists

Tool Calling & Excessive Agency

70 items · source

Copy as markdown

Filter items

Tool / Function Calling Security

For every tool:

• Authentication exists.
• Authorization exists.
• Input validation exists.
• Output validation exists.
• Rate limiting exists.
• Audit logging exists.
• Least privilege exists.
• Tenant isolation exists.
• Resource-level authorization exists.
• Destructive-operation protection exists.
• Timeout exists.
• Size limits exist.
• Error handling is safe.

Then verify:

• Model cannot invoke undeclared tools.
• Model cannot modify tool arguments after authorization.
• Tool arguments are independently validated.
• Tool names cannot be attacker-controlled.
• Tool routing cannot be manipulated.
• Tool selection cannot bypass authorization.
• Tool output is treated as untrusted.
• Tool errors cannot expose credentials.
• Tool return data cannot inject arbitrary instructions.
• Tool retries cannot duplicate destructive actions.
• Tool calls are idempotent where necessary.
• Tool invocation has per-user/per-tenant limits.
• High-risk tools have approval gates.
• Verify the approval prompt the user sees is drawn by the server from the stored pending call, and that any confirm card, button or approve link in model output is stripped — otherwise the model, or text it read, can draw its own approval.
• Verify approving replays exactly the stored call once — same tool, same arguments — bound to the user who owns it, with an expiry, and still passing the kill switch, spend ceiling and schema checks when it executes.
• Verify an exemption that lets unattended runs skip approval keys on what started the run (a schedule, an event, a webhook, read from the stored record), not on the presence of a run id — a model that can create an automation and start it in the same turn otherwise approves itself; an unknown trigger asks.
• Verify fields that carry money (prices, fees, payout details) are removed server-side from anything a model writes, so an AI edit to a page or record cannot change what a buyer pays or where the money goes.
• Tools are disabled when not required.
• Tools cannot dynamically grant themselves additional permissions.
• Tool chains cannot escalate privilege.

Dangerous Tools

Pay special attention to agents with:

• shell
• terminal
• arbitrary code execution
• filesystem access
• database SQL execution
• arbitrary HTTP requests
• browser automation
• email sending
• SMS sending
• payment operations
• user deletion
• account modification
• password reset
• role modification
• GitHub write access
• repository administration
• deployment
• infrastructure changes
• secret retrieval
• database migrations

For each:

• Tool has minimal permissions.
• Tool has strict input schema.
• Tool validates destination/resource.
• Tool validates current user authorization.
• Tool validates tenant.
• Tool validates environment.
• Tool prevents arbitrary command execution.
• Tool prevents arbitrary URLs.
• Tool prevents arbitrary SQL.
• Tool prevents arbitrary filesystem paths.
• Tool prevents arbitrary cloud resources.
• Tool prevents arbitrary recipients.
• Tool prevents arbitrary repository/branch modification.
• Tool prevents production access when not required.
• Tool supports dry-run.
• Tool supports approval workflow.
• Tool logs complete security-relevant context.