Open, Reliable, and Collective: A Community-Driven Framework for Tool-Using AI Agents

TL;DR

An agent can call the right tool and still get a wrong answer if the tool is broken, unstable, or unsafe. OpenTools addresses both tool-use accuracy and the intrinsic reliability of the tools an agent uses.

• Turns documented Python functions into standardized, reviewable tool bundles.
• Inspects contributed code without executing it, then gives maintainers evidence for evaluation and review.
• Lets people run tools and agents, inspect results, and contribute tests through a public demo.

Demo Video

Why OpenTools?

Tool-using agents depend on both good decisions about which tool to call and dependable behavior from the tool itself. APIs and dependencies change, tools may fail silently, and contributed code can introduce security or credential risks. OpenTools treats tool reliability as something to inspect, test, and maintain over time.

Paper Summary

OpenTools is an open-source toolbox for discovering, contributing, evaluating, and using tools with LLM agents. It standardizes tool interfaces, converts supported Python functions into reviewable bundles, and records evaluation evidence. Submitted code undergoes non-executing risk inspection, with optional LLM advice based on sanitized evidence. Maintainers decide which tools to test and accept. The toolbox also provides a public web demo and controlled access through the Model Context Protocol (MCP).

Framework Overview

Figure 1 from the paper: the tool maintenance and review loop (top) connects to agent execution with tool and reasoning logs (bottom).

Core Idea: Two Complementary Workflows

1. Tool accuracy and maintenance
   • Standardizes descriptions, typed JSON schemas, output contracts, and tool cards.
   • Converts supported documented Python functions into wrappers and review bundles without executing them.
   • Inspects source and potential secrets before policy-gated functional tests; optional LLM advice uses sanitized evidence.
   • Lets maintainers review submissions, run selected tests locally, and refresh reliability records and the shared inventory.
2. Agentic use
   • Exposes tool collections to ReAct, OctoTools, MultiAgent, and custom agent policies.
   • Supports controlled tool discovery and use from external applications through MCP.
   • Validates arguments and records tool calls, observations, failures, and final answers for debugging.

Try the Public Demo

The OpenTools web demo lets visitors inspect tool cards and recorded evaluations, run supported tools and predefined agents, and submit candidate test cases. Contributors can submit an open-source Python tool and README for conversion and non-executing inspection. The result is a pending-review bundle; uploaded code is not run or published by the hosted submission flow.

Evaluation

Overall averages from Table 1 of the paper:

Higher is better. The paper reports full results by task group.

• The study compares a 42-tool OpenTools toolbox with a 13-tool OctoTools toolbox across five task categories.
• OpenTools improves the overall average for ReAct, OctoTools, and MultiAgent in the reported settings, with roughly 5%–22% relative gains.
• The largest task-group improvements appear on tool-intensive agent tasks. The comparison reflects both tool coverage and tool quality.

Looking Ahead

Future work includes more domain-specific tools and regression tests, stronger isolated and long-term evaluation, and studying community maintenance at scale. Inspection and advisory review provide evidence for human maintainers; they do not guarantee that a tool is correct or safe.