hy dang.
← All blog posts

Research notes / Blog

OpenTools: Open, Reliable, and Collective Tool-Using AI Agents

Hy Dang 5 minutes

Updated: Oct 8

TL;DR

Tool-using LLM agents often fail for two different reasons:

  • the agent uses a tool incorrectly (tool-use accuracy), or
  • the tool itself is unreliable (intrinsic tool accuracy).

Most prior work focuses on the first issue. In our OpenTools project, we focus on both.

OpenTools introduces a community-driven framework that:

  • converts documented Python functions into standardized, reviewable tool bundles,
  • inspects submitted code without executing it and gives maintainers evidence for evaluation and review,
  • and provides a public demo for running tools and agents, inspecting evidence, and contributing tests.

In the paper’s evaluation, the OpenTools toolbox improves the overall average across three agent frameworks over the OctoTools toolbox. The comparison captures both broader tool coverage and tool quality.

Try Live Demo View on GitHub Read Paper

Demo Video

Why OpenTools?

An agent can select the right tool and still fail if the tool is unavailable, unstable, or wrong. APIs and dependencies change, and contributed code can introduce security or credential risks. Reliable tool use therefore requires a way to inspect, test, and maintain the tools themselves.

OpenTools is designed around a simple idea:

Reliable agents require both good tool orchestration and reliable tools.

This motivates a framework that treats intrinsic tool reliability as something to assess and maintain over time.

Core Idea: Two Complementary Workflows

The OpenTools framework has two linked workflows:

  1. Tool Accuracy / Maintenance Loop
    • Standardize tool descriptions, typed JSON schemas, output contracts, and tool cards.
    • Convert supported documented Python functions into wrappers and review bundles without executing them.
    • Inspect submitted source and potential secrets before policy-gated functional tests. An optional LLM advisor reviews sanitized evidence.
    • Have maintainers review submissions, run selected evaluations locally, and update the shared toolbox and reliability records.
  2. Agentic Workflow
    • Expose selected tools to ReAct, OctoTools, MultiAgent, or user-defined agents.
    • Provide controlled tool discovery and access through MCP for external applications.
    • Validate arguments and record tool calls, observations, errors, and final answers for debugging and reproducibility.

The public demo exposes both workflows: visitors can inspect tool cards and recorded evaluations, run supported tools and agents, and submit candidate tests. A contributor can upload an open-source Python tool and README for conversion and non-executing inspection. The hosted submission flow returns a pending-review bundle; it does not run or publish the uploaded code.

What the Figure Highlights

The system figure shows two connected workflows:

  • top half: contributions, non-executing tool scanning, optional LLM advice, human review, and selected reevaluation;
  • bottom half: user query, agent planning, tool execution, final answer, and execution logs.

The distinction matters: scanner findings and LLM advice support a maintainer’s decision, but neither automatically accepts a tool or certifies its safety.

OpenTools framework overview with tool scanning, human review, and agent execution

Figure 1. OpenTools framework overview: tool maintenance and review (top) and agentic use (bottom).

Main Experimental Takeaway

The paper compares a 42-tool OpenTools toolbox with the 13-tool OctoTools toolbox across VQA/puzzle, math/reasoning, scientific, medical, and agent tasks. It evaluates ReAct, OctoTools, and MultiAgent policies with fixed base models.

Key outcome:

  • The OpenTools toolbox improves the overall average for each evaluated agent framework in the reported settings.
  • The reported overall relative gains are approximately 5%–22%, depending on framework and model.
  • The largest task-group gains are on tool-intensive agent tasks. The experiment changes both toolbox coverage and quality, so it does not isolate either factor’s effect.

The results show why the toolbox matters alongside the agent policy, especially when tasks require external actions.

Figure: Table 1 from the OpenTools paper showing consistent gains across frameworks and task groups.

What I Think Matters Most

Three practical implications stand out:

  • Separate tool maintenance from agent decisions: the same standardized tools can be used by different agent policies.
  • Keep contributions reviewable: source inspection, test evidence, and human acceptance make community updates accountable.
  • Record what happened: tool and reasoning traces help distinguish agent-side mistakes from tool-side failures.

Looking Ahead

The paper identifies more domain-specific tools and regression tests, stronger isolated and long-term evaluation, and community maintenance at scale as next steps. Risk inspection and advisory LLM review provide evidence, not guarantees of correctness or safety.

If you are building or evaluating tool-using AI agents, I would love to hear your feedback and potential collaboration ideas.