7. Agents & Tool Use

In the previous capability layers such as vision and language, models mostly operated in a "passive answering" mode — receiving input and producing output. In many real-world business scenarios, however, what we need is an agent that can proactively plan, invoke external tools, and orchestrate workflows: it must not only see/read/hear, but also "decide what to do next" on its own — such as looking up information, running code, reading/writing files, calling internal systems, then integrating, interpreting, and feeding the results back to the user.

This layer can be understood as the critical glue that "turns a foundation model into an actionable system": through structured tool-calling interfaces, workflow orchestration, multi-agent collaboration, and human-in-the-loop mechanisms, it extends an LLM from a powerful "cognitive kernel" into a "digital employee" capable of completing end-to-end tasks.

7.1 Tool Calling / Function Calling

In the plain-text era of read-only, say-only interactions, an LLM was more like a "super conversationalist": it could understand questions, give advice, write code, and outline plans, but all the "truly executable" work — querying databases, running scripts, generating files, invoking cloud services — still had to be done manually by humans. The emergence of Tool Calling / Function Calling allows models, for the first time, to "take action" within a safety boundary: they automatically generate structured parameters from natural language to invoke external capabilities such as search engines, databases, computation engines, and image/audio/video generation services, and then organize and return the execution results, thereby forming a closed loop of "understanding → decision → execution."

From a product perspective, tool calling is the "chassis capability" of virtually all agent systems: the OpenAI Assistants API, LangChain, LlamaIndex, AutoGen, and agent platforms from various cloud providers are essentially building a runtime layer on top of LLMs centered around how to define tools, how to let the model correctly select tools, and how to handle errors and retries. Below, we examine this layer from three angles — scenarios, principles, and models — and then expand on "tool-calling interface design," "tool selection and strategy," and "typical tool types" in the subsequent subsections.

• Scenarios
  • Intelligent Q&A and retrieval augmentation: The model automatically decides, based on the user's question, whether to invoke retrieval tools (vector/keyword search), query internal enterprise knowledge bases or public web search, and integrate retrieved documents and FAQs into the final answer.
  • Data and report automation: For requests like "look up sales for this period and draw a chart" or "calculate the risk metrics for this portfolio," the model automatically generates SQL or analysis parameters, invokes databases and computation engines, and returns charts and conclusions.
  • Document and file operations: Automatically read PDF/Word/Excel/database tables, extract and summarize key information, or generate new files (reports, contracts, proposals) as instructed, and upload/store them to designated locations via tools.
  • Media generation and processing: Invoke image/audio/video/3D generation services based on text instructions, or perform editing, compression, transcoding, watermarking, and other operations on existing media, forming a one-click "copywriting + design + export" content pipeline.
• Principles The core of tool calling is: driving structured function calls with natural language.
  • First, expose external tools' names, descriptions, and parameter structures (types, required fields, enumerated values, etc.) to the LLM in the form of JSON Schema or function signatures.
  • When a user makes a request, the LLM must not only understand the semantics but also determine "whether a tool needs to be called," "which tool(s) are needed," and "how the parameters for those tools should be filled in."
  • Once the model decides to call a tool, it generates structured parameters (typically JSON), which the runtime uses to actually execute the external API/program, and returns the execution result in structured form to the model, allowing the model to continue reasoning or generate the final answer based on the result.
  • To ensure safety and robustness, the system must handle parameter validation, timeouts, error returns, retries, and fallbacks during this process, and apply permission and audit controls for calls that may involve security/privacy concerns.
• Models The models and frameworks supporting this capability fall into three main categories:
  • LLMs with native Function Calling support: such as the GPT‑4.1 / o series, which natively understand "tool signatures + JSON Schema" at the decoding level and can proactively or reactively produce structured call parameters at the appropriate moment.
  • Tool-augmented reasoning paradigms: such as ReAct and Toolformer, which weave "thinking + tool calling" into the same reasoning chain, treating tool use as part of the intermediate steps rather than simple pre/post-processing.
  • Engineering frameworks and runtimes: the OpenAI Assistants API, LangChain, LlamaIndex, AutoGen, cloud vendor agent platforms, etc., which provide infrastructure for tool definition, call routing, state management, error handling, and log auditing, allowing developers to focus on "which tools to expose" and "what business API abstractions to design" rather than building a runtime from scratch.

7.1.1 Tool-Calling Interfaces: From Natural Language to Structured Function Calls

A usable tool-calling system first requires a clear, standardized, LLM-friendly "tool interface layer." This layer is responsible for wrapping external-world APIs, scripts, and services into "functions" that the model can understand and safely invoke, enabling the model to "verbalize" the tool and its parameters it wishes to call as if writing pseudocode.

• Tool definitions and parameter schemas At the interface layer, each tool is typically defined using a structure similar to JSON Schema or a function signature: including the name, description, parameter fields (properties), types (string / number / boolean / array / object), whether they are required, and allowed value ranges or enumerations. This information is used on one hand to drive type checking in the frontend/SDK, and on the other hand is fed directly to the LLM to help the model "learn" how to correctly fill in parameters. The clearer the descriptions and the more reasonable the constraints, the more standardized the model's generated calls and the lower the error rate.
• LLM-generated structured parameters When a user makes a request like "look up Q3 2024 revenue and draw a bar chart broken down by region," the model must first reason that this requires at least a "report query tool" (to access data) and possibly a "chart generation tool" (to draw the chart). For each tool, it must extract and map structured parameters from the raw language — such as time range (start_date/end_date), dimension (region), metric (revenue), chart type (bar), output format, etc. — and then output them as JSON to be handed off to the runtime. In this process, the model is essentially performing integrated reasoning of "natural language → task planning → parameter extraction/filling," making the natural-language prompts in tool descriptions, parameter examples, and few-shot samples critically important.
• Tool execution and result feedback After receiving the JSON call produced by the model, the runtime first performs parameter validation and security checks before actually invoking the backend API or program. Once execution is complete, the result is wrapped as a structured object (such as a query result table, file URL, media resource ID, etc.) and returned to the model. The model then transforms these raw results into a user-readable explanation or performs further processing, such as summarizing reports, generating natural-language analysis, or embedding chart annotations. For the model, tool results are merely intermediate information — it is still responsible for "understanding the results + explaining the results."

7.1.2 Tool Selection and Strategy: Making Decisions in a Multi-Tool World

When there is only one tool in the system, "whether to use a tool" is the only question. But in real-world agent applications, there are often dozens or even hundreds of tools: retrieval from different data sources, business APIs from different departments, generation/analysis capabilities from different technical domains. This introduces a new challenge: how the model makes reasonable selection and orchestration in a multi-tool environment.

• Tool selection and routing First, the model must determine "whether the current request requires a tool call" and "which tool (or tools) to call." This is typically achieved by listing available tool descriptions in the system prompt and providing typical examples so the model learns to select the appropriate tool based on user intent. For scenarios with a large number of tools and high description similarity, many frameworks introduce a "tool router" (such as vector-retrieval-based or rule-based pre-filtering) that first narrows down a large list to several candidate tools, which are then exposed to the LLM for selection, thereby reducing the model's burden and the probability of misselection.
• Multi-tool sequencing and composition Complex tasks often require multiple tools working together. For example, "research the major listed companies in a certain industry and generate a report with financial comparison charts" may involve a search engine, a financial report database, a computation engine, a chart generation tool, and a document export tool. In such cases, the model needs to perform lightweight task planning: which tool to use first to obtain a list, then query detailed information for each item on the list one by one, then merge the data, perform calculations and visualization, and finally invoke the export tool to generate the report. Typical practices include ReAct/Planner-Executor approaches, where the model progressively completes combined tool calls in a cycle of "thinking (Plan) — calling (Act) — reflecting (Reflect)."

7.1.3 Typical Tool Types: The Capability Puzzle from Retrieval to Media Generation

Different types of tools provide agent systems with different dimensions of "external brains." From an engineering practice perspective, the following categories of tools are almost "standard equipment" for all complex applications.

• Retrieval tools: vector and keyword search Retrieval tools are responsible for extending "memory" to the external world:
  • Keyword search is suitable for traditional documents and business databases with good structure and clear fields.
  • Vector search uses embeddings to build semantic indexes for unstructured text, code, conversation logs, and even multimodal data, supporting "fuzzy but semantically relevant" retrieval. In RAG scenarios, the LLM pulls context relevant to the user's question via retrieval tools and then performs reasoning and generation on top of that context, significantly improving the timeliness and accuracy of answers.
• Code execution and computation engines Code execution tools (such as Python/JS sandboxes, notebook executors) allow LLMs to "write a piece of code and run it immediately" to solve complex computation, data processing, numerical simulation, visualization, and other problems. The model is responsible for producing code and input parameters, while the execution environment is responsible for security isolation, resource limiting, and result collection. These tools are critical in scenarios such as data analysis, quantitative research, automated reporting, scientific computing, and agent self-verification (where the model generates an answer and then verifies it with code).
• File and data source access File read/write tools are responsible for bringing external file systems and data sources into the agent's field of view: reading PDF/Word/Excel, accessing database tables, calling internal business APIs, etc. The model obtains real business data through these tools and then performs summarization, comparison, and report generation. Accompanying these are file writing and management tools: persisting generated reports, charts, PPTs, code, etc., and returning links or IDs for convenient subsequent access and integration by users.
• Media generation and processing tools Media generation tools add "creative" and "design" arms to agents:
  • Image/video generation and editing: automatically generate illustrations, posters, and storyboards based on copy, or crop, subtitle, watermark, etc., existing media.
  • Audio generation and processing: TTS, dubbing, music generation, audio enhancement and editing.
  • 3D / engineering tools: generate simple 3D scenes, CAD sketches, UI prototypes, etc. In content production, marketing design, education and training, gaming, and multimedia applications, these tools bring "from idea to finished product" closer to an automated pipeline.

Taken together, tool calling and execution extend LLMs from "language models" to "general-purpose controllers with action interfaces": the model understands requirements and the environment through language, performs real operations through tools, and continuously adjusts its strategy through feedback. Combined with appropriate workflow orchestration and multi-agent collaboration (see 7.2), this forms the foundational architecture for the next generation of intelligent applications.

7.2 Workflow & Orchestration

With tool-calling capabilities, an LLM is no longer just a "question answerer" but can become an "execution unit" oriented toward specific tasks. However, real-world business is often far more complex than a single conversation: a complete litigation analysis, a market research project, an A/B experiment configuration, or an end-to-end ops handling process typically requires multi-step operations, multiple tools, and even long-term participation from multiple roles. At this point, the single LLM + tools pattern begins to strain, necessitating further workflow orchestration and multi-agent collaboration.

From a systems perspective, this layer's responsibility is: abstracting a complex, multi-step, multi-stakeholder business process into a workflow graph that LLMs can understand and manipulate, then scheduling one or more agents on this graph, coordinating with human intervention, to jointly complete the task. Typical implementations include Planner-Executor agent architectures, agents with reflection/self-correction capabilities, and graph-based Workflow Orchestrators; corresponding product forms include various automated report generation and operations automation platforms, low-code workflow + LLM integration, complex business process robots, automated ops systems, and so on.

• Scenarios
  • Report and content pipelines: From "receiving requirements → retrieval and data pulling → analysis and visualization → writing reports → review and revision → export and distribution," automating or semi-automating multi-step content production processes.
  • Business process automation: Such as "product analysis → competitor monitoring → campaign strategy generation → deployment configuration" in e-commerce operations, or "monitoring alerts → root cause analysis → mitigation execution → post-mortem report" in ops scenarios.
  • Cross-role collaboration: Enabling agents from different domains (legal, finance, engineering, operations) to collaborate around a complex project, such as M&A due diligence, investment and financing material preparation, or large-scale project bid preparation.
• Principles The core of workflow and multi-agent collaboration is adding a layer of structured control and state management on top of LLMs:
  • Decompose complex tasks into several subtasks with dependencies, represent them using DAGs / state machines / directed graphs and other structures, and configure trigger conditions, inputs/outputs, and required agents/tools for each node.
  • A Planner-type agent or upper-level orchestrator decides when to trigger which node and which agent or tool to use, and dynamically adjusts subsequent paths (conditional branches, loops, error fallbacks) based on execution results.
  • Introduce Human-in-the-loop at critical junctures, performing manual confirmation and editing for high-risk decisions and key outputs, and feeding human feedback back into the system to update strategies or fine-tune models.
• Models The main technical directions supporting this layer include:
  • Planner-Executor agent architecture: A "planning agent" is responsible for task decomposition and path design, while one or more "execution agents" are responsible for implementing specific steps.
  • Reflection / self-correction agents: Continuously review their own performance during execution, reflect on and correct unreasonable intermediate results, and reduce the silent propagation of "confident errors."
  • Graph-based Workflow Orchestrator: Model the entire task flow as a graph structure, introducing node states, edge conditions, parallel/serial control, and other mechanisms, so that LLM calls become one or more nodes in the graph rather than the sole control center.

7.2.1 Task Decomposition and Planning: From "One-Sentence Requirements" to Executable Flows

What users give to agents is typically a highly compressed natural-language requirement, such as "do a market research report on the new energy vehicle industry and output a PPT," which actually encompasses numerous steps including retrieval, filtering, analysis, visualization, layout, and multiple rounds of revision. How to automatically build a clear, executable workflow from this single sentence is the first step in workflow orchestration.

• From natural language to subtask graphs A Planner-type agent must first "unfold" the requirement: combine built-in templates, historical cases, and the tool inventory to identify key phases (such as information collection, data analysis, structural design, content writing, review and export) and further refine them into executable subtasks (such as "retrieve 5 authoritative industry reports from the past year," "pull the last 3 years of sales data broken down by vehicle model," "generate 3 comparison charts," etc.). The dependencies and scheduling logic among these subtasks are explicitly represented as a graph or state machine: which ones can run in parallel, which must execute sequentially, at which nodes human confirmation is needed, and under what conditions rollback or retry is required.
• Conditional branches, loops, and exception paths Real-world processes are often not linear pipelines but contain conditional branches (e.g., "if not enough high-quality reports can be retrieved, switch keywords or data sources"), loops (e.g., "continuously attempt rewriting and compression until the report length meets the limit"), and exception paths (e.g., "if a data source is unreachable, switch to an alternative source or use an estimation method"). This requires the workflow orchestration layer to be able to express control-flow semantics such as if/else, while/for, and try/catch on the graph structure, and to allow the Planner Agent or upper-level orchestrator to make decisions based on real-time results during execution, rather than merely planning all steps upfront in a one-shot manner.
• Connection with tool calling Task decomposition and planning are tightly connected with the tool calling discussed in 7.1: when generating subtasks, the Planner often simultaneously specifies "which tools/agents this task requires" and "the input/output format of this node," laying the groundwork for subsequent automatic parameter filling and tool execution. Some systems adopt an explicit two-phase "Plan + Execute" approach: the Planner first outputs a machine-readable plan (such as a JSON workflow description), and then the Executor strictly invokes tools and agents according to the plan. Other systems adopt a ReAct style, weaving "thinking–tool calling–observation–rethinking" into the same conversation to achieve more flexible adaptive execution.

7.2.2 Multi-Agent Collaboration: Letting a "Virtual Team" Play to Their Strengths

A single large model is certainly powerful, but in complex business scenarios, different domains often require different knowledge structures, style preferences, and safety strategies. The idea behind multi-agent collaboration is to decompose a single "big and comprehensive" intelligence into multiple "specialized and refined" roles: someone is responsible for planning, someone for execution, someone for review, and someone for domain-specific professional judgment, forming a virtual team composed of agents + tools + humans.

• Role division: planning, execution, and review In a typical multi-agent workflow, common roles include:
  • Planning Agent: Responsible for understanding user requirements, designing the overall plan, decomposing subtasks, and dynamically adjusting the path based on results during execution.
  • Execution Agents: Deeply optimized around certain tools or subdomains (such as retrieval agents, data analysis agents, content writing agents), completing specific steps according to the plan.
  • Review Agent: Inspects and revises intermediate and final outputs from structural, logical, stylistic consistency, and risk-control perspectives, analogous to a "virtual editor/reviewer."
• Domain-expert agent coordination For highly specialized domains such as law, finance, engineering, and operations, domain-expert agents can be further subdivided: such as "legal advisor agent," "investment research analyst agent," "cloud-native ops agent," "advertising optimization agent," etc. They can participate in project-based collaboration based on domain-specific knowledge bases, tools, and even specially fine-tuned models: for example, in investment and financing materials, the technical agent handles the technical feasibility section, the financial agent handles the financial model and valuation, the legal agent handles compliance and risk disclosure, the operations agent handles market and growth strategy, and a master control agent consolidates and unifies the style.
• Collaboration protocols and message routing The key to multi-agent collaboration also lies in "who talks to whom and when." The system needs a message routing and coordination mechanism to:
  • Decide which agent should handle a given user request or intermediate result.
  • Maintain shared context and each agent's private memory.
  • Control parallel and serial execution, as well as conflict resolution (such as how to arbitrate when different agents propose contradictory suggestions). These capabilities are typically provided by an upper-level orchestrator or "management agent," while frameworks such as LangChain and AutoGen provide infrastructure at the engineering level for conversation routing, multi-agent sessions, role configuration, and more.

7.2.3 Human-in-the-Loop: Keeping Risk Gates in Hand

No matter how intelligent workflows and multi-agent collaboration become, real-world business still cannot completely dispense with human judgment, especially in high-risk, high-cost, high-sensitivity scenarios such as legal compliance, financial decision-making, medical advice, large-scale production changes, and public opinion response. The design of Human-in-the-loop is precisely about finding a balance between automation and controllability: automate what should be automated, and definitely pause for a human to review what requires manual confirmation.

• Manual confirmation at critical steps In the workflow graph, several "manual approval/confirmation nodes" are typically explicitly marked:
  • For example, when auto-generating a contract, dual confirmation from legal and business leads is required before issuance;
  • In automated ops systems, operations involving production environment changes, batch restarts, or configuration modifications must be confirmed by the on-call engineer with a click;
  • In content generation scenarios, content intended for broad public release or that is brand-sensitive requires manual review. The Orchestrator pauses automatic execution at these nodes, sends intermediate results to the corresponding human roles, and continues the subsequent flow only after receiving feedback.
• Feedback-driven strategy updates Humans don't just "press approve or reject" at a certain moment; more importantly, the content of their feedback can be absorbed by the system:
  • Compare human-edited versions with original outputs and record them as "positive/negative examples" for subsequent prompt optimization or model fine-tuning.
  • Based on statistical analysis, identify which types of tasks/steps are most frequently modified by humans, and then optimize the corresponding agent's prompts, tool combinations, or workflow design.
  • In extreme or anomalous cases, humans can add "blacklists / whitelists / special rules" that directly influence the system's strategy selection in similar situations.
• Risk grading and observability Finally, Human-in-the-loop also requires a clear set of risk-grading and observability mechanisms:
  • Classify processes into different risk levels based on dimensions such as task type, scope of impact, monetary scale, and sensitive information involved, corresponding to different intensities of human intervention (e.g., read-only review, mandatory approval, multi-level approval).
  • Through logs, audits, visualization dashboards, and other means, enable operations/management personnel to track at any time which tasks are running, which step they are at, where human intervention was triggered, and what failures and manual corrections have occurred historically. These capabilities not only improve the system's acceptability within the enterprise but also provide a foundation for subsequent compliance audits and responsibility attribution.

Taken together, tool calling and execution (7.1) addresses the problem of "single-step action," while workflow orchestration and multi-agent collaboration (7.2) attempts to answer "how to string many steps together so that different roles can collaborate long-term and operate in a controllable manner." The combination of both, along with Human-in-the-loop and sound engineering practices, forms the next-generation intelligent application foundation for real-world business scenarios.