Custom Tool Selection

Effective tool selection is critical for tool-using agents. It ensures that agents choose the right tool for each task, leading to accurate, efficient, and purposeful interactions. This guide explores strategies and techniques for implementing your own tool selection in Prisme.ai.

These guides provide example YAML files for educational purposes, showcasing the range of possibilities. It is up to the technical teams to implement them according to your internal tools and APIs.

Understanding Tool Selection

Tool selection is the process by which an AI agent:

Recognizes when a tool is needed to fulfill a user request
Evaluates which available tool is most appropriate
Prepares the necessary parameters for tool execution
Decides whether to use the tool results or try alternative approaches

Effective tool selection balances accuracy (choosing the right tool) with efficiency (minimizing unnecessary tool usage).

The Tool Selection Challenge

Tool selection presents several key challenges:

Ambiguous Requests

User requests may not clearly indicate which tool is needed

Multiple Valid Options

Several tools might satisfy the request in different ways

Parameter Extraction

Identifying and formatting the required parameters from natural language

Decision Complexity

Balancing multiple factors including capabilities, costs, and permissions

Tool Selection Strategies

Prisme.ai supports several approaches to tool selection, each with different strengths:

LLM-Based Selection
Rule-Based Selection
Hybrid Approach

Leverage the language model’s reasoning to select appropriate tools.How it works:

Tools are described to the LLM with clear purposes and capabilities
The LLM analyzes user requests and determines tool requirements
Function calling capabilities enable structured tool invocation
The LLM formats parameters based on tool schemas

Best for:

General-purpose agents with diverse tools
Complex decision-making with nuanced criteria
Adapting to varied user request formulations
Scenarios where selection logic is difficult to formalize

Implementing LLM-Based Selection

For many applications, LLM-based selection provides the best balance of flexibility and effectiveness:

Craft Clear Tool Descriptions

Provide the LLM with detailed information about each tool.Effective description elements:

name: CustomerLookup
description: |
  Use this tool to find customer information in the CRM system.
  This tool should be used when the user asks about specific customer details,
  account status, or purchase history.
  Do NOT use this tool for general product inquiries or non-customer-specific questions.
  The tool requires either a customer ID or an email address to perform the lookup.

Key components include:

Clear purpose statement
When to use the tool
When NOT to use the tool
Required parameters
Expected outcomes

Provide Selection Guidelines

Include explicit guidance for tool selection in your agent instructions.Example guidelines:

When choosing between available tools:

Always use the most specific tool for the task
Prefer internal data sources over external ones when both contain the information
Use the ProductSearch tool for general product inquiries and ProductDetails for specific product information
Only use the CustomerUpdate tool when explicitly asked to change customer information
If multiple tools could work, explain your selection reasoning to the user

These guidelines help the LLM make consistent decisions aligned with business preferences.

Implement Parameter Extraction

Guide the LLM in extracting and formatting parameters from user requests.Example guidance:

When extracting parameters for tools:

For customer lookups, identify any customer ID (format: CUS-XXXXX) or email address
For product searches, extract product type, category, price range, and features
For date ranges, convert relative dates (e.g., "last week") to specific date formats
Always confirm ambiguous parameters with the user before tool execution
Format numeric parameters according to each tool's requirements

Effective parameter extraction ensures tools receive correctly formatted inputs.

Design Error Recovery

Prepare for tool selection errors with recovery strategies.Example recovery instructions:

If a tool returns an error or no results:

First, check if parameters were correctly formatted and try again with adjustments
If parameters are correct but no results found, try an alternative tool if available
For customer lookups that fail, try searching by alternate identifiers
If all tool attempts fail, explain to the user what was tried and ask for additional information
Never make up information if tools cannot retrieve it

These strategies ensure resilience when initial tool selections don’t succeed.

Example: LLM-Based Tool Selection

Here’s a complete example of custom implementation of LLM-based tool selection:

slug: llm-tool-selection
name: "LLM-Based Tool Selection"
description: "Uses the LLM to select appropriate tools based on user input"
do:
  - AgentAnalyzer.chat-completion:
      prompt: |
        Select the most appropriate tool for this user request:
        User request: "{{input.text}}"

        Available tools:
        1. WeatherTool - Gets current weather information for a location
        2. CalendarTool - Manages calendar events and appointments
        3. StockTool - Retrieves current stock prices and market information
        
        Respond with a JSON object containing:
        - selectedTool: name of the selected tool
        - reasoning: why this tool was selected
        - parameters: parameters to pass to the tool
      output: toolSelection
  
  - conditions:
      '{{toolSelection.selectedTool}} == "WeatherTool"':
        - WeatherTool.execute:
            location: "{{toolSelection.parameters.location}}"
            output: result
      '{{toolSelection.selectedTool}} == "CalendarTool"':
        - CalendarTool.execute:
            action: "{{toolSelection.parameters.action}}"
            date: "{{toolSelection.parameters.date}}"
            output: result
      '{{toolSelection.selectedTool}} == "StockTool"':
        - StockTool.execute:
            symbol: "{{toolSelection.parameters.symbol}}"
            output: result
      default:
        - set:
            name: result
            value:
              error: "No appropriate tool selected"

output: "{{result}}"
when:
  events:
    - user-request

Implementing Rule-Based Selection

For more controlled environments, rule-based selection provides predictability and reliability:

Pattern-Based Routing

Implement explicit patterns that map user requests to specific tools.Example implementation:

slug: rule-based-tool-selection
name: "Rule-Based Tool Selection"
description: "Uses explicit rules to determine which tool to use"
do:
  - EntityExtractor.extract:
      text: "{{input.text}}"
      output: entities
  - set:
      name: selectedTool
      value: "None"
 
  - conditions:
      '{{input.text}} matches "weather" && {{entities.locations.length}} > 0':
        - set:
            name: selectedTool
            value: "WeatherTool"
        - set:
            name: toolParameters
            value:
              location: "{{entities.locations[0]}}"

  - conditions:
      '{{input.text}} matches "schedule" && {{entities.dates.length}} > 0':
        - set:
            name: selectedTool
            value: "CalendarTool"
        - set:
            name: toolParameters
            value:
              action: "{{entities.action}}"
			  date: "{{entities.dates[0]}}"
  
  - conditions:
      '{{input.text}} matches "stock" && {{entities.companies.length}} > 0':
        - set:
            name: selectedTool
            value: "StockTool"
        - set:
            name: toolParameters
            value:
              symbol: "{{entities.companies[0].symbol}}"  
  - conditions:
      '{{selectedTool}} == "WeatherTool"':
        - WeatherTool.execute:
            location: "{{toolParameters.location}}"
            output: result
      '{{selectedTool}} == "CalendarTool"':
        - CalendarTool.execute:
            action: "{{toolParameters.action}}"
            date: "{{toolParameters.date}}"
            output: result
      '{{selectedTool}} == "StockTool"':
        - StockTool.execute:
            symbol: "{{toolParameters.symbol}}"
            output: result
      default:
        - set:
            name: result
            value:
              message: "I couldn't determine which tool would help with your request."

output: "{{result}}"
when:
  events:
    - user-request

This approach:

Uses pattern matching on user input text
Extracts and validates required entities
Maps patterns to specific tool selections
Provides a clear fallback for unrecognized patterns

Intent-Based Routing

Use intent classification to determine appropriate tools.Example implementation:

do:
  - IntentClassifier.classify:
      input: '{{input.text}}'
      output: intent
  - conditions:
      '{{intent.primary}} == "get_account_balance"':
        - AccountBalance.execute:
            input: '{{input.text}}'
            output: toolResult
      '{{intent.primary}} == "transfer_funds"':
        - FundsTransfer.execute:
            input: '{{input.text}}'
            output: toolResult
      '{{intent.primary}} == "report_fraud"':
        - FraudReport.execute:
            input: '{{input.text}}'
            output: toolResult
      default:
        - conditions:
            '{{intent.confidence}} < 0.7':
              - IntentClarification.execute:
                  input: '{{input.text}}'
                  intent: '{{intent}}'
                  output: toolResult
            default:
              - GeneralAssistant.execute:
                  input: '{{input.text}}'
                  output: toolResult

This approach:

Uses a dedicated intent classification model
Routes requests based on classified intent
Considers confidence scores in decision-making
Includes clarification flows for low-confidence cases

Example: Hybrid Selection Approach

Most enterprise applications benefit from hybrid approaches that combine LLM flexibility with rule-based governance:

slug: hybrid-tool-selection
name: "Hybrid Tool Selection"
description: "Combines rule-based filtering with LLM ranking for optimal tool selection"
do:
  - EntityExtractor.extract:
      text: "{{input.text}}"
      output: entities
  - set:
      name: eligibleTools
      value: []
  
  - conditions:
      '{{input.text}} matches "weather" && {{entities.locations.length}} > 0':
        - set:
            name: eligibleTools[]
            value:
              name: "WeatherTool"
              parameters:
                location: "{{entities.locations[0]}}"
  
  - conditions:
      '{{input.text}} matches "schedule" && {{entities.dates.length}} > 0':
        - set:
            name: eligibleTools[]
            value:
              name: "CalendarTool"
              parameters:
                action: "{{entities.action}}"
                date: "{{entities.dates[0]}}"
  
  - conditions:
      '{{input.text}} matches "stock" && {{entities.companies.length}} > 0':
        - set:
            name: eligibleTools[]
            value:
              name: "StockTool"
              parameters:
                symbol: "{{entities.companies[0].symbol}}"
  
  - conditions:
      '{{eligibleTools.length}} > 1':
        - Knowledge Client.chat-completion:
            prompt: |
              Rank these tools for the user request: "{{input.text}}"
              
              Available tools:
              {{eligibleTools}}
              - {{tool.name}}              
                Respond with a JSON array of tool names, ranked from most to least appropriate.
            output: rankedTools
        - set:
            name: selectedTool
            value: "{{rankedTools[0]}}"
      '{{eligibleTools.length}} == 1':
        - set:
            name: selectedTool
            value: "{{eligibleTools[0].name}}"
        - set:
            name: toolParameters
            value: "{{eligibleTools[0].parameters}}"
      default:
        - Knowledge Client.chat-completion:
            prompt: |
              Select the most appropriate tool for this user request:
              User request: "{{input.text}}"
              
              Available tools:
              1. WeatherTool - Gets current weather information for a location
              2. CalendarTool - Manages calendar events and appointments
              3. StockTool - Retrieves current stock prices and market information
              
              Respond with a JSON object containing:
              - selectedTool: name of the selected tool
              - parameters: parameters to pass to the tool
            output: llmSelection
        - set:
            name: selectedTool
            value: "{{llmSelection.selectedTool}}"
        - set:
            name: toolParameters
            value: "{{llmSelection.parameters}}"
 
  - conditions:
      '{{selectedTool}} == "WeatherTool"':
        - WeatherTool.execute:
            location: "{{toolParameters.location}}"
            output: result
      '{{selectedTool}} == "CalendarTool"':
        - CalendarTool.execute:
            action: "{{toolParameters.action}}"
            date: "{{toolParameters.date}}"
            output: result
      '{{selectedTool}} == "StockTool"':
        - StockTool.execute:
            symbol: "{{toolParameters.symbol}}"
            output: result
      default:
        - set:
            name: result
            value:
              message: "I couldn't determine which tool would help with your request."

output: "{{result}}"
when:
  events:
    - user-request

Example: Tool Definition

Here’s an example of how to define a weather tool in AI Builder:

slug: get-weather
name: "Get Weather Information"
description: "Returns current weather information for a specified location"
do:
  - fetch:
      url: "https://api.weatherservice.com/v1/current"
      method: GET
      headers:
        Authorization: "Bearer {{secret.weatherApiKey}}"
      query:
        location: "{{location}}"
      output: weatherData
  - set:
      name: output
      value:
        temperature: "{{weatherData.current.temperature}}"
        condition: "{{weatherData.current.condition.text}}"
        humidity: "{{weatherData.current.humidity}}"
        location: "{{weatherData.location.name}}, {{weatherData.location.country}}"
        updated: "{{weatherData.current.last_updated}}"

validateArguments: true
arguments:
  location:
    type: string
    description: "The city or location to get weather for"
  format:
    type: string
    enum: 
     - detailed
     - simple
    default: simple
    description: "Level of detail to return"
output: "{{output}}"
when:
  endpoint: true
labels:
  - weather
  - tools

Best Practices for Tool Selection

Balance Precision and Recall

Optimize selection logic to balance false positives and false negatives.Recommendations:

Evaluate both incorrect tool selections (false positives)
And missed opportunities to use tools (false negatives)
Track and analyze selection accuracy metrics
Iterate on selection logic based on real usage patterns

Implementation example:

# Confidence thresholds for tool selection
toolSelectionThresholds:
  # High-impact tools (e.g., making purchases, updating records)
  highImpact: 0.85
  # Medium-impact tools (e.g., retrieving information)
  mediumImpact: 0.75
  # Low-impact tools (e.g., informational lookups)
  lowImpact: 0.65

Provide Selection Transparency

Make tool selection reasoning visible to users when appropriate.Recommendations:

Explain which tool was selected and why
Indicate when the agent is using external systems
Show progress during tool execution
Provide context for tool results

Example agent response:

I'll check our product inventory system to find that information for you.

[Checking inventory...]

Based on our inventory system, the XPS 15 laptop is currently in stock at 3 locations:
- Downtown Store: 5 units
- Westside Mall: 2 units
- Online Warehouse: 15 units

Implement Progressive Tool Access

Start with limited tool access and expand based on need and performance.Recommendations:

Begin with core, high-reliability tools
Add tools incrementally as selection logic matures
Implement usage limits for new or sensitive tools
Monitor performance before expanding capabilities

Implementation approach:

# Tool tiers with progressive access
toolTiers:
  tier1: # Core tools available to all agents
    - ProductSearch
    - FAQlookup
  tier2: # Added after baseline performance validation
    - CustomerLookup
    - OrderStatus
  tier3: # Limited to specific use cases with approval
    - PaymentProcessing
    - AccountUpdate

Continuous Improvement

Use actual usage data to refine selection logic over time.Recommendations:

Implement usage logging for selection decisions
Analyze patterns in successful and unsuccessful selections
Identify common error cases and add specific handling
Periodically review and update selection criteria

Monitoring focus areas:

Tool selection accuracy
Tool usage distribution
User satisfaction with tool results
Error rates and types
Response time impacts

Tool Selection in AI Knowledge

For AI Knowledge agents, implement effective tool selection through these configurations:

Configure Tool Descriptions

Create detailed, LLM-friendly descriptions for each tool.Effective descriptions include:

Clear purpose and capabilities
When to use and when not to use
Example scenarios
Required parameter information
Expected result formats

Set Tool Selection Instructions

Add specific guidance for tool selection in agent instructions.Example instructions:

Available Tools:

1. Web Browsing: Use when the user needs current information not in our knowledge base. Only use for factual information that would be publicly available.

2. Customer Database: Use when the user asks about specific customer records. Requires a customer ID or email address. Never use for general inquiries.

3. Product Catalog: Use when the user needs detailed product specifications or inventory information. Prefer this over Web Browsing for any product-related queries.

When choosing tools:
- First check if the information is available in our knowledge base
- Only use Web Browsing for current events or information that changes frequently
- Always use the most specific tool for the task
- If multiple tools could work, explain your choice to the user

Add Usage Examples

Provide concrete examples of correct tool selection.Example:

Example 1:
User: "What's the current price of the XPS 15 laptop?"
Appropriate tool: Product Catalog
Reasoning: This is a specific product query about current pricing, which is exactly what the Product Catalog tool is designed for.

Example 2:
User: "Did we release any new products this month?"
Appropriate tool: Knowledge Base first, then Product Catalog
Reasoning: Check our knowledge base first for recent product announcements. If not found or if the information might be outdated, use the Product Catalog to check for products with recent release dates.

Example 3:
User: "What were the key announcements at yesterday's tech conference?"
Appropriate tool: Web Browsing
Reasoning: This is about a recent external event, so the information is likely not in our knowledge base and requires current web information.

Test and Refine

Evaluate tool selection performance with real-world scenarios.Testing approaches:

Create a test set of diverse queries
Evaluate tool selection accuracy
Identify patterns in incorrect selections
Refine descriptions and instructions
A/B test different instruction formats

Tool Selection in AI Builder

For advanced tool selection logic in AI Builder, implement these patterns:

Intent-Based Selection

Use a dedicated intent classification system for routing.Example implementation:

slug: intent-based-tool-selection
do:
  - MyApp.classify:
      input: '{{event.data.text}}'
      classes:
        - product_inquiry
        - order_status
        - account_question
        - technical_support
        - general_inquiry
      output: classification
  - emit:
      event: intent.classified
      data:
        originalText: '{{event.data.text}}'
        classification: '{{classification}}'
  - conditions:
      '{{classification.class}} == "product_inquiry" && {{classification.confidence}} > 0.7':
        - ProductTool.execute:
            input: '{{event.data.text}}'
            output: toolResult
      '{{classification.class}} == "order_status" && {{classification.confidence}} > 0.7':
        - OrderTool.execute:
            input: '{{event.data.text}}'
            output: toolResult
      default:
        - MyApp.generateResponse:
            input: '{{event.data.text}}'
            context: 'The user intent was classified as {{classification.class}} with confidence {{classification.confidence}}.'
            output: toolResult
  - emit:
      event: tool.result
      data:
        result: '{{toolResult}}'

Multi-Stage Selection Pipeline

Implement a pipeline approach for complex selection logic.Example implementation:

slug: multi-stage-tool-selection
do:
  - TextAnalysis.extract:
      text: '{{event.data.text}}'
      output: analysis
  - ToolFilter.apply:
      analysis: '{{analysis}}'
      tools: ['CustomerTool', 'ProductTool', 'OrderTool', 'SupportTool']
      output: eligibleTools
  - conditions:
      '{{eligibleTools.count}} > 0':
        - ToolRanker.rank:
            analysis: '{{analysis}}'
            tools: '{{eligibleTools}}'
            output: rankedTools
        - set:
            name: selectedTool
            value: '{{rankedTools[0].name}}'
      default:
        - set:
            name: selectedTool
            value: 'GeneralAssistant'
  - conditions:
      '{{selectedTool}} == "CustomerTool"':
        - CustomerTool.execute:
            input: '{{event.data.text}}'
            analysis: '{{analysis}}'
            output: result
      # Other tool conditions...
  - emit:
      event: tool.selected
      data:
        originalText: '{{event.data.text}}'
        selectedTool: '{{selectedTool}}'
        eligibleTools: '{{eligibleTools}}'
  - emit:
      event: tool.result
      data:
        result: '{{result}}'

Context-Aware Selection

Consider conversation history and context in the selection process.Example implementation:

slug: context-aware-selection
do:

  - ConversationManager.getContext:
      conversationId: '{{event.data.conversationId}}'
      output: context
  - ContextAnalyzer.analyze:
      currentInput: '{{event.data.text}}'
      context: '{{context}}'
      output: analysis
  - conditions:
      '{{analysis.isFollowUp}} && {{context.lastTool}} == "ProductTool"':
        - ProductFollowup.execute:
            input: '{{event.data.text}}'
            previousResult: '{{context.lastToolResult}}'
            output: result
      '{{context.activeWorkflow}} == "ordering"':
        - OrderWorkflow.continueFlow:
            input: '{{event.data.text}}'
            currentState: '{{context.workflowState}}'
            output: result
      default:
        - StandardSelection.select:
            input: '{{event.data.text}}'
            output: result
  - ConversationManager.updateContext:
      conversationId: '{{event.data.conversationId}}'
      updates:
        lastTool: '{{result.toolUsed}}'
        lastToolResult: '{{result.data}}'
        activeWorkflow: '{{result.workflow}}'
        workflowState: '{{result.workflowState}}'
  - emit:
      event: tool.result
      data:
        result: '{{result}}'

Next Steps

Ready to implement effective tool selection for your agents? Continue your journey with these resources:

Memory Management

Learn how to maintain state across tool interactions

Execution & Activity

Explore monitoring and debugging tool execution

Error Handling

Implement robust error management for tools

Multi-Agent Systems

Learn about systems with specialized agent collaboration

Getting Started

Prompting & RAG Agent

Tool-Using Agents

Custom Tool Selection

Understanding Tool Selection

The Tool Selection Challenge

Ambiguous Requests

Multiple Valid Options

Parameter Extraction

Decision Complexity

Tool Selection Strategies

Implementing LLM-Based Selection

Example: LLM-Based Tool Selection

Implementing Rule-Based Selection

Example: Hybrid Selection Approach

Example: Tool Definition

Best Practices for Tool Selection

Tool Selection in AI Knowledge

Tool Selection in AI Builder

Next Steps

Memory Management

Execution & Activity

Error Handling

Multi-Agent Systems

​Understanding Tool Selection

​The Tool Selection Challenge

Ambiguous Requests

Multiple Valid Options

Parameter Extraction

Decision Complexity

​Tool Selection Strategies

​Implementing LLM-Based Selection

​Example: LLM-Based Tool Selection

​Implementing Rule-Based Selection

​Example: Hybrid Selection Approach

​Example: Tool Definition

​Best Practices for Tool Selection

​Tool Selection in AI Knowledge

​Tool Selection in AI Builder

​Next Steps

Memory Management

Execution & Activity

Error Handling

Multi-Agent Systems

Understanding Tool Selection

The Tool Selection Challenge

Tool Selection Strategies

Implementing LLM-Based Selection

Example: LLM-Based Tool Selection

Implementing Rule-Based Selection

Example: Hybrid Selection Approach

Example: Tool Definition

Best Practices for Tool Selection

Tool Selection in AI Knowledge

Tool Selection in AI Builder

Next Steps