Chat-based Knowledge Retrieval Agent

Agent Description:

The Chat-based Knowledge Retrieval Agent provides a conversational interface for querying specific knowledge bases. It uses the Knowledge Retriever and an MCP Client tool to search designated PDF datasets for relevant information. The agent retrieves and ranks documents by Confidence Score (threshold ≥ 0.65) to determine result reliability. Finally, the Knowledge Responder generates a summarized, conversational answer for high-confidence matches or informs the user if no sufficiently confident result is found.

Purpose and Components
  • Purpose: To provide a reliable chat interface for users to query specific knowledge bases, receive ranked results based on relevance, and get summarized, conversational answers, while also indicating the confidence level of the information found.
  • Components:

    • Knowledge Retriever: An agent that searches a specified PDF dataset based on a user query and ranks results by confidence.

    • Knowledge Responder: An agent that formulates a user-friendly answer based on the retrieved documents and their confidence scores.

    • MCP Client: A tool used to access and read the PDF dataset.

Supported Capabilities

  • Receiving user queries in natural language.

  • Reading specified PDF datasets via an MCP Client connection.

  • Searching the dataset and retrieving relevant documents based on the query.

  • Ranking retrieved documents by Confidence Score.

  • Applying a confidence threshold (0.65) to classify results as "High Confidence" or "Low Confidence".

  • Summarizing the content of high-confidence documents into a natural, conversational answer.

  • Providing a specific response indicating low confidence if no strong match is found.

  • Optionally showing the closest low-confidence match with a disclaimer.

  • Delivering user-friendly, concise, and context-aware responses.

LLM Used

  • GPT_4O_MINI

    Note: To learn more about the LLM and to modify its behavior, refer to the Configuring LLM settings section.

Sub-Agents

1. Knowledge Retriever

  • Role: Data Extractor

  • Scope: Interacts with users to fetch accurate, context-aware information from databases or knowledge bases in real time.

  • Description: This agent takes the user's query and uses the MCP Client tool to search within a specific PDF dataset (identified by dataset_id). It returns a ranked list of matching documents, including their title, content snippet, ID, and a calculated ConfidenceScore. It then applies a threshold (0.65) to flag each result as "High Confidence" or "Low Confidence" before passing them to the next agent.

  • LLM Used: Default (Inherits from parent agent).

  • Tools Used: MCP Client

2. Knowledge Responder

  • Role: Query Responder

  • Scope: Generate a conversational, context-aware answer using retrieved documents.

  • Description: This agent receives the ranked documents and confidence flags. If it receives "High Confidence" results, it summarizes the most relevant document(s) into a conversational answer. If all results are "Low Confidence," it informs the user that a confident match wasn't found, potentially offering the closest match with a disclaimer.

  • LLM Used: Default (Inherits from parent agent).

  • Tools Used: None

Tools Used:

  • MCP Client: Connects via SSE to access and query the specified PDF knowledge base.

Note: For details on modifying the Tools, refer Tools Library section.
Agent Workflow Behavior Summary

  1. A user submits a query to the Knowledge Retriever (start node).

  2. The Retriever uses the MCP Client tool to search the designated PDF dataset based on the query.

  3. It retrieves and ranks documents by relevance, calculating a ConfidenceScore for each and flagging them as "High" or "Low" confidence (threshold ≥ 0.65).

  4. These ranked results and flags are passed to the Knowledge Responder.

  5. The Knowledge Responder (end node) checks the confidence flags. If high confidence matches exist, it generates a summarized, conversational answer. If not, it informs the user about the low confidence or lack of a match.

Sample Questions:

  • What is our company's policy on remote work?

  • Explain the procedure for submitting expense reports.