Summary

For my role supporting a product marketing team, I wanted to create a RAG (retrieval-augmented generation) agent to help the program's stakeholders find content and be a resource for FAQs on process, policies, resources, etc.

I quickly ran into several issues — while consumer-facing LLMs (large language models) can be great at interpreting intent and using the context within their environments, developing agents to do the same is a completely different ballgame.

The key lessons I learned were that you need some form of unified, intelligent orchestration to interpret user queries, and you need to insert LLM intelligence at key decision points in your architecture or you will end up with a glitchy, ineffective agent. In this post, I will share how I implemented this with Copilot Studio, Power Automate, and the OpenAI ChatGPT LLM connector.

Initial setup

Based on the available tech stack in my corporate environment, Copilot Studio was the best available resource to build the agent — so I set about configuring a new agent and experimenting with using it to answer FAQs. I started by adding files as a Knowledge Source. Then I experimented with separate Power Automate Flows to allow the agent to conduct different kinds of data lookups.

These attempts resulted in several issues that made it clear I needed to rethink my approach:

Early issues

• Incomplete answers
• Inaccurate answers
• Multiple chained responses to the same question, with no clear way to determine which was correct
• Agent inability to choose correct data source to answer a question

Solution

The solution I landed was a unified search architecture that merged all the lookup Flows into one master Flow. I then implemented an initial intent parsing/classification step that uses the OpenAI ChatGPT LLM connector in Power Automate to determine the type of question being asked and provide it as a JSON object used to filter the request into the correct data source.

The next step was to implement a switch router that takes the classified intent (JSON object) generated in the previous step and routes the request to the correct RAG pipeline. Each RAG pipeline has its own connected data sources and its own LLM connectors with custom prompts for intelligent processing and retrieval based on the question type, and formatted responses optimized for consumption by the agent.

• Single Power Automate Flow connected as a Tool in the agent configuration
• Initial intent parsing step that uses an LLM connector to determine the type of question being asked and provides it as a JSON object used to route the request in the next step
• Switch router that takes the intent generated in the previous step (JSON object) and routes the request to the correct RAG pipeline
• Several separate RAG pipelines (branches of a single Power Automate flow) that each have their own connected data sources, own LLM connectors with custom prompts for intelligent processing and retrieval, and formatted responses optimized for consumption by the agent
• Guidance in the agent Instructions to treat follow up questions as net-new, while retaining context for refinement of user intent, to avoid re-use of previous data lookups resulting in potentially inaccurate information