Azure-Samples/azure-search-openai-demo: The RAG Template That Shows Its Work

A deep look at the Azure sample that turns enterprise search, semantic reranking, secure citations, and multimodal ingestion into one production-minded reference app.

10 min read • View on GitHub • More from Azure-Samples

A developer studies a split-screen terminal where one side rewrites a user question into a cleaner search prompt and the other side fans out retrieved documents, citations, and ranked results. A transparent overlay shows the final answer assembling from those parts, making the internal flow visible instead of hidden.
This sample’s real trick is not the chat box. It is the way it exposes the machinery that turns a question into a grounded answer.

We've combined the power of Azure OpenAI Service with Azure Cognitive Search to create a unique experience that allows you to chat with your own data.

Pamela Fox, Cloud Advocate at Microsoft · ChatGPT + Enterprise Data with Azure OpenAI and Cognitive Search
Key Takeaways

Most RAG demos promise a chat interface. This one promises something harder to ship: a system that can explain why an answer exists. The moment you open the app’s Thought Process view, the project stops feeling like a toy and starts feeling like an audit trail.

The app does not just answer. It explains.

That is the central idea behind Azure-Samples/azure-search-openai-demo. A user asks a question, but the app surfaces the rewritten query, the retrieval path, and the citations that shaped the final response. For enterprise teams, that is not a nice-to-have. It is the difference between a demo you admire and a system you can debug.

Why Microsoft built the whole stack, not just a library

This is a Microsoft sample, but it behaves like a reference architecture. The repo ties together frontend, backend, search, identity, ingestion, and deployment so the hard parts live in one place. That matters because enterprise RAG fails in the seams: auth, citations, observability, and infrastructure are where nice ideas go to die.

Hedcut-style portrait of Pamela Fox, based on her GitHub avatar. The portrait identifies the maintainer associated with the sample and anchors the article’s origin story in a real person rather than an anonymous project.

This sample demonstrates how to implement the RAG (Retrieval-Augmented Generation) pattern, which is a powerful way to enhance the capabilities of large language models by grounding them in your own data.

Liam Cavanagh, Principal Program Manager, Azure AI Search · GitHub Repository README

The real core is a three-step retrieval loop

At the heart of the backend is a simple pattern: rewrite, retrieve, read. The user’s question and conversation history are first condensed into a better search query. That rewritten query then fans out into Azure AI Search. The best matches are passed into the model, which composes a grounded answer from the retrieved context.

The app’s signature flow is not a single search call. It is a staged pipeline that rewrites the question, retrieves evidence, and then reads that evidence back into a grounded answer.

# Conceptual flow from the backend approach layer
rewritten_query = rewrite_question(chat_history, user_question)
results = search(rewritten_query, modes=["vector", "keyword", "hybrid"])
ranked = semantic_rank(results)
answer = synthesize_answer(ranked, chat_history)
return {
    "answer": answer,
    "citations": extract_citations(ranked),
    "thought_process": build_thought_steps(rewritten_query, ranked),
}
A close-up conveyor system moves document pages through three stations. One station slices and labels pages, another injects embeddings into a search vault, and the last deposits ranked fragments into an answer box with citation tags attached. The image explains how ingestion and retrieval become one continuous production pipeline.
The boring part of RAG is usually the real product. Chunking, labeling, embedding, and indexing decide whether the model ever sees the right context.

Retrieval is not one search. It is a stack of search behaviors.

The repo teaches a decision system, not just an API call. It can combine keyword and vector retrieval, then let Azure AI Search’s semantic ranker reorder the results before the model sees them. That layered approach is the difference between "found something" and "found the right thing."

LayerPlain vector demoThis Azure sample
Query handlingSend the user prompt as-isRewrite the question before retrieval
RetrievalSingle vector lookupHybrid search with keyword and vector signals
RankingTop-k similarity onlySemantic reranking before answer generation
TransparencyHidden chain of thoughtThought Process UI shows the path
OutputAnswer textAnswer text plus citations and provenance

The implementation also leans on strategy classes under approaches/, which makes the retrieval logic swappable instead of hardwired. That is useful in production because search behavior changes faster than UI expectations.

The ingestion pipeline is where enterprise becomes real

If the frontend is the promise, prepdocs.py is the machinery that keeps it honest. The pipeline parses files, chunks content, generates embeddings, and loads documents into Azure AI Search. Integrated vectorization can even push part of the embedding work into the search service itself, which is exactly the kind of detail teams need when the corpus is large or messy.

This is also where the sample stops looking like a notebook and starts looking like a platform. Documents are not an afterthought. They are the product, and the ingestion path is part of the user experience because it determines what the model can know.

Security is not bolted on here

The repo treats identity and access as part of retrieval, not a separate concern. Microsoft Entra ID, RBAC, and permission-aware citations help ensure users only see what they are allowed to see. That is a major differentiator versus generic RAG demos, which often discover security only after the first internal pilot goes sideways.

ConcernGeneric RAG demoThis Azure sample
IdentityOptional or deferredIntegrated with Microsoft Entra ID
Citation accessAssumed publicPermission-aware content delivery
DeploymentAd hocBicep and azd friendly
ObservabilityUsually missingOpenTelemetry and Azure Monitor aligned

Multimodal support hints at where RAG is going

The sample is not frozen in text-only search. Image embeddings and media description support point toward a broader model of document understanding, where charts, screenshots, and layouts can become first-class retrieval targets. That shift matters because many enterprise documents are visual before they are textual.

The result is a sample that keeps moving up the abstraction ladder. It starts with chunks and citations, then pulls in images and richer document interpretation without abandoning the core RAG discipline.

Why this sample still matters compared with LangChain, LlamaIndex, Semantic Kernel, and AWS

The cleanest way to read this repo is as an application template, not a framework. LangChain and LlamaIndex are broader libraries. Semantic Kernel is a middleware layer. AWS has its own enterprise search solution. Azure-Samples/azure-search-openai-demo is different because it gives you an opinionated Azure-native stack with a working UI, deployment path, security defaults, and a retrieval story you can inspect.

OptionWhat it isWhy this repo is different
LangChainGeneral-purpose LLM frameworkThis repo is a full application template, not a library
LlamaIndexData framework for LLM appsThis repo is more deployment-ready and Azure-specific
Semantic KernelMicrosoft orchestration SDKThis repo is simpler and more transparent about the underlying calls
AWS enterprise searchCloud-specific enterprise solutionThis repo is tightly aligned to Azure identity, search, and deployment

That is why the sample has endured. It does not just show what is possible. It shows what a serious Azure RAG app needs before anyone can trust the answer.