help-calc

08 Numbers and Units

LlamaIndex RAG Docs: How It Works & Setup

Card 08-0245 · Updated · Standard formula, shown below

Total tokens processed 2,560 tokens
Effective context per chunk
90.2 %
Result, rounded to 1 decimal place
Number of chunks
5
Input
Chunk length (tokens)
512 tokens
Input
You enter
Number of chunks, Chunk length (tokens) (tokens), Overlap (tokens) (tokens)
You get
Total tokens processed (tokens), Effective context per chunk (%)
Rounding
0 dp · 1 dp

LlamaIndex RAG docs refer to the official documentation and guides for building Retrieval-Augmented Generation (RAG) applications using the LlamaIndex framework. You use these docs to learn how to connect your own data (PDFs, databases, APIs) to a large language model so it can answer questions based on that data. This article explains what the docs cover, how the core RAG pipeline works, and how to get started without getting lost in the details.

Formula

The calculator above estimates the token load of a simple RAG retrieval step. The total tokens processed is simply the number of chunks multiplied by the chunk length:

Total tokens = n × cl

Where:

  • n = number of chunks retrieved (integer, no unit)
  • cl = chunk length in tokens (e.g., 512 tokens per chunk)

The effective context percentage shows how much of each chunk is new information after accounting for overlap:

Effective context % = (cl − ol) / cl × 100

Where ol is the overlap in tokens between consecutive chunks. This matters because overlapping chunks are often used to preserve context across boundaries, but they also mean the same text is counted twice in the total.

Worked Example

Using the default values from the calculator: 5 chunks, each 512 tokens, with 50 tokens of overlap.

  1. Total tokens = 5 × 512 = 2,560 tokens
  2. Effective context per chunk = (512 − 50) / 512 × 100 = 462 / 512 × 100 = 90.2%

So the retrieval step would process 2,560 tokens, and each chunk contributes about 90% new content relative to its neighbor. This is a rough planning number – actual tokenizers vary by model.

What LlamaIndex Actually Is

LlamaIndex (formerly GPT Index) is an open-source data framework designed to connect large language models (LLMs) to your own data. The official docs at docs.llamaindex.ai cover everything from basic installation to advanced agentic workflows. The core idea is that instead of sending your entire database to the model (which is impossible due to context limits), you index your data into chunks, retrieve the most relevant chunks for a query, and feed only those to the LLM along with the question.

The docs are organized into several major sections: getting started, core concepts (nodes, indices, retrievers), advanced features (agents, workflows, observability), and integrations with vector databases like Pinecone, Weaviate, or Chroma. Most users start with the “High-Level Concepts” page, which explains the five stages of a RAG pipeline: loading, indexing, storing, querying, and evaluating.

How the RAG Pipeline Works in the Docs

The RAG pipeline in LlamaIndex is described as a series of steps, each with its own documentation page:

  • Loading: Ingest documents from PDFs, HTML, Notion, Google Drive, or even SQL databases using data connectors (called “readers”).
  • Indexing: Parse the documents into “nodes” (chunks of text) and build an index. The most common index is the vector store index, which embeds each chunk into a high-dimensional vector.
  • Storing: Save the index and metadata to a storage backend. In-memory works for demos; production uses a vector database.
  • Querying: For a user query, embed the query, find the most similar chunks (top-k), and pass them to the LLM with the prompt.
  • Evaluating: Measure response quality using metrics like faithfulness or relevancy. The docs include an evaluation module for this.

A key detail in the docs is the difference between “retrieval” and “query engine”. The query engine is the end-to-end interface that takes a natural language query and returns a response. Under the hood, it uses a retriever (which fetches nodes) and a response synthesizer (which formats the prompt and parses the LLM output). The docs walk through customizing each piece.

How to Choose the Right Index Type

The docs describe several index types, and choosing the wrong one is a common beginner mistake. The main ones are:

  • VectorStoreIndex: Best for semantic similarity search. It embeds chunks and retrieves by cosine similarity. Use this for most question-answering over unstructured text.
  • SummaryIndex: Retrieves all chunks and lets the LLM summarize. Good for queries that require the whole document, like “what is the main argument?”
  • TreeIndex: Builds a hierarchical tree of nodes. Useful for large documents where you want to navigate from summary to detail.
  • KeywordTableIndex: Extracts keywords from each chunk and matches queries by keyword. Faster but less accurate for synonyms or paraphrases.

The docs recommend starting with VectorStoreIndex for most use cases because it balances accuracy and simplicity. For structured data like SQL, you would use a database-specific index or the “structured” query engine instead.

Common Problems and Misconceptions

Reading the docs, you’ll notice three frequent pain points:

  1. Chunk size tuning: Too small chunks lose context; too large chunks dilute relevance. The docs suggest starting with 512 tokens and an overlap of 50, then adjusting based on evaluation results.
  2. Embedding model mismatch: The embedding model used for indexing must match the one used for querying. If you change models, you must rebuild the index. The docs emphasize this because it’s a silent source of poor results.
  3. Hallucination vs. missing data: If the retriever doesn’t find the right chunk, the LLM will still answer, often confidently wrong. The docs recommend setting a similarity cutoff or using a “reranker” to improve precision.

A common misconception is that RAG eliminates the need for prompt engineering. In reality, the prompt template that wraps the retrieved context is crucial. The docs provide examples of custom prompt templates and show how to include instructions like “answer only from the context” to reduce hallucination.

Costs and Performance Considerations

Using LlamaIndex itself is free (open-source), but you pay for the underlying LLM API calls and vector storage. The docs include a cost estimation page that breaks down token usage. For a typical RAG query, you incur:

  • Embedding cost for indexing (one-time per document) and for each query (to embed the question).
  • LLM cost for the final generation, which depends on the number of retrieved chunks and prompt template length.
  • Vector database hosting cost if you use a cloud service like Pinecone or Supabase.

To reduce costs, the docs suggest caching embeddings, using smaller models for retrieval, and limiting the number of chunks retrieved (top-k). The calculator above helps estimate the token load of the retrieval step, which is the main driver of cost.

Safety and Privacy in RAG Applications

When building RAG with your own data, the docs emphasize that you are responsible for data privacy. Unlike a public chatbot, your documents may contain sensitive information. Key safety practices covered in the docs include:

  • Never send private data to a public LLM API unless you have a data processing agreement.
  • Use local or self-hosted models (e.g., Ollama, vLLM) if you need full control.
  • Implement access controls on the retrieval layer – the index should only return chunks the user is authorized to see.
  • Sanitize prompts to prevent prompt injection from malicious documents (e.g., a PDF containing “ignore previous instructions”).

The docs do not provide legal advice, but they do link to security best practices for LLM applications. For production, they recommend using the built-in tracing and evaluation tools to monitor for unexpected behavior.

Alternatives to LlamaIndex

LlamaIndex is not the only RAG framework. The docs themselves mention that you can use it with LangChain, but there are also standalone alternatives:

  • LangChain: A broader framework with more integrations for agents and chains. It has its own RAG modules, but many users find it more complex.
  • Haystack: An older framework focused on search and QA, with a strong pipeline abstraction.
  • Semantic Kernel: Microsoft’s SDK that integrates with Azure OpenAI and has RAG patterns.
  • Custom code: For simple use cases, you can write your own RAG with just an embedding model and a vector database, but you would have to handle chunking, indexing, and retrieval yourself.

The choice often comes down to how much control you need. LlamaIndex is praised for its fine-grained control over the indexing and retrieval steps, while LangChain offers more pre-built agent tools. The docs include a comparison page that highlights these differences.

Frequently Asked Questions

Do I need to read all the docs before using LlamaIndex? No. Start with the “Getting Started” tutorial, which builds a simple RAG pipeline in about 20 lines of code. Then consult specific sections only when you need to customize something.

Can I use LlamaIndex with local models? Yes. The docs have a “Local Models” guide that shows how to use Ollama or HuggingFace models for both embeddings and generation. This keeps your data on your own machine.

What is the difference between a node and a document? A document is the original file (PDF, text, etc.). A node is a chunk of text from that document, often with metadata like source page or heading. The index stores nodes, not whole documents.

How do I update my index when my data changes? The docs recommend either rebuilding the index (for small datasets) or using the “insert” and “delete” methods on the index to update specific nodes. For large datasets, you typically use a vector database and update only the changed documents.

Why are my RAG answers so bad? Most likely the retriever is not finding the right chunks. Check your chunk size, embedding model, and similarity threshold. The docs have a “Evaluating” section that shows how to measure retrieval accuracy separately from generation quality.

Is LlamaIndex production-ready? Yes, but you need to add your own monitoring, error handling, and scaling. The docs cover deployment patterns with FastAPI, Docker, and cloud services, but they do not provide a one-click production setup.

Related

Formula used
in n | Number of chunks | - | 5 | 1 | 50
in cl | Chunk length (tokens) | tokens | 512 | 50 | 4096
in ol | Overlap (tokens) | tokens | 50 | 0 | 500
out total | Total tokens processed | tokens | n * cl | 0
out eff | Effective context per chunk | % | (cl - ol) / cl * 100 | 1

Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More on this shelf

The whole Numbers and Units shelf →