RAG with Markdown Chunks: Best Practices for Vector Stores

Retrieval-Augmented Generation (RAG) is the standard pattern to feed language models with your own knowledge. The setup sounds simple: split documents into chunks, load them into a vector store, find the most relevant chunks per query, send them as context to the LLM. In practice, however, chunk quality determines whether your RAG system produces useful answers or hallucinations. Markdown is the ideal input format here, but only if you prepare it correctly.

Why Markdown is better than plain text for RAG

Plain text has no structure. If you extract a PDF as running text and cut it into 500-character chunks, you slice sentences in the middle of arguments, mix topics, lose hierarchy. Markdown gives you explicit structure markers that you can use during chunking.

Concretely, Markdown has three advantages for RAG:

First, heading-based chunking. You can split documents along their H2 or H3 sections. Each chunk is then semantically coherent because it covers one topic area completely.

Second, preservation of lists and tables. In plain text, a table becomes a chaotic sequence of characters. In Markdown, it stays structured, the embedding model recognizes it as a table, retrieval scores improve.

Third, metadata tagging. You can use Markdown frontmatter (YAML at the file start) or structured headers to extract metadata per chunk: source, date, author, topic area. This metadata lands in the vector store and enables filtered searches.

The right chunk size

There is no universally right chunk size. It depends on your documents, your use case, and the embedding model. But there are reliable rules of thumb.

Very small chunks (100 to 300 tokens). Good for fact databases, short FAQs, definition searches. Bad for complex reasoning because context is lost.

Medium chunks (500 to 1,000 tokens). The sweet spot for most use cases. Enough context for meaningful arguments, small enough for precise retrieval hits.

Large chunks (1,500 to 3,000 tokens). Useful for narrative content like whole chapters or reports. Risk: the embedding becomes "diluted" because one chunk contains multiple topics. Retrieval precision drops.

Very large chunks (5,000+ tokens). Usually suboptimal because one chunk then mixes multiple unrelated topics. Exception: when you only want top-1 retrieval and the LLM works with large context windows.

Using chunk overlap correctly

Overlap means that consecutive chunks partially overlap. If an argument starts at the end of one chunk and continues in the next, the context is preserved.

Recommendation in practice: 10 to 20 percent of chunk size as overlap. For 800-token chunks, that's 80 to 160 tokens overlap.

Why not more? Higher overlap inflates the vector store and produces more redundant hits in retrieval. At 50 percent overlap, a search result can return the same content three times.

Why not less? Without overlap, sentences and arguments break at chunk boundaries. The corresponding chunk gets a low retrieval score because it lacks context.

Heading-based chunking

The smartest chunking strategy for Markdown is heading-based. Instead of cutting at a fixed 800 tokens, you cut at Markdown headings.

Approach:

1. Parse the Markdown file and identify all H1, H2, H3.
2. Each chunk starts at a heading and ends at the next heading of the same or higher level.
3. If a chunk gets too large (over 2,000 tokens), split it further at H3 headings or paragraph boundaries.
4. If a chunk gets too small (under 200 tokens), merge it with the next.

This approach produces semantically clean chunks because each one covers a complete topic section.

import re

def chunk_by_headings(markdown_text, min_tokens=200, max_tokens=1500):
    sections = re.split(r'^(#{1,3} .+)
#x27;, markdown_text, flags=re.MULTILINE)
    chunks = []
    current_chunk = ""
    
    for i in range(0, len(sections), 2):
        heading = sections[i+1] if i+1 < len(sections) else ""
        content = sections[i] if i > 0 else ""
        
        section = heading + "\n" + content
        token_count = len(section.split()) * 1.3
        
        if token_count < min_tokens and current_chunk:
            current_chunk += "\n\n" + section
        elif token_count > max_tokens:
            chunks.extend(split_by_paragraphs(section, max_tokens))
        else:
            if current_chunk:
                chunks.append(current_chunk)
            current_chunk = section
    
    if current_chunk:
        chunks.append(current_chunk)
    
    return chunks

This is a simplified version. In production code, you would use a real tokenizer like tiktoken instead of the word-count approximation.

Attaching metadata to chunks

Every chunk in the vector store should have metadata. This enables filtered searches ("only chunks from the 2024 handbook, only sections about the API") and significantly improves retrieval precision.

Sensible metadata per chunk:

• source: filename or path of the source document
• section: H2 heading of the chunk
• parent_section: H1, if the H2 sits within a larger structure
• date: document date or last update
• category: topic category (manually or automatically classified)
• chunk_position: order within the document (1, 2, 3, ...)

If you use Markdown with frontmatter, you can read this metadata directly from the document:

---
source: product-handbook-v3.md
category: API documentation
last_updated: 2026-04-15
author: Adriano Caruso
---

# API Authentication

Content here...

The frontmatter is ignored during chunking (not embedded as content) but stored as metadata per chunk.

Embedding strategy

The embedding model converts your chunk into a high-dimensional vector. The choice of model influences retrieval quality but less than often assumed.

Proven models in 2026:

OpenAI text-embedding-3-large (3072 dimensions). Very good, especially for English and mixed-language content. High storage needs due to large dimension.

OpenAI text-embedding-3-small (1536 dimensions). Balanced, significantly cheaper than large, sufficient for most use cases. My default recommendation for beginners.

Cohere embed-multilingual-v3 (1024 dimensions). Strong with multilingual content, especially for German source documents.

Voyage AI voyage-3 (1024 dimensions). Very good, also for long contexts, often better than OpenAI for technical content.

Sentence Transformers (all-MiniLM-L6-v2, 384 dimensions). Open source, free, runs locally. Quality-wise weaker than commercial ones, but for GDPR-critical setups often the only option.

A practical workflow to copy

A typical RAG setup workflow for a new project:

1. Collect source documents. PDFs, Word files, Excel sheets, anything relevant.
2. Convert to Markdown. With MarkItDown in the browser or the Python library for batches.
3. Add frontmatter. Five to ten lines of YAML metadata at the start of each document.
4. Chunk heading-based. With chunk size 600 to 1,000 tokens and 10 percent overlap.
5. Generate embeddings. With text-embedding-3-small or comparable model.
6. Load into vector store. Popular: Pinecone, Weaviate, Qdrant, ChromaDB.
7. Test with ten typical queries. Check whether retrieval finds the right chunks.
8. Iteratively adjust. When search results are poor, adjust chunk size, overlap, or metadata.

Common mistakes

Mistake one, chunks too large out of convenience. "I'll take the whole chapter as one chunk" sounds easy but kills retrieval precision. Stick to 500 to 1,500 tokens.

Mistake two, no metadata. Without metadata you cannot filter. At 10,000 chunks without filters, retrieval gets less accurate than at 1,000 chunks with good filters.

Mistake three, original formats instead of Markdown. If you parse PDFs directly, you lose structure. Convert to Markdown, then chunk.

Mistake four, one embedding model for everything. If you have both technical documentation and marketing texts, a single model may not perform optimally for both. Consider separate embeddings per document type.

Mistake five, never re-embedding. Embedding models evolve. If your vector store is two years old, you should fully re-embed with the next model update. It costs money but pays off.

Mistake six, no hybrid search. Pure vector search loses on exact terms (product names, codes, IDs). Hybrid search combines vector search with classic BM25 search and is better than pure vector search in 90 percent of cases.

Conclusion

RAG quality comes from data preparation, not from the embedding model alone. Clean Markdown as input format, heading-based chunking, good metadata, and a deliberate chunk size are the four levers with the biggest effect. If you implement these four points consistently, your RAG system beats a poorly built system with premium embeddings even using free embedding models.

Read more

• Use MarkItDown directly to convert your source documents
• Convert PDF to Markdown for PDF-specific conversion
• What is Markdown?, if you haven't read the basics guide yet
• Feeding Claude Projects with documents, as a RAG alternative for smaller setups