Local PDF Q&A Without Hallucinations: Ollama + LangChain

Local document Q&A promises privacy and zero per-token bills—run a model on your own hardware and query your PDFs. But “local” alone doesn’t stop hallucinations. A sloppy retriever or permissive prompt will still trigger confident fabrications. Building a grounded system demands a precise retrieval pipeline and a prompt that refuses to guess.

The Pipeline at a Glance

Every local document Q&A system follows the same steps:

1. Load documents and split them into chunks.
2. Embed those chunks and store them in a vector database.
3. Accept a user question, retrieve the most relevant chunks, and hand them to the LLM with a prompt that includes the retrieved text.
4. Return the answer.

When hallucinations happen, the cause is almost always in step 2 or step 3: chunks lose important context, the retriever misses the right passage, or the prompt fails to constrain the model. Solve those, and you sharply reduce made-up answers.

Setting Up Ollama and LangChain

Assume Ollama is installed and a model is pulled. A 7B or 8B parameter model hits the sweet spot for local Q&A: fast enough on a consumer GPU, large enough to follow instructions when the prompt is tight. For hardware requirements, see Running Open-Source LLMs on a $300 Mini PC. llama3.1:8b or mistral:latest work; start with llama3.1:8b—its instruction-following is more predictable for retrieval-augmented generation.

ollama pull llama3.1:8b

LangChain connects the pieces. The community edition is enough:

pip install langchain langchain-community chromadb pypdf sentence-transformers

ChromaDB stores vectors in memory or on disk. For a small collection, an in-memory instance works. For repeated queries, point it at a persist directory.

from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

That embedding model is small, runs on CPU, and yields 384-dimensional vectors. It’s fine for most PDFs. Swap in bge-small-en-v1.5 if you need better retrieval quality; it’s twice the size but still cheap.

PDF Ingestion and Chunking

Schemes that blindly split every N characters shred context. Retrieval quality starts here. pypdf loads the text; RecursiveCharacterTextSplitter breaks it into chunks that respect paragraph boundaries.

from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = PyPDFLoader("manual.pdf")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ".", " ", ""]
)
chunks = splitter.split_documents(documents)

Chunk size and overlap matter more than most settings. A 1000-character chunk with 200 characters of overlap keeps the retriever from slicing a sentence clean in half. For dense technical documents, drop chunk_size to 500 and bump overlap to 100. For legal contracts, where every clause is self-contained, 1500-character chunks with minimal overlap often work better. There is no universal number; test retrieval on a handful of known questions before committing.

Scenario	Chunk Size (chars)	Overlap (chars)	Notes
Technical manuals	500–800	100–150	Preserves code blocks and procedure steps
Narrative reports	1000–1200	200	Keeps paragraphs intact
Legal contracts	1500–2000	200–300	Each chunk captures a full clause or section
Dense reference material	400–600	50–100	Avoids mixing unrelated definitions

After splitting, feed the chunks to the vector store:

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

Building a Retrieval-Augmented Q&A Chain

The simplest chain that minimizes hallucinations is a “stuff” chain: all retrieved documents are stuffed into the prompt at once. It works for collections under a few hundred pages. For larger sets, consider MapReduce or Refine chains, but they add latency and can still lose context across steps. For most local Q&A, “stuff” is the right default.

from langchain.chains import RetrievalQA
from langchain_community.llms import Ollama

llm = Ollama(model="llama3.1:8b", temperature=0)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    return_source_documents=True
)

A couple of sharp edges: temperature=0 removes randomness that often pushes the model into inventing specifics. k=4 fetches up to four chunks. More chunks increase the chance the answer is in the context, but also increase the token count and, if they are irrelevant, confuse the model. For dense documents where the right chunk is almost always in the top three, k=3 is enough.

Then query:

result = qa_chain.invoke({"query": "What are the power requirements?"})
print(result["result"])
for doc in result["source_documents"]:
    print(doc.metadata)

Where Hallucinations Creep In

Even with retrieval, a local model will happily invent when:

• The retriever returns chunks that talk about a tangentially related topic but not the answer.
• The chunks contain the information but in an ambiguous way, and the model draws the wrong inference.
• The prompt allows the model to “fill in blanks” when the context is insufficient.
• The model’s pre-training knows a plausible-sounding answer that conflicts with the document.

A classic example: you ask “What is the default timeout?” and your chunks mention timeouts for different protocols but not the default. A model at temperature 0 may still answer “30 seconds” because that’s a common default in its training data. Local doesn’t fix that; grounding does.

Strategies to Keep the Model Grounded

1. Rewrite the user query for retrieval

A naive query might not overlap with the document vocabulary. Use the LLM to rephrase the question, but keep it simple and deterministic:

from langchain import LLMChain, PromptTemplate

rephrase_prompt = PromptTemplate(
    input_variables=["question"],
    template="""Given the user question, rephrase it to be a stand-alone question optimized for search in a technical document.
User question: {question}
Rephrased question:"""
)
rephrase_chain = LLMChain(llm=llm, prompt=rephrase_prompt)
rephrased = rephrase_chain.run(question)
docs = vectorstore.similarity_search(rephrased, k=4)

Make sure the rephrasing is brief; if the model adds invented details, you’ve introduced a hallucination before retrieval begins.

2. Force citations in the prompt

Modify the retrieval QA prompt to require citations and refuse to answer when information is missing. LangChain’s default prompt is permissive. Override it:

from langchain.prompts import PromptTemplate

custom_prompt = PromptTemplate(
    template="""Use the following pieces of context to answer the user's question.
If you don't know the answer, just say that you don't know. Don't try to make up an answer.
Include a citation to the relevant source document in your answer.

Context:
{context}

Question: {question}

Answer (with citations):""",
    input_variables=["context", "question"]
)

Then pass it:

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    return_source_documents=True,
    chain_type_kwargs={"prompt": custom_prompt}
)

This prompt does two things: it explicitly tells the model to say “I don’t know” rather than guess, and it asks for a citation. The model might still hallucinate a source, but because you’re returning the actual source_documents, you can verify its claims. In practice, the threat of citation makes the model far less likely to invent entire claims.

3. Post-processing guard: Verify against sources

Even with the right prompt, models sometimes extract wrong numbers. A blunt but effective guard: after generating an answer, check if any of the claimed facts appear verbatim in the retrieved documents. A simple string search for key numbers or terms catches the worst of it.

def verify_answer(answer, source_docs):
    for doc in source_docs:
        if any(keyword in doc.page_content for keyword in ["default timeout", "30 seconds"]):
            return True
    return False

This is crude; you can go further by extracting the cited sentence and fuzzy-matching it in the retrieved text. For a local deployment handling a bounded document set, a simple keyword check often flags enough hallucinations to be worth the few lines of code.

4. Tune retrieval parameters ruthlessly

Retrieval quality is the strongest lever. similarity_search uses cosine distance by default. ChromaDB supports maximal marginal relevance (MMR), which improves diversity—useful when your top chunks come from the same section and are nearly identical.

docs = vectorstore.max_marginal_relevance_search(question, k=4, fetch_k=20)

fetch_k=20 grabs 20 candidates then selects 4 diverse ones. The default is fetch_k=20, but setting it explicitly makes the behavior clear. For documents with a lot of repetition, MMR often retrieves the critical chunk that simple similarity misses.

Also, bump the embedding model if retrieval feels weak. The bge-small-en-v1.5 model runs on CPU, scores significantly higher on retrieval benchmarks, and adds only a fraction of a second to embedding time per chunk.

embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5")

5. Keep chunks atomic

A chunk should contain one idea. If the answer to a question is spread across two chunks with no overlap in their embedding, the retriever won’t retrieve both. Solutions:

• Use a sliding window with enough overlap to capture cross-boundary concepts, as already configured.
• For very intertwined technical prose, consider a “parent document” retriever: retrieve small chunks but feed the model larger parent chunks that contain surrounding context. LangChain’s ParentDocumentRetriever does this, though it adds complexity. For most documents under a few hundred pages, good chunking and k=4 with MMR are enough.

Putting It All Together

A full working script that loads a PDF, creates a retriever, and queries with grounded prompts and source verification:

from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_community.llms import Ollama
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

# Load and chunk
loader = PyPDFLoader("document.pdf")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200, separators=["\n\n", "\n", ".", " ", ""])
chunks = splitter.split_documents(docs)

# Embed and store
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = Chroma.from_documents(documents=chunks, embedding=embeddings, persist_directory="./db")
vectorstore.persist()

# LLM
llm = Ollama(model="llama3.1:8b", temperature=0)

# Custom prompt
prompt_template = PromptTemplate(
    template="""Use the following pieces of context to answer the user's question.
If you don't know the answer, just say that you don't know. Don't try to make up an answer.
Include a citation to the relevant source document in your answer.

Context:
{context}

Question: {question}

Answer (with citations):""",
    input_variables=["context", "question"]
)

# Retriever with MMR and k=4
retriever = vectorstore.as_retriever(search_type="mmr", search_kwargs={"k": 4, "fetch_k": 20})

# QA chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=retriever,
    return_source_documents=True,
    chain_type_kwargs={"prompt": prompt_template}
)

# Query
question = "What are the minimum system requirements?"
result = qa_chain.invoke({"query": question})
print("Answer:", result["result"])
print("\nSources:")
for doc in result["source_documents"]:
    print("-", doc.metadata.get("source"), "page", doc.metadata.get("page"))

The output should cite a source. If the model says it doesn’t know, the document likely lacks the answer—a correct refusal is a win.

Conclusion

Local document Q&A can be genuinely private and responsive, but only when retrieval and prompting are tuned to suppress the model’s instinct to embellish. Chunking strategy, retrieval parameters, a citation-enforcing prompt, and a crude verification step catch the majority of hallucinations. The rest is up to the model’s training, but these levers get you from a demo that looks magic to a tool you can rely on. Start with the exact code above, swap in your PDF, and only complicate things when the simple version fails on a specific document.