Return the source chunks with every answer from your local PDF Q&A pipeline. With Ollama, LangChain and Chroma, you can inspect both what retrieval selected and how the model used it. Fluent answers still need that check.
What Changed in LangChain v1
Older examples import RetrievalQA, LLMChain, text splitters, Ollama, and Chroma from the top-level langchain package. LangChain v1 reduced that namespace. Legacy chains moved to langchain-classic, while the Chroma, Ollama, and text-splitter integrations have separate packages. The older PyPDFLoader belongs to langchain-community, which was sunset in 2026[1]. For a text PDF, using pypdf directly avoids that sunset package.
This guide avoids the legacy RetrievalQA abstraction and uses current integration packages directly. Use Python 3.10 or newer with these pins:
python3 -m venv pdf-qa-env
source pdf-qa-env/bin/activate
python -m pip install --upgrade pip
python -m pip install \
langchain-core==1.6.6 \
langchain-chroma==1.1.0 \
langchain-ollama==1.1.0 \
langchain-text-splitters==1.1.3 \
pypdf==6.19.0
Pull one model for answers and one for embeddings:
ollama pull llama3.1:8b
ollama pull nomic-embed-text
Keep the package pins with your project, and record ollama --version and the model IDs shown by ollama list. A model tag can change without the Python code changing. Review the pins before a later deployment.
Extract the PDF and Preserve Page Metadata
pypdf extracts text from text-based PDFs. It cannot read an image-only scan. If a page is empty or garbled, add an OCR step and review the result before embedding it. Set the zero-based page index explicitly so the citation map does not depend on a loader’s metadata conventions.
from pathlib import Path
from pypdf import PdfReader
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
pdf_path = Path("document.pdf")
reader = PdfReader(str(pdf_path))
pages = [
Document(
page_content=page.extract_text() or "",
metadata={"source": str(pdf_path), "page": index},
)
for index, page in enumerate(reader.pages)
]
if not any(page.page_content.strip() for page in pages):
raise ValueError("No extractable text; OCR and inspect this PDF first")
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=150,
separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_documents(pages)
for index, chunk in enumerate(chunks):
chunk.metadata["chunk_id"] = f"chunk-{index:05d}"
The explicit source and page index travel with each chunk. The application can show the retrieved passage beside its actual PDF page; the page number is not something the model has to guess.
Tune chunk sizes against the PDFs you intend to use. Tables, code listings, footnotes and multi-column layouts often need their own handling. Keep a small set of questions with known page-level answers so you can see whether a change helps retrieval.
Create a Persistent Chroma Collection
Use the maintained Chroma and Ollama integrations:
import hashlib
from langchain_chroma import Chroma
from langchain_ollama import OllamaEmbeddings
embeddings = OllamaEmbeddings(model="nomic-embed-text")
pdf_hash = hashlib.sha256(pdf_path.read_bytes()).hexdigest()[:16]
vector_store = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
ids=[chunk.metadata["chunk_id"] for chunk in chunks],
collection_name=f"local_pdf_qa_{pdf_hash}",
persist_directory="./chroma_db",
)
The example gives each PDF its own collection. A file change produces a different collection name; fixed chunk IDs prevent duplicate rows when you index the same file again with the same settings. If you change chunking or the embedding model, rebuild the collection or use a new name: the file hash alone does not distinguish those settings. Remove obsolete collections deliberately, since they still use disk. For distance comparisons, Chroma defaults to squared L2 and also supports cosine and inner product. Choose a space supported by the embedding function, then evaluate it on your questions before changing the default.
Calibrate refusal thresholds on labeled retrieval examples. Distance scores depend on the function and embedding model, so their direction and scale must be understood before they can inform a decision. An arbitrary score cannot serve as a universal confidence percentage.
Retrieve and Label the Exact Context
Retrieve a bounded set of chunks and give each one a temporary citation ID:
def retrieve_context(question: str, k: int = 5):
docs = vector_store.similarity_search(question, k=k)
sources = {}
blocks = []
for number, doc in enumerate(docs, start=1):
citation = f"S{number}"
page_index = doc.metadata.get("page")
page = page_index + 1 if isinstance(page_index, int) else "unknown"
source = doc.metadata.get("source", "document.pdf")
chunk_id = doc.metadata.get("chunk_id", "unknown")
sources[citation] = {
"source": source,
"page": page,
"chunk_id": chunk_id,
"text": doc.page_content,
}
blocks.append(
f"[{citation}] source={source} page={page} chunk={chunk_id}\n"
f"{doc.page_content}"
)
return "\n\n".join(blocks), sources
The model sees the same IDs that the application maps back to retrieved text. This is stronger than asking it to invent a filename or page number from memory.
Ask for a Grounded Answer
Use ChatOllama from langchain-ollama and make refusal explicit:
import re
from langchain_ollama import ChatOllama
llm = ChatOllama(model="llama3.1:8b", temperature=0)
SYSTEM_PROMPT = """Answer only from the numbered source blocks.
Every factual statement must cite at least one source ID such as [S1].
End each factual sentence with a bracketed ID from the supplied blocks; for example: The warranty is 24 months [S1].
If the sources do not contain the answer, reply exactly:
Not found in the retrieved sources.
Do not use outside knowledge and do not invent source IDs."""
def answer_question(question: str):
context, sources = retrieve_context(question)
response = llm.invoke([
("system", SYSTEM_PROMPT),
("human", f"Question: {question}\n\nSources:\n{context}"),
])
answer = str(response.content).strip()
cited = set(re.findall(r"\[(S\d+)\]", answer))
invalid = cited.difference(sources)
if invalid:
return {
"answer": "Answer rejected: it cited a source that was not retrieved.",
"sources": sources,
}
if answer != "Not found in the retrieved sources." and not cited:
return {
"answer": "Answer rejected: no retrieved source was cited.",
"sources": sources,
}
return {"answer": answer, "sources": sources}
The validation checks [S<number>] IDs against the retrieved map. Apart from the exact refusal text, an answer needs at least one recognized citation, and any recognized ID outside the map causes rejection. Other citation formats and sentence-level source support remain unchecked. Display the cited text beside the answer and require human review for decisions where an error matters.
A validation function that searches sources for hard-coded phrases does not verify the generated answer. Verification must be tied to the current question, answer, and returned source chunks.
Test Retrieval Separately from Generation
When an answer is wrong, first ask whether the relevant source was retrieved.
Build a small evaluation file containing:
question
expected document
expected page or section
answerable: yes/no
required facts
For each question, record whether a top-k result includes the expected passage. If it does not, changing the prompt will not solve the retrieval problem. Try better extraction, different chunk boundaries, a different embedding model, metadata filters, or a carefully evaluated distance function.
Then score answer behavior:
- Did every factual sentence cite a returned source?
- Does each citation actually support the sentence?
- Did the model refuse when the answer was absent?
- Were page numbers displayed from metadata rather than generated by the model?
- Did the answer distinguish direct text from inference?
Keep an “unanswerable” set. A system tested only on questions with obvious answers never demonstrates that it can refuse.
PDF Failure Modes to Expect
PDF is a presentation format, and extracted reading order can be wrong. Common problems include:
- scanned pages with no text layer;
- two-column pages interleaved in the wrong order;
- repeated headers and footers polluting every chunk;
- tables flattened into ambiguous prose;
- diagrams whose meaning is not present in text; and
- page labels that differ from zero-based PDF page indexes.
Inspect representative chunks before trusting retrieval. For high-stakes documents, preserve a link to a rendered page and let reviewers compare the extraction against the original visual.
Know What the Citations Prove
The returned context lets you inspect why the model answered. The pipeline rejects [S<number>] IDs that were not retrieved and permits “not found” when that context is insufficient, but retrieval can still miss a better passage and the model can still misread the one it sees. Read the cited page before relying on a consequential answer.