Document Loaders

The first step of indexing — turning raw files into one common Document object, and the quiet ways that step can fail

Why

Every source format — PDF, HTML, a database row, a plain text file — stores and structures content differently. A document loader's job is simple: take a raw file and convert it into a standardized Document object that the rest of the pipeline can work with, regardless of where the content originally came from. Loading is also where more real-world RAG quality is lost than anywhere else in the pipeline, because a bad extraction fails silently — the pipeline keeps running, it just runs on garbage.

Explanation
The Document object has exactly two parts
  • page_content — the actual extracted text
  • metadata — extra info about that text, e.g. {"source": "policy.pdf", "page": 4}
from langchain_community.document_loaders import PyPDFLoader loader = PyPDFLoader("insurance_policy.pdf") documents = loader.load() print(documents[0].page_content) # the raw text of page 1 print(documents[0].metadata) # {'source': 'insurance_policy.pdf', 'page': 0}
A loader per format
Text-native formats
TextLoader (.txt), Docx2txtLoader / UnstructuredWordDocumentLoader (Word), UnstructuredMarkdownLoader (Markdown), BSHTMLLoader / WebBaseLoader (HTML/web pages) — these formats store text as text, so extraction is close to lossless.
Structured formats
CSVLoader (one Document per row, columns become metadata or content), JSONLoader (with a jq-style schema to pick which fields become content), SQLDatabaseLoader (database tables, one Document per row or per query result).
Connectors
NotionDBLoader, ConfluenceLoader, GoogleDriveLoader, S3DirectoryLoader, Slack/GitHub loaders — pull content directly from a live system rather than a local file, usually returning the same metadata the system itself tracks (author, last-modified, URL).
General-purpose extractors
Unstructured.io and LlamaIndex's readers handle dozens of formats behind one interface and add layout-aware parsing (detecting titles, list items, and tables as distinct elements rather than one flat text blob) — a strong default when the source formats are mixed or unknown ahead of time.
PDFs are the hard case

PDF is a page-layout format, not a text format — it stores where characters are drawn on a page, not the logical reading order of a paragraph. This causes specific, common problems:

  • Multi-column layouts — a naive extractor reads left-to-right across the whole page width, interleaving two unrelated columns into nonsense text. Layout-aware extractors (Unstructured, PyMuPDF with layout detection, or a vision-model-based parser) detect column boundaries first.
  • Tables — a table's rows and columns collapse into a wall of numbers with no structure once naively extracted; specialized table extraction (e.g., camelot, pdfplumber, or Unstructured's table detection) is often needed to preserve the row/column relationship as markdown or structured data.
  • Scanned pages (images, not text) — a scanned PDF has no text layer at all; the loader returns an empty string, and the document is silently indexed as nothing. OCR (e.g., Tesseract, or a vision-capable LLM used as an OCR step) is required first.
  • Headers, footers, and page numbers — get extracted as regular body text unless explicitly filtered, and can pollute every chunk on every page with repeated boilerplate.
Designing a metadata schema

Metadata isn't an afterthought — it's the field you'll filter on, cite from, and debug with, so it's worth deciding up front what every document type needs to carry:

{ "source": "handbook.pdf", # always: where this came from "page": 12, # location within the source, if applicable "section": "Leave Policy", # logical grouping, if the format has one "doc_type": "policy", # for metadata filtering across document types "last_modified": "2025-03-01",# for freshness filtering / re-indexing "access_level": "internal", # for access-control filtering (see Vector Databases) "doc_id": "hb-2025-v3" # a stable id used to update/delete this doc later }

The last field — a stable document id — matters more than it looks: without it, updating or deleting a source document means you have no reliable way to find and remove its old chunks from the vector database (see Production Concerns → incremental indexing).

Failures that hide silently
  • Encoding issues — a file saved in Latin-1 loaded as UTF-8 produces garbled characters that still "load successfully," embed as noise, and never surface as an error.
  • Duplicate content — the same policy PDF uploaded twice under different filenames doubles up in the index, skewing retrieval toward whichever copy happens to embed slightly closer to a given query.
  • Truncated extraction — some PDF parsers silently stop partway through a very large or malformed file rather than raising an error.
  • Boilerplate pollution — repeated headers/footers/legal disclaimers extracted on every page waste chunk budget and dilute the embedding of the surrounding real content.

None of these throw exceptions. The only reliable way to catch them is spot-checking extracted page_content against the original source for a sample of documents before trusting the rest of the pipeline — treat loader output as untrusted until verified, the same way you'd validate any other data ingestion step.

Why metadata matters downstream

Metadata survives all the way through chunking, embedding, and storage in the vector database — that's what makes it possible to filter retrieval to a specific source or date range, and to cite exactly which page of which document a final answer came from, which is exactly what fixes the explainability gap described in What Is RAG?.

← Previous Next →