A practical guide to preparing structured content for chatbots, RAG pipelines, and intelligent retrieval. AI-readiness is a content engineering problem — retrieval precision is upstream of model selection, and the seven dimensions of content engineering below determine whether your AI pipeline returns the right answer or the closest keyword match.
AI systems don't magically understand your documentation. They need structure, metadata, and semantic boundaries to retrieve the right answer — not just the closest keyword match. But most enterprise content sits in unstructured formats. When it lands in a vector store, the AI treats every paragraph as undifferentiated text. The failure pattern is consistent: undifferentiated text → unreliable retrieval → either missed citations or, worse, hallucinated ones the user follows.
The problem
Most enterprise content sits in unstructured formats — Word, PDF, wiki pages, Markdown. When that lands in a vector store, the AI treats every paragraph as undifferentiated text. It can't distinguish a safety warning from a marketing blurb, a step-by-step procedure from a concept explanation, or a beginner topic from an expert reference.
The cost of inaction
Organizations that deploy chatbots on unstructured content see retrieval precision around 25–35%. Users get wrong answers, lose trust, and escalate to human support. Worse, AI hallucinations fill gaps where metadata should have guided the system — and a hallucinated citation is materially worse than a missed one.
The opportunity
Content that is semantically typed, metadata-enriched, and section-addressable delivers 85%+ retrieval precision. The AI knows what each piece of content is, who it's for, and when to surface it. That's the difference between a chatbot that guesses and one that answers — and it's an engineering problem, not a model-selection problem.
Rate your content on each dimension below: 0, 1, or 2 points. Total range is 0–14. Sum your score and consult the interpretation rubric below. Honest self-scoring is the prerequisite — the dimensions compound, so a single 0 in a foundational dimension caps what's achievable downstream.
Content typing
Is every document formally typed — concept, task, reference, troubleshooting — or is everything a generic 'page'?
Why it matters
AI intent routing depends on type. 'What is X?' → concept. 'How do I X?' → task. 'List of X' → reference. Without types, the AI cannot match question intent to answer format.
Scoring
Metadata coverage
Does every topic have machine-readable metadata — audience, difficulty, domain, intent, duration?
Why it matters
Metadata enables filtering. Without it, the AI retrieves beginner content for experts, developer docs for writers, and overview content when the user needs a step-by-step procedure.
Scoring
Section addressability
Does every major section have a stable, unique ID?
Why it matters
Vector stores chunk content at section boundaries. Without IDs, chunks break at arbitrary points — mid-sentence, mid-procedure, mid-table. The AI retrieves fragments instead of complete answers.
Scoring
Vocabulary control
Are metadata values drawn from a controlled vocabulary, or is it free-text tagging?
Why it matters
Free-text tags produce inconsistency: 'dev', 'developer', 'software engineer', 'eng' all mean the same thing. AI filters break when vocabulary is uncontrolled. A subject scheme or taxonomy enforces consistency.
Scoring
Semantic markup richness
Do topics use semantic elements — shortdesc, prereqs, steps, result, tables, notes — or is everything paragraphs?
Why it matters
Semantic elements tell the AI the role of each block. A <shortdesc> becomes the chatbot's one-line answer. <steps> become numbered instructions. <prereq> tells the AI what the user needs before starting.
Scoring
Reuse architecture
Is content reused via conref / keyref / transclusion, or duplicated by copy-paste?
Why it matters
Copy-paste duplication means the AI retrieves multiple slightly-different versions of the same content. Users see conflicting answers. Proper reuse means one authoritative source — one answer.
Scoring
Output pipeline maturity
Can the pipeline produce HTML, PDF, JSON, and chatbot-ready JSONL from a single source automatically?
Why it matters
AI pipelines consume JSON / JSONL — not PDF. If your publishing pipeline only produces PDF and HTML, you need a separate export step for AI ingestion. A mature pipeline includes chatbot-ready output as a first-class format.
Scoring
Five phases to take content from unstructured to AI-ready. Each phase builds on the previous one; skipping the foundation phases is the most common way these programs fail. Total duration scales with corpus size, not with project ambition — the timeline below shows typical ranges for small, medium, and large corpora.
Score your current content on the seven dimensions above. Identify the biggest gaps. Inventory content by type, volume, language, and update frequency.
Deliverable: Content audit report with AI-readiness score and gap analysis.
Convert content to a typed format (DITA concept / task / reference, or equivalent). Split monolithic documents into focused single-purpose topics. Add section IDs to every major block.
Deliverable: Typed, modular topics with section-level addressability.
Define a controlled vocabulary (subject scheme or taxonomy) with facets for audience, difficulty, content domain, content intent. Apply metadata to every topic via batch tooling — not manual entry.
Deliverable: 100% metadata coverage with enforced vocabulary.
Configure the publishing toolchain to produce chatbot-ready output alongside traditional formats. JSONL export with section-aware chunking, metadata inheritance, and glossary extraction.
Deliverable: Automated build producing HTML5, PDF, and JSONL from a single source.
Connect the JSONL output to your vector store and chatbot. Configure metadata filters for intent routing and audience matching. Run retrieval tests with real user queries.
Deliverable: Working AI retrieval with measured precision rate.
Typical timeline
Concrete examples of the transformations the scorecard measures. Each pair shows the failure mode at score 0 and the working pattern at score 2. The three transformations compound each other — metadata filters the chunks, section IDs keep them coherent, and a controlled vocabulary makes both consistent across the corpus.
Metadata that AI systems can filter
Audience filtering
A developer asks 'How do I create a custom plugin?' — the AI filters for audience=developer and intent=how-to, skipping the manager-level overview of the same feature. Without metadata, both would be returned with equal rank.
Difficulty matching
A beginner asks 'What is DITA?' — the AI returns the difficulty=beginner concept topic, not the advanced architecture reference. The right answer at the right level prevents overwhelm and helpdesk escalation.
Intent routing
'What is X?' routes to concepts. 'How do I X?' routes to tasks. 'Show me the parameters' routes to reference tables. Topic typing plus intent metadata makes this automatic — no prompt engineering required.
Section-aware chunking
Without section IDs
A 2,000-word topic gets split into three chunks of ~667 words each. Chunk 2 starts mid-paragraph, references "the step above" (which is now in Chunk 1), and includes half a table. The AI retrieves Chunk 2 and the user sees a fragment that doesn't make sense.
With section IDs
The same topic has five sections, each with a stable ID and self-contained heading. Chunking happens at section boundaries. Each chunk is a complete, citable unit. The AI retrieves the exact section, and the user sees a coherent answer with a deep-link back to source.
Controlled vocabulary vs. free-text tags
Free-text chaos
Topic A is tagged "developer". Topic B is tagged "dev". Topic C is tagged "software engineer". Topic D has no audience tag. A filter for audience=developer returns only Topic A — missing 75% of relevant content. The chatbot appears to have knowledge gaps that don't actually exist.
Controlled vocabulary
A DITA subject scheme defines exactly seven valid audience values: technical-writer, developer, build-engineer, manager, localization-specialist, editor, all. Validation rejects any other value. The same filter returns 100% of relevant content — zero false negatives.
Each section becomes a JSONL record with the text content, topic title, section heading, topic type (concept / task / reference), all metadata fields, the section ID for deep-linking, and the parent map path for breadcrumb context. A glossary index maps terms to definitions with synonyms. A content manifest provides a searchable index of all topics.
Submit a 20-topic sample. We'll score it on all seven dimensions, return a gap analysis with recommended next steps, and indicate the engineering effort required to reach production-grade retrieval. No obligation to proceed.
Submit a sample →