Ask a generic AI about your business and you'll get generic advice. The gap isn't intelligence — it's context.
Every leader who's pasted a company question into a public chat tool has felt it: plausible, fluent, wrong for your reality. Earlier articles in this series covered context in each prompt and team practices. This one steps up a level: context windows, retrieval (RAG), and your own business data — the layer that turns AI into an operational tool.
At a glance
- Context window = how much text (measured in tokens, units of text) the model can "see" in one request — with real limits
- RAG (retrieval-augmented generation) = fetch relevant docs from your corpus, then generate an answer
- Without curated data, AI guesses from public training — risky for operations and compliance
- Context strategy connects to governance and data safety
Why context windows matter
LLMs (large language models) process a finite amount of text per call — policies, emails, manuals, chat history all compete for space. When you exceed limits, something gets truncated — often silently.
| Symptom | Likely cause |
|---|---|
| Answer ignores part of your upload | Truncation or poor prioritization |
| Inconsistent answers same day | Different chunks retrieved or session drift |
| "Forgot" earlier instructions | Window filled; earlier context dropped |
Practical rule: send what matters for this task, not everything you have. Summarize long material; link to canonical sources. A manager who pastes 80 pages of project notes and asks for an executive summary often gets output that ignores the budget section — not because the model is "dumb," but because the window is saturated.
RAG in plain language
Retrieval-augmented generation follows four steps:
- User asks a question
- System searches your approved knowledge base (policies, wikis, project files)
- Top relevant passages are injected into the prompt
- Model answers using those passages — ideally with citations
Done well, RAG turns AI from "smart stranger" into "well-briefed colleague." Done poorly, it retrieves wrong chunks and confidently misquotes your own policies — often a stale version still indexed.
Building the corpus is operational work, not a standalone IT project. Article 8 in this series details how to build a knowledge base.
Business data: the asset and the risk
Your data is why AI could be valuable for you specifically:
- Pricing history, SOPs, past proposals, ticket resolutions
- Meeting archives, inspection templates, regulatory checklists
It's also why privacy and security matter. Classify before indexing:
- Green — internal, low sensitivity, good for early RAG
- Yellow — personal or confidential — strict access and logging
- Red — excluded until legal and security sign off
Never index "everything in the drive" as v1. Noise drowns signal — and multiplies risk.
A professional firm I advised indexed HR policies before client-facing SOPs — the wrong priority for their pilot. Staff kept asking operational questions; the system kept returning vacation policy excerpts. Reordering the corpus around the top 30 weekly questions fixed retrieval quality faster than switching models.
Context for agents and workflows
Autonomous agents multiply context needs — they pull from CRM, email, and tools in one run. Without:
- Clear source of truth per data type
- Permissions matching human roles
- Logging of what was retrieved
…agents amplify confusion faster than chat. System context isn't optional when AI acts, not just answers.
How to improve context without a big project
- One canonical folder or wiki for the pilot domain
- Remove duplicates and obsolete versions — RAG hates stale PDFs
- Metadata — owner, date, language (fr/en), status
- Test questions — 20 real queries your team asks weekly; score answer quality
- Human review loop — "wrong doc retrieved" is a product signal, not user error
Context and prompting together
Even with RAG, good prompts specify:
- Which sources to prefer
- What to do if evidence is missing ("say you don't know")
- Output format and review flags
"Use only the provided documents" is a minimum — enforce it in workflow design, not hope.
Where you are
You've standardized team prompting practices; this article explains why your data and history matter more than this week's model. Next: Multimodal AI: text, audio, images, and video — operational use cases beyond text alone.
Before you buy another chat tool, Let's sanity-check your data readiness. We'll sanity-check whether your data is ready for RAG — or what to fix first.