Build an internal AI assistant that answers company process questions (Slack/Telegram + RAG)
TL;DR
Architecture: internal docs → chunk + embed → vector store; Slack/Telegram question → retrieve relevant chunks → Claude answers with source links, says “I don't know” when nothing matches. Doable with n8n or a few hundred lines of code. Permissions and doc freshness matter most.
“How do I request leave?”, “Where is staging deployed?”, “Where’s the latest quote template?” — these repeated questions eat time for both asker and answerer. An internal AI assistant that answers from company documents, right in Slack or Telegram, is one of the AI use cases with the clearest ROI. This article covers the RAG architecture, the key design decisions and a three-step rollout.
Architecture overview (RAG)
INGESTION (scheduled)
Confluence / Google Drive / Notion / internal files
→ clean and chunk
→ (optional) add context to each chunk
→ embeddings + keyword index → vector store (pgvector, Qdrant…)
ANSWERING (per question)
Question from Slack/Telegram
→ retrieve top-k relevant chunks (filtered by the asker's permissions)
→ Claude answers ONLY from those chunks + source links
→ nothing in the sources → say "no documentation yet"
→ log unanswered questions → backlog for documentation updatesRAG (Retrieval-Augmented Generation) means that before the model answers, the system finds relevant document chunks and puts them in the prompt. The model answers from your documents rather than general knowledge.
Do you even need RAG?
Anthropic suggests that if your whole knowledge base is small (under roughly 200,000 tokens, a few hundred pages), you can put the entire corpus in the prompt together with prompt caching — far simpler than building RAG. Cached reads cost a small fraction of normal input. Retrieval only becomes necessary once the corpus is larger.
Better retrieval: Contextual Retrieval
RAG’s classic problem: a chunk cut out of a document loses its context — a chunk saying “revenue grew 3%” doesn’t say which company or quarter. Anthropic’s Contextual Retrieval prepends a short model-generated context sentence to each chunk before creating embeddings and the BM25 index. Anthropic’s reported results:
| Technique | Reduction in failed retrievals |
|---|---|
| Contextual embeddings | 35% |
| Contextual embeddings + Contextual BM25 | 49% |
| Plus reranking | up to 67% |
The cost of generating per-chunk context drops sharply with prompt caching (cache the whole document, vary only the chunk being described).
Key design decisions
| Issue | Recommendation |
|---|---|
| Permissions | Tag each chunk with permission metadata (department, level) and filter before it reaches the prompt — don’t rely on telling the model “don’t reveal” |
| Made-up answers | System prompt: answer only from provided sources; otherwise say “no documentation yet” and suggest who to ask |
| Stale documents | Sync on a schedule; show each source’s last-updated date in answers |
| Sensitive data | Keep salaries, HR records and customer data out of the shared corpus |
| Citations | Every answer links to the original document so users can verify |
| Measuring impact | 👍/👎 buttons, unanswered-question rate, most repeated questions |
Sample system prompt
You are the company's internal assistant for processes and documentation.
- Use only the information inside below. No outside knowledge.
- If the documents aren't enough to answer, say "There's no documentation on this yet"
and suggest which team to ask.
- End with the source links you used and their last-updated dates.
- Keep answers short; use bullet points for multi-step processes. A three-step rollout
- MVP (2–3 weeks): one department, 20–50 documents, one Slack channel. Build it in n8n (AI Agent + vector store) or a few hundred lines of code.
- Measure for 2 weeks: which questions repeat, which answers are wrong, which topics lack documentation → that’s your documentation backlog.
- Expand: more sources, per-department permissions, and integration with new-joiner onboarding.
How to apply it
- Developers: Start by putting a small corpus straight into a cached prompt; build a vector store only when the corpus outgrows it. When you do build RAG, try Contextual Retrieval + reranking before switching models.
- Tech Leads: Design permissions from day one — retrofitting is expensive and the leak risk is real.
- PMs / HR / Ops: The “unanswered questions” list is gold: it shows which documentation is missing. Assign an owner to work through it weekly.