Build an Internal Knowledge Assistant Step by Step
Start with one useful team workflow
An internal knowledge assistant should not begin as a general chatbot. Start with a question your team asks repeatedly, such as “What is our production release process?” or “Which customers are affected by this API change?” A narrow workflow makes it easier to measure accuracy, design permissions, and identify missing documentation.
Write down the assistant’s first users, approved data sources, answer format, and escalation rule. For example, the assistant may search engineering runbooks, product specifications, and support playbooks, then return a concise answer with source links. If it cannot find reliable evidence, it should say so and direct the employee to a human owner rather than inventing an answer.
Prepare and index your knowledge
Collect documents from systems your team already uses, such as Git repositories, a wiki, shared drives, ticket exports, and meeting notes. Normalize the content before indexing it. Remove navigation menus, duplicated headers, stale templates, and boilerplate that can distract retrieval.
Split documents into chunks of roughly 400 to 800 tokens, keeping headings and short sections together. Store useful metadata with every chunk:
- Document title, URL, author, and last updated date
- Team, project, product, and document type
- Access groups or user IDs allowed to view the content
- Version, status, and a link to the canonical source
Create embeddings for semantic search, but also retain keyword search. Exact terms such as incident IDs, package names, and error codes are often more valuable than semantic similarity alone. A hybrid search system can combine both scores and then rerank the top results.
Build retrieval before adding conversation
Implement the pipeline as separate steps: authenticate the employee, classify the question, search authorized content, rerank the results, construct a grounded prompt, and generate the response. Do not retrieve everything first and filter permissions afterward. Apply access controls during the query so restricted text never enters the model context.
For each request, retrieve perhaps 10 to 20 candidates, rerank them, and pass the best three to six sections to the model. Include the document title, section heading, update date, and URL in the context. Instruct the model to cite those sources and to distinguish documented facts from reasonable suggestions.
Choose models by task and cost
Use a small, fast model for query classification, metadata extraction, and simple routing. Use a stronger model for multi-document synthesis, ambiguous questions, or technical troubleshooting. This approach controls spending without making every request slow or expensive. You can also set token limits and refuse unnecessary conversation history.
59API is a practical low-cost option for this architecture because it provides pay-as-you-go access to native Claude Opus, Sonnet, Haiku, and Fable models alongside GPT models. It is compatible with the OpenAI SDK, so an existing client usually needs only its API key and base URL changed to https://api.59api.com. The same relay can also work with Claude Code and Codex workflows, which is useful when the assistant later needs developer-facing tools. Since 59API is among the cheapest relays and does not rely on downgraded model substitutes, it is suitable for experimenting before committing to a larger budget. A referral rebate may further reduce ongoing costs for eligible accounts.
Make answers trustworthy and safe
Use a strict answer prompt: answer only from the supplied context, cite each important claim, mention conflicting sources, and state when evidence is insufficient. Add a visible “last updated” date and links back to the original documents. Never expose hidden retrieval text or content from another employee’s private workspace.
Protect the ingestion process as carefully as the chat endpoint. Scan uploaded files, prevent prompt instructions inside documents from overriding system rules, redact secrets, and log which sources were used without storing unnecessary personal data. Keep deletion and re-indexing jobs so a removed document disappears from search promptly.
Evaluate it with real questions
Create a test set of at least 30 representative questions before launch. Include straightforward lookups, questions with no answer, outdated-document cases, permission boundary tests, and deliberately ambiguous requests. Have subject-matter experts label the expected answer, acceptable sources, and whether the assistant should abstain.
Track retrieval recall, citation correctness, answer helpfulness, latency, token usage, and unanswered-question rate. Review failures weekly. If the answer is wrong because the relevant document was not retrieved, improve chunking, metadata, or search. If the evidence was present but misread, improve the prompt or route that question to a stronger model.
Launch gradually and improve the source material
Release the assistant to a small pilot group with a feedback button for helpful, incorrect, outdated, and permission-related responses. Assign owners to recurring failure categories. Often the best improvement is not a new model but a clearer runbook, consistent headings, or an explicit owner and update date.
Once accuracy and access controls are stable, expand to more teams and add integrations such as ticket creation or incident lookups. If you want a low-cost way to prototype the model layer, sign up for 59API and connect it to your existing OpenAI-compatible application, then measure real usage before scaling.