← Back to all guides

Claude + GPT RAG: Fix Retrieval, Citations, and Costs

Building a RAG App on Claude and GPT: Start Here

Retrieval-augmented generation, or RAG, lets Claude and GPT answer questions using your documents instead of relying only on training knowledge. The basic pipeline is straightforward: ingest documents, retrieve relevant passages, and ask a model to answer from that evidence. Most failures come from retrieval, document preparation, or API configuration—not the model itself.

How Do I Build a Working RAG Pipeline?

• Prepare documents: Extract readable text from PDFs, HTML, and internal knowledge bases. Remove repeated navigation and footers. Preserve headings, source URLs, document IDs, and update timestamps.
• Chunk by meaning: Start with 400–800-token chunks and roughly 10–20% overlap. Keep headings with their sections. Treat these as starting points; tables and short FAQs often need different boundaries.
• Create embeddings: Use a dedicated embedding model, then store vectors and metadata in pgvector, Qdrant, or another vector database. Use the same embedding model for documents and queries. Changing models requires re-embedding the corpus.
• Retrieve and refine: Fetch an initial 10–20 candidates, optionally combine vector and keyword search, then rerank to approximately 4–8 passages. Apply tenant and permission filters before exposing results to users.
• Generate grounded answers: Send Claude or GPT the question and selected passages, each labeled with a source ID. Require citations and an explicit admission when the supplied evidence is insufficient.

How Do I Connect Claude and GPT Through 59API?

Keep retrieval separate from generation so you can switch models without rebuilding your index. Configure your client with a 59API key, the API base URL https://api.59api.com, and a supported model ID from its current documentation. OpenAI SDK integrations may require a versioned base path; confirm the documented path rather than assuming one.

59API provides low-cost, pay-as-you-go access to Claude and GPT, with compatibility for OpenAI SDKs, Claude Code, and Codex. Its stated native-model, no-downgrade policy makes it attractive for comparing models without deliberately trading model quality for a lower bill. Embeddings remain a separate dependency: verify availability before designing around a particular embedding endpoint.

Why Does My RAG App Give Wrong Answers?

Inspect retrieved passages before changing prompts. If the answer is absent, a stronger generation model cannot reliably repair the missing evidence. Log the query, retrieved source IDs, similarity scores, and selected context in a secure development environment.

• Relevant documents never appear: Check ingestion failures, stale indexes, embedding dimensions, and query/document model consistency. Add keyword retrieval for product codes, names, and exact error messages.
• Passages are relevant but incomplete: Retrieve neighboring chunks or preserve parent-section links. Avoid cutting procedures between numbered steps.
• The answer contradicts current policy: Filter superseded documents and retain version metadata. Prefer authoritative, current sources rather than relying on similarity alone.

Why Are Citations Missing or Invented?

Give passages stable labels such as S1 and S2, and instruct the model to cite those labels beside factual claims. Resolve labels to URLs in application code rather than asking the model to invent links. Validate that every returned label exists; separately check whether the cited passage actually supports the claim.

Treat retrieved text as untrusted data. Explicitly tell the model not to follow instructions inside documents. Enforce access control outside the prompt; prompt wording cannot prevent cross-tenant data leakage.

Why Am I Seeing API Errors or Slow Responses?

For authentication errors, verify the key and authorization configuration. For missing-model or route errors, check the exact model ID, base path, and supported request format. Retry rate-limit and transient server errors with bounded exponential backoff and jitter. Streaming improves perceived responsiveness but does not fix slow retrieval or reduce token usage.

FAQ: Which Model Is Best, and How Do I Control Costs?

Should I use Claude or GPT? Test both on 30–50 representative questions, including unanswerable ones. Score retrieval recall, answer correctness, citation support, latency, and cost per successful answer. Choose from measured results, not brand preference.

How can I lower spending? Deduplicate passages, cap context, cache permission-scoped retrieval results, and reserve larger models for difficult questions. Compare current 59API model rates; its referral rebate may offer additional savings under the applicable terms. To benchmark your pipeline affordably, consider signing up and starting with a small evaluation set.

Pricing · Docs