How to Compare RAG Development Companies: What to Ask and a Real Retrieval Trace

How do you compare RAG development companies?

Compare RAG development companies by the evidence they show on your own documents. In Neaptide’s October 7, 2026 trace over 542 passages from four service pages, the R04 reference passage ranked first of three retrieved. Ask each provider for the same trace on your questions, then for four separate results: retrieval, answer support, permissions and freshness.

• Eight requirements make RAG proposals comparable: a source trace, a separate retrieval result, tests on your documents, permission checks, freshness rules, handling of unsupported questions, a named handover and scoped support.
• Four results should be reported separately: retrieval, answer support, permissions and freshness. A single accuracy figure hides which part failed.
• Six tests belong before signature: a source trace, the reference rank, a restricted role, an obsolete document, a missing answer and a handover rehearsal.
• Neaptide’s R04 trace (October 7, 2026, four public service pages) ranked the published $825–1,650 GEO audit fee first among the top three retrieved chunks.

RAG (retrieval-augmented generation) is a way of building an assistant that retrieves relevant passages from approved documents and supplies them to a language model, so each answer can be checked against its sources. A RAG development company designs the retrieval, the answer rules and the access checks around your documents.

What should you require from a RAG provider before hiring?

Before hiring, require eight things from a RAG provider, each with a check you can run yourself. The table lists what to ask for, how to check it and when the answer is a red flag. Use the same rows for every shortlisted provider so the proposals can be compared line by line.

Our provider brief turns these rows into blank fields: seven scoping questions, six items of acceptance evidence and three handover lines. Send the same brief to every provider and compare the completed answers.

How do you decide between RAG proposals?

Decide between RAG proposals by running the same checks in a fixed order and stopping at the first one a proposal fails. The first four checks remove proposals that cannot be evaluated; only the remaining proposals are worth comparing on price. Stop at the first match:

1. If the provider cannot trace one of your questions to a source passage, ask for that trace before you continue.
2. If the evaluation uses only demo or public documents, ask for a test on a redacted sample of your approved documents.
3. If permissions were tested only on public content, require the same question to be run under an ordinary role and a restricted role.
4. If the handover does not name code, configuration, prompts, ingestion jobs, evaluation files and operating instructions, agree that list before signing.
5. If all four checks pass, compare quotes on the same scope: source count and quality, access complexity, integrations, hosting constraints and evaluation coverage.

A guaranteed accuracy percentage quoted before the provider has seen your documents is a red flag. Accuracy depends on the questions and sources in the acceptance set, so it belongs in the acceptance criteria.

Which results should a RAG provider report separately?

A RAG provider should report four results separately: retrieval, answer support, permissions and freshness. Each failure needs a different fix: if the current document was never retrieved, retrieval needs correcting; if the model ignored evidence it received, the answer rules do.

• Retrieval: whether the annotated reference passage appeared among the chunks passed to the model, and at what rank.
• Answer support: whether the generated answer states only what the retrieved passages support, with the source shown.
• Permissions: whether each user role retrieves only the documents that role may open.
• Freshness: whether an updated, deleted or revoked document stops appearing after the agreed refresh.

Ask for the evidence packet behind those four results: the approved source versions, the questions with annotated references, the retrieved passages with source IDs, the actual answers when generation is included and the configuration used for the run. The acceptance report should also list failed and excluded cases and keep a separate set of questions for later tests.

What did Neaptide’s own retrieval trace show?

On October 7, 2026, Neaptide’s selected R04 trace ranked the published $825–1,650 standalone GEO audit fee first among the top three retrieved chunks, with the source URL and surrounding terms. Neaptide, a digital marketing studio that also implements AI, ran the trace over 542 passages extracted from four live English service pages. The trace tests retrieval only: no language model generated an answer, and one selected case is not an accuracy score.

• Research data and method · EN
• Questions and answers

Which question and source did the R04 trace use?

The R04 trace asked “How much is the standalone Neaptide GEO audit?” against four Neaptide service pages captured on October 7, 2026: SEO, GEO, web development and AI implementation. The annotated reference was the sentence “A standalone audit costs $825–1,650” on the GEO service page. Each question in the set asked about a published offer, and the passage that answers it was annotated as the reference.

• Current GEO service and terms
• Selected successful retrieval trace · JSON

What did the retriever return for R04?

For R04, the retriever returned the GEO service page passage containing the reference sentence as its first-ranked chunk. The table maps each recorded field to the evidence a RAG provider should keep for every acceptance case.

The retriever was a BM25-style lexical ranker (k1 = 1.2, b = 0.75) without a reranker, returning the top three chunks. The source URL and the hash of the original HTML stayed attached to each chunk.

Why should a retrieved price keep its surrounding terms?

A retrieved price is usable only when the same passage says which service it belongs to and what it excludes. In the R04 chunk, the $825–1,650 standalone audit sits next to the monthly ranges for Core ($1,650–2,475) and Growth ($2,475–3,300), excluding taxes and agreed extras. Those neighbouring sentences let a reviewer confirm that the fee belongs to the one-off audit and not to a monthly programme.

In your own evaluation, require the provider to keep the document identity, version and relevant surrounding text with every retrieved passage. Then check the answer separately: it must name the intended service and keep the conditions that make the statement accurate.

What should you test before signing a RAG contract?

Run six tests before signing a RAG contract, each with a pass condition agreed in writing. A proposal that fails any of the first three on a sample of your documents is not ready for a price comparison.

1. Source trace: one of your real questions is traced to a retrieved passage with its source ID and version; fail if the trace starts at the answer.
2. Reference rank: the annotated reference passage appears within the stated cut-off (R04 used the top three chunks); fail if the provider cannot name the cut-off.
3. Restricted role: the same question asked by an ordinary user returns no passage from a restricted document; a single leak blocks release.
4. Obsolete document: a superseded procedure is in the test set; pass only if the current version is retrieved or the conflict is flagged.
5. Missing answer: a question your documents do not answer is declined or escalated by the agreed rule; fail on any invented answer.
6. Handover rehearsal: another team deploys the system from the runbook and handover files; make this part of acceptance if your team will maintain the system.

Neaptide’s public R04 trace covers tests 1 and 2 on public content only; tests 3–6 need evidence from the actual project environment and its user roles.

What should a RAG handover include?

A RAG handover should name six assets: application code, configuration, prompts, ingestion jobs, evaluation files and operating instructions. Third-party components, hosted model services and vendor platforms come with their own licences and subscriptions, so list them separately: a provider cannot transfer them as its own code.

The operating walkthrough should cover source updates, access changes, failed requests and the route for reporting incorrect answers. Continuing support is a separate agreement with a named owner and scope; get it in writing before you compare total cost.

What should you prepare before asking for a RAG quote?

Before asking for a RAG quote, prepare three inputs: the approved documents, the user roles and the recurring questions from the workflow the assistant will serve. Redacted samples and a role list are enough for a first discussion; keep confidential documents in your approved environment until access arrangements are agreed. The first scoping decision is which sources are authoritative and what an acceptable answer must show.

• Preparing company documents
• AI implementation cost categories
• Provider brief with blank evidence fields · TXT