Victor Cuellar
ENES
All selected work

go-ms-rag

Document ingestion and evidence-bounded answers through a staged Go retrieval service.

Allowed evidence shapes the answer.
Interface
HTTP API
Language
Go
Vector index
Pinecone
Database
PostgreSQL
Model provider
AWS Bedrock
View repository
Query console
Keep scores ≥ 0.55 × the top score, up to the context budget.Context budget: 5 · Retained: 3
Tool loansPassage 1

Members may borrow one repair stand for two days. Return the stand clean before the workshop closes.

Illustrative similarity0.92
Tool loansPassage 2

Return the stand clean before the workshop closes. Reserve a stand at the cooperative desk before collection.

Illustrative similarity0.78
Tool loansPassage 3

Reserve a stand at the cooperative desk before collection. A member must sign the loan record at collection.

Illustrative similarity0.62
Loan renewalsPassage 1

A tool loan may be renewed once before its return date. Renewal is available only when no other member has reserved the tool.

Illustrative similarity0.49
Workshop accessPassage 1

Members may use a workshop bench during opening hours. Bench use requires a safety orientation before the first visit.

Illustrative similarity0.31
Answer from the excerpts

Members may borrow one repair stand for two days. It must be returned clean before the workshop closes.

Fixed illustrative scores, not measured retrieval quality.

Illustrative simulation with fixed example data. No live model or service is called.

Figure 1. An illustrative interface for the document API, not an interface shipped by the repository.

Problem

A question about a document collection needs evidence, not just a fluent answer. Retrieval-augmented generation (RAG) finds relevant passages first, then uses them to answer. go-ms-rag exposes that workflow as an HTTP API for raw text and uploaded documents.

Find a line in the collection.

How it works

Ingestion extracts and normalizes text, splits it into passages, creates numeric representations called embeddings, and stores them. Queries retrieve candidate passages, apply the caller’s document allowlist, and only then generate an answer.

Document, passages, vectors, answer.
Service flow
IngestionQueryIngest requestExtract andnormalizeChunk andclassifyEmbedPersistRelationaldatabaseVector indexObject storageDecomposeRouteRetrieveDocumentallowlistSynthesizeEarly exitIngestionStorageQueryIngestrequestExtract andnormalizeChunk andclassifyEmbedPersistRelationaldatabaseVector indexObjectstorageDecomposeRouteRetrieveDocumentallowlistSynthesizeEarly exit

No passages means a fixed response, zero tokens and no model call.

All components

  • Ingest request

    Raw text or uploaded documents enter the service.

  • Extract and normalize

    Extract file text, remove HTML and normalize whitespace.

  • Chunk and classify

    Overlapping passages receive kind and quality labels.

  • Embed

    Numeric text representations support similarity search.

  • Persist

    Write vectors and store document, passage and fingerprint records.

  • Relational database

    PostgreSQL stores documents, passages and fingerprints.

  • Vector index

    Pinecone stores per-document and default namespaces; queries currently use the default.

  • Object storage

    Uploaded originals go to S3 when configured.

  • Decompose

    Names and multi-need words supply a heuristic routing signal.

  • Route

    Explicit mode, probe results and query signals select LOCAL or GLOBAL.

  • Retrieve

    Retrieve from the default namespace; GLOBAL may widen sparse results.

  • Document allowlist

    Filter retrieved passages before they reach synthesis.

  • Synthesize

    Select bounded context and answer from the available excerpts.

  • Early exit

    No passages means a fixed response, zero tokens and no model call.

Figure 2. Ingestion and queries share stored documents and vectors; the allowlist sits before synthesis.

Ingest a document

Raw text or an uploaded file enters a bounded asynchronous queue. Text is extracted and normalized, split into classified passages, embedded and persisted.

From documents to searchable passages
  1. Documents
  2. Extract + normalize
  3. Sentence-aware passages
  4. Embeddings
  5. Persist
  6. Two vector writes

More detail Illustrative examples

Asynchronous queue202 processing

DocumentsPDF / DOCX / text
Extract + normalizeStrip HTML, normalize text
Sentence-aware passagesShared sentence
EmbeddingsAbstract vector glyphs
Persist
Document namespace
Default namespace
Postgres metadata
Uploaded originalsUploads only, when configured

An asynchronous queue accepts documents, builds overlapping sentence-aware passages and embeddings, then writes vectors twice, metadata to Postgres and uploaded originals to object storage when configured.

Figure 3. A document moves through extraction, chunking, embedding and persistence.

Read query signals

The service matches known names and checks words that suggest multiple needs. These are deterministic rules, not a language-model call.

Two inexpensive signals
  1. Question
  2. Word check
  3. Named-entity match
  4. Routing

More detail Illustrative examples

Question
Word check“compare” / “y”
Synthetic count: 2Only if no entity matches
Named-entity matchRegistered name
Entity foundSynthetic count: 1
Routing

No decomposition model call

Word cues and named-entity matching feed routing without a model call; a word cue creates a synthetic count of two only when no named entity matches.

Figure 4. Names and multi-need words provide routing signals.

Choose a retrieval mode

An explicit mode takes precedence. Otherwise the service uses the probe, multi-need count, names and score threshold to choose LOCAL or GLOBAL.

The first matching rule decides
  1. Explicit mode?
  2. Probe failed?
  3. Several needs?
  4. Named entity?
  5. Probe score ≥ 0.70?
  6. Otherwise

More detail Illustrative examples

  1. Explicit mode?Requested mode
  2. Probe failed?GLOBAL
  3. Several needs?GLOBAL
  4. Named entity?LOCAL
  5. Probe score ≥ 0.70?LOCAL
  6. OtherwiseGLOBAL
If not, continue

Explicit mode takes priority, followed by probe failure, multiple needs, a named entity and finally the inclusive 0.70 similarity threshold.

Figure 5. The routing decision combines explicit mode and query signals.

Apply the document allowlist

Retrieved passages are filtered before synthesis. An omitted list allows all documents; an empty list allows none.

An access gate before synthesis
  1. Omitted / null
  2. Only document A
  3. Empty list

More detail Illustrative examples

Omitted / null
ABC
Access gate
ABC
All pass
Only document A
ABC
Access gate
ABC
Only matches pass
Empty list
ABC
Access gate
∅
Nothing passes

After retrieval and before synthesis, an omitted allowlist preserves every passage, a document list preserves only matches, and an empty list preserves none.

Figure 6. The allowlist decides which passages can enter the answer context.

Answer from bounded evidence

The service selects passages using a relative score floor and a fixed context budget. With no passages, it returns a fixed answer without calling a model.

Evidence in, grounded answer out
  1. Retrieved passages
  2. Relative score floor
  3. Synthesis
  4. Early exit

More detail Illustrative examples

Retrieved passages
0.900.640.30
Relative score floor0.90 × 0.55 = 0.495
SynthesisAnswer from excerpts only
At most 5 passages; 8 for several needsEach excerpt: at most 1,200 runes
No passages → Early exit

No relevant information was found to answer this question.

No synthesis call

Only sufficiently strong excerpts enter the capped context; no remaining passage produces the fixed response instead of a synthesis call.

Figure 7. Evidence leads to synthesis; no evidence leads to an early exit.

Details

Chunking

A document becomes overlapping passages so adjacent context is not lost at every boundary. The service also classifies each passage by kind and quality.

Sentence-aware passages
Original document
Tool loans

Members may borrow one repair stand for two days. Return the stand clean before the workshop closes. Reserve a stand at the cooperative desk before collection. A member must sign the loan record at collection.

Passage 1Kind: bodyQuality: high

Focus or hover a passage to inspect its labels. Highlighted sentences appear in both neighbouring passages.

Passage sizes are reduced for this illustration: two sentences per passage, with one shared sentence. Backend defaults are approximately 512 tokens with 64-token overlap.

Illustrative simulation with fixed example data. No live model or service is called.

Figure 8. A document split into overlapping passages with kind and quality labels.

Routing rule

LOCAL and GLOBAL are routing labels, not separate document-security boundaries. Both currently query the default namespace; GLOBAL can widen a search when fewer than three results are returned.

Retrieval routing
LOCAL

The probe reaches the 0.70 threshold

0.82 ≥ 0.70

In the current code both modes query the default namespace; GLOBAL widens and deduplicates the search when fewer than 3 passages come back.

Illustrative simulation with fixed example data. No live model or service is called.

Figure 9. Query signals change the routing decision without changing the document allowlist.

Allowlist before generation

The allowlist is applied after retrieval and before synthesis. An omitted list does not restrict documents; an empty list permits none. Without passages, the service returns a fixed no-evidence answer without calling a model.

Access before synthesis

allowed_document_idsOmitted

An omitted or null allowlist is unrestricted; an empty list allows nothing. Filtering happens after retrieval, before synthesis.

Retrieved: 9 Permitted passages: 9
Tool loansPassage 1

Members may borrow one repair stand for two days. Return the stand clean before the workshop closes.

Illustrative similarity0.86
Workshop accessPassage 1

Members may use a workshop bench during opening hours. Bench use requires a safety orientation before the first visit.

Illustrative similarity0.80
Tool loansPassage 2

Return the stand clean before the workshop closes. Reserve a stand at the cooperative desk before collection.

Illustrative similarity0.76
Workshop accessPassage 2

Bench use requires a safety orientation before the first visit. Shared tools stay inside the workshop after each session.

Illustrative similarity0.68
Tool loansPassage 3

Reserve a stand at the cooperative desk before collection. A member must sign the loan record at collection.

Illustrative similarity0.63
Workshop accessPassage 3

Shared tools stay inside the workshop after each session. Clean the bench before leaving space for the next member.

Illustrative similarity0.59
Loan renewalsPassage 1

A tool loan may be renewed once before its return date. Renewal is available only when no other member has reserved the tool.

Illustrative similarity0.52
Loan renewalsPassage 2

Renewal is available only when no other member has reserved the tool. Request a renewal at the desk during workshop opening hours.

Illustrative similarity0.49
Loan renewalsPassage 3

Request a renewal at the desk during workshop opening hours. The desk records the new return date in the loan record.

Illustrative similarity0.40
Answer from the excerpts

Members may borrow one repair stand for two days. It must be returned clean before the workshop closes. A tool loan may be renewed once before its return date, only when no other member has reserved it. Members may use a workshop bench during opening hours after a safety orientation. Shared tools stay inside the workshop.

Illustrative simulation with fixed example data. No live model or service is called.

Figure 10. Permitted documents determine which retrieved passages can reach synthesis.

Decisions and tradeoffs

Filter before generation

The allowlist is enforced before retrieved text reaches the synthesis context. The document boundary is separate from the model prompt.

Filter before text reaches the prompt
  1. Retrieved passages
  2. Access gate
  3. Prompt

More detail Illustrative examples

Chosen order
Retrieved passages
AB
Access gateBefore synthesis
PromptOnly passage AA
Rejected alternative
Retrieved passages
AB
PromptA + B already exposed
Access gateToo late to remove exposure

The accepted order keeps denied document text out of the prompt, unlike the rejected alternative of filtering after synthesis.

Figure 11. Document filtering happens before synthesis.

Stop when evidence is absent

Empty evidence produces a fixed response and zero token usage. The service does not ask a model to fill the gap.

No evidence is a stopping condition
  1. No passages
  2. Early exit
  3. Fixed response

More detail Illustrative examples

Chosen order
No passages
Early exitZero token usage
Fixed response
Rejected alternative
No passages
Model call
Risk of an unsupported answer

No relevant information was found to answer this question.

No synthesis call

Stopping on zero evidence avoids asking the synthesizer for an unsupported answer and returns the exact fixed response with zero token usage.

Figure 12. An empty passage set skips the model call.

Write vectors to two namespaces

Ingestion writes each vector to its document namespace and to the default namespace. This maintains both layouts, while the current query path uses the default one.

Two destinations for each vector
  1. One vector
  2. Document namespace
  3. Default namespace
  4. Two vector writes

More detail Illustrative examples

One vector
Document namespace
Per-document deletion
Default namespace
Global retrieval
Two vector writes
More writes + storage

A document namespace and the default namespace require more vector writes and storage in exchange for global retrieval and per-document deletion.

Figure 13. Each vector is written to a document namespace and the default namespace.

Use inspectable query rules

Name matching and multi-need expressions avoid an extra model call. They also limit decomposition to the patterns those rules recognize.

A heuristic, not generated sub-questions
  1. Word + entity checks
  2. Synthetic count: 2
  3. Model-based decomposition
  4. Can miss or over-trigger

More detail Illustrative examples

Word + entity checksOne fewer model call
Synthetic count: 2Can miss or over-trigger
Model-based decomposition

“Where do the stand and its case go?”

One intended needCue found: “and” → Counted: 2

Word and entity checks avoid the latency and cost of a decomposition call, but a simple conjunction can overstate the number of information needs.

Figure 14. Fixed rules trade broader interpretation for inspectable signals.

Keep a local test path

Fake mode substitutes deterministic providers and an in-memory vector store. PostgreSQL remains required.

An offline substitute, not a missing database
  1. Deterministic hash
  2. In-memory vectors
  3. Fake synthesizer
  4. Offline checks
  5. Postgres still required

More detail Illustrative examples

Deterministic hash
In-memory vectors
Fake synthesizer
Offline checks
Postgres still required

Fake mode substitutes deterministic hash embeddings, in-memory vectors and a fake synthesizer for offline checks, while Postgres is still required.

Figure 15. Local provider substitutes do not remove the database dependency.

Limitations

LOCAL still uses the default namespace

LOCAL and GLOBAL currently query the default vector namespace. LOCAL does not isolate a per-document namespace; the allowlist remains a separate filter.

Two modes, one search namespace
  1. LOCAL / GLOBAL
  2. Default namespace
  3. Widened search

More detail Illustrative examples

LOCALGLOBAL
Default namespace
ABC
GLOBALFewer than 3 passagesWidened search: 20

Both modes currently query the default namespace; GLOBAL alone adds a widened search when fewer than three passages return.

Figure 16. Both retrieval modes reach the default vector namespace.

Widening is a bounded fallback

When GLOBAL returns fewer than three results, the service widens the search to a limit of 20 and merges duplicates. This is not a reranker; there is no reranker in the current path.

A wider search is not a reranker
  1. Initial search
  2. Search limit: 20
  3. Merge + deduplicate
  4. Unique results

More detail Illustrative examples

Initial searchA · 0.91B · 0.82
Search limit: 20A · 0.91C · 0.68
Merge + deduplicate
Unique resultsA · 0.91B · 0.82C · 0.68
Vector score onlyNo reranker

The widened search merges and deduplicates results up to the requested limit, but ranking still uses vector similarity rather than a reranker.

Figure 17. A low result count widens GLOBAL retrieval, without reranking.

Heuristics can miss query intent

Multi-need detection recognizes a fixed set of expressions. Context score floors, passage counts and truncation can also leave out retrieved material.

Words are imperfect proxies for intent
  1. Counted: 2
  2. Counted: 1

More detail Illustrative examples

“Where do the stand and its case go?”

One intended need
Cue found: “and”Counted: 2

“Explain loan renewals; describe workshop access.”

Two intended needs
No cue wordCounted: 1

A word-based rule can count one intent as several needs or miss several needs that do not use its cue words.

Figure 18. Query rules and context budgets bound what reaches the answer.

Answer text does not embed citations

The response includes source metadata with scored passages. The generated answer text itself does not embed source citations.

Sources beside the answer, not inside it
  1. Answer text
  2. Source metadata

More detail Illustrative examples

Answer text

Return the stand clean before the workshop closes.

No inline citations
Source metadatascored_chunks

Document A · passage 0 · score 0.92

Alongside the answer

The response carries document and passage identities in scored_chunks metadata, not inline citations attached to the answer’s individual claims.

Figure 19. Source metadata accompanies the answer rather than being embedded in it.