Parth TiwariStart a project
MenuClose

Case 04 / Evidence-bound retrieval

MedRAG

MedRAG answers from a bounded drug-guidance corpus, keeps citations attached, and treats refusal as a valid result when the retrieved evidence is insufficient.

Read the evidence record View repository
Role
AI engineer
Built at
Personal project
Contribution
Retrieval, evaluation, refusal policy, and application
Status
Shipped
Recorded evaluationEvidence before answer
Corpus
Bounded FDA and NICE drug guidance
Retrieval
Vector search with recorded cited chunks
Evaluation
20 questions
Refusals
4 recorded
Source: Committed RAGAS run, 2026-02-03
Real evaluation recordCommitted main at a741897

Measured, with the denominator attached

Proof before polish.
4 of 20 refused

MedRAG refused 4 of 20 questions in its final recorded evaluation.

20 evaluation questionsVerified 2026-08-28
What it is

01 / What it is

The answer is allowed only inside the evidence boundary.

MedRAG retrieves from a bounded collection of FDA and NICE drug documents and writes an answer with citations.

When the retrieved evidence does not support the question, the system is designed to stop instead of filling the gap.

Built for

A reader exploring drug guidance who needs the supporting document context kept beside the answer and a visible stop when that context is not available.

I built the document parsing, chunking, vector and BM25 retrieval experiments, evidence gate, cited generation path, Streamlit surface, API, and the recorded 20-question RAGAS evaluation.

Contribution record

02 / Product proof

Read the source evidence before the argument.

Evaluation trace

Evaluation traceOne supported question, one refusal
Query 01
Common side effects of warfarin
Retrieved
8 chunks; 3 cited
Query 07
Ibuprofen with lisinopril
Retrieved
8 chunks; 0 cited
Outcome
Refused
Source: data/evaluation/ragas_results.json at a741897
This is a DOM rendering of the committed evaluation metadata, not simulated product UI. It shows the system's supported and refused paths without publishing medical advice.
01
Workflow recordDocument boundary
Sources
FDA and NICE guidance
Parsing
Document text with source metadata
Output
Bounded chunks
Source: Committed ingestion and corpus source
Bound the corpus

FDA and NICE documents are parsed into source-aware chunks before any question is answered.

02
Retrieval recordCandidate evidence
Candidate A
Vector retrieval
Candidate B
BM25 lexical retrieval
Risk
Hybrid context contamination
Source: Committed retrieval scripts and evaluation
Retrieve and test

Lexical and semantic paths were compared; hybrid retrieval was not accepted as automatically better because mixed context could contaminate an answer.

03
Policy recordEvidence gate
Enough evidence
Generate with citations
Insufficient evidence
Refuse
Not allowed
Unsupported medical completion
Source: Committed generation and refusal path
Answer or refuse

Generation is downstream of evidence sufficiency, and every supported response keeps citations attached.

03 / Problem and responsibility

Fluent medical prose is dangerous when the supporting context is missing.

Medical language makes a fluent unsupported answer more dangerous, not more useful.

The system needed a retrieval and refusal contract that could be evaluated separately from how confident the prose sounded.

Corpus
Source selection, parsing, chunking, metadata, and bounded retrieval context.
Retrieval
Vector and BM25 comparison, reranking experiments, evidence sufficiency, and citation retention.
Evaluation
Recorded RAGAS run, refusal observation, query metadata, and honest interpretation limits.

04 / Research that changed the build

Research only matters here when it changed a decision.

  1. 01

    FDA and NICE document boundaries

    The system can only support what its indexed guidance contains, regardless of how plausible a broader answer sounds.

    The corpus boundary and citation source became visible parts of the answer contract.
  2. 02

    Parsing and chunking experiments

    Document structure and source metadata affect whether retrieval returns a usable evidence unit.

    Parsing and chunk provenance were treated as retrieval architecture, not preprocessing trivia.
  3. 03

    Vector versus BM25 tests

    Semantic retrieval and lexical matching fail differently, while combining them can also introduce unrelated context.

    Retrieval methods were evaluated separately and hybrid contamination remained an explicit risk.
  4. 04

    RAGAS and refusal review

    Answer quality metrics do not replace an evidence-sufficiency decision for questions outside the retrieved support.

    Refusal became a first-class outcome and is reported separately from fluent answer quality.

05 / Architecture

Make evidence sufficiency a gate before generation, and keep the cited source beside every supported answer.

The pipeline combines lexical and semantic retrieval over a bounded corpus, reranks evidence, and produces cited answers only when the evidence clears the gate.

Evaluation records faithfulness, relevance, context precision, and refusals as separate outcomes.

A bounded path from guidance document to cited answer or refusal
  1. 01GuidanceBounded FDA and NICE documents with source metadata
  2. 02ChunksParsed evidence units stored for lexical and semantic retrieval
  3. 03RetrieverSelects and reranks candidate evidence for the question
  4. 04Evidence gateAllows cited generation or returns a refusal
  5. 05EvaluationRecords per-query support, citations, refusal, and RAGAS metrics

06 / Decisions and trade-offs

What I chose, what I rejected, and what the choice costs.

DecisionRejectedTrade-off
Gate generation on retrieved support and retain citations.

Let the model answer from general knowledge when the bounded corpus is thin.

More questions end without an answer, while unsupported completion becomes less likely.

Evaluate retrieval and generation as separate surfaces.

Use one aggregate answer score as proof that the entire pipeline works.

The evaluation is harder to summarise, but failure location stays visible.

Keep hybrid retrieval as an experiment, not an assumed upgrade.

Merge semantic and lexical results and call the larger context better by default.

The selected path may miss useful context, while contamination is easier to reason about.

07 / Evidence

Four refusals are observed. They are not proof of clinical safety.

4 of 20 refused

MedRAG refused 4 of 20 questions in its final recorded evaluation.

20 evaluation questionsVerified 2026-08-28

Only the 4-of-20 refusal observation is published here. Other aggregate metrics remain in the source artifact and are not promoted without separate claim records and interpretation review.

08 / What failed

The failures are part of the architecture record.

  1. 01

    Readable answers hid weak support

    What I saw
    An answer could sound complete even when the retrieved context did not establish it.
    Why
    Early review weighted prose quality more heavily than evidence sufficiency.
    What changed
    Retrieval quality, faithfulness, citations, and refusal were separated into explicit outcomes.
    Still true
    The recorded evaluation is internal and does not establish clinical safety.
  2. 02

    More retrieval could make context worse

    What I saw
    Hybrid retrieval could add related-looking but unsuitable chunks to the answer context.
    Why
    Lexical and semantic results were combined without assuming every additional chunk improved support.
    What changed
    Vector and BM25 behaviour were compared independently and contamination stayed visible in the decision record.
    Still true
    The best retrieval policy is not proven across a larger medically reviewed test set.
  3. 03

    The application depends on external services

    What I saw
    A committed local surface cannot be treated as a durable public demo without its model and retrieval services.
    Why
    Generation and evaluation use external model services beyond the static portfolio boundary.
    What changed
    This page uses the committed evaluation artifact as the primary proof instead of fabricating a successful live answer.
    Still true
    A fresh end-to-end local capture remains unverified in this batch.
Read why refusal is a product feature

09 / Limits and future

What the product cannot do yet, beside what may come next.

Current limitations

  • MedRAG is not clinically validated, is not a diagnostic tool, and must not replace a clinician or primary guidance.
  • The 20-question internal evaluation is too small to establish broad medical reliability.
  • The published refusal count is an observed outcome, not proof that every unsupported question will be refused.
  • The local Streamlit application was not used as publication proof when external services could not be verified.

Deliberate boundaries

  • Diagnose a person or replace a clinician.
  • Treat one internal evaluation as proof of medical safety.
planned

Clinician-reviewed evaluation

Create a larger question set with reviewed support and refusal expectations before making stronger quality claims.

investigating

Retrieval contamination tests

Test vector, BM25, and bounded hybrid policies against cases where adjacent drug context could mislead generation.

planned

Reproducible local evidence capture

Package a service-independent recorded path that demonstrates citations and refusal without exposing credentials or medical user data.

10 / Stack and sources

The systems underneath it, and where this account comes from.

  • Python
  • FastAPI
  • RAGAS
  • ChromaDB
  • BM25
  • Streamlit
Audited internallyMedRAG committed repository, evaluation, and retrieval artifacts
Audited internallyPortfolio claim record for the final refusal count