Parth TiwariStart a project
MenuClose

Case 09 / Operational fraud scoring

UPI Fraud Engine

UPI Fraud Engine evaluates a frozen model at the alert volume an operations team can actually review, then keeps held-out testing and seven-day replay as separate evidence.

View the product proof View repository
Role
Machine-learning engineer
Built at
Personal project
Contribution
Features, model, thresholding, serving, and replay
Status
Shipped
Real UPI Fraud Engine evaluation chart showing precision and recall across the seven-day replay
Real evaluation visualisationReplay evidence, not a mockup

Measured, with the denominator attached

Proof before polish.
92.06% precision

On an 85,429-row held-out test set, the UPI model reached 92.06% precision and 12.81% recall at a 0.5% alert budget.

85,429 held-out transactionsVerified 2026-08-28
75.22% replay precision

Across a 22,071-transaction seven-day replay, the UPI system recorded 75.22% precision, 12.13% recall, and no alert-budget violations.

22,071 transactions across 7 daysVerified 2026-08-28
What it is

01 / What it is

The threshold is an operational decision, not a decoration on the model.

The UPI Fraud Engine scores transactions and turns the highest-risk fraction into alerts.

The operating point is constrained by an alert budget, so precision and recall are reported at the threshold the team could actually review.

Built for

A fraud operations or risk team that needs a reviewable alert queue, paired precision and recall, and a clear separation between offline evaluation and time-based replay.

I built the temporal feature pipeline, leakage controls, model evaluation, frozen serving artifacts, alert-budget thresholding, real-time API, dashboard visualisations, and seven-day operational replay.

Contribution record

02 / Product proof

See the actual product before reading the argument.

Real replay visualisation

Real UPI Fraud Engine chart showing daily alert-budget compliance during the seven-day replay
This committed evaluation visualisation shows the alert-budget constraint across the replay. It is evidence from the repository, not simulated dashboard data.
01
Feature recordPast only
Input
Transaction and historical behaviour
Split
Temporal holdout
Guard
No future leakage
Source: Committed feature pipeline at dbc43ad
Build time-aware features

Behavioural and temporal features are computed without letting future information leak into an earlier transaction.

02
Real UPI Fraud Engine fraud breakdown visualisation from the evaluation repository
Choose an alert budget

The frozen model's score distribution is cut at the 0.5 percent review budget, then precision and recall are reported together.

03
Real UPI Fraud Engine daily precision and recall trend across the operational replay
Replay through time

A separate seven-day replay applies the chosen operating policy and records daily precision, recall, and budget compliance.

03 / Problem and responsibility

A strong model metric can still create an alert queue no team can review.

Fraud datasets are imbalanced, and a strong ROC-AUC can still produce an unusable queue.

The model, temporal features, threshold, and replay all need separate evidence so an offline score is not mistaken for operational performance.

Data
Temporal splitting, leakage prevention, behavioural features, imbalance handling, and reproducible artifacts.
Model
XGBoost evaluation, probability scoring, frozen preprocessing, and threshold selection.
Operations
Alert-budget policy, serving surface, daily replay, paired metrics, and evidence visualisation.

04 / Research that changed the build

Research only matters here when it changed a decision.

  1. 01

    Temporal leakage review

    Random splits and future-derived aggregates can make fraud detection look stronger than it will behave on later transactions.

    Feature construction and evaluation follow time, and the replay remains separate from the held-out model result.
  2. 02

    Alert-budget evaluation

    Optimising ROC-AUC or a default threshold does not determine whether the resulting queue is operationally reviewable.

    The operating point is selected and reported at a fixed 0.5 percent alert budget with precision and recall together.
  3. 03

    Dynamic-threshold experiments

    Score distributions can shift across time, so a static threshold may violate volume or quality expectations.

    Threshold behaviour and daily budget compliance became explicit replay outputs.
  4. 04

    Seven-day operational replay

    The same system produced different precision and recall on the replay than on the held-out model test.

    The two evaluations became separate claims with separate denominators and could no longer be collapsed into one number.

05 / Architecture

Choose and evaluate the threshold under an explicit alert budget, then replay it across time.

The pipeline builds temporal and behavioural features, trains the model, freezes production artifacts, and scores transactions through an API.

A separate replay measures the chosen operating point across consecutive days. Its result is never merged with the held-out test result.

Temporal features to frozen model to alert-budget replay
  1. 01TransactionsChronological rows enter with historical behaviour only
  2. 02FeaturesTemporal and behavioural transforms follow the frozen contract
  3. 03ModelXGBoost returns a risk score rather than a final verdict
  4. 04Budget thresholdThe highest-risk 0.5 percent becomes the review queue
  5. 05ReplaySeven consecutive days record paired quality and budget compliance

06 / Decisions and trade-offs

What I chose, what I rejected, and what the choice costs.

DecisionRejectedTrade-off
Select the operating point under an explicit alert budget.

Use the default probability threshold or optimise one model metric in isolation.

Recall remains low, while the alert volume stays tied to an operational capacity.

Freeze the model and preprocessing artifacts before replay.

Recompute or retune the pipeline while evaluating the same replay window.

The replay cannot benefit from later tuning, while its result remains attributable to one production candidate.

Publish held-out and replay evidence separately.

Choose the more flattering precision value as the system's headline result.

The story contains more context, while each number keeps the dataset and question it actually answers.

07 / Evidence

Held-out model quality and operational replay answer different questions.

92.06% precision

On an 85,429-row held-out test set, the UPI model reached 92.06% precision and 12.81% recall at a 0.5% alert budget.

85,429 held-out transactionsVerified 2026-08-28
75.22% replay precision

Across a 22,071-transaction seven-day replay, the UPI system recorded 75.22% precision, 12.13% recall, and no alert-budget violations.

22,071 transactions across 7 daysVerified 2026-08-28

Held-out: 92.06 percent precision and 12.81 percent recall on 85,429 rows. Replay: 75.22 percent precision and 12.13 percent recall on 22,071 rows across seven days. These records remain separate.

08 / What failed

The failures are part of the architecture record.

  1. 01

    Two precision values became one claim

    What I saw
    Earlier copy quoted a precision number without saying whether it came from held-out evaluation or operational replay.
    Why
    Different datasets and purposes were collapsed into one system-level headline.
    What changed
    The held-out and replay records now keep separate precision, recall, dates, and denominators.
    Still true
    Readers may still overvalue precision if recall and alert budget are separated visually.
  2. 02

    A model metric ignored queue capacity

    What I saw
    Offline optimisation could select a threshold that produced more alerts than a team could review.
    Why
    The evaluation treated classification quality independently from the operating budget.
    What changed
    Threshold selection and reporting are anchored to a fixed 0.5 percent alert budget.
    Still true
    The chosen budget is a project assumption, not evidence from a real fraud operations team.
  3. 03

    Replay performance fell below held-out performance

    What I saw
    Precision moved from 92.06 percent on held-out rows to 75.22 percent across the replay.
    Why
    Time-based operational conditions differ from the static held-out test distribution.
    What changed
    The replay result is published as a separate operational record rather than hidden or merged.
    Still true
    Seven days are not enough to establish long-term stability or drift behaviour.
Read why the two precision results stay separate

09 / Limits and future

What the product cannot do yet, beside what may come next.

Current limitations

  • The 0.5 percent alert budget is an explicit project operating assumption, not a customer-validated staffing limit.
  • Recall is 12.81 percent on held-out rows and 12.13 percent in replay, so most fraudulent transactions are not captured at this budget.
  • The seven-day replay is too short to establish seasonal drift, production stability, or long-term financial impact.
  • The risk score prioritises review and is never presented as a final fraud verdict.

Deliberate boundaries

  • Call a risk score a final fraud verdict.
  • Hide recall to make precision look stronger.
planned

Longer temporal replay

Run the frozen operating policy across a larger period and report drift, daily volume, precision, recall, and budget violations together.

investigating

Adaptive thresholds

Test dynamic threshold policies without allowing daily retuning to erase the precommitted alert-budget contract.

planned

Human-review feedback

Model analyst decisions and queue outcomes before claiming operational impact or a final fraud determination.

10 / Stack and sources

The systems underneath it, and where this account comes from.

  • Python
  • XGBoost
  • FastAPI
  • feature engineering
  • backtesting
Audited internallyUPI Fraud Engine committed repository and frozen evaluation artifacts
Audited internallyReal held-out and replay visualisations