Parth TiwariStart a project
MenuClose

Case 07 / Evidence-bound career automation

SecondSelf

SecondSelf prepares career work from verified evidence, removes unsupported material before drafting, and makes human review a durable state before consequential outbound action.

Read the evidence record
Role
Product and AI engineer
Built at
Personal project
Contribution
Evidence model, retrieval, drafting, review, and automation
Status
Running
Safe evaluation recordEvidence reaches review before action
Retrieval
Hybrid evidence search
Drafting
Unsupported material removed
State
Durable human-review queue
Private data
Excluded from this page
Source: Committed v2-dev at 4a1447c
Publication-safe recordNo personal KB or application data

Measured, with the denominator attached

Proof before polish.
0.9753 faithfulness

SecondSelf recorded 0.9753 faithfulness on a 20-question internal RAGAS evaluation.

20 evaluation questionsVerified 2026-08-28
What it is

01 / What it is

The system prepares the work. It does not get to become me.

SecondSelf gathers job material, matches it against verified experience, and prepares an application packet.

Outbound work enters a Telegram review queue, where a person can inspect and approve it.

Built for

A job seeker who wants assistance researching roles and preparing accurate applications while retaining final authority over every claim and outbound action.

I built the evidence store, hybrid retrieval, gentle reranking, evidence-first drafting, RAGAS evaluation, preparation state, review workflow, browser-assistance boundary, and the safeguards around unsupported claims and outbound actions.

Contribution record

02 / Product proof

Read the source evidence before the argument.

Safe fixture trace

Safe fixture traceA packet stops at review
Request
Prepare an application packet
Evidence
Retrieve verified projects, skills, and experience
Filter
Remove unsupported material
Durable state
Awaiting human review
Outbound
Not sent automatically
Source: Committed evaluation and test fixtures at 4a1447c
This DOM record uses publication-safe committed fixtures. Personal knowledge entries, applications, emails, credentials, and Telegram content are deliberately absent.
01
Retrieval recordEvidence candidates
Lexical
Exact skills and project terms
Semantic
Related experience evidence
Reranking
Gentle weighting, no invented claim
Source: Committed safe evaluation fixtures
Retrieve evidence gently

Hybrid search finds candidate material, then a bounded reranker improves ordering without turning weak similarity into a new fact.

02
Draft recordSupport before prose
Supported
Included with provenance
Unsupported
Removed before packet
Output
Reviewable draft
Source: Committed drafting tests
Prepare from support

Drafting receives evidence packets and removes material that cannot be traced to a verified record.

03
Review recordConsequential action is pending
State
Awaiting review
Reviewer
Human owner
Action
Approve, revise, or reject
Source: Committed preparation-state fixtures
Wait for a human

The prepared artifact and its provenance enter durable review state instead of relying on one paused process or hidden browser step.

03 / Problem and responsibility

Plausible career copy is still false when the evidence does not support it.

Career automation is especially prone to turning weak evidence into confident claims.

The hard part is not filling a form. It is maintaining one evidence contract across research, resume variants, generated answers, and the final action.

Knowledge
Evidence schema, provenance, hybrid retrieval, reranking, and privacy boundaries.
Application
Research, packet preparation, evidence-first drafting, unsupported-claim removal, and review state.
Automation
Assisted form handling, browser boundaries, reCAPTCHA stop conditions, durable state, and safe failure semantics.

04 / Research that changed the build

Research only matters here when it changed a decision.

  1. 01

    Hybrid retrieval evaluation

    Exact terms and semantic similarity surface different useful evidence, while aggressive reranking can overstate a weak match.

    Retrieval combines lexical and semantic candidates with gentle reranking and keeps provenance attached.
  2. 02

    Evidence-first drafting tests

    A plausible sentence can introduce unsupported scope, ownership, or outcome even when the source packet is mostly correct.

    Unsupported material is removed before drafting and checked again before review.
  3. 03

    20-question RAGAS evaluation

    The retrieval-answer path recorded high faithfulness on a bounded internal set, but that says nothing about job outcomes.

    The metric is published with its 20-question denominator and no placement implication.
  4. 04

    Assisted application and browser failure review

    Brittle scraping, reCAPTCHA, and uncertain form state make fully autonomous submission an unsafe default.

    Preparation and assisted handling stop at durable human review, with uncertain outbound state treated as failure.

05 / Architecture

Remove unsupported claims before drafting and make human review a durable state, not a pause in one process.

Structured evidence feeds retrieval and drafting. A preparation record stores provenance, current stage, and the artifact awaiting review.

Apply mode is deliberately gated. The system may automate preparation, but it does not get silent authority to represent me.

Evidence retrieval to prepared packet to durable human review
  1. 01Evidence storeVerified experience and project records with provenance
  2. 02RetrievalHybrid candidate search and gentle reranking
  3. 03PacketRole-specific support with unsupported material removed
  4. 04Review stateDurable prepared artifact awaiting a human decision
  5. 05Assisted actionProceed only inside verified browser and submission boundaries

06 / Decisions and trade-offs

What I chose, what I rejected, and what the choice costs.

DecisionRejectedTrade-off
Use verified evidence packets before drafting.

Let the language model infer missing skills or outcomes from a job description.

Some applications are less expansive, while every public claim remains attributable.

Represent human review as durable state.

Pause one browser process and assume the user will return before it expires.

The workflow requires persistence and recovery, while approval can survive restarts and delays.

Keep application handling assisted at uncertain boundaries.

Bypass reCAPTCHA or guess whether a brittle page successfully submitted.

More work returns to the user, while the system avoids silently misrepresenting an outbound action.

07 / Evidence

Faithfulness was measured on 20 questions. Job outcomes were not.

0.9753 faithfulness

SecondSelf recorded 0.9753 faithfulness on a 20-question internal RAGAS evaluation.

20 evaluation questionsVerified 2026-08-28

The published 0.9753 faithfulness score belongs to one 20-question internal RAGAS run. It is not evidence that SecondSelf improves job outcomes or can submit applications autonomously.

08 / What failed

The failures are part of the architecture record.

  1. 01

    Drafting outran the evidence

    What I saw
    A plausible answer could include a skill or result that the stored record did not support.
    Why
    The generation prompt was asked to produce complete prose before unsupported material was removed.
    What changed
    Evidence selection and unsupported-claim filtering moved ahead of drafting and remain visible in review.
    Still true
    New prompt and document paths still need the same provenance tests.
  2. 02

    Fire semantics were unsafe

    What I saw
    A retry could make an outbound action uncertain or duplicate because the system did not know whether the previous attempt completed.
    Why
    Preparation state and side-effect state were not separated strongly enough.
    What changed
    Consequential work waits in durable review and uncertain action state is not reported as success.
    Still true
    Every supported application platform needs an idempotent, observable completion contract.
  3. 03

    Browser automation met reCAPTCHA

    What I saw
    The assisted application path could not reliably continue or prove completion on protected pages.
    Why
    The target platform intentionally requires human verification and changes its markup independently.
    What changed
    The workflow stops and returns control instead of attempting to bypass verification or invent a submitted state.
    Still true
    Scraping and page selectors remain brittle even before a verification challenge appears.
Read why the human gate is durable

09 / Limits and future

What the product cannot do yet, beside what may come next.

Current limitations

  • The 0.9753 faithfulness result covers 20 internal questions and does not measure applications, interviews, offers, or job placement.
  • Personal knowledge-base entries, applications, email, credentials, and Telegram content are private and absent from the rendered site.
  • Browser-assisted application handling is vulnerable to markup changes, authentication, reCAPTCHA, and uncertain submission state.
  • Committed v2-dev at 4a1447c is the source boundary; current personal KB working-tree changes are excluded and may only inform future work.

Deliberate boundaries

  • Invent a skill or result to improve a match.
  • Describe human-gated apply mode as fully autonomous job application.
planned

Publication-safe evidence export

Create redacted evaluation fixtures that can be regenerated without touching personal knowledge, credentials, messages, or applications.

investigating

Idempotent assisted actions

Model action receipts and recovery so a retry cannot duplicate a consequential submission or hide an uncertain result.

planned

Evidence coverage review

Measure unsupported-claim removal and retrieval misses on a larger privacy-safe application set before adding outcome claims.

10 / Stack and sources

The systems underneath it, and where this account comes from.

  • Python
  • PostgreSQL
  • RAGAS
  • Telegram
  • browser automation
Audited internallySecondSelf committed v2-dev source, tests, and evaluation
Audited internallySafe RAGAS evaluation record
Audited internallyPortfolio claim record for faithfulness