How VoiceSafeKit was built

01 - The Problem

Voice assistants and the data they leak

Voice assistants are becoming more capable because they can now pass what a user says directly to a large language model. That creates a real risk. When someone speaks to an assistant, they often include private details without thinking: an email address, a phone number, a password reset flow, a medical question. The assistant records all of it and sends it upstream.

There is usually nothing in between. The raw transcript goes to a cloud model, often unfiltered. Most developers building voice assistants focus on intent detection and response quality. Privacy filtering at the speech input layer is rarely a priority and off-the-shelf solutions for it are almost non-existent.

VoiceSafeKit addresses that gap. It is a safety layer that checks what was said before it leaves the device.

02 - Architecture

Three stages: Capture, Check, Protect

The system is split into three clearly separated stages. Each one has a single responsibility.

The data flow within Stage 1 is worth showing explicitly:

Nothing in that chain touches a server. The audio is captured in the browser, decoded in the browser and transcribed in the browser. The only thing that ever leaves the device is the cleaned transcript and only if the user chooses to forward it.

03 - In-Browser Speech Recognition

Why Whisper and how it runs in a browser

The first version of VoiceSafeKit used the browser's built-in Web Speech API. It was unreliable. The API stops recognising speech after a short pause, varies widely between browsers and in some Chrome versions silently fails unless a separate microphone permission call has already been made. It is also inherently online: Chrome's implementation sends audio to Google's servers.

The replacement is Whisper, a speech recognition model published by OpenAI in 2022 and widely regarded as one of the most accurate open speech models available. The variant used here is whisper-tiny.en, an English-only model around 75 MB in size.

The transcription pipeline, simplified:

// Decode the audio blob to 16kHz mono - Whisper's required format
const audioCtx = new AudioContext({ sampleRate: 16000 });
const decoded  = await audioCtx.decodeAudioData(arrayBuffer);
const float32  = decoded.getChannelData(0);

// Run Whisper
const model  = await pipeline("automatic-speech-recognition",
                              "Xenova/whisper-tiny.en");
const result = await model(float32, { sampling_rate: 16000 });
const text   = result.text.trim();

The model is loaded lazily: it starts downloading as soon as the user presses Record, so by the time they finish speaking, it is usually ready. On repeat visits, the cached version loads in under a second.

Why tiny over base? The tiny model (75MB) completes transcription in roughly 2-4 seconds on a typical laptop. The base model (145MB) is more accurate but takes longer to load and process. For a tool where speed matters for usability, tiny is the right trade-off.

04 - The Detection Engine

What it looks for and how it decides

Once the transcript exists as plain text, VoiceSafeKit runs it through a set of detection rules. Each rule has a pattern (a regular expression), a severity level, a human-readable explanation and a replacement string for the redacted output.

Severity and scoring

Every finding contributes to a risk score. The score is additive, capped at 100.

Decision labels

Overlap resolution

When two rules match overlapping text (for example, a 16-digit number that could be both a phone and a card), the higher-severity rule wins. The overlapping lower-severity finding is discarded. This prevents double-counting and ensures the more serious label is shown.

function removeOverlaps(findings) {
  const accepted = [];
  findings.sort((a, b) =>
    severityRank[b.severity] - severityRank[a.severity]
  );
  for (const f of findings) {
    if (!accepted.some(x => f.start < x.end && x.start < f.end))
      accepted.push(f);
  }
  return accepted;
}

05 - Technical Stack

What was used and why

06 - Design Decisions

The reasoning behind key choices

Why no API key for transcription? Any API key embedded in a public GitHub repository is public. Anyone can read your source, extract the key and use it. Running the model entirely in the browser avoids this problem entirely. It also means the audio never touches a third-party server, which is directly relevant to the privacy use case.

Why not use the Luhn algorithm to validate card numbers? The Luhn checksum is used by card issuers to catch typos. It does not distinguish real cards from made-up ones that happen to pass. More importantly, a user dictating a card number to a voice assistant may speak the digits in a way that produces a transcript with a typo. The card would fail Luhn but should still be flagged. A safety tool should err on the side of flagging too much, not too little.

Why flag intent (medical, legal, financial) separately from data? A transcript might not contain any personally identifiable information but still be problematic. Someone asking an AI for a medication dose change or investment advice needs different handling to someone asking for a restaurant recommendation. Intent rules capture that distinction.

Why client-side everything? The thesis of the project is that sensitive voice data should not leave the device unless it has been checked and cleaned. If the checking itself required sending data to a server, the whole premise would collapse. Everything runs locally by design.

Why encrypt saved exports? The filter has to read plain text while it is checking the transcript. That part happens in memory. Saved files are different. They can include the redaction map, which may contain the exact email address, SSN or account number that was removed from the safe transcript. Browser exports now use the Web Crypto API to save encrypted JSON. The Python CLI uses AES-256-GCM through the cryptography package. TLS/HTTPS is still useful when data travels over a network, but it does not encrypt a file sitting on disk.

How the export encryption was built. The analysis engine was left unchanged: it still produces the same structured result with decision, findings, safe_transcript and redaction_map. The encryption layer sits after analysis, at the moment a user chooses to save the result. That keeps detection simple while making the saved artifact safer.

In the browser, the Export JSON button requires a local export key, derives an AES-GCM key through Web Crypto and saves only the encrypted bundle. In the CLI, --out writes encrypted files by default using VOICESAFEKIT_EXPORT_KEY. The explicit --plain-out option is kept for local debugging, and voicesafekit decrypt restores an encrypted export when the same passphrase is available. Tests cover the round trip and confirm that sensitive redaction-map values are not visible in the stored JSON.

07 - Limitations

What it does not do well

Accent and noise sensitivity. Whisper tiny is accurate for clear English speech in a quiet environment. Heavy accents, background noise, or fast speech can produce transcription errors. Those errors may cause sensitive content to be missed. The base model would improve accuracy but at the cost of speed.

Keyword rules miss novel phrasing. The detection rules use fixed patterns and keywords. Someone saying "give me a new dose" rather than "change my medication" would not trigger the medical rule. A neural classifier trained on intent would handle this better.

No streaming transcription. The current design records a full clip and then transcribes it. Real-time word-by-word transcription is possible with a sliding-window approach to Whisper, but it was not implemented here.

Not a security product. VoiceSafeKit is a demonstration of the concept. It should not be used as the sole privacy control in a production voice assistant. It is a starting point and a reference implementation.

08 - Future Work

Where this goes next

The most natural next step is integration with real open-source voice assistant frameworks. Two candidates are already referenced in the project:

OpenVoiceOS is an open-source voice operating system built on the Mycroft platform. A VoiceSafeKit adapter would intercept the STT output before it is passed to the intent parser, giving privacy filtering at the right layer of the pipeline.

Wyoming protocol is a lightweight protocol used by Home Assistant for communicating between voice satellite devices and processing servers. A Wyoming microservice wrapping the VoiceSafeKit detection engine would allow any Wyoming-compatible assistant to use it as a filter stage.

Other areas worth exploring include streaming transcription, neural intent classification to replace keyword rules and a dashboard for assistant developers to tune sensitivity thresholds per deployment context.

09 - Confidence Scoring

Why every finding needs a probability

The original engine returned binary findings: a pattern either matched or it did not. That approach overstates certainty. A phone number matched near a credit card number is less likely to be a real phone number than one found in isolation. A medical keyword in a clearly technical context is less alarming than the same keyword in an emotionally charged message. Binary matching discards this context entirely.

Version 0.2.0 adds a confidence: float field to every Finding. Each rule declares a base confidence that reflects how precise its pattern is. Context-aware validators then adjust it upward or downward.

Confidence-weighted scoring

The risk score is now computed as:

def _score(findings: list[Finding]) -> int:
    total = sum(SEVERITY_POINTS[f.severity] * f.confidence for f in findings)
    return min(100, round(total))

A 70%-confidence high-severity finding contributes 25 points (36 × 0.70) rather than the full 36. A Luhn-verified payment card at 99% confidence contributes 35.6 points. This means the score reflects genuine risk rather than a count of regex matches.

10 - Multi-Turn Conversation Analysis

Why a single transcript is not enough

Voice assistants do not work in isolated utterances. A user might give their name in the first turn, their email address in the second and their date of birth to confirm their identity in the third. Analyzed individually, turns two and three each get a medium-severity result. Seen together, the session reveals a structured identity disclosure that carries much higher real-world risk.

The analyze_conversation function takes an ordered list of utterances and returns a ConversationResult with per-turn analysis, a peak decision and the union of all entity kinds detected across the session.

from voicesafekit import analyze_conversation

result = analyze_conversation([
    "Set a timer for ten minutes.",
    "My SSN is 471-55-8843 - please verify me.",
    "Thank you, that is all.",
])

result.peak_decision    # "BLOCK"
result.cumulative_score # 95
result.seen_kinds       # ("ssn",)
len(result.turns)        # 3

The web app exposes this as a Conversation mode toggle. Each turn is added explicitly, a turn history panel tracks the sequence and a timeline view below the main panels shows every turn's findings and decision alongside a cumulative summary bar.

11 - New Detector Classes (June 2026)

Financial identifiers, context signals and human vulnerability

The v0.2.0 detector set adds three categories that were entirely absent from the original engine.

Financial identity numbers

SSNs, IBANs and CVV codes are the three most common financial identifiers that a voice assistant user might accidentally speak. Each has a distinct detection challenge.

SSN - The pattern \b(?!000|666|9\d{2})\d{3}[-\s]\d{2}[-\s]\d{4}\b excludes area numbers that the Social Security Administration has never issued. This eliminates a class of false positives from account numbers and tracking codes that happen to match the XXX-XX-XXXX format.

IBAN - The pattern matches the standard two-letter country code, two check digits and 11-30 alphanumeric BBAN characters. A mod-97 checksum then validates the matched text. Without this step, any sufficiently long alphanumeric string near a country code would be flagged.

CVV - Unlike card numbers, CVV codes are short (3-4 digits) and indistinguishable from any other number without context. The detector is keyword-triggered: it only fires when a CVV/CVC/security-code phrase precedes the number.

Emotional and situational context

Two new intent categories address situations that no PII detector can catch but that require fundamentally different assistant behavior.

The emotional distress signal fires on phrases expressing hopelessness, worthlessness, or a desire to disappear. When detected, the assistant guidance changes: instead of "redact and proceed," the guidance tells the assistant to respond with empathy and direct the user to appropriate support. The confidence is set at 70% because emotional language is genuinely ambiguous - "I feel like giving up" can mean quitting a project as easily as it can signal a crisis.

The coercion signal fires on phrases suggesting the user is being directed by a third party: "they made me say this," "don't tell anyone I said this." This is the lowest-confidence detector in the system (65%) because coercive language and ordinary reporting language overlap substantially. A higher false negative rate is acceptable here because the guidance produced is non-destructive - it simply tells the assistant not to act on instructions that may have been forced.

12 - Closing

What this project is really about

Voice assistants are getting smarter quickly. The gap between what someone says and what an AI model is allowed to act on needs careful thought. Most of the safety work in this space focuses on model outputs (preventing harmful responses) rather than model inputs (preventing harmful data going in).

VoiceSafeKit is an argument that input filtering matters too and that it can be done locally, in a browser, with no infrastructure and with a good user experience. The code is open-source. The approach is intentionally simple so that it can be understood, copied and improved.