piimask: a 0.5B model that redacts PII locally — in English and Turkish

Today we released piimask, our first public small language model. It's a 0.5B model that detects personally identifiable information (PII) in English and Turkish, and it runs fully on your machine — no cloud, no API key, no data leaving the device. The model and the evaluation set are both on Hugging Face.

This post is the story behind it — including the part where the model wasn't the answer.

Why a tiny local model

Redacting PII is a narrow, well-defined task. Yet the reflex today is to send the text to a frontier LLM and ask it to strip the sensitive parts. That's slow, expensive at volume, and — this is the punchline — it ships your PII to a third party before you've redacted it. Sending data to a remote model in order to remove private data defeats the point.

The classic alternative, hand-written regexes, avoids the round-trip but breaks the moment text gets informal — or the moment it's Turkish, where suffixes attach to names and places and a pattern written for one form misses ten others.

There's a third option that only recently became practical: fine-tune a sub-1B model on the one task you care about. On a narrow task, a 0.5B student routinely matches or beats a general-purpose giant, because all of its capacity is spent on your distribution. It runs offline on a laptop, costs nothing per request, and — crucially — its failure modes are visible in a test set you control.

piimask is that model. Given a message, it returns a JSON list of entities across nine types — PERSON, EMAIL, PHONE, NATIONAL_ID, IBAN, CREDIT_CARD, ADDRESS, DATE_OF_BIRTH, IP_ADDRESS:

{"entities": [{"type": "PERSON", "value": "Ayşe Yılmaz"}, {"type": "EMAIL", "value": "ayse@firma.com"}]}

It's built for short, informal text — chat messages, support tickets, emails, form fields. The obvious use: scrub input locally before it ever reaches a third-party LLM, or cut PII out of developer logs. For Turkish in particular, good open options were thin on the ground — and that gap is a lot of why we built it.

The lesson that hit harder than the model

Here's the part worth your time.

On our own internal test set, piimask scored an F1 of 0.75 on Turkish addresses. Respectable. Then we ran it against a separate, independently-built benchmark with adversarial distractors — and Turkish address recall collapsed to 0.16.

The model had memorized one address format. The Turkish addresses in our training data mostly followed a single shape, and the model learned that shape rather than the concept. On street-first addresses it had never seen, it simply didn't fire.

The instinct is to throw more data at it. We tried. It made the independent score worse — more examples of the same format only deepened the memorization without teaching structure.

The fix wasn't the model at all. Turkish addresses have a strong, recognizable structure: a neighborhood/street anchor (Mahallesi, Caddesi, Sokak…), a number component (No, Daire, Kat…), and a province name from a closed set of 81. That's not a job for a probabilistic model; it's a job for a grammar. We wrote a small, order-invariant deterministic layer that recognizes that structure directly. Recall went from 0.16 to ~1.0 (F1 0.79) — with zero retraining, and zero impact on English, because the grammar only fires on Turkish structure.

The takeaway generalizes well beyond addresses: not every quality problem is a model problem. The strongest system is a small model doing the fuzzy, context-dependent work — names, free-text addresses — with a deterministic layer doing the structured, checkable work: emails via format checks, cards via Luhn, IBANs via mod-97, Turkish IDs via TCKN. Each does what it's good at. Reaching for a bigger model when the real answer is a fifty-line grammar is a common and expensive mistake.

The benchmarks, kept honest

On that independent bilingual set, micro-F1 came out:

• piimask (0.5B): 0.81
• ai4privacy DeBERTa: 0.65
• Microsoft Presidio: 0.51

Two honest notes, because a benchmark without caveats is just marketing:

One — the comparison isn't apples to apples. The baselines aren't built for Turkish, so their near-zero Turkish scores are a scope limitation, not a fair head-to-head. On clean English structured fields they stay perfectly competitive.

Two — we tuned against this set. We read the errors and iterated, so these numbers are a strong development signal, not a pure held-out result. A genuinely untouched slice would score a little lower. We'd rather tell you that than round it away.

Use it as one layer, not the only one

One more thing we say loudly on the model card: redaction is a recall problem — a single miss is a leak. piimask misses a few percent even on its own short-text distribution, and much more on the long, formal documents it was never trained on. So it is not a compliance-grade safeguard on its own. Use it as one layer in defense-in-depth, with a stricter gate or human review on high-stakes flows.

That's not a disclaimer bolted on at the end — it's the same philosophy as the address fix. Layers that each do one thing well, and honesty about where each one stops.

Try it — and break it

The model and eval set are live on Hugging Face under Apache-2.0. Run it, throw hard inputs at it, and send us the formats we still miss — especially Turkish ones. That feedback is exactly how the deterministic layer grew.

• Model & eval set: huggingface.co/promptrails/piimask-qwen2.5-0.5b

---

piimask is one of the guardrails behind PromptRails' operating layer — PII masking is a built-in step in every workflow. Book a 30-minute call and we'll map where it fits in yours.