100 seed sentences, one vector, and a ranked queue of suspect labels

Why this matters

Most production NLP runs on silver-labeled data — automatically tagged at scale by translation pipelines, distant supervision, weak heuristics. Silver is how teams get to enough training examples to make a model work; it's also noisy by design. Auto-translation drift, label-rule edge cases, brittle keyword heuristics — these accumulate into double-digit error rates that are invisible until something downstream breaks.

The contrast is gold-labeled data — human-annotated, multi-annotator, adjudicated. AfriSenti is gold. It's carefully built and considerably cleaner. But gold doesn't scale: a 100K-tweet gold corpus is a year of annotator work; a 100M-sentence silver corpus is a weekend of pipeline runs. Every team has both. And every team's silver corpora are quietly contaminating their training and evaluation.

Two questions follow:

1. Can a model find the silver mislabels without being trained on the labels it's auditing?
2. Does the answer transfer across languages without sentiment supervision in each?

Yes to both, with caveats sized to the evidence. With 100 hand-curated Zulu archetypes and Bhala's multilingual embedding space — not trained on sentiment — we hit 100% top-10 precision identifying the real silver errors on a held-out human-verified validation set of 80 sentences. (At that n, top-10 precision of 100% is strong but has wide confidence bounds; the AUROC of 0.732 over the full set is the more stable number.) The same Zulu-built sentiment direction, applied cross-lingually, surfaces clear errors and annotation-policy boundary cases in AfriSenti Swahili gold.

To stress-test the audit, we ran it on a worst-case substrate: a 10K-sentence Bhala-built bulk Zulu sentiment corpus auto-labeled at scale. Random review of 175 sentences from that corpus showed silver-labeling produced ~55% disagreement with human re-review — close to the public ceiling for what poorly-supervised auto-labeling produces in low-resource languages. The audit found the real errors in that noise.

The setup, in one paragraph

Bhala's multilingual encoder produces sentence embeddings in which a sentiment direction can be estimated from a handful of examples — the same mean-difference construction used throughout the concept-direction literature (think word2vec's "king − man + woman ≈ queen", one level up at sentence scale). The encoder was never trained on sentiment — we just need to find the sentiment direction in its space, which is what the seed pairs do for us. We hand-curated 50 unmistakably positive and 50 unmistakably negative Zulu sentences, embedded them through the Bhala API, and computed a single direction:

sentiment_axis = unit_norm( mean(embed(positive_seeds))
                          - mean(embed(negative_seeds)) )

That's the entire learned object. One unit vector. No classifier head, no fine-tuning, no labeled training data outside those 100 seeds.

For any new sentence:

score = embed(sentence) · sentiment_axis

inconsistency = -score    if declared_label == "positive"
                +score    if declared_label == "negative"
                |score|   if declared_label == "neutral"

Sort by inconsistency descending. The top of the list is the candidate-mislabel queue.

Three experiments

Exp A — 40 hand-curated archetypes (baseline)

20 positive + 20 negative archetypal Zulu seeds. Validate on 175 random-sampled silver-labeled sentences from the bulk corpus that we hand-corrected. Top-10 precision: 90% — of the 10 sentences Bhala flagged loudest, 9 were ones the human reviewer corrected. AUROC across all 175: 0.571 — the signal lives at the extremes, not uniformly.

Exp B — 175 hand-labels as seed pool, 5-fold CV

We tried scaling the seed pool by using the 175 hand-corrected sentences themselves as seeds (with cross-validation to avoid leakage). Mean top-10 precision dropped to 58%.

The lesson: more isn't better if more isn't clean. The 175 hand-labels include genuinely-ambiguous sentences (some at conf=2 — annotator unsure). Using messy real-world sentences as seeds creates a noisier sentiment axis. Archetypes are clean prototypes; hand-labeled in-the-wild text is not.

Exp C — 100 clean seeds (40 archetypes + 60 highest-confidence hand-labels)

We took the 40 hand-curated archetypes and added 60 hand-corrected sentences filtered to confidence ≥ 4 and label ∈ {positive, negative} — dropping the borderline conf<4 cases and neutrals. Total: 100 clean seeds. Validate on the held-out portion of the 175 (lower-confidence and neutral cases) — no seed/test contamination.

More clean seeds did help. AUROC rose +0.16, top-K saturated at 100%. The compositional geometry has more signal as the seed pool gets larger and stays clean.

What this catches in the wild

Top-15 most-inconsistent sentences from the 9,924 unlabeled bulk Zulu sentences (silver-labeled positive, Bhala says highly inconsistent):

A spot-check of 15 surfaces ~10 obvious errors. The silver auto-labeler appears to have systematically miscategorized somber/biblical/news content as positive — Bhala caught the pattern.

The cross-lingual finding

We took the same sentiment axis built from 100 Zulu seeds and applied it to AfriSenti Swahili gold (3,011 native-speaker-annotated tweets, per Muhammad et al., 2023). Bhala has zero Swahili sentiment supervision. Bhala's encoder transfers across Bantu languages (FLORES Zulu↔Swahili 58.2% on translation) — the question is whether the sentiment direction transfers along with the language coverage.

It does — but the right framing for what gets surfaced is more nuanced than "mislabels." AfriSenti was carefully built (multi-annotator with majority vote and adjudication, native-speaker reviewers per language). What Bhala's cross-lingual probe surfaces is closer to "places where two reasonable annotators or two annotation policies would disagree" — sometimes that is a clear annotation error, sometimes it's a boundary case, sometimes Bhala itself is wrong. We split the top-15 honestly:

Clear annotation errors (Bhala correct, AfriSenti wrong):

Annotation-policy boundary cases (defensible either way):

Bhala's own false positives (Bhala wrong, AfriSenti correct):

The honest takeaway — a small fraction of the top flags are clear AfriSenti errors (the customer-service-apology mislabels are systematic and worth a heads-up to the maintainers), most are annotation-policy boundary cases that surface where two reasonable annotators would disagree (which is exactly what an auditor wants to know about), and a small fraction are Bhala over-flagging short or context-light strings.

The encoder generalized the meaning of pole sana (sympathy/regret) from Zulu seeds without ever being told what pole sana means in a Swahili sentiment-labeled training pair — that's the cross-lingual transfer claim, and it holds. The downstream value isn't "Bhala will tell you which AfriSenti labels are wrong." It's "Bhala will rank your dataset for human reviewers, and the top of the rank surfaces a productive mix of clear errors, boundary cases, and policy ambiguities."

Why the AUROC is 0.732 and not 0.95

Honest framing: this is a precision-at-top tool, not a uniform classifier. The signal is strong at the extremes (top 5–10% most-inconsistent are 80–100% real mislabels) and weaker in the middle (medium-inconsistency is mostly noise). That's exactly the right shape for a reviewer-prioritization tool:

• Random review of 1,000 silver labels → 547 confirmed mislabels (54.7% base rate)
• Bhala-prioritized review of top 100 most-inconsistent → ~80–100 confirmed mislabels (80–100% precision)
• 5–10× reviewer-time multiplier, depending on the cutoff

That's the product. Not "Bhala will fix your dataset." Not "Bhala will tell you the right label." Bhala will rank your dataset so a human reviewer's first hour finds 5–10× as many real errors as random sampling would.

How to use it on your dataset

The recipe is short:

1. Pick a 1-D sentiment axis you care about (positive↔negative; or polite↔abusive; or any binary).
2. Hand-curate 30–50 clear archetypes per side. Short, unambiguous, varied topics. ~30 minutes of human work.
3. Encode all your data with Bhala's API.
4. Compute axis = unit_norm( mean(embed(positive_seeds)) - mean(embed(negative_seeds)) ) — one unit vector defining the direction.
5. Score each sentence by the inconsistency between its position on axis and its declared label.
6. Sort descending. Review the top 5%. Expect 70–100% of them to be real errors.

For Bantu-language datasets you can curate seeds in any one Bantu language and apply the resulting axis cross-lingually to other Bantu languages — at least as a strong starting point. The Zulu→Swahili transfer above shows it works empirically.

Where this fits

This post is one slice — data curation. The pattern it demonstrates is small manual input, frozen encoder, ranked output for a human reviewer: 100 seeds and ~30 minutes of curation bought a review queue whose top is 80–100% real errors against a ~55% base rate, a roughly 5–10× improvement in reviewer yield on this corpus (the multiplier is a projection from the measured precision-at-top numbers above, not an independently measured time study). The same seed-and-rank recipe should apply to other binary axes — polite↔abusive, on-topic↔off-topic — but each is a claim to verify, not one we've verified here.

Three larger threads run through this small experiment:

Explainability by construction. Every flag this audit raises comes with its evidence attached: a named axis, a score on that axis, and the disagreement with the declared label. A reviewer never has to trust a black-box verdict — they see why the sentence was flagged and can overrule it (as they should for the false positives we showed above). That's the difference between explainable AI and a model that emits confident labels: the explanation is the output, not a rationalization generated afterwards.

The regulatory tie. Data governance is not a nice-to-have in the frameworks now taking effect — the EU AI Act makes training-data quality and relevance an explicit obligation for high-risk systems, and NIST's AI RMF puts data provenance in the measurement core. "We reviewed a random sample" is the weakest possible answer to that obligation; a ranked, axis-scored, human-adjudicated review queue — with the seed set and scores retained as artifacts — is a defensible one. Label auditing is what a data-governance obligation looks like when it's implemented rather than attested.

The middleware position. Notice what the encoder did and didn't do here: it never became the classifier of record and it never touched the labels — it sat between the data and the humans, converting a hopeless review task into a tractable one. That's the same middle-layer role our model audits and steering work play between foundation models and applications: not the model, not the app — the explainable, auditable layer in between, for whichever backbone or dataset sits on either side. Explainable AI + safety and bias tooling + middleware is one product surface; this post is what it looks like pointed at data instead of models.

What's next

• Multi-axis extension: same recipe applied to bias axes (28 dimensions) for fairness audits, intent axes for chatbot training data, hate axes for moderation pipelines. Each axis needs its own ~50 archetypes — that's the only manual cost.
• Human-in-the-loop API: POST /v1/audit/labels accepting a dataset + 50 seed sentences + a label field, returning a confidence-ranked list of suspect labels with margin scores. Coming.
• AfriSenti coordination: the customer-service-apology pattern looks systematic enough that we'll reach out to the AfriSenti maintainers with a candidate list — framed as "here's a probe-flagged subset for the maintainers' own review," not as unilateral correction. Their annotation policy is theirs to set.

Reproduce

The full audit ran in ~2 minutes on a single CPU. To reproduce on your own data:

• 100 Zulu seeds (50 positive + 50 negative, hand-curated archetypes)
• A Bhala API key — self-serve here
• Your labeled corpus

Code to compute the sentiment axis and rank your dataset is ~30 lines. We're publishing the seed list and the AfriSenti Swahili audit JSON alongside the API; if you want early access to either, talk to us.

---

This work was conducted by the Bhala research team in May 2026. The cross-lingual transfer result (Zulu seeds → Swahili audit) is consistent with Bhala's prior published cross-lingual numbers on Bantu sentiment and intent.