# Gemini's Local Replacement Scored 0.83 Against Gemini Itself

**Summary:** Gemini labeled 4,290 Reddit comments for $9, then a local GLiNER model reached 0.83 F1 against those labels. I like the economics, but I would not trust that score without a small human-checked test set.

- Canonical: https://markhuang.ai/news/gemini-ner-teacher-needs-audit
- Language: en
- Author: [Mark Huang](https://markhuang.ai/about)
- Published: 2026-09-17
- Section: News
- Tags: Gemini 3.1 Pro, GLiNER, Named Entity Recognition, Weak Supervision, Model Evaluation
- Source: [Peter Vijeh](https://www.petervijeh.com/projects/reddit-ner)
- License: https://creativecommons.org/licenses/by-nc/4.0/

---

![A cloud AI labels forum comment cards that feed a compact local model while a second connection from the same cloud controls the evaluation gauge](https://cdn.markhuang.ai/news/gemini-ner-teacher-needs-audit/hero.webp)

*The cloud model labels the cards, trains the local model, and feeds the gauge. The loop is cheap, but it still grades itself.*

[Peter Vijeh had Gemini 3.1 Pro label 4,290 Reddit comments for $9](https://www.petervijeh.com/projects/reddit-ner), then used those labels to fine-tune GLiNER to recognize knife brands, product models, and materials. The local model reached 0.83 F1 on a locked set of 225 comments.

I like the economics. One paid labeling pass can replace an API call on every new comment, and the resulting model runs locally. I would still put one more step between this experiment and a claim about accuracy: a small set of comments labeled by a person who knows the domain.

The reason is sitting inside the reported score. Gemini wrote the training labels, and Gemini's answers also became the test target. The 0.83 result shows that GLiNER learned to imitate its teacher on unseen comments. It does not tell us how often either model is right about the knives.

## What the 0.83 F1 actually measures

Vijeh is unusually clear about this limit. Nobody manually checked Gemini's labels. If Gemini mislabeled a steel, GLiNER earns credit for repeating that mistake and loses credit for correcting it. F1 still helps compare training runs, but its meaning is narrower than the usual headline reading of "83 percent accurate."

The pipeline itself is sensible. Gemini returned exact substrings rather than character offsets, and TypeScript found the positions in the original comment. Comments with broken spans were dropped and logged. Roughly 30 percent of the training examples contained no target entity, so the model also saw words such as "carbon" and "handle" without learning to tag every mention as a product.

That produced a useful specialist. Zero-shot GLiNER had scored about 0.65 F1 against Gemini's answers. Fine-tuning raised the result to 0.83. The labeling pass took 25 minutes, and Vijeh reports about $2.50 of GPU time across ten training runs. For a site that continuously processes new comments, turning a recurring inference bill into a small local model is a practical win.

But this is weak supervision: a machine created labels that another machine learned. [Research on NER with weakly labeled data](https://arxiv.org/abs/2106.08977) found that label noise can hurt model performance and used a smaller strongly labeled set to correct it. That is the missing comparison I want here. It need not replace Gemini's 4,290 comments. It needs to tell us where Gemini's version of the task differs from a knowledgeable person's.

## Five failed runs are part of the price

The cheap final model hides a lot of expensive attention. Five of the ten runs produced no usable model. Three failed on configuration. Two completed with a flat loss because a tensor named `words_mask` had been filled like a binary attention mask. GLiNER expected word indices instead, so the broken input effectively collapsed the comment into one enormous word.

Nothing crashed. Checkpoints appeared on schedule, gradients looked ordinary, and the evaluation score stayed near zero. That is the part of Vijeh's account I would carry into another fine-tuning job. A successful training process can still be consuming structurally wrong data.

The public GLiNER support trail shows similar edges. In [an issue about examples with no entities](https://github.com/urchade/GLiNER/issues/139), the maintainer explains that those examples still need an explicit list of entity types. One of Vijeh's early failures came from missing labels on negative examples. These are fixable problems, but they make the "about $11.50" compute-and-label total a poor estimate of the work required to reproduce the result.

I would add assertions before another training run: confirm that word indices rise as expected, decode sampled spans back to the source text, and inspect batches containing both entities and empty examples. The checks are boring, but cheaper than discovering after a full run that the tensor had the right shape and the wrong meaning.

## The human test can stay small

I would keep the Gemini-labeled validation set because it is useful for tuning and regression. Then I would add a separate human-checked set drawn from the ugly cases: punctuation-heavy steel names, ambiguous abbreviations, comments with no products, and terms that change meaning with context. A domain expert should settle disagreements without seeing either model's answer first.

That second set answers a different question. The Gemini set asks whether the local model preserves the behavior being replaced. The human set asks whether that behavior is useful. I need both because a system can preserve a recurring mistake and make it cheaper without becoming more accurate.

Reddit text deserves this extra check. In its own [engineering account of named-entity recognition](https://www.reddit.com/r/RedditEng/comments/1igvq6d/ner_you_ok/), Reddit describes jargon, humor, misspellings, and word forms as reasons not to rely entirely on a neural model. A knife community adds model numbers, steel codes, nicknames, and ordinary words that can become product references. That is exactly where a compact gold set can expose a teacher's blind spots.

> **Info:**
>
> My rule: use teacher labels to buy scale, then reserve human labels for the test you use to make accuracy claims. Do not let the teacher write every answer key.

## I would deploy the model with a narrower promise

Vijeh's model now powers New Knife Day, which tracks what knife enthusiasts discuss and buy. For that kind of exploratory product signal, local inference with periodic review looks reasonable. The site can process everything instead of skipping comments to control an API bill, and a bad tag is recoverable.

I would be more cautious if the extracted entities fed inventory, pricing, compliance, or an automated purchase decision. The same model can be useful in both settings, but the cost of an error changes what the evaluation must prove.

I came away convinced that a repetitive language task can move to a smaller local model. I just read the score narrowly: 0.83 agreement with Gemini after $9 of Gemini labels, five failed runs, and no human answer key. I would ship the local model, measure it against people, and keep those two results separate.