How Accurate Is This AI Detector?

How we know the human samples are human

Every sample was published between 2010 and 2019, i.e. before any general-purpose LLM was publicly available. Human authorship here is a date fact, not a stylistic judgement. Each sample records how the date was verified, preferring identifiers that cannot be back-dated: DOIs (checked against Crossref), PMC ids and Europe PMC firstPublicationDate, Federal Register document numbers, MediaWiki revision ids, git tags, and Stack Exchange server-assigned creation timestamps.

On the non-native English samples

READ THIS BEFORE PUBLISHING ANY ESL NUMBER. The five "esl-nonnative" samples are NOT learner-corpus essays. No learner corpus with a license compatible with commercial use could be sourced: PELIC is CC BY-NC-SA (NC excludes us), the Cambridge Learner Corpus FCE set, ICLE, ICNALE, NUCLE and the BEA-2019 W&I+LOCNESS data are all behind restrictive or registration-gated licenses, and the ETS TOEFL11 corpus is a paid LDC product. WHO and FAO publications were also rejected as CC BY-NC-SA. Rather than substitute something unlicensed, these five are CC BY / CC BY-SA scholarly prose written by authors whose institutional affiliations are all in non-anglophone countries (Indonesia, Turkey, Iran, China), selected because visible L2-transfer features survived peer review uncorrected. That is evidence of non-native authorship, not proof of any individual author's first language, and the register is academic rather than the undergraduate essay that produces our worst false positives. Treat any FPR computed on this slice as a floor, not as the ESL-essay false-positive rate.

Which models the AI samples came from

Read this before quoting any FNR computed from this file. WHICH MODELS ARE ACTUALLY IN HERE (24 samples): - claude-opus-5 (Anthropic), generated 2026-08-26 for this benchmark: 18 samples. - OpenAI gpt-4o-mini, as the engine of the Codaone production humanizer, rewriting claude-opus-5 text: 3 samples. Mixed lineage — Claude wrote it, GPT rewrote it. - OpenAI ChatGPT, December 2022 web release (gpt-3.5 era), via the HC3 dataset: 3 samples. WHICH ARE NOT: Google Gemini. Meta Llama. Mistral. DeepSeek. Qwen. xAI Grok. Cohere. Every current-generation OpenAI model (GPT-4o, GPT-5 family) except gpt-4o-mini in its humanizer role. Every current-generation Claude except opus-5. Nothing from a consumer "undetectable AI" service (StealthGPT, Undetectable.ai, Phrasly) — those are the strongest evasion tools in the wild and we have zero samples of their output. THEREFORE: an FNR computed over this corpus is an FNR *for Claude Opus 5 output, for our own humanizer's output, and for three-year-old ChatGPT output*. It is not an estimate of how often we miss AI text in general, and it must not be published as one. If the number is quoted on /ai-detector/accuracy, this sentence has to travel with it. A SECOND BIAS, LESS OBVIOUS: 18 of the samples were written by the same model that assembled this corpus, while it knew it was building a detector benchmark. That is a demand-characteristic risk in both directions — the model may have written more stereotypically "AI" prose for the default-register samples, and may have tried harder than a naive user would on the evasion samples. The prompts are recorded verbatim so a third party can rerun them on any model and check. Doing that on a non-Claude model is the cheapest available fix for both biases. THIRD, THE REGISTER MIX IS NOT A USAGE DISTRIBUTION: {"default-assistant":7,"instructed-casual":5,"instructed-varied-rhythm":5,"humanizer-output":3,"domain-specific":4}. It is deliberately weighted toward evasion (13 of 24 samples are instructed-casual, instructed-varied-rhythm, or humanizer output) because §1.4 of AI-DETECTOR-COMPETITIVE-2026-08-25 measured that all of our false negatives come from disguised AI. A corpus weighted this way will report a WORSE FNR than a corpus of default-register text would. That is intentional and honest; it is not comparable to a competitor's headline accuracy number, which is measured on whatever mix flatters them. FOURTH, ON THE HUMANIZER SAMPLES: the free anonymous quota is 3/day/IP and all three were used on 2026-08-26, so there are exactly three and no reruns. They are the highest-value rows in the file — a false negative on any of them means our own paid product defeats our own free product, which is the central question the accuracy page exists to answer honestly.

On how few voices some registers have

Registers are unevenly hard to source under an open license. Government and technical-documentation samples come from a small number of institutional voices (three Federal Register rules, two Kubernetes docs pages, three English Wikipedia revisions), so within-register variance understates the real world.

Why it fails: Statistical detection reads simpler vocabulary and regular sentence rhythm as machine-like. This is the best-documented bias in the entire detector category, and it lands on the people with the most at stake. On our internal adversarial set, a real ESL student essay scored 42.

What we do: Mid-range scores get a hedged verdict ("Mixed or AI-assisted", "Possibly AI-generated") and low confidence — never a flat accusation. If you write in English as a second language and got flagged, that context outweighs our number.

Why it fails: Connectors like "Furthermore" and "In conclusion" are also the strongest LLM tells, so a regulation, a contract, or a press release can out-score actual AI text on the pattern model. On our internal set, a human-written regulatory passage scored 68 and a 2019 press release scored 62.

What we do: In this band the verdict language stays at "Possibly" with low confidence, and the per-sentence view shows exactly which phrases drove the score — so a human can judge whether they came from a lawyer or a language model.

Why it fails: Good documentation is deliberately uniform: same sentence shape, same structure, restrained vocabulary. A variance-based model reads that discipline as generation. A human-written scheduler doc scored 50 ("Mixed") on our internal set.

What we do: Uniform prose caps out at hedged verdicts with low confidence. If you are checking documentation, weigh the per-sentence breakdown over the headline score.

Why it fails: We used to over-weight punctuation habits as an AI tell: an em-dash-heavy human essay scored 80 on an earlier version of the scoring. Plenty of humans write like that; so do LLMs.

What we do: We downgraded punctuation from primary evidence to corroboration. The same essay now scores 13 on the benchmark below, and it stays in the corpus permanently so a regression here fails the benchmark before it ships.

Why it fails: Prompt an LLM to write casually — or run its output through a paraphraser — and the formal tells our pattern model keys on are stripped away. Both disguised-AI samples in the benchmark below slipped through (scored 45 and 9). This is the main way to beat us, and every other statistical detector.

What we do: Mostly, we cannot fix this within our approach, and we say so. Treat a low score on text you already suspect as weak evidence of anything. Uneven per-sentence scores on a polished piece are sometimes the residue of partial rewriting — a lead worth following, not a verdict.

Why it fails: Statistics computed over a handful of sentences are close to noise. Sentence-length variance, vocabulary richness, and starter diversity all need material to measure.

What we do: Under 30 words we refuse to give a verdict at all. Just above that line, read the score as a hint, not a finding.

Why it fails: The scoring is built for English. Other Latin-script languages get degraded analysis; Chinese, Japanese, and Korean text does not get meaningful statistical analysis at all.

What we do: Do not use this tool on non-English text. A score produced there is not evidence of anything, and we would rather tell you that here than let a number imply otherwise.