Is it really Opus?

In August Anthropic said they will put invisible watermarks in text written by Claude, a hidden pattern that only their own tools can detect. In October OpenAI said the same for the EU. So the labs want to know which text came from their models. My first thought was, OK, can we do the opposite? Send some requests to an API, look at what comes back, and know which model is behind it.

A few days later I found a reseller selling claude-opus-5-5 and six GPT models for a cheap price. Cheap is nice, but is it really Opus? Nothing stops a seller from putting a cheaper model behind the expensive name. Also model gateways sometimes list “stealth” models with a nickname, and you have no idea who made them. So I wanted one tool for both: is this the model they say, and if nobody says, which model is it?

Disclaimer: I dictated this post and directed every test. AI did the grunt work: code, data and drafts.

Others tried this before

Yes, a lot. Papers like LLMmap and One Token Is Enough ask a model many small questions and compare the answers with models they know. There are also GitHub tools that check API resellers, and claude-relay-verifier checks Claude’s hidden reasoning, which Claude signs so only Anthropic can verify who wrote it.

There is an obvious problem here: siblings, models from the same lab like Opus and Sonnet. If a seller swaps Opus with Sonnet, both write almost the same answers. I tried the One Token method first: ask “name a random number between 1 and 100” and other one-word questions many times, then compare the mix of answers with real Opus and real Sonnet. The result is a score, 0 means the same mix and 1 means completely different. A common reading (from the ToSea checker) is: 0.25 or less is the same model, 0.35 or more is a different model, in between it’s unsure.

0.417 is above 0.35, so the reseller’s model is not Opus 5, fine. (One catch: this reseller drops the system prompt, which moves these scores a bit.) But real Opus 5.5 vs real Sonnet 5.5 scores 0.195, under 0.25, so this method calls two different models “the same”. If the seller swapped them, it can’t see it. GPT was even worse: GPT-6 Astra and GPT-6.1 Sol gave the same answer mix, reported the same knowledge cutoff, and gave identical answers to 20 out of 20 hard questions.

I can hear you thinking already, why not just ask the model who it is? Because a seller can put “You are Claude Opus” in a hidden prompt, and the answer is just more text. So no single test was enough. I put the existing ideas together in one tool, fpx, and added my own tests where they failed.

How the tool works

First I measured 40 models from 12 companies and saved the results in a file. To check an unknown model, the tool sends it 42 small requests and compares what comes back with that file. It doesn’t call the 40 models again, so one check costs about one cent for most models, up to 12 cents for the expensive ones.

The 42 requests check two things:

• the server: how many tokens some fixed texts cost, error messages, tool call IDs, how the reasoning comes back
• the model itself: 26 short questions at temperature 0 (temperature is the randomness setting, 0 means the model gives its most likely answer every time)

If that’s still not enough to separate two siblings, a second step sends 100 more requests: 10 one-word questions, 10 times each at temperature 1, the normal randomness.

The most important part is when the tool gives a name. There is always some model in the file that is “closest”, even for a model I never measured. A tool that always prints the closest name will be confidently wrong for every new model. So mine says “unknown” unless the winner beats every sibling in at least 2 separate kinds of evidence (for example the server and the answers), and changing any single reply wouldn’t give a different name.

Let’s see three real examples from my runs.

Example 1: count the tokens

Models don’t read letters, they read tokens, small pieces of text. Every lab cuts text into tokens in its own way, and the API bill tells you how many tokens your request was. The tool sends this Arabic greeting: السلام عليكم ورحمة الله وبركاته يا أصدقائي (“peace be upon you, and God’s mercy and blessings, my friends”), and checks how many more tokens it costs than the word “ok”:

One short sentence already puts the labs in groups. GPT is alone at 8, Claude alone at 25. You can even see xAI changed its tokenizer between Grok versions, 19 down to 13, and Mistral Large uses a different one from its smaller siblings. But some labs tie (DeepSeek, GLM, Llama and Muse all say 16), so the tool sends eight different texts, not one. And most siblings, like the five Claude models, give the same number, so this can’t tell Opus from Sonnet.

Example 2: repeat after me

For siblings the tool has to ask the model itself. One of the 26 questions is “Repeat exactly, nothing else:” plus some weird Unicode text, with two invisible right-to-left characters inside:

Same lab, same tokenizer, same question. Grok 4.6 copies it, Grok 4.7 refuses, every single time. Other questions have one right answer, like a multiplication or a word written backwards. Smaller models, like Gemini 2.5 Flash-Lite next to Gemini 2.5 Flash, get some of them wrong, and always the same way. The tool also reads how long a model thinks before it answers, because the API reports the hidden reasoning tokens. GPT-6 Sol and GPT-5.6 Sol use 0 of them before they give the square root of 2, GPT-6 Luna uses 57 to 82.

Example 3: ask again, and again

Some siblings give the same answer to everything at temperature 0. Well, ask them many times at temperature 1 and count the answers. This chart is from my saved data, “name a random animal” asked 24 times to each model, over three different runs:

GPT-5.6 Terra and GPT-5.6 Sol look the same on every server check. Ask once and both can say “axolotl”. Ask many times and Terra says it every time, Sol only 3 times out of 24.

This is the second step from above, the One Token idea again, with their 10 questions. In a real check the tool asks each one 10 times, compares the mix with the saved mixes, and counts it as evidence only if the mix is clearly closer to one sibling than to the others. Before using it, I tested this on my saved runs, each time leaving out the run being checked: in 276 checks it never pointed to the wrong model. In 48 of them, about 1 in 6, the difference was too small and it gave no answer, which is fine, “unknown” is better than wrong.

My own test: read the bill

The examples above recognise a model. But for “did they swap my model”, the best test I found doesn’t look at the answers at all. It looks at the bill.

Providers cache long prompts: the second time you send the same long prompt it is cheaper, and the bill shows how many tokens came from the cache. This cache is built by one model and only that model can use it, Sonnet can’t reuse what Opus computed. So the test is:

• make a new long prompt that nobody sent before
• send it with the name you paid for, this fills the cache
• send it again with the same name, the bill must show cached tokens
• send it with the cheaper sibling’s name
• do it all again with another new prompt

If the cheaper name also gets cached tokens, it’s using the same cache as the expensive name, so most probably it’s the same model behind both names. Six requests per sibling, few cents. The picture shows one round with real numbers. The prompt is the same text for every model, GPT counts it as about 2,300 tokens and Claude as about 3,900, because Claude cuts text into more tokens (Example 1 again).

To test the test, I wrote a small proxy that lies on purpose: it sends the request to the cheaper model but keeps the expensive name. In the last round it caught all 5 swaps, even Astra sold as 6.1 Sol, the pair nothing else could separate, and it cleared all 5 honest pairs. Over 16 honest runs there was no false alarm. That’s a small test, but a clear result. Two honest limits: it shows two names share one server, not that they are the same model, and a seller who uses a different account for each name would not share a cache, so the swap would pass. The closest work I found, CacheTracer, uses a shared cache to see which resellers buy from whom, and it assumes different models never share a cache. I used that same idea to catch swaps. I didn’t find anyone doing it this way, but maybe I missed it.

The weak point: a dishonest reseller can just set the cache numbers to zero. I tried timing instead (is a cached 20,000-token prompt faster?), but the difference was smaller than normal waiting in the queue. So if the cache is hidden, this test gives no answer.

Off to the market

All of that was on my own proxy and one big gateway. So I went shopping: a marketplace with dozens of sellers, some selling “Opus 5.5” for about 4% of the official price. I picked 8 sellers that sell both Opus 5.5 and Sonnet 5.5. Only 3 answered, the rest were offline, refused, or wanted a bigger deposit.

Then something exciting: one seller, I’ll call it S2, showed a shared cache on both pairs, Opus with Sonnet and GPT-6.1 Sol with GPT-6 Luna. A swap, caught in the wild! Well, not so fast. Before I believed it I sent S2 a few brand new prompts, and they already came back with cached tokens. A tiny one-line prompt to its “GPT-6 Luna” was billed as 9,061 tokens, 8,665 of them cached. So S2 adds its own long hidden text to every request, and that text is always cached. My test only asks “any cached tokens under the cheaper name?”, so it saw a swap that wasn’t there. I took the result back.

This is not rare. Anyone reselling a Claude Code or Codex subscription has to send the official app’s long instructions with every request, and many relays add their own system prompt. So if you try the cache test on a seller, first send a brand new prompt under the cheaper name. If that already shows cached tokens, the cache test can’t help you there.

But S2 had one more surprise, and it came from Example 1, not the cache. The same long text cost this many tokens:

S2’s “Sonnet 5.5” doesn’t count text like Claude at all, it counts like GPT. So either a GPT-type model is behind that name, or the bill is made up. I can’t tell which model it is, token counts can be faked, but it’s not what the label says. Amazing, the simplest check in this whole post found the only real problem.

Does it work? Let’s put this to the test

A test here means: run the tool against all 40 models, twice, at different times (two sessions). Before each test I locked the tool and the saved file, and wrote down what counts as a pass: at least 32 of the 40 models named correctly in both sessions, and no wrong name. Each test used new runs only, never data the tool was built on.

The first test named 25. With example 3 added it named 34. Before I believed it, I asked a separate AI agent to attack the tool, and it found two real problems. Some names came from the gateway repeating the model name I asked for, that’s not a fingerprint at all. And a provider that changes its replies on purpose can make Haiku look like Opus. I removed the name repeating from the error texts. I didn’t try to make the tool cheat-proof, that dropped the named models from 34 to 26, and this tool is for honest endpoints anyway. The next test failed at 21 because of a bug in my new error reader, a short and a long copy of the same error looked different. Fixed it, locked again, ran a new test:

65 of 80 is the 32 models twice, plus GLM-5.3 Flash, named in one session only. The other sessions came back “unknown”, never a wrong name.

32 of 40 and no wrong name. To be fair, three of the 32 (MiniMax M2.5, MiniMax M2.7 and GLM-5.3 FlashX) were named only from an error message written by the gateway, not by the model’s server. Without those gateway messages it’s 29 of 40, under my own pass line, still with no wrong name. And it took four tries to get one pass, so some luck is in there too. Also 0 mistakes doesn’t prove it never makes mistakes. With 33 named models the math (95% confidence) can only say the real mistake rate is below about 9%. Good, not perfect.

Eight models were not named in both sessions. GPT-6 Astra and GPT-6.1 Sol gave the exact same answers in every test I tried, but they sit in separate caches and OpenAI lists slightly different settings for them, so I can’t say if it’s one model or two. GPT-6 Luna and 6 Sol, MiMo 2.5 and 2.6 Flash, and GLM-5.3 and 5.3 Flash are a little different, but not enough for the rule.

What didn’t work: counting reasoning tokens alone (the host’s billing changed them within hours), harder questions (all siblings got them right) and asking “what’s the newest software version you know?” (reasoning models gave a different answer every time, even at temperature 0).

So, was it Opus?

Yes, most probably. Here is why. Claude’s API has a thinking setting, how the model thinks before it answers. Some values work only on some models, the rest get an error before the model even runs. Anthropic’s docs list these rules, I just used them as a test:

The reseller’s claude-opus-5-5 behaved like Opus 5.5 in every check:

• it gave the same errors as Opus 5.5 in the table, and the same error text
• Claude signs its hidden reasoning, and when you send that reasoning back, each model keeps blocks only from certain models. Real Opus 5.5 keeps Sonnet 5.5’s blocks, real Sonnet 5.5 drops Opus 5.5’s. The reseller’s “Opus” kept them, like Opus does
• one error message came straight from Anthropic’s side, and it named claude-opus-5-5
• it doesn’t share a cache with the same reseller’s Sonnet name

No check alone is proof, a careful seller could fake some of them. All of them agreeing makes me about 97% sure. That’s my own gut number, not a calculation, and part of it came from a speed check I later dropped.

For the six GPT names, each one’s answer mix (the One Token score) was closest to the same model bought direct. That’s weaker evidence. That route doesn’t show caching, so the cache test couldn’t run there.

What it can’t do, AKA the fine print

• If a reseller detects my test requests, sends only them to the real model and everything else to a cheap one, all of this fails. None of my tests can catch that.
• A provider that changes the replies on purpose can fool it.
• I tested the naming only through one gateway. Through another gateway or reseller it may do worse.
• If a provider turns thinking on for Sonnet 5.5, the tool reads it as Opus 5.5. For that pair, trust the error table above, not the name.
• It only knows the 40 models I measured. Models from NVIDIA, Tencent, Cohere, ByteDance and Amazon get “unknown” until I add them.
• Providers change their software, so the saved file needs to be measured again from time to time.

Final thoughts

Token counts alone were never enough, and Token Counts Are Not Model Lineage found the same. The server, the model’s own habits and the bill together got it to 32 of 40 (29 if you’re strict), on honest endpoints, with no wrong name :)

If you buy from a cheap reseller, give the cache test a try. It’s six requests and a few cents, and the bill can tell you more than the answers.

The tool and the full technical report are on GitHub.