Across every shared forecast we have settled, the answers that declared under 60% confidence landed inside their stated range 22 times out of 29. The answers that declared 60% or more landed 11 times out of 21.
76% against 52%. The confident half was wrong roughly twice as often.
That is fifty settled answers across ten questions — a starting point, not a finding, and we will keep saying so. But the mechanism behind it is not subtle, and it shows up in every question we have asked.
SOXL 4 hours Prediction @ 2026-08-24 12:28 UTC
Each model has a confidence setting, and it barely moves
Eighteen questions, all five models on every one, five assets and four horizons. Here is what each declared:
| Model | Stated confidence | Its whole range, 18 forecasts |
|---|---|---|
| ChatGPT — GPT-5 | 70% ± 4.3 | 63–78% |
| Gemini — Pro | 64% ± 6.7 | 50–80% |
| DeepSeek | 52% ± 6.8 | 40–65% |
| Claude — Opus | 51% ± 6.4 | 40–64% |
| Grok | 39% ± 7.1 | 28–53% |
The five averages are spread over 10.8 points. Each model's own confidence, across eighteen different questions about different assets over different horizons, moves by 6.3 points. The number varies almost twice as much between models as it does between forecasts.
GPT-5 was the most confident of the five on sixteen of the eighteen questions. Grok was the least confident on seventeen of them. Whatever that figure measures, it is mostly measuring which model you asked.
There is a cleaner demonstration. We asked the same FAS 30-day question twice, eight minutes apart. Three of the five returned a byte-identical range the second time — and moved their confidence anyway:
| Model | First ask | Second ask, 8 minutes later |
|---|---|---|
| ChatGPT — GPT-5 | $145–215, 66% | $145–215, 71% |
| Claude — Opus | $162–194, 58% | $162–194, 62% |
| DeepSeek | $155–210, 55% | $155–210, 55% |
Same question, same answer to the dollar, different confidence. Whatever moved between those two runs, it was not an opinion about FAS.
And the number points the wrong way
A confidence figure attached to a range ought to mean something like "the probability the close lands inside it". If it does, then for the same asset over the same window, a wider range must carry a higher probability — there is simply more room to land in.
The models do the reverse. Within a single question, the more confident model draws the narrower range: correlation −0.47 across the settled answers. Gemini's ranges are the tightest of the five at 78% of the group average width, and it is the second most confident. Grok is the least confident and draws ranges a third wider than Gemini's.
So confidence is not being traded against precision. It is a second, independent way of sounding sure.
Width, meanwhile, does exactly what you would expect it to:
| Answers that were… | Closed in range |
|---|---|
| wider than their group's average | 19 of 23 — 83% |
| narrower than their group's average | 14 of 27 — 52% |
Put the two together and the outcome table stops being a surprise. Ordered by how sure each model said it was:
| Model | Said | Range width vs the five | Closed in range |
|---|---|---|---|
| ChatGPT — GPT-5 | 70% | 96% | 6 of 10 |
| Gemini — Pro | 64% | 78% | 5 of 10 |
| DeepSeek | 52% | 115% | 7 of 10 |
| Claude — Opus | 51% | 106% | 8 of 10 |
| Grok | 39% | 106% | 7 of 10 |
The two most confident models are the two least accurate. Not because confidence causes error — because the models that sound most certain also commit to the tightest ranges, and tight ranges get missed.
None of which makes Claude or Grok the better forecaster. A wide range is an easier target, and "I hit a wide band" is a weaker claim than "I hit a narrow one". That is exactly why our scoring counts width against the score rather than counting hits: a 24% band around a leveraged ETF has not said very much, however often it contains the close.
What consensus does not buy you
Twice now, every model has missed the same question in the same direction. On 24 August we asked all five about SOXL over four hours. It closed at $111.82:
| Model | Range | Result |
|---|---|---|
| ChatGPT — GPT-5 | $116.20 – $125.80 | missed |
| Gemini — Pro | $115.20 – $123.80 | missed |
| DeepSeek | $113.50 – $122.50 | missed |
| Claude — Opus | $115.20 – $124.30 | missed |
| Grok | $114.20 – $122.80 | missed |
Three days later, the same question produced the same outcome: five ranges, five floors above the $113.51 close, five misses. The confident models and the cautious ones failed together, in the same direction, on the same day.
This is the part worth carrying away. Asking five models is not five opinions; it is one opinion in five accents, because they read the same pages and inherit the same recent-price anchor. Agreement between them tells you they agree. It does not tell you they are right.
What to actually do with the number
- Read the width, not the confidence. The width is a claim you can check. The confidence is a house style.
- Compare a model against itself, never against another model. A 64% from GPT-5 and a 64% from Grok are not the same statement — one is that model at its most hesitant in eighteen tries, the other is the most certain it has ever been.
- Treat unanimity as a warning, not a confirmation. When five models agree, go and find what they are all anchored to.
Every figure above was computed from shared forecasts you can open and read in full — the range, the confidence, the reasoning and the settlement are all on the record.
👉 How a forecast is scored, in full
👉 Read the shared forecasts these numbers came from
Ten settled questions is a starting point, not a finding, and we will keep saying so until the sample says otherwise. Content on ForecastNest Insights is for research, educational and informational purposes only. Nothing here is financial, investment, legal or tax advice, and nothing here is a recommendation to buy or sell anything. We publish how models perform on a disclosed task; we do not publish price targets or trading signals.