Of the 59 settled forecasts we can read, 27 do not say which way the asset is going.
Not "they were vague". The midpoint of the stated range sits within half a percent of the price when the question was asked — the forecast is a statement about how far the asset might travel, not about where it ends up. On SPY it is worse than that: 13 of 14 SPY forecasts express no direction at all.
Which leaves 32 answers that actually take a side. This post is about those, and about how little we can yet conclude from them.
How direction is measured here
From the midpoint of the range, against the price when asked — both printed on every forecast page, so you can redo any number below by hand.
Our prompt does ask each model for an explicit BULLISH/BEARISH signal, and we store it. We are not using it, because it is not shown anywhere a reader could check it, and a claim you cannot audit is one we would rather not make. Midpoint above the starting price is an up call, below is a down call, and within ±0.5% is no call.
The headline, and why it is not good news
Across those 32 calls, 21 were right. 66%. Against a coin that would be encouraging.
The trouble is that the market did not behave like a coin over these twelve questions. It fell on eight of them. So a rule that says "down" every single time, reads nothing, and costs nothing, scores 56%.
Ten points of edge over a rule with no brain in it. On 32 calls, that is four extra correct answers. It is not nothing, and it is nowhere near enough to conclude anything.
And it gets worse when you split the questions by how much the asset actually moved:
| Calls | The five models | "Always down" | "Always up" | |
|---|---|---|---|---|
| Moves over 2% | 17 | 12 — 71% | 17 — 100% | 0 — 0% |
| Moves under 2% | 15 | 9 — 60% | 1 — 7% | 14 — 93% |
| All | 32 | 21 — 66% | 18 — 56% | 14 — 44% |
Every large move in this sample was downward, and every trivial one was, on balance, upward. So on each half of the data, a one-line rule with no information in it beat all five models — 100% against 71%, then 93% against 60%. The models only come out ahead overall because they sit between two baselines that each win on their own half.
That is what a twelve-question sample looks like when you interrogate it honestly. We are not claiming the models have no directional skill. We are saying this data cannot demonstrate that they do.
By asset
| Asset | Questions | Took a side | Right |
|---|---|---|---|
| SOXL | 5 | 19 of 25 | 12 — 63% |
| IONQ | 2 | 6 of 10 | 3 — 50% |
| MSTR | 1 | 5 of 5 | 4 — 80% |
| ETH | 1 | 1 of 5 | 1 |
| SPY | 3 | 1 of 14 | 1 |
Only SOXL has enough behind it to be worth a number, and 63% there is a shade above the 56% a permabear would have managed on the same questions.
The SPY row is the interesting one, and it is not about accuracy. Three SPY questions, fourteen model-answers, and one of them took a directional position. The models will happily write two thousand characters about the S&P over the next day and then place their range symmetrically around today's close. That is a defensible answer — a day of SPY is mostly noise — but it is worth seeing plainly, because it is not what most people think they are getting when they ask an AI where the market is going.
MSTR is the opposite: a single 7-day question where all five took a side and four were right. One question is an anecdote.
By model
| Model | Took a side | Right | Up calls | Down calls |
|---|---|---|---|---|
| Gemini — Pro | 5 of 12 | 4 — 80% | 1 | 4 |
| Grok | 7 of 12 | 5 — 71% | 3 | 4 |
| DeepSeek | 9 of 13 | 6 — 67% | 6 | 3 |
| ChatGPT — GPT-5 | 5 of 11 | 3 — 60% | 3 | 2 |
| Claude — Opus | 6 of 11 | 3 — 50% | 0 | 6 |
Five to nine calls each. These are not rankings and should not be read as any; the gap between 80% and 50% here is three or four answers.
One column does say something, though. Claude — Opus called "down" on every single directional forecast it made — six for six, no up calls in eleven forecasts. It was right on all three occasions the asset fell and wrong on all three occasions it rose:
| Forecast | Called | Market did | |
|---|---|---|---|
| SOXL 4h | down | −7.27% | right |
| SOXL 4h | down | −7.75% | right |
| SOXL 4h | down | −5.08% | right |
| MSTR 7d | down | +1.82% | wrong |
| IONQ 1d | down | +0.26% | wrong |
| SOXL 1d | down | +0.39% | wrong |
That is not a forecasting record. That is the "always down" rule from the table above, with an 8,000-character rationale attached. In a period that mostly fell, it scored 50% — and it would have scored 50% with no reasoning at all.
DeepSeek is the mirror image: six up calls against three down, in a market that fell two questions out of three. It is the only model of the five that leaned against the tape, and it finished at 67%.
Does it help when they all agree?
The intuitive answer is yes: if five models independently reach the same conclusion, that conclusion is probably right. It is the reason people ask several models in the first place.
The data does not support it. Taking the six questions where at least three models committed to a side, and splitting them by whether those models agreed:
| Questions | Individual calls | Right | |
|---|---|---|---|
| All agreed | 3 | 13 | 9 — 69% |
| They disagreed | 3 | 13 | 9 — 69% |
Identical. Being part of a unanimous group made a model's call no more likely to be right than being part of a divided one.
What unanimity does change is the shape of the outcome:
| Question | Consensus | Result |
|---|---|---|
| SOXL 4h, fell 7.27% | unanimous — down | 4 of 4 right |
| SOXL 4h, fell 7.75% | unanimous — down | 5 of 5 right |
| SOXL 1d, rose 0.39% | unanimous — down | 0 of 4 right |
| MSTR 7d, rose 1.82% | split 4–1 | 4 of 5 right |
| SOXL 4h, fell 5.08% | split 2–2 | 2 of 4 right |
| IONQ 1d, rose 0.26% | split 3–1 | 3 of 4 right |
When they agreed, the result was total: two clean sweeps and one complete wipeout. When they disagreed, every question came back mixed — some models right, some wrong, never nothing.
So consensus does not raise your hit rate. It removes your diversification. Five agreeing models are not five confirmations; they are one position held five times, and when that position is wrong you own the whole error. The SOXL question that rose 0.39% is the case in point — all four models that took a view said down, and the asset closed higher.
The smaller IONQ question makes the same point from the other side: only two models committed, both said up, and it fell 3.77%. Two for two, wrong.
Three unanimous questions is not enough to settle this and we are not pretending otherwise. But the mechanism is not mysterious, and it is the same one visible in the correlated-failure section of our confidence post: these models read the same pages, inherit the same recent-price anchor, and are trained on overlapping corpora. Their errors are correlated by construction. Agreement between them is evidence that the inputs agreed, not that the answer is right.
If anything, the practical reading inverts the intuition. A split verdict is the more informative outcome — it tells you the question is genuinely uncertain, and it guarantees some of your models are on the right side. Unanimity tells you the models found the same story, and leaves you no hedge if the story is wrong.
What we would need before saying anything firmer
- Questions where the market went up. Eight of twelve fell, and every large move fell. Until that balances out, any directional number here is partly a measurement of August.
- More assets like MSTR and ETH. One question each is an anecdote, and the two liquid-index questions produce almost no directional calls to score.
- Thirty or so calls per model, rather than five to nine. At this size a single answer moves a model's percentage by more than ten points.
Until then, the honest summary is short. These models rarely take a directional position at all — 27 of 59 forecasts decline to, and 13 of 14 on SPY. When they do take one they are right about two-thirds of the time, which is better than a coin and not clearly better than a rule you could write on a napkin.
Every figure above comes from shared forecasts you can open, and every direction was computed from the two numbers printed on each page: the price when asked, and the range.
👉 How a forecast is scored, in full
👉 Read the shared forecasts these numbers came from
Twelve settled questions and 32 directional calls is a starting point, not a finding, and we will keep saying so until the sample says otherwise. Content on ForecastNest Insights is for research, educational and informational purposes only. Nothing here is financial, investment, legal or tax advice, and nothing here is a recommendation to buy or sell anything. We publish how models perform on a disclosed task; we do not publish price targets or trading signals.