How many days ahead is a weather forecast better than the guess you could make without it?
Every weather app shows a number and never shows whether that number was worth printing. This scores 7,358,400 archived forecasts against a competitor that needs no model at all, across thirty Turkish cities and two years.
For rain the competitor is the sentence "it will not rain" — true most hours, because rain is rare. For temperature it is "tomorrow will be like today." Both are free. The trust horizon is the lead time at which the forecast stops beating them.
| 8 cities | where the hourly rain forecast never beats "it will not rain", at any lead from 1 to 7 days |
| 5.74 days | rain trust horizon in Rize, the widest measured |
| 2 cities | where temperature never beats yesterday: Antalya and Mersin |
| 7.0 days | in Trabzon, still winning at the last lead the archive carries |
The eight are Ankara, Konya, Kayseri, Eskişehir, Denizli, Isparta, Kars and Van — all inland and dry. In every one of them measured skill is negative at all seven leads. In Isparta and Kars even the bootstrap interval leaves no room for a positive horizon; in the other six it reaches about a day, so the honest claim is "measured negative at every lead", not "provably never".
Ankara makes the point twice: its temperature horizon is 5.92 days and its rain horizon is zero. Same city, same models, same two years — the answer depends on which question you ask.
The same care applies to the temperature zeros. Antalya's interval is [0.00, 0.00]; Mersin's
reaches 2.33 days, so Antalya is the stronger of the two claims.
It rains in Ankara about one hour in twenty, so "it will not rain" is right roughly 95% of the time before anyone looks at anything. To clear that bar a forecast has to catch enough real rain to pay for the hours it cries wolf. In Rize, where it rains far more often, the naive rule is weak and the forecast is worth reading almost six days ahead.
The result is mostly not about model quality. It is about how good the free guess already is where you live.
Python fetch + shape network-bound, nothing else matters
Rust skill + bootstrap 30 cities x 2 variables x 7 leads x 4,000 resamples
Python analysis + charts
Skill is 1 - MAE_forecast / MAE_reference. It is positive while the forecast adds value and
crosses zero when it stops; the crossing is interpolated between whole days. A city still winning
at day 7 is reported as 7 rather than extrapolated.
The resampling detail changes the answer. Consecutive hourly errors are strongly correlated — a warm afternoon is followed by a warm evening — so drawing hours independently would report a precision the data does not have. Whole days are resampled as blocks instead.
Before any scoring runs, the assembled table passes a data contract: column types, physical ranges, and exactly one row per city-hour-variable-lead. A silent ingestion bug produces numbers that look entirely reasonable, which is what this kind of study dies of.
No API key, no signup. The archive is public.
pip install pyarrow proofframe
cargo build --release
cp target/release/fe_metrics.dll python/fe_metrics.pyd # .so on Linux/macOS
PYTHONPATH=python python -m forecastedge.fetch # ~40 min, cached to data/raw
PYTHONPATH=python python -m forecastedge.build # 7.36M pairs -> parquet
PYTHONPATH=python python -m forecastedge.validate # the contract gate
PYTHONPATH=python python -m forecastedge.analyse # horizons + intervals
PYTHONPATH=python python -m forecastedge.banner # redraws the banner from the resultsDisagree with the baseline? Change it. The argument lives in one function in
analyse.py, and a different reference forecast gives a different
and equally legitimate answer. That is the useful thing about scoring against a competitor — you can
always name a better one.
- The truth is a model. ERA5 reanalysis, not rain gauges. ERA5 reports noticeably more precipitation events than gauges do, with a wet, high-false-alarm bias. That inflates the count of rainy hours, which makes the free guess look worse and the forecast's job easier — and the forecast still loses. The bias runs against the conclusion, so the result is likely conservative.
- The scorer and the scored share ancestry. ERA5 is ECMWF's reanalysis and much of the forecast data is ECMWF too. Where a model is graded partly against itself, skill is flattered. Again: flattered, and still negative.
- A hard metric. Binary hourly rain against a majority-class baseline is a known-difficult scoring setup. It is the right question for "should I have checked the app at 3pm?" and the wrong one for "is this model any good?"
- One grid point per city. A grid cell is not a city, and rain is local.
| City | Rain horizon | 95% CI | Temp horizon | 95% CI |
|---|---|---|---|---|
| Ankara | 0.00 | 0.00 – 1.00 | 5.92 | 5.58 – 6.26 |
| Denizli | 0.00 | 0.00 – 1.29 | 4.27 | 4.11 – 4.44 |
| Eskisehir | 0.00 | 0.00 – 1.32 | 6.25 | 5.88 – 6.64 |
| Isparta | 0.00 | 0.00 – 0.00 | 4.23 | 4.11 – 4.34 |
| Kars | 0.00 | 0.00 – 0.00 | 4.35 | 3.74 – 4.91 |
| Kayseri | 0.00 | 0.00 – 1.21 | 4.97 | 4.62 – 5.35 |
| Konya | 0.00 | 0.00 – 1.37 | 6.13 | 5.88 – 6.31 |
| Van | 0.00 | 0.00 – 1.41 | 4.15 | 3.88 – 4.32 |
| Sanliurfa | 1.18 | 0.00 – 2.47 | 5.99 | 5.55 – 6.21 |
| Erzurum | 1.39 | 0.00 – 2.25 | 3.97 | 3.32 – 5.43 |
| Diyarbakir | 1.72 | 0.00 – 2.84 | 5.02 | 4.51 – 5.52 |
| Mersin | 1.75 | 0.00 – 3.71 | 0.00 | 0.00 – 2.33 |
| Sivas | 1.81 | 0.00 – 2.91 | 5.55 | 5.15 – 5.97 |
| Balikesir | 2.03 | 1.30 – 2.70 | 5.50 | 5.08 – 5.94 |
| Canakkale | 2.22 | 1.66 – 2.89 | 5.68 | 5.08 – 7.00 |
| Antalya | 2.28 | 0.00 – 4.18 | 0.00 | 0.00 – 0.00 |
| Tokat | 2.63 | 1.52 – 3.40 | 5.77 | 5.34 – 6.31 |
| Gaziantep | 2.64 | 1.44 – 3.71 | 6.17 | 5.80 – 6.51 |
| Aydin | 2.71 | 1.84 – 3.42 | 4.16 | 4.04 – 4.28 |
| Malatya | 2.71 | 1.80 – 4.09 | 4.80 | 4.56 – 5.09 |
| Istanbul | 2.82 | 1.78 – 3.59 | 5.51 | 4.99 – 6.13 |
| Izmir | 2.86 | 1.72 – 4.11 | 5.73 | 5.39 – 6.50 |
| Samsun | 3.18 | 1.96 – 3.57 | 4.72 | 4.51 – 4.96 |
| Adana | 3.36 | 1.45 – 4.06 | 4.97 | 4.69 – 5.40 |
| Bursa | 3.43 | 2.89 – 3.99 | 5.98 | 5.62 – 6.55 |
| Zonguldak | 4.40 | 3.90 – 4.90 | 4.84 | 4.69 – 5.01 |
| Trabzon | 4.45 | 3.95 – 4.88 | 7.00 | 5.75 – 7.00 |
| Ordu | 4.59 | 4.14 – 5.06 | 4.90 | 4.74 – 5.19 |
| Mugla | 5.61 | 4.55 – 6.55 | 4.50 | 4.35 – 4.66 |
| Rize | 5.74 | 5.11 – 6.71 | 4.17 | 4.06 – 4.28 |
Sources: Open-Meteo Previous Runs API for the fixed lead-time forecast archive, and the ERA5 reanalysis archive for observations. Rain counted at
0.1 mm per hour.
Apache-2.0