Climate Record Platform
An observational climate data platform on public NOAA daily station records: quality-controlled pipeline, dimensional warehouse, automated tests, and a fast multi-chart explorer for long-record stations across the United States — with transparent metric methods.
Problem & Outcome
Public daily weather archives are large, station-oriented, and quality-flagged. Turning them into trustworthy analytics needs a real pipeline — not a one-off spreadsheet — if the goal is reproducible heating/cooling metrics, extremes, and freeze-season summaries with honest methods.
Outcome: an end-to-end warehouse (raw files → cleaned daily data → quality control → star schema and analytic summaries), dbt models and tests, and this page’s live explorer for long-record stations nationwide — degree-days, extremes, freeze season, climate series, completeness, ranks, and a station map.
Live Explorer
Charts load station-level mart summaries from the warehouse — not millions of daily rows in the browser. Filter by state, then station and years.
Heating & cooling degree-days
Base 18 °C · quality-controlled temperatures only
Hot days & freeze days by year
Days with high ≥ 32 °C and low ≤ 0 °C in the selected range
Wet days ≥ 25.4 mm by year
Days with daily precipitation ≥ 1 inch (25.4 mm)
Annual max high & min low
Hottest daily maximum and coldest daily minimum (°C) each year
Max daily precipitation by year
Largest single-day total (mm) in each year
Precipitation
Monthly totals or yearly sums
Average max & min temperature
Average daily max and min · °C
How complete is the daily high-temp record?
For each year: usable daily maximum temperatures ÷ days in the year (after quality control). Low % means averages/extremes for that year are less trustworthy.
Freeze days & growing-season length
Freeze day = daily low ≤ 0 °C
Hottest / coldest / wettest years (selected station)
For the year range selected above
| Rank | Hottest years by max high °C |
Coldest years by min low °C |
Wettest years by max daily rain mm |
|---|---|---|---|
| Load a station to see ranks… | |||
Network snapshot — hot days (latest year available)
All long-record sample stations · one year · top 40 by hot-day count. Independent of the station dropdown above.
Station map
Hover for day count · click to select. Color follows the metric below.
Architecture
Python owns ingest, quality control, and metric methods. Gold data (Parquet) is the analytical core. dbt packages the star schema and summaries into DuckDB with tests. This page is static HTML with interactive charts from precomputed files — not a live link to the full warehouse.
flowchart TB A["NOAA GHCNd daily files"] --> B["Cleaned rows + QC"] B --> C["Star schema"] C --> D["Monthly and extremes summaries"] D --> E["dbt + DuckDB"] D --> F["JSON on this explorer"]
Hosting & Operations
Public site: this domain is served from Cloudflare Pages (static HTML). A home power cut does not take the brochure down.
Explorer JSON: station files live on Cloudflare R2
(data.dunleavyorganization.com), not in the Pages deploy.
Data product: the warehouse and weekly NOAA refresh stay in ClimateRecordPlatform on the Windows workstation. The public site only receives exported JSON.
Deploy: HTML via deploy/publish_pages.ps1; climate JSON via R2 upload.
The warehouse is not opened to the public internet.
Tech Stack
| Layer | Choice | Why |
|---|---|---|
| Source | NOAA GHCNd daily bulk | Public, documented daily station archive |
| Pipeline / QC | Python, Pandas, Parquet | Reproducible transforms; quality flags retained |
| SQL layer | dbt + DuckDB | Models and data tests |
| Analytics model | Star schema + marts | Flexible warehouse; fast product summaries |
| Web demo | Static HTML + Chart.js + Leaflet | Interactive charts without a browser database |
Methods
Loading summary…
- Scope: long-record USW/USC stations nationwide (all states + D.C.) with ≥50-year overlapping TMAX + TMIN + PRCP inventory coverage — the same rules used for the earlier SC/NC/GA regional sample.
- Quality control: missing values, NOAA quality flags, temperature range gates, and same-day high < low pairs are excluded from gold.
- Degree-days: daily mean ≈ (maximum + minimum) / 2; heating/cooling degree-days vs 18 °C; summed by month.
- Extremes: days with maximum ≥ 32 °C / 35 °C; minimum ≤ 0 °C; precipitation ≥ 25.4 mm.
- Freeze season: freeze day if minimum ≤ 0 °C; growing-season length between last spring freeze and first fall freeze when both exist.
Takeaways
End-to-end ownership of a data platform: land public source data, enforce quality rules, model a dimensional warehouse, test it, and serve a fast interactive explorer from summaries — with transparent climate methods.