Critical Materials Atlas
Transparency · project map

How this atlas is built

Every figure here traces back to a public source through code you can run. This is the whole machine in one view — the sources, the reconciliation engine, the data layers, the deterministic generators, and all the pages — and how they connect. Nothing is hand-typed into a page; rebuild it and it regenerates.

The pipeline — from public data to the page you’re reading

It flows one way: public sources → a reconciliation engine → JSON data layers → deterministic generators → this static site.

flowchart LR
  subgraph SRC[" Public data sources "]
    direction TB
    S1["UN Comtrade
raw bilateral trade"]
    S2["CEPII BACI
HS02+HS17 · 2002-2024"]
    S3["USGS · IEA
mine / refine shares"]
    S5["World Bank WGI
governance"]
    S6["Maus 2022 · Sepin 2025
satellite mine polygons"]
  end
  ENG["Reconciliation engine
comtrade-reconcile
CI green · pre-registered"]
  J[("out/*.json
data + 25 flow-years
+ 15 analytical layers")]
  GEN["~20 generators
deterministic · reproducible"]
  SITE["This static site
28 pages + material & country profiles"]
  S1 --> ENG
  S2 --> ENG
  ENG --> J
  S3 --> J
  S5 --> J
  S6 --> J
  J --> GEN
  GEN --> SITE
  classDef src fill:#eaf3f1,stroke:#0e7c74,color:#15323a;
  classDef eng fill:#15323a,stroke:#0e7c74,color:#ffffff;
  classDef data fill:#ffffff,stroke:#b8c6c3,color:#15323a;
  classDef gen fill:#f4f7f6,stroke:#7d9b97,color:#15323a;
  classDef site fill:#0e7c74,stroke:#0a3b39,color:#ffffff;
  class S1,S2,S3,S5,S6 src; class ENG eng; class J data; class GEN gen; class SITE site;
  click S1 "map.html#stage-sources" "What these data sources are"
  click S2 "map.html#stage-sources" "What these data sources are"
  click S3 "map.html#stage-sources" "What these data sources are"
  click S5 "map.html#stage-sources" "What these data sources are"
  click S6 "map.html#stage-sources" "What these data sources are"
  click ENG "map.html#stage-engine" "What the reconciliation engine does"
  click J "map.html#stage-data" "What the data layers are"
  click GEN "map.html#stage-generators" "What the generators do"
  click SITE "map.html#stage-site" "This static site"

Every box above is clickable — click it to jump to its plain-English explanation below (which links onward to the code and data). Or just read them in order.

What happens at each stage

  1. Public data sources. Everything starts from datasets anyone can download — UN Comtrade (raw trade), CEPII BACI (the reconciled reference), USGS & IEA (mine and refining shares), the World Bank’s governance indicators, and peer-reviewed satellite mine-polygon datasets. No private or paid data. Sources & method →
  2. Reconciliation engine. The two sides of every trade disagree — the exporter and the importer report different numbers. The engine mirror-matches them into one figure per flow (a Gaulier–Zignago-style reconciliation) and validates the result against the official BACI dataset. It is open-source, runs in CI on every push, and the forecast is pre-registered.
  3. Data layers (out/*.json). The reconciled trade, plus every metric computed from it, is frozen as plain JSON files — the material data, one file per trade-year (2002–2026), and 15 analytical layers (risk, network, trends, satellite…). This is the single source of truth every page reads; you can fetch them directly, no key.
  4. Generators. About 20 small, deterministic Python/R scripts turn those JSON layers into each page — the charts, tables and the numbers written into the prose. The same input always produces the same site, so nothing is hand-typed and every figure is reproducible.
  5. This static site. The 28 pages and the material & country profiles you are browsing, served straight from GitHub Pages — no backend, nothing to trust but the code and the public data. Open the atlas →

The analytical dependency tree — and where the atlas corrects itself

The diagram above is the plumbing — how data becomes pages. This one is the reasoning: each analytical layer built on the one before it, from the reconciliation engine out to the final “hardest cases”. What makes it unusual is the three dashed red loops — places where a later result reached back and overturned an earlier claim of our own. Every box is clickable and opens its page.

flowchart TD
  classDef spine fill:#eef3f2,stroke:#9db4b0,color:#15323a;
  classDef sat fill:#eaf0f8,stroke:#3f6fb0,color:#1c3a63,stroke-width:1.5px;
  classDef supply fill:#e7f4f1,stroke:#0e7c74,color:#0b4a45,stroke-width:1.5px;
  classDef demand fill:#fbf1dd,stroke:#b07a18,color:#7a5310,stroke-width:1.5px;
  classDef systemic fill:#fbe9e7,stroke:#c0392b,color:#8f2a20,stroke-width:1.5px;
  classDef data fill:#e6f4ec,stroke:#2f8f6b,color:#1d5a43,stroke-width:1.5px;
  classDef rigor fill:#f0eafa,stroke:#7d5fb0,color:#4c3a75,stroke-width:1.5px;
  classDef synth fill:#f7e6ea,stroke:#8a2f3f,color:#6a2331,stroke-width:2px;

  SRC["Public data
Comtrade · BACI · USGS · IEA · WGI"] --> ENG["comtrade-reconcile
engine"]
  ENG --> CORE["Core atlas
concentration · origin · risk · trends"]

  CORE --> ATT["1 · Commodity attribution
13 registers · 63%→73% via clustering"]
  ENG --> REF["6 · Refining wedge
ore↔refined HHI · BACI"]
  ATT --> COMP["2 · Companionality
hostage metals"]
  COMP --> ADJ["3 · Adjusted risk"]
  COMP --> HOST["4 · Host shock"]
  COMP --> REC["5 · Recycling"]
  COMP --> DEM["7 · Demand & squeeze"]
  DEM --> PRICE["8 · Price test
falsifier"]
  PRICE --> VOL["9 · Volatility retest
real USGS prices · size control"]
  VOL -.->|retracts the volatility claim| PRICE
  PRICE --> COUP["10 · Host coupling"]
  DEM --> BLOC["11 · Demand by bloc"]
  BLOC --> NET["12 · Net demand
re-exports out"]
  COMP --> CASC["13 · Supply-shock cascade
interactive · tonnes"]
  HOST --> CASC
  CASC --> PROD["14 · Production in tonnes
WMD cross-check"]
  PROD -.->|tonnage feedback| CASC
  PROD --> GEO["15 · Concentration in tonnes
GeoPolRisk · production HHI"]
  GEO -.->|re-bases value-HHI on tonnes| ENG

  COMP --> SYN["16 · Synthesis
the hardest cases"]
  REC --> SYN
  DEM --> SYN
  PROD --> SYN

  SYN --> UNC["17 · Uncertainty
Monte-Carlo"]
  SYN --> GA["18 · Gallium deep-dive
mass balance"]
  PROD --> GA
  SYN --> CHAL["19 · Break this atlas
open challenge"]

  class SRC,ENG,CORE spine;
  class ATT sat;
  class COMP,ADJ,HOST,REC,REF supply;
  class DEM,PRICE,COUP,BLOC,NET demand;
  class VOL rigor;
  class CASC systemic;
  class PROD,GEO data;
  class UNC,GA,CHAL rigor;
  class SYN synth;

  click ATT "commodity-attribution.html" _blank
  click COMP "companionality.html" _blank
  click REF "refining.html" _blank
  click ADJ "risk-adjusted.html" _blank
  click HOST "host-shock.html" _blank
  click REC "recycling.html" _blank
  click DEM "demand.html" _blank
  click PRICE "price-squeeze.html" _blank
  click VOL "price-volatility.html" _blank
  click COUP "host-coupling.html" _blank
  click BLOC "bloc-demand.html" _blank
  click NET "net-demand.html" _blank
  click CASC "cascade.html" _blank
  click PROD "production.html" _blank
  click GEO "geopolrisk.html" _blank
  click SYN "synthesis.html" _blank
  click UNC "uncertainty.html" _blank
  click GA "gallium.html" _blank
  click CHAL "challenge.html" _blank
  click ENG "https://github.com/Varcolacus/critical-materials-atlas" _blank

Those red dashed edges are the atlas testing itself: the volatility retest retracted the volatility claim on the price page; real tonnes re-run the supply-shock cascade; and concentration in tonnes re-bases the engine’s dollar-based concentration. The full record of every headline that broke — and how — is the error log.

Every page, grouped

The same groups you see in the navigation. Click any to jump there.

Core

  • Atlas · interactive table / flow / map / globe
  • Insights · state of supply
  • Findings · the origin gap (headline)

Analysis

Risk

Satellite

Rigor & honesty

Reference

Sources & reproducibility

Data sources — all public

UN Comtraderaw bilateral trade (reconciled)
CEPII BACIHS02 + HS17, 2002–2024
USGSmine-production shares
IEArefining shares
World Bank WGIgovernance
Maus et al. 2022satellite mine polygons
Sepin et al. 2025tropical mining expansion
Eurostat ComextEU import-origin lens

How it reproduces

Enginethe reconciliation engine is open-source with CI that regenerates the numbers from raw Comtrade — no API key needed.
Data layers41 plain JSON files (the material data, 25 flow-years, and 15 analytical layers), fetched straight from GitHub Pages.
Generators~20 deterministic builders turn the JSON into every page — the same input always gives the same site.
The betthe 2025 nowcast’s accuracy is pre-registered — scored against the next data release, pass or fail.
What could be wrongevery known limitation, with its direction of bias and a falsifier, is on the limitations page.

The point of all this: you shouldn’t have to trust the author. The code checks itself, and every claim is one click from how it was derived.