# Thinking Text data API

Indexes of what the news says, read by a language model from thousands of articles a day, published as static
CSV and JSON files. Free for research and non-commercial use, with attribution ("Source: Thinking Text,
thinkingtext.com").

**Not investment advice.** The indexes are news analysis for research and entertainment only: a description of the
news flow, not a forecast, not a trading signal and not a recommendation to buy or sell anything. Terms of service:
https://thinkingtext.com/legal/terms. Privacy policy: https://thinkingtext.com/legal/privacy.

- **Oil News Index F6 (OIL-F6)**: crude oil. Six fundamental questions (demand, supply, inventories, OPEC, disruption,
  margins) put every day to about a thousand oil articles, next to the WTI and Brent moves. Page: https://thinkingtext.com/oil-index
- **Equity News Index (EQ-S10)**: twelve US large caps and a general market read, one folder per name.
  Page: https://thinkingtext.com/equity-index
- **Polymarket News Index**: for each of the top ~250 Polymarket markets, what the news says about every outcome, next
  to the price: news-implied probabilities, how far the news tracks each market, rule-based alerts with links to the
  articles. Every hour, with daily, weekly and monthly files since January 2026. Page:
  https://thinkingtext.com/polymarket-news-index

Get a free key (name, company, email; the key is emailed): https://thinkingtext.com/data-api

## Access

- Base URL: `https://thinkingtext.com/api/v1`
- Authentication, one of:
  - header `X-API-Key: <key>`
  - header `Authorization: Bearer <key>`
  - query `?key=<key>` (spreadsheets, plain links)
- `GET /api/v1/` returns the catalog of files (and checks the key).
- Errors are JSON `{"error": "...", "message": "..."}`: `401 missing_api_key`, `403 invalid_api_key`, `404` no such file.
- Updates: oil and equity, final values once a day, in the morning (UTC); `oil/today.json` is a provisional read of the
  current press day, refreshed every six hours. Download once and cache; there is nothing to gain from polling.
  Polymarket: a release every hour, about 22 minutes past (UTC), and the final day after 01:20 UTC; poll
  `polymarket/manifest.json` (cached 30 seconds), every other Polymarket file is immutable.
- Keys look like `tt1_` followed by 48 hexadecimal characters. Keep yours out of public repositories; if it leaks,
  write to info@thinkingtext.com for a new one.

```bash
curl -H "X-API-Key: $TT_API_KEY" https://thinkingtext.com/api/v1/oil/latest.json
curl -o oil.csv -H "X-API-Key: $TT_API_KEY" https://thinkingtext.com/api/v1/oil/history/all.csv
```

```python
import io, os
import pandas as pd, requests

API, HEADERS = "https://thinkingtext.com/api/v1", {"X-API-Key": os.environ["TT_API_KEY"]}
oil = pd.read_csv(io.StringIO(requests.get(f"{API}/oil/history/all.csv", headers=HEADERS, timeout=30).text))
```

Spreadsheet: `=IMPORTDATA("https://thinkingtext.com/api/v1/oil/history/all.csv?key=<key>")`

## Conventions (oil and equity)

- Values are **z-scores**: each question's daily value is standardised on its own trailing year (units: standard
  deviations of that year) and lightly smoothed. The index is the mean of its parts, each signed toward a higher price.
- **States** map a z-score to a word: -2 below -1.5, -1 below -0.5, 0 from -0.5 to 0.5, +1 to 1.5, +2 above. The words
  per question are in `spec.json`.
- **Press lag**: the read attributed to market day *t* is the press of the next price day (`press_day`), so a row
  never uses news written after the close it is compared with.
- **Cards** (`k1`, `k5`, `k20`): one OLS beta of the k-day log return on the k-day index change, fitted on the trailing
  250 price days, refit every 20 days and applied forward. `k<K>_implied` is the news-implied k-day move (percent),
  `k<K>_beta` / `k<K>_b0` the coefficients used that day; residual = actual - implied.
- `status` is `final` or `provisional`; rows before `live_since` (in `spec.json` / `index.json`) are a back-fill.
- Read `spec.json` before interpreting values: it is the authoritative description, versioned (`spec_version`).

## Oil files: `oil/`

| file | content |
|---|---|
| `oil/latest.json` | the last complete press day: `f6` (index z, state, label, 1/5/20-day changes), `slots` (the six questions), `context`, `price`, `cards` (attribution at 1, 5, 20 days, with hit rates), `level` |
| `oil/latest.csv` | the same as one flat row |
| `oil/today.json` | provisional read of the current press day (index and slots, no price yet); 404 when there is none |
| `oil/days/<YYYY-MM-DD>.json` | every final `latest.json`, kept by press day |
| `oil/history/all.csv`, `.json` | one row per market day since 2022 (JSON: `{"rows": [...]}`) |
| `oil/history/<year>.csv`, `.json` | the same, one year |
| `oil/spec.json` | the questions, their sentences and state words, the conventions, the price source |
| `oil/backtest.json` | the back-analysis behind the page's charts (out of sample where a fit is involved) |

History columns: `market_day`, `press_day`, `status`; `f6_z`, `f6_state` (the index); `<q>_z`, `<q>_state` for the six
questions `demand`, `supply`, `inventories`, `opec`, `disruption`, `margins`; context states (not in the index)
`growth`, `rates`, `dollar`, `stocks`, `mood`, `inflation`, `fedcuts`, `gold`, `euro`, `geopolitics`, `trade`,
`volatility`; WTI `cl_close`, `cl_r1`, `cl_r5`, `cl_r20` (log returns, percent), `cl_ref60`, `cl_pz`; Brent the same
with `bz_`; cards `k1_implied`, `k1_beta`, `k1_b0` and the same for `k5`, `k20`.

## Equity files: `stocks/`

| file | content |
|---|---|
| `stocks/index.json` | every name (`names`: ticker, company, published, latest value), `published`, `kept_back`, `market`, `unavailable` |
| `stocks/summary.json` | per-name numbers of the heat map and the scatter |
| `stocks/spec.json` | the slot families, the market read and peers rules, the selection, card and level rules, the price source |
| `stocks/<name>/latest.json`, `latest.csv` | the last complete press day for one name: `value`, `parts`, `slots`, `price`, `cards`, `level` |
| `stocks/<name>/days/<YYYY-MM-DD>.json` | every final `latest.json`, kept by press day |
| `stocks/<name>/history/all.csv`, `.json`, `<year>.csv`, `.json` | one row per market day since 2022 |
| `stocks/<name>/backtest.json` | the back-analysis of that name |

`<name>` is the lower-case code from `index.json` (`nvda`, `aapl`, `msft`, ...; `market` is the general market read).
Only the `published` names form the index; `kept_back` names are computed and served but not steady enough.

History columns: `market_day`, `status`; `index_z`, `index_state`; parts `company_z`, `market_read_z`, `peers_z`;
slot families `competition_z`, `demand_z`, `macro_z`, `market_z`, `outlook_z`, `profits_z`, `reflectors_z`,
`surprise_z`; price `close` (adjusted), `r1`, `r5`, `r20`, `px_ref60`, `index_ref60`; cards `k1_implied`, `k1_beta`,
`k1_b0` and the same for `k5`, `k20`.

## Polymarket files: `polymarket/`

The full format (every file and field, events, versions, how to poll) is `polymarket/SCHEMA.md`
(https://thinkingtext.com/api/v1/polymarket/SCHEMA.md, format `pmnews/1`). Read it before interpreting values. In short:

- **Start from `polymarket/latest.json`** (cached 30 s): the paths of the current `list`, `index`, `state`, `events`
  and `top` lists. Every path in it is relative to `polymarket/` and **immutable** (its content never changes; cache it
  forever). File names carry a release number (`<seq>`): take them from `latest.json` or the list, never build them.
- **Markets** are keyed `pm-<event_id>` (polymarket.com), `us-<event_id>` (Polymarket US), `pm-<event_id>-<market_id>`
  (one event read as several questions). The list (`lists/markets.<seq>.json`) has one row per market: question,
  outcomes, polymarket.com URL, volume, rankable numbers, `status` (open / closed, `closed_on`; closed markets stay listed),
  ready-made top-10 lists `views.top20 / top40 / all` (and `views.all.recently_closed`), and the paths of its `state`
  (current values per outcome) and `history` (one row per final day and outcome since January 2026).
- **Per day, all markets**: `daily/<YYYY-MM-DD>.json`, `.csv`, `.parquet` (one row per market and outcome, final days),
  `events/<YYYY-MM-DD>.jsonl`; closed periods `weekly/<monday>.*`, `monthly/<YYYY-MM>.*`; top lists
  `top/{daily,weekly,monthly}/<period>.json`.
- **Polling**: read `manifest.json`; when `seq` is higher than the last you processed, read `files.events.path` and the
  states of the keys in `changed`; releases you skipped are in `manifests/<seq>.json`.

| file | content |
|---|---|
| `polymarket/latest.json` | the pointer for pages and scripts: current list, index, state, events, top lists |
| `polymarket/manifest.json`, `manifests/<seq>.json` | the current release (`seq`, `final_through`, `files`, `changed`, `markets`); every release kept |
| `polymarket/lists/markets.<seq>.json` | every market, its rankable numbers, top-10 views and detail paths |
| `polymarket/markets/<key>/v<N>/state.<seq>.json`, `history.<seq>.json` | one market now; one market's daily rows |
| `polymarket/daily/<date>.{json,csv,parquet}`, `events/<date>.jsonl` | all markets, one final day; that day's alerts |
| `polymarket/weekly/<monday>.{json,csv,parquet}`, `monthly/<YYYY-MM>.{json,csv,parquet}` | closed weeks and months |

Per outcome: `p_market` (market probability), `p_news` (news-implied probability), `gap` = log(p_news / p_market),
`flag`, `news_tilt` and `news_z_1d` / `news_z_7d` (how the news moved), `news_ema7/14/30` and `news_ema14_scaled` (the
smoothed news read; the scaled one on the price's 0-1 range), `news_rank`, `market_rank`, `favorites_agree`,
`articles`, `answer_share` and `prior_share` (the model's answer with and without the articles). Per market:
`news_price_r` (correlation of the smoothed news with the price over the market's history), `news_price_r_null` (the
same against unrelated markets), `same_favorite_share`, `trust`. The news tracks some markets closely and others not
at all: check these before using a market's numbers. A gap is a description of the press, not a forecast.

Events are rule-based: `type`, `k`, `severity`, `tier` (`major`, `normal`, `all`), a pre-worded `message` and up to
three `links` to the articles behind the move: `outlet`, `url`, `title` and `published` (from the article page's
metadata), `seen` (when GDELT first saw it), `outlets_carrying`. No article text is published.

## Not published

No article text, outlet list, question wording, prompt or per-article score (the Polymarket feed links to the
articles behind its alerts). The scoring engine behind the indexes is offered
separately as an API by spinf: https://spinf.com
