Replicating this platform on free GitHub

A complete blueprint for running the entire Atlas data platform — gathering, cleansing, validation, storage, refresh and serving — on free GitHub + GitHub Actions, pulling directly from the primary sources. No intermediary API (not even Atlas's own), no server, no database, no paid tier of anything. Every number on this page is measured from the running Atlas pipeline, not estimated.

Written for sibling projects (e.g. london-buses.farhan.app) that want to reproduce the stack independently. For what each field means and how it is validated, this page links into the field-level reference — the per-dataset rules there apply unchanged; only the runner and the store differ.

Overview & architecture

The design principle: GitHub Actions is the scheduler and compute, the git repo is the database, and static hosting serves the API. Every dataset is fetched from its primary source on a schedule, cleaned and validated in the workflow run, committed as JSON/NDJSON files, and served as static assets (optionally wrapped by serverless functions for a queryable API).

 TfL Unified API ─┐
 BODS · DfT ······│    ┌────────────────────┐     ┌───────────────────┐    ┌─────────────────┐
 DVLA · DVSA VOL ─┼──▶ │ GitHub Actions     │ ──▶ │ git repo          │ ─▶ │ static hosting   │
 londonbusroutes ·│    │ scheduled workflows │     │ data/*.json       │    │ (+ functions =   │
 bustimes · OSM ──┘    │ fetch→clean→validate│     │ data/history/*    │    │  a public API)   │
                       └────────────────────┘     └───────────────────┘    └─────────────────┘
                         the compute                 the database             the serving layer

• Fetch → Clean → Validate → Store → Serve, with validation as a hard gate: a run whose output fails row-count/sanity checks commits nothing, so bad upstream data can never overwrite last-good. This single rule is what makes unattended operation safe.
• Each push of refreshed data triggers the static-host rebuild — data deploys are just git pushes.
• Warm caches (HTTP validators, one-lookup-ever registries like DVLA) persist between runs via actions/cache, replacing a server's disk.

The free-tier budget — measured, not guessed

Repo growth from committing data daily, measured on the live Atlas repo and by simulation (676 routes × 90 days of realistic history rows, extrapolated):

GitHub is comfortable to ~1 GB and warns approaching 5 GB — day-partitioned history buys years of headroom. Static-host ceilings (Cloudflare Pages free): 20,000 files/site · 25 MiB/file · 500 builds/month. A nightly + 5×-daily push cadence is ~180 builds/month; a per-route file layout for ~5 history datasets lands ~6,000 files growing ~1,800/year — inside all three, but the build count is the limit to watch if you add more intraday refreshes.

Workflows to set up

Two scheduled workflows cover everything. Cadence rule: match the source, don't hammer it — most reference data changes at most daily; TTLs inside the pipeline make re-runs cheap no-ops (a dataset within its TTL is skipped; conditional requests via ETag/If-Modified-Since skip unchanged downloads).

1 — refresh-data.yml (nightly, the workhorse)

name: refresh-data
on:
  schedule: [{cron: "17 3 * * *"}]      # 03:17 UTC — off-peak for TfL, before the day's users
  workflow_dispatch: {}                 # manual runs for backfills/debugging
concurrency: {group: refresh, cancel-in-progress: false}   # never two refreshes at once (TfL 429s)
permissions: {contents: write}         # lets GITHUB_TOKEN push the data commit
jobs:
  refresh:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: {node-version: 22}
      - uses: actions/cache@v4          # HTTP validator cache + one-lookup-ever registries
        with: {path: .cache, key: "http-cache-${{ github.run_id }}", restore-keys: http-cache-}
      - run: npm ci --omit=dev
      - run: node pipeline/run.js       # fetch → clean → validate per dataset
        env:
          TFL_APP_KEY: ${{ secrets.TFL_APP_KEY }}
          DVLA_API_KEY: ${{ secrets.DVLA_API_KEY }}
      - run: node pipeline/validate-atlas.js    # HARD GATE — fail here = no commit
      - name: commit refreshed data
        run: |
          git config user.name  "data-bot"
          git config user.email "bot@users.noreply.github.com"
          git add data/
          git diff --cached --quiet && exit 0   # no change → no commit → no rebuild
          git commit -m "chore(data): scheduled refresh $(date -u +%FT%TZ)"
          for i in 1 2 3 4 5; do git push && break || { git pull --rebase; sleep $((i*3)); }; done

2 — diversions.yml (intraday freshness)

on:
  schedule: [{cron: "17 7,11,15,19,23 * * *"}]   # 5×/day; diversions are the one fast-moving reference set
# identical job, but:  run: node pipeline/run.js --only=diversions
# same concurrency group as refresh-data — they must never overlap

Secrets to configure Settings → Secrets → Actions

No push token needed: the workflow's own GITHUB_TOKEN with permissions: contents: write commits to the repo it runs in. Note one quirk: pushes made with GITHUB_TOKEN do not trigger other Actions workflows (loop protection) — but they do trigger external webhooks, so the static-host rebuild still fires.

Scheduling honesty

• Cron is best-effort. Scheduled runs routinely start 10–30+ minutes late at busy times and are occasionally skipped entirely. Fine for daily/4-hourly reference data; disqualifying for anything that must sample at precise intervals (see the boundary). Schedule at odd minutes (:17, not :00) to dodge the top-of-hour stampede.
• Auto-disable: GitHub disables scheduled workflows after 60 days without repo activity — a data-committing pipeline keeps itself alive, but know the rule.
• Never two runs at once: TfL throttling punishes concurrency; one concurrency group across both workflows replaces a server's lock file.

Per-dataset source map — plug into these directly

Every dataset, its primary source, and the cadence to run it at. The cleansing/validation rules for each are specified field-by-field in the reference (linked per row) — they are runner-independent: apply them identically in an Actions job. Global rules: timeout every request; retry 429/5xx with backoff + jitter; treat every source as fallible; a failed fetch keeps last-good, never writes empty.

Files as the database

Current-state datasets are one JSON file each, overwritten per run — exactly the Atlas static store. Time-series (history) needs a layout decision, and the measured difference is 5×:

data/history/<dataset>/day/<YYYY-MM>/<YYYY-MM-DD>.ndjson   # write-once → git-cheapest (~5 MB/yr), answers "all routes on a day"
data/history/<dataset>/route/<id>.ndjson                   # append per run (~25 MB/yr), answers "one route over time" in one read

• Dual-write both axes. Storage is cheap (~30 MB/yr/dataset for both); cross-file scanning at query time is not — serverless functions allow ~50 subrequests per invocation, so a year-long range over day-files would break. One query = one file read.
• NDJSON, stable field order, sorted rows — line-oriented diffs keep git deltas small and reviews readable.
• Idempotent upserts: re-running a day rewrites that day's file identically (write-once semantics survive retries).
• Validation before commit, always: row-count floors, not-all-null checks, plausibility bands — the gate step in the workflow is the entire integrity story. Quarantine failures in the log; never let them reach git add.
• Roll up old partitions (day-files → monthly bundles after a year) to stay far from the 20,000-file ceiling.

Serving an API from files

A database is not required to have a real API. The pattern (Atlas's own /api/v1 current group runs exactly this way):

• Static hosting serves data/** directly — that alone is a versioned, cacheable, CORS-able read API.
• A thin serverless function in front adds: a discovery index, parameter validation against a strict whitelist, filter/order/limit application in memory on one fetched file, consistent JSON envelopes and error shapes, and edge caching (5–10 min for reference data, 10–120 s for live proxies).
• Query params map to the file layout: ?route=25&from=…&to=… → read route/25.ndjson, filter rows; no route → read the day partition(s). Design the layout from the queries, then denormalise until every query is one file.
• Keyed upstreams (BODS live GPS) hide behind the function with the key as a host secret and a shared short-TTL cache, so unlimited callers collapse to one upstream poll per TTL.

What GitHub Actions cannot do — the honest boundary

One class of workload does not fit, at any repo visibility: continuous, stateful observation. Atlas's own daily EWT/OTD, lost-mileage and first-party fleet tracking come from a daemon polling BODS every 25 seconds, holding thousands of open bus trips in memory between polls. Actions cannot host that:

Consequences for a pure-GitHub replica: consume reliability figures rather than compute them (TfL QSI now; the placeholder above if licensed), and take fleet moves from bustimes' vehiclejourneys — the community instance of exactly this collector. Everything else on this page fits comfortably.

Getting-started checklist

• ① Create the repo (private is fine at ~1,350 min/month; public removes the minutes question). Add data/ and .cache/ conventions; gitignore raw intermediates.
• ② Add secrets: TFL_APP_KEY (recommended), DVLA_API_KEY (fleet enrichment), BODS_API_KEY (live serving only).
• ③ Land the two workflows above; run refresh-data once via workflow_dispatch as the cold-start backfill (first run is the slow one — DVLA/tender caches fill, then steady-state is quick).
• ④ Check the run summary: fetched/updated/skipped/failed per dataset, and the validation gate green before the commit step.
• ⑤ Point static hosting at the repo (no build command needed if the repo root is the site); confirm a data push triggers a rebuild and the JSON is publicly fetchable.
• ⑥ Watch the first week: Actions minutes used vs 2,000, repo size growth vs the measurements above, host build count vs 500/month.
• ⑦ Politeness pass: every fetcher sends an identifying User-Agent, respects per-source rate caps, uses conditional requests, and backs off on 429 — you are a guest on every one of these feeds.

Licensing reminder for anything you republish: TfL open data terms (attribution — "Powered by TfL Open Data"), OGL v3 (DfT/DVLA/DVSA), ODbL (OSM — attribution + share-alike), community sources credited. The full per-source table is in the reference appendix.