JIMMY OKOTH 30%

Writing / Technical / Build Log · 8 OCT 2026

Fake Data, Real Discipline: What a Test Data Generator Taught Me About Data Engineering

How I built a seeded synthetic data generator for my credit data platform, where AI genuinely helps, what you must give it, and the file, testing and Git habits I'm carrying into every future project.

  • data engineering
  • python
  • testing
  • git
  • synthetic data
  • ai

Let me be honest about today first. I will probably never write a synthetic data generator from scratch at work. If I need fake data, I'll ask an AI to write most of it, and so will everyone else.

But building one taught me something I didn't expect. Generating data well is almost entirely discipline, and that discipline is the same discipline that makes real pipelines trustworthy. Fixed boundaries. Exact money. Files that behave like contracts. Tests that prove promises. A Git workflow that keeps main safe.

So this post's about what I'm taking from it into every project after this one, and into whatever team I join.

Context: This is part two of building the Newcomer Credit Data Platform, a platform that turns financial records into fair, point-in-time affordability features for people with little or no local credit history. Part one, Before the First Pipeline, covers setting up the machine and project base. The code is on GitHub.

What I ended up with: one command that builds a synthetic town of 1,000 customers and about 60,000 bank transactions in seconds, a manifest that fingerprints every file, twelve tests that prove the data keeps its promises, and the whole thing shipped through a reviewed, merged pull request.

Part 1: Why fake data exists at all

Real financial data is private, and it should be. You can't put it on a laptop, in a public repo, or in a demo. But you still need data to build and test a pipeline against.

At work, this comes up constantly. Development environments need data that isn't production. Demos need data you can show a client. Tests need data where you already know the right answer.

That last point is the one that matters most. In my project, every synthetic customer gets a hidden affordability score that the pipeline never sees. Later, I'll check whether my pipeline worked out who can afford a loan, and the hidden score is the answer key.

Think of it like a teacher writing an exam. The teacher writes the answer key first, then the questions. If a student's answers match the key, the teaching worked. My pipeline is the student. The hidden score is the answer key, locked in a separate drawer.

Part 2: Where AI helps, and what you must give it

I'll put it plainly, AI is very good at writing data generator code. Describe a dataset clearly and a model will produce a working Python generator in minutes. The code for this project was drafted with Claude, then I tested it, broke it on purpose and walked through every line.

The catch is the word "clearly." AI fills every gap in your description with a guess, and its guesses are where the bugs live. It doesn't know your business rules. It doesn't know which values must never exist. It will happily write code that looks perfect and quietly produces different data every time it runs.

Your job isn't to write the generator. Your job is to write the specification and to verify the result.

What to give the AI

Here's what I'd include in any request for a test data generator, at work or on a side project:

  • The entities and how they connect. Customers, transactions, applications. Which table points to which, and through what key. "Every transaction belongs to exactly one existing customer."
  • The keys and their format. CUST-00001, TXN-00000001. Unique, stable, and in a format the next system expects.
  • The time window, as fixed dates. "The simulation runs from 1 January 2024 to 31 December 2025." Never "the last two years," because that moves every day.
  • Value ranges and shapes. Salaries mostly near £2,300, between £1,400 and £3,500, with a longer tail of higher earners. Rent between 30% and 50% of salary. A range alone isn't enough; say what's common and what's rare.
  • The business rules. No transaction before the customer arrived. Salary between the 25th and 28th, occasionally a day or two late. No loan application in the first 90 days.
  • What must never be generated. For a lending dataset, that means no nationality, no immigration status, no ethnicity, nothing that could become a discriminatory feature. And no real company or personal names. If it shouldn't exist in the model, it shouldn't exist in the data.
  • The repeatability requirement. "One seeded random generator, passed through every function in a fixed order. The same seed must produce byte-identical output files."
  • The output contract. Column order, date and timestamp formats, how money is represented, where files go. (Part 4 covers this in detail.)
  • The defects you want, if any. Dirty data should be planted deliberately, at known rates, and logged. Not left to chance.
  • The tests you expect. Ask for them in the same request, named after the promises they protect.

A prompt template I'll reuse

Write a Python test data generator.

Entities and keys:
- customers (customer_id like CUST-00001)
- bank_transactions (transaction_id like TXN-00000001), each linked to one customer
- applications (application_id like APP-00001), one per customer

Time window: fixed dates, 2024-01-01 to 2025-12-31. Never use today's date.

Values and rules:
- [ranges, distributions, and business rules here]

Never generate: [protected attributes, real names, anything that must not exist]

Hidden ground truth: [what it is], written to a separate file in a separate folder,
never present in the other tables.

Repeatability: one numpy Generator created from a seed and passed through every
function in a fixed order. Same seed must give byte-identical files.

Money: integers in minor units (pence) internally, formatted as exact decimal strings.
Timestamps: timezone-aware UTC, written as ISO 8601 with a Z.

Structure: keep logic (building DataFrames) separate from file writing.
Output: fixed column order, UTF-8, header row, no index column, atomic writes,
plus a manifest.json with row counts and SHA-256 checksums.

Tests (pytest): same seed gives identical data and identical file checksums;
IDs unique; no orphan records; [each business rule]; hidden truth never leaks.

What to check in AI-generated generator code

These are the mistakes I'd look for first, because they look harmless and break things later:

Trap Why it hurts
Uses today's date anywhere Same seed, different data tomorrow. Repeatability is gone
No seed, or a separate seed per function Results drift between runs, or change when you reorder code
Iterates over a Python set Sets have no guaranteed order, so the sequence of random draws can change
Money as floats 0.1 + 0.2 is 0.30000000000000004. Totals drift
Timestamps without a time zone "9am" where? Ambiguous the moment data crosses systems
df.to_csv() without index=False A mystery unnamed first column appears in every file
Hidden truth in the same table as visible data The answer key leaks into the exam
Real-looking personal data from libraries like Faker Realistic fake names and addresses can still cause privacy and compliance headaches
Distributions that make no sense together Rent larger than salary, bills before arrival. Code runs; data lies

The rule I'm keeping: AI-generated code is a first draft from a fast junior colleague. It's often good. It's never trusted until a test proves it.

Part 3: The ideas inside the generator

You don't need to write a generator to use these ideas. They show up everywhere.

A seed: the same shuffle every time

Randomness in code isn't truly random. It comes from a mathematical sequence, and the seed is where that sequence starts.

Think of a card-shuffling machine. Give it the same starting number and it produces the same shuffle, every time. I proved this to myself in the Python REPL:

>>> import numpy as np
>>> rng = np.random.default_rng(42)
>>> rng.integers(1, 7, size=5)
array([1, 5, 4, 3, 3])
>>> rng = np.random.default_rng(42)
>>> rng.integers(1, 7, size=5)
array([1, 5, 4, 3, 3])
>>> rng.integers(1, 7, size=5)
array([6, 1, 5, 2, 1])

Same seed, same five dice. Roll again without resetting and you get the next cards in the deck.

That last part is the subtle one, that's because the order of random draws matters. Add one extra draw at the start of the program and every value after it shifts. The data is still repeatable, but it's different data. When results "randomly" change between runs, my first question becomes "was there a seed, and did the order of random calls change?"

Logic separate from files: the chef and the waiter

The generator is two files with two jobs.

  • build.py is the chef. It creates the data in memory and never touches the disk.
  • run.py is the waiter. It asks the chef for the data, then handles everything about files: formatting, writing, fingerprinting.

Why split them? Because a test can walk into the kitchen and taste the food directly, without waiting for it to be plated and served. My tests call build_all() and inspect the tables in memory, with no files to create or clean up. Code that mixes logic and file handling is far harder to test, and this separation shows up in every well-built pipeline.

Fixed boundaries, never "now"

SIM_START = date(2024, 1, 1)
SIM_END = date(2025, 12, 31)

Anything that must be repeatable can't depend on the current date. This applies far beyond generators. A backfill job, a monthly report or a test that uses "now" gives different answers on different days. Pass dates in explicitly.

Money in whole pence

>>> 0.1 + 0.2
0.30000000000000004
>>> 0.1 + 0.2 == 0.3
False

Computers store most decimals in binary and can't store 0.1 exactly. That's fine for physics. With money, tiny errors compound into totals that don't reconcile. So every amount lives as a whole number of pence until the moment it's written out:

def pence_to_str(pence: int) -> str:
    sign = "-" if pence < 0 else ""
    pence = abs(int(pence))
    return f"{sign}{pence // 100}.{pence % 100:02d}"

// is whole-number division and % is the remainder, so -125050 becomes -1250.50 with no floats anywhere. In a database, the equivalent is NUMERIC, never FLOAT.

Every timestamp gets a time zone

Every timestamp is created in UTC and written like 2024-02-25T06:43:00Z. The Z means UTC. A timestamp without a time zone is like a meeting invite that says "3pm" with no city. Someone will turn up at the wrong time.

Three timestamps, even when they're equal

Each transaction records when it happened, when the bank booked it, and when the lender could see it. Today all three are identical. In a later version some transactions will arrive late, and that's when point-in-time correctness gets tested. The columns exist from day one, so nothing needs restructuring later. Designing the shape of data for where it's going, not just where it is, saves painful migrations.

Stable order, stable IDs

df = df.sort_values(["customer_id", "event_at", "description"])
df.insert(0, "transaction_id", [f"TXN-{i:08d}" for i in range(1, len(df) + 1)])

Sort first, then number. TXN-00000001 is the same transaction on every run. Stable IDs are what let a loader recognise a row it has already loaded, which is the foundation of loading data without duplicates.

Part 4: Writing files other systems can trust

Anyone can call df.to_csv(). Writing a file that another system can depend on takes more care.

The analogy I keep coming back to is shipping containers. Before standard containers, every ship, crane and truck handled cargo differently, and everything was slow and fragile. Standard sizes and fittings meant any crane could lift any box. A data file is the same kind of handover between systems, and these are the standards I'm keeping:

1. Fixed column order. Written out explicitly in code, so a refactor can never shuffle columns by accident.

2. Exact money. Whole minor units internally, exact decimal strings in the file.

3. ISO 8601 timestamps in UTC, like 2024-02-25T06:43:00Z. Year first, so text sorting is also time sorting.

4. Dates as YYYY-MM-DD. Never 02/03/2024, which is February in the US and March in Kenya and the UK.

5. UTF-8, a header row, no index column.

df.to_csv(tmp, index=False, encoding="utf-8", lineterminator="\n")

6. Atomic writes. Write to a temporary file, then rename it into place:

tmp = path.with_name(path.name + ".tmp")
df.to_csv(tmp, index=False, encoding="utf-8", lineterminator="\n")
os.replace(tmp, path)

Think of a shop repainting its sign. It doesn't scrape the old sign off and paint the new one in front of customers. It paints the new sign in the back room, then swaps them in one move. If the program crashes halfway, I'm left with a broken .tmp file, never a half-written CSV pretending to be complete.

7. A manifest with fingerprints.

{
  "generator_version": "1.0.0",
  "seed": 42,
  "generated_at": "2026-10-08T07:39:55Z",
  "files": {
    "bank_transactions": {
      "path": "data/generated/bank_transactions.csv",
      "rows": 59801,
      "sha256": "e48c4841bfe8815c..."
    }
  }
}

A manifest is the packing slip on a delivery: what's in the box, how many, and how it was made. The SHA-256 checksum is a fingerprint of the file's exact contents. Change one character and the fingerprint changes completely. When the next stage loads a file, it can record exactly which version it loaded.

Notice where generated_at lives: in the manifest, not in the data files. If the files contained the current time, two runs with the same seed would differ, and I'd lose the ability to prove repeatability. Metadata that changes every run belongs beside the data, not inside it.

8. Keep the answer key in its own drawer.

data/generated/truth/generator_truth.csv

The hidden scores live in a separate folder, so a "load every CSV in this folder" loop can't pick them up by accident. At work, the same thinking applies to anything sensitive: physical separation beats remembering to be careful.

And one rule outside the code: generated data never goes in Git. data/generated/ is in .gitignore. Anyone can recreate it exactly with one command, so storing it would only bloat the repository.

Part 5: Look at the data, even data you made

Running the generator:

make seed

Generated with seed 42 (generator v1.0.0)
  customers             1,000 rows  ->  data/generated/customers.csv
  bank_transactions    59,801 rows  ->  data/generated/bank_transactions.csv
  applications          1,000 rows  ->  data/generated/applications.csv
  generator_truth       1,000 rows  ->  data/generated/truth/generator_truth.csv
  manifest           ->  data/generated/manifest.json

make seed runs python -m generator.run --seed 42 --customers 1000. The -m flag runs the code as a module from the project root, which is what lets run.py import generator.build. Running python generator/run.py from the wrong folder is the classic cause of ModuleNotFoundError: No module named 'generator'.

Then I looked, because I never trust data I haven't looked at.

From the terminal:

wc -l data/generated/*.csv                                  # rows, plus one header line each
head -10 data/generated/bank_transactions.csv | column -s, -t   # first rows, lined up

From Python:

>>> import pandas as pd
>>> tx = pd.read_csv("data/generated/bank_transactions.csv")
>>> tx[tx["customer_id"] == "CUST-00001"].head(5)[["event_at", "amount", "description"]]
            event_at  amount                  description
2024-02-25T06:43:00Z 1770.00 SALARY ALDERWOOD HOSPITALITY
2024-03-01T09:00:00Z -835.00          RENT STANDING ORDER
2024-03-08T07:22:00Z -134.96           BLUEFLAME POWER DD
2024-03-16T07:08:00Z  -17.00              PULSE MOBILE DD
2024-03-28T06:04:00Z 1770.00 SALARY ALDERWOOD HOSPITALITY

Following one customer through time is the fastest sanity check I know. Paid at month end, rent on the 1st, bills mid-month. It looks like a life.

Then I found something odd. Counting transaction types:

>>> tx["description"].str.split().str[0].value_counts()
SALARY        15400
RENT          14541
...

About 860 more salaries than rent payments. Every customer pays rent monthly, so why?

The answer is in the rules, not a bug. Customers arrive on a random day. Anyone arriving after the 1st misses that month's rent, but still receives that month's salary if they arrived before payday. The generator did exactly what it was told. I just didn't know what I'd told it until I looked.

That's the lesson I'll carry to real work. Count rows, eyeball a few, follow one entity through time, and explain every number that surprises you before trusting it.

Part 6: "Repeatable" is a claim. A fingerprint is proof.

shasum -a 256 data/generated/bank_transactions.csv
make seed
shasum -a 256 data/generated/bank_transactions.csv

Both fingerprints are identical. Sixty thousand rows regenerated and not one byte changed. Then, with a different seed:

python -m generator.run --seed 43 --out /tmp/seed43
shasum -a 256 /tmp/seed43/bank_transactions.csv

A completely different fingerprint. The seed is genuinely in control.

Linux and WSL: use sha256sum instead of shasum -a 256. Windows PowerShell: Get-FileHash data\generated\bank_transactions.csv -Algorithm SHA256.

Part 7: Tests are promises. Fire drills prove they work.

make test

12 passed in 0.84s

Each test protects one promise, and its name says which:

Test The promise
test_same_seed_gives_identical_data Same seed, identical tables
test_different_seed_gives_different_data The seed actually matters
test_ids_are_unique No duplicate IDs anywhere
test_every_transaction_belongs_to_a_known_customer No orphan records
test_no_transaction_before_arrival_or_after_sim_end Nobody has a history before they existed
test_applications_have_enough_history At least 90 days of history before any application
test_hidden_truth_never_appears_in_visible_tables The answer key never leaks
test_salary_arrives_roughly_monthly Pay gaps stay between 24 and 37 days
test_v1_timestamps_are_all_equal Documents today's assumption on purpose
test_money_formatting_is_exact -125050 pence is exactly -1250.50
test_cli_writes_every_file_and_a_matching_manifest The manifest's counts match the real files
test_same_seed_gives_byte_identical_files Identical fingerprints on disk, not just in memory

The determinism test is short enough to show in full:

def test_same_seed_gives_identical_data():
    first = build_all(seed=7, n_customers=50)
    second = build_all(seed=7, n_customers=50)
    for name in first:
        pd.testing.assert_frame_equal(first[name], second[name])

Two pytest features worth knowing. pytest.ini tells pytest where things are:

[pytest]
pythonpath = .
testpaths = tests

And tmp_path, used in the file tests, is a fresh temporary folder pytest creates for each test and deletes afterwards. Tests never touch the real data/ folder.

The fire drill

A smoke alarm you've never tested is a smoke alarm you hope works. So I planted bugs on purpose.

I removed the arrival-date check from the generator:

# before
if d < p["arrival_date"] or d > SIM_END:
# after (bug)
if d > SIM_END:
FAILED tests/generator/test_generator_build.py::test_no_transaction_before_arrival_or_after_sim_end
1 failed, 11 passed

Exactly the right test failed, and its name told me exactly what broke. Then I removed the seed:

rng = np.random.default_rng()   # no seed: different every run
FAILED ...::test_same_seed_gives_identical_data
FAILED ...::test_same_seed_gives_byte_identical_files
2 failed, 10 passed

Both determinism tests caught it. I restored the file and was back to 12 passed.

The habit: name every test after the promise it protects, and watch each important test fail at least once. A test that has never failed hasn't proven anything yet.

Part 8: The Git workflow I'll use everywhere

Yesterday I learned Git's basics. Today I used the workflow almost every software team runs, and this is the part I'm most sure I'll use at work.

The daily start ritual

Every session starts the same way:

cd ~/projects/newcomer-credit-data-platform
source .venv/bin/activate
make check
git status
git pull

Go to the project, enter its Python environment, confirm the setup is healthy, check for leftover changes, and pull anything new from GitHub. Ten seconds of checks saves an hour of debugging something that was broken before I started.

Windows (native PowerShell): activate with .venv\Scripts\Activate.ps1. Most of this guide assumes a Unix shell, so on Windows I'd use WSL and the VS Code WSL extension, which runs VS Code against the Linux side. Setup: Install WSL and VS Code in WSL.

Branches: draft copies of the book

git switch -c feat/generator-v1

main is the published edition of a book. Nobody scribbles in the published edition. You write in a draft copy, get it reviewed, and merge it into the next edition. A branch is that draft copy.

git switch -c creates a branch and moves onto it. (Older guides use git checkout -b, which does the same thing.) The feat/ prefix is a common convention: feat/ for features, fix/ for bug fixes, docs/ for documentation.

Small commits that tell a story

Instead of one giant commit, I made three:

git add generator/
git commit -m "Add synthetic customer generator v1 (salaried, rent, two bills)"
git add tests/ pytest.ini
git commit -m "Add generator tests: determinism, integrity and leakage checks"
git add Makefile
git commit -m "Add make seed and make test targets"
git log --oneline

Each commit is one idea. If a reviewer, or me in three months, wants to know when the tests arrived and why, the history answers in one line. git add -A stages deletions as well as new and changed files, which matters when you remove placeholder files like .gitkeep.

Before every commit, two commands:

git status          # what's changed, and is anything there that shouldn't be?
git diff --staged   # exactly which lines am I about to commit?

Push, then open a pull request

git push -u origin feat/generator-v1

-u links my local branch to the one on GitHub, so later a plain git push is enough. If I forget the branch name, git push -u origin HEAD pushes whatever branch I'm on.

A pull request is the formal request: "please merge my draft into the published edition." On a team, this is where colleagues review the code. With the GitHub CLI:

gh pr create --title "Synthetic customer generator v1" --body "Adds a seeded generator for 1,000 synthetic customers, writing CSVs and a manifest with checksums. Includes 12 tests covering determinism, integrity and leakage of the hidden truth. Run with make seed and make test."
gh pr view --web

Without the CLI, GitHub shows a yellow "Compare & pull request" banner on the repository page right after you push a branch. Same result.

I read my own diff before merging. On the PR's "Files changed" tab, every line I'm about to add to main is visible. It's where leftover debug code, accidental files and secrets get caught.

Merge, tidy up, return to main

gh pr merge --squash --delete-branch
git switch main
git pull

--squash combines the branch's commits into one clean commit on main. Think of it as an editor merging three drafts into one tidy edition. --delete-branch removes the draft now that its work is published. Then I switch back to main and pull, so my laptop matches GitHub.

The same workflow in VS Code

Not everyone lives in the terminal, and VS Code can do all of this:

  • Branches: click the branch name in the bottom-left status bar, then "Create new branch."
  • Staging and committing: the Source Control panel (Ctrl+Shift+G, or Cmd+Shift+G on Mac). Click + next to a file to stage it, type a message, then click Commit.
  • Seeing diffs: click any changed file in Source Control to see before and after side by side.
  • Pushing: "Publish Branch" or "Sync Changes" in the same panel.
  • Pull requests: the official GitHub Pull Requests extension creates, reviews and merges PRs without leaving the editor.

The concepts are identical. Only the buttons change.

My Git rescue kit

The commands I'll want when something goes slightly wrong. All of these are safe as long as the work hasn't been pushed yet, unless noted.

Situation Command What it does
I changed a file and want it back as it was at the last commit git restore path/to/file Discards uncommitted changes to that file. They're gone, so be sure
I staged a file by mistake git restore --staged path/to/file Unstages it; the changes stay in the file
I want to undo my last commit but keep the work git reset --soft HEAD~1 Removes the commit; the changes go back to staged
I committed to main instead of a branch (not pushed) git branch feat/my-work, then git reset --hard origin/main, then git switch feat/my-work Saves the commits on a new branch, puts main back to match GitHub, and moves you onto the branch. --hard deletes uncommitted changes, so commit or stash first
My branch is behind main git switch main, git pull, git switch feat/my-work, git merge main Brings the latest main into your branch
I want to see what changed before committing git diff and git diff --staged Unstaged and staged changes, line by line
Old merged branches are piling up locally git branch -d branch-name Deletes a local branch, but only if it's merged

One rule overrides all of these: never rewrite history that's already been pushed and shared (no reset --hard or force-push on a shared branch) unless the whole team agrees. Undoing pushed work is done with a new commit (git revert), not by erasing the old one.

Part 9: Traps worth knowing before they bite

You see What it means Fix
ModuleNotFoundError: No module named 'generator' Wrong folder, or run as a file instead of a module cd to the project root and use python -m generator.run
ModuleNotFoundError: No module named 'numpy' Virtual environment isn't active source .venv/bin/activate
No module named generator.run even from the root The folder isn't a package Make sure generator/__init__.py exists (it can be empty)
pytest reports no tests ran It can't find tests Files must be named test_*.py, and pytest.ini should set testpaths
ImportError inside a test Tests can't see the project code pythonpath = . in pytest.ini
Fingerprints differ between two identical runs Something non-repeatable crept in Check for missing seeds, today's date, iteration over sets, or timestamps written into the data
data/generated/ shows in git status It isn't ignored Add data/generated/ to .gitignore
zsh commands stop working and the prompt is >>> You're still inside Python exit()
fatal: The current branch has no upstream branch First push of a new branch git push -u origin HEAD
gh: command not found (Linux/WSL) GitHub CLI isn't installed Follow the official install guide
python: command not found (Linux/WSL) Some systems only have python3 outside a virtual environment Use python3, or activate the environment first, where python always exists

The lessons I'm carrying forward

  • AI writes generator code quickly. My value is the specification and the verification.
  • Tell the AI what must never exist in the data, not just what should.
  • A seed makes randomness repeatable, and the order of random draws matters.
  • Keep logic and file handling apart, so logic can be tested alone.
  • Anything repeatable uses fixed dates, never "now."
  • Money is whole minor units until the moment it's displayed.
  • Every timestamp carries a time zone.
  • Design columns for where the data is going, not just where it is.
  • Sort before numbering, and IDs stay stable.
  • A file is a contract: fixed columns, ISO formats, no index column.
  • Write atomically: temporary file, then rename.
  • A manifest with checksums turns "I think it's the same" into proof.
  • Metadata that changes every run belongs beside the data, never inside it.
  • Keep the answer key in a separate drawer.
  • Never commit data that code can regenerate.
  • Look at the data and explain every surprising number.
  • Name tests after promises, and watch each one fail at least once.
  • Never commit to main. Branch, small commits, pull request, read the diff, merge, pull.
  • Never rewrite history that's already been shared.

A checklist for the next time I need test data

At work or on a new project, this is the sequence:

  1. Write the specification first: entities, keys, fixed time window, value ranges, business rules, forbidden attributes, hidden truth, output contract.
  2. Give it to an AI along with the prompt template from Part 2, and ask for tests in the same request.
  3. Read the code against the trap table in Part 2 before running it.
  4. Generate a tiny version (--customers 3) and read the whole file by eye.
  5. Generate the full version and follow one entity through time.
  6. Explain every count that surprises you.
  7. Prove repeatability by comparing checksums across two runs.
  8. Plant at least one bug and confirm a test catches it.
  9. Branch, commit in small pieces, open a pull request, read the diff, merge.

What's next

The data exists, but it's just files. The next stage of the Newcomer Credit Data Platform loads it into PostgreSQL: creating the raw tables, writing a loader that can run twice without creating a single duplicate, recording every load against the manifest's fingerprints, and querying the synthetic town with SQL for the first time. Same format, written up the same way.