Production replay

The promiserecord once · replay anywhere

While serving, a Tin server writes a capsule for every request that ends in a 5xx or a panic: the request bytes plus the ordered log of everything the handler asked the outside world. tin replay runs that request again from the capsule.

# production: record failures into a spool (encrypted with the key)
TIN_REPLAY_DIR=/var/spool/tin TIN_REPLAY_KEY=$(openssl rand -hex 32) ./app

# your laptop: the same request, same answers from Redis, the clock, the payment API
TIN_REPLAY_KEY=... tin replay 00001700000000000000-003-41.tcap --against app.tin

• No Redis, no database, no upstream, no network: every effect is answered from the capsule.
• The build you replay against can be newer than the one that recorded: that is the point.

What it solvesproduction → laptop

The bugs that are expensive today are the ones you cannot reproduce. A capsule is the reproduction.

Switchesread once, before the cores start

Recording is on when both the spool directory and the key are set. A malformed key prints one line and leaves recording off.

# Kubernetes: a secret for the key, an emptyDir for the spool
env:
  - name: TIN_REPLAY_DIR
    value: /spool
  - name: TIN_REPLAY_KEY
    valueFrom: { secretKeyRef: { name: tin-replay, key: key } }
  - name: TIN_REPLAY_SAMPLE
    value: "0.001"

• Requests ending in a 5xx or a panic are always kept; the sample adds a slice of the rest.

What is recordedkind@version · key · outcome

An effect is anything whose result does not follow from the program and its input. Each one is a record: its kind, what was asked, and what came back, in the order they completed on the core.

• The request itself: method, path, headers, body as received (secrets as handles).
• DNS is inside wire.http and wire.dial: replay performs neither.

// what one record holds (the tape and the capsule share the layout)
word   seq        0, 1, 2 ... completion order on the core
string kind       "redis@1"
string key        "3:GET 6:cart:7\n"           secrets as handles
word   outcome    0 result, 1 fault
word   ident      the runtime sentinel in a fault's chain, or 0
string data       the encoded result, or the fault's message

CapsulesWALL-CORE-N.tcap

When a request task ends, its tape becomes a capsule: written on the core, after the task, via a .tmp file and a rename, so a reader never sees a partial file.

spool/
  00001700000000000000-000-17.tcap     # wall clock, core, sequence
  00001700000000000000-003-41.tcap

• The body (request, status, panic message, effects) is encrypted with an HMAC-SHA256 keystream and authenticated by a tag: a wrong key or a changed byte is refused.
• The spool is bounded; the oldest capsules are deleted first, at start and per core.
• Shipping capsules anywhere is operations, not the runtime.

Secrets never reach a capsuletin-secret:…

A secret header, a secret value in a client call, or a secret written into a query is stored as a keyed handle, never its text. Logins happen inside the live call and are not recorded.

// stored in the capsule
Authorization: tin-secret:3f9c0a7e2d1b4c5a6e8f9d0c1b2a3f4e

// a secret in a query: sent to Redis as itself, stored as its handle
fn session(token secret str) ! {
	try cache.Do("GET session:{token}")
}

• A handle is tin-secret: plus 16 bytes of HMAC-SHA256 under a key derived from TIN_REPLAY_KEY.
• On replay the handler reads the handle as the header's value; it hashes to itself, so recorded keys still match.

tin replaytin replay CAPSULE --against BUILD

BUILD is a binary or a .tin file that tin builds first. Its Serve does not listen: it opens the capsule, checks the tag, and sends the recorded request once through the handler or router on one core.

TIN_REPLAY_KEY=$KEY tin replay spool/00001700000000000000-003-41.tcap --against app.tin

replay: status 500 (recorded 500)
charge failed: payments answered 503

• Run it with the same configuration the server recorded under: effect keys hold what the program sent, so another upstream URL is a divergence.
• Every effect gets its recorded result or fault; none is performed.

tin replay cap.tcap --against app.tin --live wire.http

Divergencenever a live call by accident

Replay stops at the first effect whose kind or key differs from the next recorded one, or that comes after the last. From then on every effect of the request fails with the divergence.

replay: status 500 (recorded 500)
replay: divergence at effect 0: got wire.http@1 "POST https://pay.example/charge ...", recorded redis@1 "3:GET 6:cart:7"
replay: 2 recorded effects not served

• A different call, a different order, or an extra call all diverge. Replay never falls through to the network.
• Divergence is information: the new build asks the world a different question than the old one did.

Scheduling replays tooselect · resume order · cancels

Tasks on a core switch only at waits, so with the events recorded a request's concurrency replays exactly. The winning select arm, the order tasks resumed in, and each cancel with its reason are on the tape.

select {
	let r = fast.wait() => use(r)       // recorded: this arm won
	after(50ms) => fail "slow"          // replayed: this arm loses, whatever the clock says
}

• A recorded 5 ms deadline replays under a 10 s one and the other way round: deadlines come from the tape, not the clock.
• Children whose effects complete in another order than issued get their results in the recorded order.
• A replay in which no task can take the next record diverges instead of hanging.

From capsule to regression test--save-test NAME --issue N

When a replay against the fixed build exits 0, one flag turns the capsule into a case CI replays on every change.

tin replay cap.tcap --against fixed.tin --save-test checkout-total --issue 242

tests/regressions/checkout-total.tin     # a copy of fixed.tin
tests/regressions/checkout-total.tcap    # the capsule, sealed again under a public test key
tests/regressions/cases.json             # + an entry with the expected report and body

• The runner sets the replay switches, runs the program and checks the exit status and output.
• The case names its capsule and the public key it is sealed under; the production key never leaves production.

{
  "source": "replay-checkout-total.tin",
  "issue": 242,
  "replay": { "capsule": "replay-checkout-total.tcap", "key": "6663…8989" },
  "expected": { "phase": "run", "exit": 0,
                "stdout": "replay: status 200 (recorded 500)\ntotal 3800\n" }
}

Record a failure, replay it offlineexamples/checkout.tin

POST /checkout/{id} reads the cart from Redis and charges it. The payment service declines, the request answers 500, and anvil keeps the capsule.

fn checkout(q anvil.Req, w mut anvil.Out) {
	let id = q.PathParam("id")
	let (items, found) = cache.Get("cart:" + id) catch err {
		w.Status(502)
		w.Text("cart: {fault.Message(err)}\n")
		return
	}
	if !found {
		w.Status(404)
		w.Text("no cart {id}\n")
		return
	}
	let r = wire.Post(quarry.Getenv("PAYMENTS_URL") + "/charge", "text/plain", "cart=" + items) catch err {
		w.Status(500)
		w.Text("charge failed: {fault.Message(err)}\n")
		return
	}
	if r.Status != 200 {
		w.Status(500)
		w.Text("charge failed: payments answered {r.Status}\n")
		return
	}
	w.Text("paid {r.Body} for {items}\n")
}

# 1. record, against a real Redis and a declining payment service
REDIS_ADDR=127.0.0.1:6379 PAYMENTS_URL=http://127.0.0.1:9999 \
TIN_REPLAY_DIR=spool TIN_REPLAY_KEY=$KEY tin examples/checkout.tin
curl -X POST localhost:9186/checkout/7          # -> 500 charge failed: payments answered 503

# 2. stop Redis and the payment service. Replay:
TIN_REPLAY_KEY=$KEY PAYMENTS_URL=http://127.0.0.1:9999 \
tin replay spool/*.tcap --against examples/checkout.tin
# replay: status 500 (recorded 500)
# charge failed: payments answered 503

# 3. fix the handler, replay against the fix, keep it as a test
tin replay spool/*.tcap --against examples/checkout.tin --save-test checkout-total --issue 242

What replay does not dohonest limits

Replay reproduces a request, not a process.

• Background work is outside it: detach tasks, ticks, relay handlers and on handlers have no tape.
• A later build must ask for the same effects in the same order; a refactor that reorders calls diverges, by design.
• A capsule replays on later Tin versions while its effect schemas are supported; an unsupported kind is a clear error, not a wrong replay.
• Capsules are stored locally and bounded; shipping, retention and access are yours to operate.
• Packages with the unsafe capability can reach the OS behind the hook and are marked not replay-safe.