Skip to main content
Vexlo bounds shape and values on two separate clocks in two different units. Shape (traces, steps, attempts, tags) is kept for a number of days, 7 by default. Values (the captured bytes behind each slot) are kept against a byte budget, 20 GiB by default. Nothing ties the two together, so they do not run out together. Values go first, always. The byte budget only ever removes values, and a trace keeps its shape until its own day count passes, so a trace can lose every value it was captured with and still answer every other question about itself. The default configuration surprises people, so here is the arithmetic of it. The default 20 GiB covers a 7-day shape count only below about 1.24 traces per second: vexlo check prints 168h58m of value retention at 1.23 traces/s and 167h37m at 1.24. Above that rate the budget runs dry first, and the check exits 1 rather than let the pair pass. At the 2 traces/s it assumes by default, the budget holds 103h55m, a little over four days against a seven-day shape count. At the 30 traces/s of a busy service it holds 6h55m, and then a trace from yesterday is still searchable, still has every step, every timing and every status, and answers expired for the input you wanted to read. A reader who was told “7-day retention” and goes looking for yesterday’s inputs is right to read that as data loss. Flags are spelled with a single dash, as -h prints them; the binary accepts both spellings.

​
The two clocks

vexlo serve prints the first two on startup, and they are the only retention settings it prints:
One 10-minute tick runs three passes, each bounded to 1,000 rows per table or 1,000 value files so ingest is never stalled behind it. The shape sweep deletes a trace and its children once its first record reached the server more than the day count ago, measured on the server’s own clock rather than on the started_at the SDK declared: the trace stops being searchable and its id returns 404. The same pass applies the function index’s own 90 days. It takes one chunk per tick rather than looping until a table is drained, so at 1,000 rows a table it clears about 26 traces a tick at the 38 slot rows the captured trace below carries, and a trace past its day count stays searchable until a tick reaches it. The byte-budget pass removes whole oldest hour segments of the value store until usage is back under the budget, never a partial segment and never a single file. A value’s segment is the hour its trace was first seen, so one trace’s values go together. The third pass deletes value files no live slot row points at any more, skipping any file written less than an hour ago: that is how the values of a trace the shape sweep has already deleted leave the disk. Both of the passes that free value bytes run on that one tick, so between two ticks nothing frees any, and “oldest first” describes the byte-budget pass rather than the whole story. Ingest checks the same byte budget on every value it commits, and once its own measurement has reached that budget it commits the slot and discards the bytes. Between two ticks, then, the values being discarded are the newest ones. That measurement is a cached directory walk refreshed at most once every 5 seconds, so the discarding can also outlast by that much a pass that has already freed the space. The 1,000-file bound is what the byte-budget pass runs into first. It takes a segment whole or not at all, so once the oldest hour holds more than 1,000 value files that pass frees nothing: it reports the segment as stuck, logs a warning, and leaves the store over budget. No flag moves the bound. The trace captured further down put 28 value files on disk by itself, so an hour holding more than 35 traces like it is already past the bound, and 2 traces/s is 7,200 traces an hour. What a reader meets on such a server is not oldest-first reclamation. The store stays over budget, ingest discards the bytes of every value it commits, and GET /v1/health answers 503 with mode: "shape_only". The pass that can still thin a stuck segment is the unreferenced one, and it follows the day count rather than the budget, because a value file stops being referenced when the shape sweep deletes the trace that held it. It deletes at most 1,000 files a tick, 144,000 a day, against the 4.8 million a day that 2 traces/s at 28 files a trace would write. Ingest refuses a trace-open whose declared started_at is older than the configured shape retention, so the same day count also bounds how late a backfilled trace can arrive. The byte budget is one number for the whole server, not one per team. One team’s traffic spike shortens every other team’s value horizon on the same instance.

​
What the byte budget buys

vexlo check resolves the configuration vexlo serve would run with, then prints the arithmetic. It opens neither the database nor the value store, so it has no traffic to measure and takes the rate as a flag. -expected-write-rate defaults to 2 traces/s:
The effective figure is the budget divided by the value bytes one trace costs times the rate: 21474836480 / (28700 x 2) seconds, the 103h55m printed above against a configured 168h. Of the 31,000 bytes a fully valued trace costs, 28,700 are its values, which is why the budget is denominated in bytes and why value sampling is the lever that moves it. Pass a production rate and the gap widens:
Just under 7 hours of values against a week of shape. Five things about that number matter before you plan against it:
  • The per-trace figure is an assumption, not a measurement. The 2,300 and 28,700 byte figures are fixed constants in the binary, a worked example of a 40-step trace with 2.2 valued slots per step and a 900 B mean value after gzip and deduplication. vexlo check never measures your traffic, and it has no flag for a measured average.
  • The rate is whatever you pass. -expected-write-rate’s own help text says vexlo check has no live traffic to measure it from, so the horizon is only as good as the rate you supply.
  • The arithmetic assumes every trace keeps its values. There is no sampling input. Lowering value_sample_rate through a capture policy stretches the real horizon past what the figure says, and nothing reconciles the two for you. It stretches it by excluding traces, and an excluded trace’s ok attempts read expired on a fetch like a reclaimed value.
  • The arithmetic counts bytes and never files. vexlo check knows nothing about the 1,000-file bound each reclaim pass runs under, and on a server whose hour segments hold more than that, the bound is what decides what happens when the budget fills.
  • vexlo serve applies no overcommit check of its own. It starts on the configuration vexlo check rejects, and prints nothing about it beyond the startup line above.
GET /v1/health does not close that gap. Its value_store.effective_retention_estimate_hours field reads 0, because the route has no write-rate input to compute an estimate from, and its shape_store.effective_retention_days field reads 7 whatever -shape-retention-days is set to. The run below is a server started with -shape-retention-days 3650:
bytes_used is a fresh walk of the value store on every call, and byte_budget is the configured flag echoed back. Neither retention field is computed from anything, so vexlo check is the only place either horizon is worked out.

​
What a read returns once the bytes are reclaimed

The byte budget reclaims bytes. It never touches the shape store, and it never rewrites a slot’s recorded state. Those two facts together are what a reader runs into. Below is one step of a trace read twice from one server storage directory, before and after the byte-budget pass reclaimed that trace’s 28 value files. Nothing was reingested and no shape write happened in between. Forcing that pass took one change, and it is worth naming rather than implying a week passed: the server was stopped and restarted on the same storage directory with -value-budget-bytes 1, which puts every byte already on disk over the budget, and the first retention tick ten minutes later removed the lot. The step below is the one a debugger would open, the one whose input produced the error:
The shape read of that same step is byte-identical across the two reads. It still reports stored, the size the bytes were, their encoding, and the 64-code-point preview:
That is not a stale cache. A trace-detail read answers from the shape store alone and never looks in the value store, and preview was computed at capture time and travels on the step-attempt record rather than with the bytes. So the state on a shape read is the state the SDK reported, and GET /v1/traces/{id}/values is the one read that reports expired at all. One other route reaches the bytes, and it reports their absence differently. GET /v1/functions/examples?include=values reads the same value store, and on a miss it returns that example’s handle as the shape store recorded it, with no value field beside it. It never answers expired. Below is the same step read as one example of its own function, either side of one byte-budget pass, from a second run of the scenario above on its own port and its own storage directory. The first read carries no preview, because the route clears it wherever it sets value; the output handle is empty on both because that step has no output slot.
The state is the same word in both reads. On that route the missing value field is the only signal, so do not read a handle’s state there as an answer about the bytes. Plan on all of that: shape says stored until the day count deletes the step, and a fetch through the values route is the only way to learn whether bytes are there.

​
expired and missing

Both are handle states a reader sees only at read time; neither is ever emitted by the SDK. They describe different events: Telling them apart takes a store that records a value record’s arrival. The shipped SQLite store keeps no such record: the bytes are the only evidence a slot ever had any. Every slot row carrying a content id is read against the value store, and a miss resolves to expired whatever state the SDK reported, so this store never returns missing at all. Five different events reach a reader as expired, and only the first is the one the table above describes:
  1. The byte budget reclaimed bytes that were there. This one leaves no trail. GET /v1/health reports the value store’s current bytes_used against its byte_budget and keeps no record of what a pass reclaimed. On the first of the two runs behind this page, that route answered 200 with mode: "normal" and bytes_used: 0 seconds after the pass had reclaimed all 28 of the trace’s value files.
  2. A value record never arrived, because its export failed for good after the step-attempt record had already shipped. No wire state records that, since a record is written once and never revised. The undelivered record is counted as loss, by the SDK in that trace’s own records_lost or by the relay daemon in the per-service records_lost_24h, and neither count says which slot it was: records_lost > 0 does not imply a missing step, attempt or value.
  3. The value arrived while the value store was already over budget, so ingest committed the slot and discarded the bytes. Shape keeps being ingested under that pressure, and this is the one of the five the API names while it is happening: GET /v1/health answers 503 with mode: "shape_only", captured here from a server whose budget was set to one byte. The one-byte budget is a way to force it on demand, not what makes it reachable. Ingest tests the same byte budget the byte-budget pass works against, so a spike that fills the budget between two ticks discards values for the rest of that window, and the traces reading expired afterwards are the newest ones rather than the oldest.
  4. No value record was ever owed. Two things end there. The sampling roll excluded the trace, in which case its root attempt ships sampled_out and its ok non-root attempts keep the pending they shipped with, while any error attempt on that same trace keeps its values whatever the rate is. Or the trace’s root never returned, and the SDK holds a trace’s values until its root closes, so they were never shipped and never will be.
  5. The outbox evicted the bytes before the step-attempt record was built. That slot ships dropped and gets no value record either.
So on this server, expired asserts only what those five share: the slot’s own row carries a content id, and no bytes are behind it now. It does not assert that the server ever held them, and it does not mean the function returned nothing. A durable value-arrival record is what it would take to separate the first case from the rest, and the shipped store has none. Causes 2, 4 and 5 need neither a pass nor pressure, and a reader meets them on an idle server. Below is a values read of a second trace, posted from the conformance fixture that carries one step per handle state, on a server holding 1,678 bytes against the default 20 GiB budget. The root’s two slots shipped sampled_out, AADA’s shipped dropped, AABA’s shipped pending, and AAAQ’s shipped stored with no value record behind them. All seven read the same thing:
A shape read is where the difference survives, because a read miss rewrites the state only on the values route. The trace read of that same trace still reports sampled_out on the root, dropped on AADA and pending on AABA. A slot whose bytes never formed keeps its own state on both routes, since no content id sends it to the value store: redacted, too_large and not_captured answer as themselves. So shape separates sampled_out and dropped from a reclaimed value. It does not separate pending. pending is the state that says least. A slot the SDK reported as pending keeps it in the shape store for good, because a record is written once: on the first trace above, the two slots of the step at path AABA read pending while their bytes were there, and the values route returned the bytes anyway. After the pass both read expired. So pending says nothing about whether bytes are available, on either route. Where no value record was ever owed, case 4 above, pending is also the state the slot keeps for good.

​
The function index keeps 90 days

The function index (observed callers and callees per function, with counts) runs on its own 90-day clock and has no flag. Its rows outlive shape: a row is deleted 90 days after the last time that caller/callee pair was observed, so GET /v1/functions/detail still answers for a function last called weeks ago. What does not survive is the example trace ids on that row, which are deleted as soon as their trace is, well inside 90 days at a 7-day shape retention. An index row with no reachable example is the ordinary state of this index, not a fault.

​
If you want values to last as long as shape

Two flags move the two horizons: raise -value-budget-bytes until the arithmetic clears, or lower -shape-retention-days until it does. Either makes vexlo check exit 0 and print OK instead of OVERCOMMITTED. Both clear the byte arithmetic alone: neither moves the 1,000-file bound, so a server whose hour segments hold more than that still reaches its budget around the horizon the check prints and then stops storing new values. The cheap direction is to stop promising a week of shape you have no values for. At 2 traces/s the default budget holds 103h55m, so -shape-retention-days 4 (96h) clears it with no extra disk:
Keeping the week costs disk instead. Covering 7 days of values at 2 traces/s takes 28700 x 2 x 604800 bytes, about 32.3 GiB:
At 30 traces/s the same 7 days takes 15 times as much, about 485 GiB. That is more free disk than the filesystem in this run had. vexlo check prints that as a warning above the arithmetic and still exits 0: free disk is informational there and never fatal.
Half a terabyte to keep a week of values at 30 traces/s is the real price of the default day count, and it is why lowering value_sample_rate is the third lever. Value sampling is decided once per trace, on the trace id, and only ever excludes the values of a trace that closes ok: an attempt whose own status is error keeps its values whatever the rate is. So the saving depends on how many of your traces close ok, the figure vexlo check prints does not account for it, and a sampled deployment has to redo the division itself. That lever is paid for on reads rather than on disk. On an excluded trace, the root’s slots and every slot of its ok attempts answer expired, in the same word a reclaimed value gets, so a lower rate buys horizon by making more traces read as though their values had been taken away. The last option is to accept the two horizons. -allow-overcommit makes vexlo check exit 0 on an overcommitted configuration and changes none of the arithmetic it prints: the budget fills at the effective figure, shape lasts until the day count, and a fetch in between reads expired. What that costs depends on the file bound. Below it the store reclaims oldest first, so the traces reading expired are the oldest ones. Above it the store stops storing values instead, so the traces reading expired are the ones arriving now, and the operator accepting the overcommit is accepting a shape-only server from the moment the budget fills.