vexlo relay run, is the one process that talks to the Vexlo
server: it tails the spool directory, batches and gzips what it finds, posts it to
POST /v1/records, sends a heartbeat per live writer (one that has written in the current or
previous UTC hour), and writes the capture policy the server answers with back into the spool
directory for the SDK to read. An instrumented process therefore never blocks on network I/O
to get a record captured, and an unreachable server becomes a backlog on disk, up to the spool
directory’s byte budget (-spool-budget-bytes, default 2 GiB). One daemon serves one spool
directory, one destination, and one token, so a host runs one daemon unless it splits services
or environments across spool directories. The ingest route’s own limits are on
Ingest.
Flags are spelled with a single dash, as -h prints them; the binary accepts both spellings.
Start it
Two flags are required:-destination (or VEXLO_ENDPOINT), the server’s base URL, and
-token (or VEXLO_INGEST_TOKEN), an ingest token. Use the environment variable for the token
outside a one-off manual run; a flag value is visible in the process list.
0700 and
corrects it (the directory here was created at 0755 first). The second names seven settings:
the spool directory, the destination, the environment, the three intervals, and the health
address. The other eight flags, -token and -spool-budget-bytes among them, are not logged.
-destination-file and -token-file change nothing at startup; they name the two files a
SIGHUP reload re-reads, covered in
Reload the destination or token. Ctrl-C (SIGINT) or
SIGTERM stops the daemon; the last line is msg="relay run: shutting down".
Without a token the daemon does not start:
The destination must be https or loopback
A plaintexthttp:// destination is refused unless its host is 127.0.0.1, ::1, or
localhost, matched as a string once it is lowercased and a trailing dot is dropped, never
resolved through DNS. The refusal names the ways forward:
-allow-insecure-http (or VEXLO_ALLOW_INSECURE_HTTP=1) accepts any http:// destination and
is meant for a same-network development server. The same rule governs vexlo policy set and
vexlo policy show.
-environment
-environment <name> (or VEXLO_ENVIRONMENT) is sent as X-Vexlo-Environment on every request
the daemon makes. Environment is declared here, once per daemon, never by the SDK: a host that
serves two environments runs two daemons against two spool directories. The value is 1 to 64
printable ASCII characters with no comma, checked at startup:
-environment rejects a batch that carries no header, so a daemon forwarding
under a pinned token runs with -environment set to the pinned value; see
Authentication and tokens.
The spool directory
The daemon and the SDK share one directory,-spool-dir (or VEXLO_SPOOL_DIR), default
~/.vexlo. After one demo run and one daemon pass it holds:
Every directory is created at
0700 and every file at 0600, whichever side creates it first.
At startup, and again on every retention sweep (-retention-interval, default 1 minute), the
daemon re-asserts 0700 on the spool root and on spool/, cursor/, writers/, and
policy/, and logs relay: corrected directory permissions for each one it changed. A
chmod that moves one of those five off 0700 is reverted within one sweep. The modes
after this run:
What it sends
The daemon polls the spool directory every 250 ms (-poll-interval) and sends what is ready as
one batch of at most 512 records or 1 MiB, whichever binds first. A batch is a gzip-compressed
JSON array of records with Authorization: Bearer <token> and, when -environment is set,
X-Vexlo-Environment. The server answers 204 when every record is accepted, or 200 with a
rejected list naming the records it refused. A record in that list is terminal: it is counted
as dropped and never resent.
A connection failure, a 429, and 500, 502, 503, or 504 are retried, 3 attempts in
all, with a full-jitter backoff starting from a 200 ms ceiling and doubling per attempt; a
429’s Retry-After is added to the delay and capped at 60 seconds. Every other status is
terminal on the first response: the whole batch is counted daemon_terminal_reject, never
resent, and the daemon logs send batch: terminal rejection with the status. 400, 413,
and 422 fall here, and so do 401 and 403, so a revoked or mistyped token drops batches
rather than building a backlog. Rotate a token in that order: write the new one into
-token-file, SIGHUP the daemon, then revoke the old one. Every batch the daemon sends while
it holds a revoked token is dropped.
Every 30 seconds (-heartbeat-interval) the daemon sends one POST /v1/heartbeat per live
writer, carrying that writer’s records_sent and dropped counters. A writer is live when it
has a spool file in the current or the previous UTC hour and an identity file on disk. A writer
that stops writing stops heartbeating, and once every writer of a service has stopped, nothing
refreshes that service’s capture policy file until one of its processes writes again. The
response is the capture policy for that writer’s service, which the daemon writes to
policy/<service>.json;
Capture policy covers what the SDK does with it.
Reload the destination or token
SIGHUP reloads the destination and the token, and nothing else. The daemon cannot re-read its own flags or environment (a running process’s copy of both is fixed at start), so a reload reads files:-destination-file (or VEXLO_ENDPOINT_FILE) and -token-file (or
VEXLO_INGEST_TOKEN_FILE) each name a file holding one bare string, trimmed of surrounding
whitespace. Neither affects startup; -destination and -token still start the daemon. Write
the current destination and token into those files before the first signal: a reload that
cannot read a configured file is refused at ERROR and the live destination and token stay
in place. On SIGHUP the daemon re-reads whichever files are configured, validates the result
(the https-or-loopback rule included), and swaps the new values in. A batch already in flight
finishes against the old values. The log names which value changed, never the value. Two
signals, the second after token.txt was rewritten:
Health and metrics
The daemon servesGET /healthz and GET /metrics on -health-addr (or VEXLO_HEALTH_ADDR),
default 127.0.0.1:9101. Neither route is authenticated, and neither carries a service name, an
instance id, a path, or a value: every field is an aggregate count. The daemon below bound
127.0.0.1:19100.
/healthz is 200 with {"status":"healthy"}, or 503 with a reason, when one of three
conditions holds: unsent records exist and the server has accepted nothing in the last 5
minutes (a daemon that has never had a send accepted counts as stale from its first backlog),
the spool directory holds more than 80% of its byte budget, or reading the spool directory’s
own state failed or ran past 5 seconds. An idle daemon with no backlog is healthy. Match an
alert on the status, not on the reason text: the first two reasons carry the measured
numbers, the third is the fixed string spool status unavailable.
relay: healthz spool status failed with the underlying error at the same moment:
/metrics is Prometheus text exposition with eight metrics. The three counters
(records_sent_total, records_dropped_total, panics_recovered_total) and
last_send_success_timestamp_seconds are process-lifetime state that a restart resets to
zero, so a zero there means “nothing since this process started”, never “nothing ever”. The
backlog, backoff, and spool-byte gauges are read from the spool directory on each scrape, and
spool_budget_bytes reports the configured -spool-budget-bytes. Every metric is present
from the first scrape, at zero if nothing has happened yet.
vexlo relay status
vexlo relay status reads the same files a running daemon would and prints one summary per
writer. No daemon has to be running. Before the daemon’s first pass over this spool directory:
oldest unsent age is printed only while a backlog exists, and is measured from the start of
the oldest hour that still holds unsent records (records carry no parsed timestamp), so it
overstates the oldest record’s age by up to an hour. A writer with no spool file in the
current or the previous UTC hour shows state: paused, the condition that also stops its
heartbeat; a writer with no identity file shows
identity: missing (no heartbeat can be attributed). The command repeats the daemon’s own
startup warning when the configured spool budget exceeds the free space on the spool
filesystem.
Flag reference
-spool-budget-bytes, and -max-writers do not.
vexlo relay status takes -spool-dir and -spool-budget-bytes only.