Deploy the fleet

Every role runs from the same Docker image, and the command picks the role. Your orchestrator decides how many workers run and where. The control plane decides which job runs on which worker.

Before you begin

• Docker, on a linux/amd64 host. Chrome for Testing is published for x86-64 only; on Apple silicon Docker emulates it, which is slow but works for a trial.
• An S3-compatible bucket for the archive, or a directory for a single machine.
• The spider versions you want to run, built and tested with graphdom build.

Build the image

docker build --platform linux/amd64 -t graphdom .

The image is about 700 MB, most of it Chrome. It holds Node.js 24, Chrome for Testing and Graphdom’s source, runs as user node (uid 1000), and its entry point is graphdom:

docker run graphdom fleet serve --tokens /etc/graphdom/tokens.json

Try it with Compose

The repository’s docker-compose.yml starts a control plane, two workers with two Chrome slots each, and MinIO for the archive.

docker compose up
curl localhost:7400/healthz

Its tokens (admin-dev-token, worker-dev-token) and MinIO keys are for a laptop. Override them in an .env file with ADMIN_TOKEN, WORKER_TOKEN, S3_ACCESS_KEY and S3_SECRET_KEY.

Configure the control plane

The control plane needs a tokens file and a place for its database.

graphdom fleet serve --tokens tokens.json --db /data/fleet.db --host 0.0.0.0 --port 7400

The tokens file declares the users the control plane starts with, each with its roles, and the user each bearer token acts as:

{
  "users": {
    "ops": { "name": "Operations", "roles": ["admin"] },
    "giulia": { "name": "Giulia Neri", "email": "giulia@acme.it", "roles": ["approver"] },
    "desk": { "name": "Service desk", "roles": ["operator"] },
    "workers": { "name": "Workers", "roles": ["worker"] }
  },
  "tokens": {
    "3f8c…": "ops",
    "9a1e…": "giulia",
    "c27d…": "desk",
    "5b04…": "workers"
  }
}

Users are kept in the control plane’s database, and the file’s are put there again at every start, so editing a user’s roles in the file and restarting changes them. An admin can also add users and change their roles through the API (PUT /users/<id>). Approvals, hand-over decisions and jobs record the id of the user who made them. The control plane refuses to start when a token names a user it does not have.

Role	Can
admin	Everything: configure sites, accounts and spiders, add and test versions, submit jobs
approver	Approve a tested version
operator	Read sites, jobs, runs, drift and the audit log; take over and resume or abort hand-overs
worker	Claim jobs and report on runs. Give it to workers only.

Put the database on a volume. The control plane is one process with its own database: run exactly one replica, and replace it rather than rolling it.

Configure the workers

graphdom fleet worker \
  --control http://graphdom-control:7400 \
  --vault /vault/vault.json \
  --storage "s3://graphdom/archive?endpoint=https%3A%2F%2Fs3.internal" \
  --slots 2 --host 0.0.0.0 --port 7401 --advertise http://10.0.3.17:7401

Pass the worker token in GRAPHDOM_TOKEN rather than --token, so it stays off the process list.

Option	Use it to
--control <url>	Reach the control plane
--vault <file>	Read secrets. See below.
--storage <dir | s3://bucket/prefix?endpoint=url>	Store the archive of every run
--slots <n>	Run this many Chromes at once. Default 2; budget about 1 CPU and 1 GB per slot.
--id <name>	Name the worker in runs and logs, such as the pod name
--host, --port	Serve the live endpoint that hand-overs use
--advertise <url>	Tell the control plane where it reaches this worker’s live endpoint
--spool <dir>	Keep runs here until they are pushed to storage

The vault

The vault is a JSON file of secrets by name. Spiders, accounts and storage refer to these names, never to the values.

{
  "portal/ops@acme.it/password": "…",
  "portal/ops@acme.it/totp": "JBSWY3DPEHPK3PXP",
  "storage/access-key-id": "…",
  "storage/secret-access-key": "…"
}

A worker reads the file again on every lookup, so a rotated password needs no restart. It refuses a vault other users can read: mount it read-only, owned by uid 1000, mode 0400 or 0600. S3 credentials come from storage/access-key-id and storage/secret-access-key, or else from the usual AWS_* variables.

Network

From	To	For
Worker	Control plane :7400	Claiming jobs, heartbeats, reports
Control plane	Worker :7401	A hand-over’s live browser
Worker	Object storage	The archive of each run
Worker	The sites	The runs, through each site’s proxy when it has one
Prometheus	Control plane /metrics	Scraping

Everything is plain HTTP. Terminate TLS in front of the control plane, and keep /metrics and /healthz inside the cluster.

Kubernetes

docs/DEPLOY.md in the repository has a starting manifest: a control plane Deployment with one replica and a volume, a Service, and a worker Deployment. It is a starting point, not a tested chart. Three settings matter:

• Chrome’s own sandbox is off. It needs user namespaces, which Docker’s default seccomp profile and Kubernetes’ restricted pod security deny. The container is the sandbox: run it non-root, drop every capability, and give each run its own profile.
• Reap Chrome’s helpers with shareProcessNamespace: true, or an init process in Compose.
• Give /dev/shm room, an emptyDir with medium: Memory of 1 Gi.

Scaling

Workers need no coordination: add replicas. The scheduler still enforces each site’s limits, so more workers only help when jobs wait for a free slot, not for a site’s interval or window. GET /jobs?state=queued shows why each job waits.

To autoscale, scale workers on sum(graphdom_queue_depth), for example with KEDA and about one replica per four queued jobs. Cap the replicas at what the sites’ limits make useful.

Stopping a worker

On SIGTERM a worker stops claiming, lets its runs finish for up to 25 seconds (GRAPHDOM_DRAIN_MS), then hands the rest back to the queue. A run that may have sent a submit is not handed back: its job ends failed for a person to check. Set Kubernetes’ terminationGracePeriodSeconds above the drain, such as 40 seconds for 25.

A worker that dies without draining stops sending heartbeats. After 30 seconds the control plane marks its runs lost and applies the same rule.

Metrics

GET /metrics on the control plane, in Prometheus format:

Metric	Meaning
graphdom_queue_depth{site}	Jobs waiting for a worker, per site
graphdom_runs{state}	Runs running, and runs paused for an operator
graphdom_workers_alive	Workers that claimed work in the last 30 seconds
graphdom_incidents_open	Open drift incidents

GET /healthz answers {"ok":true} on the control plane. On a worker’s live port it also says whether the worker is draining and how many runs it has.