# Spider IR

The format of spider.json, the file a spider is. Every field, generated from its JSON Schema.

A spider is one JSON file, `spider.json`. `graphdom build` writes it to `spiders/<task>/v<n>/`, beside the Playwright script it compiles to. The runtime, the compiler and Studio all read this file, and the model that repairs a spider edits this file and nothing else. You can edit it by hand too; check the result with `graphdom runtime test`.

The format is `graphdom-ir@1.0`. Its JSON Schema is at [`/schema/spider.json`](/schema/spider.json), and the reference below is generated from it.

## Example

A spider that searches a quotes site and returns every quote on the results page.

```json
{
  "format": "graphdom-ir@1.0",
  "spider": "search_quotes",
  "version": 1,
  "origins": { "site": "https://quotes.example.com" },
  "inputs": { "query": { "type": "string", "example": "love" } },
  "outputs": { "quotes": { "type": "list", "of": { "text": "string", "author": "string" } } },
  "targets": {
    "search": { "is": { "role": "searchbox" }, "by": [{ "role": "searchbox", "name": "Search" }, { "css": "#q" }] },
    "results": { "many": true, "by": [{ "css": ".quote" }] }
  },
  "start": "home",
  "states": {
    "home": {
      "recognise": { "url": "{site}/", "present": ["search"] },
      "edges": [
        {
          "id": "search",
          "steps": [
            { "id": "s1", "do": "fill", "target": "search", "value": { "input": "query" }, "effect": "read" },
            { "id": "s2", "do": "press", "target": "search", "keys": ["Enter"], "effect": "read",
              "until": [{ "navigation": "GET {site}/search" }], "timeoutMs": 5000 }
          ],
          "to": ["results"]
        }
      ]
    },
    "results": {
      "recognise": { "url": "{site}/search?q=*" },
      "edges": [
        {
          "id": "read",
          "steps": [
            { "id": "s3", "do": "extract", "target": "results", "into": "quotes",
              "fields": { "text": { "css": ".text" }, "author": { "css": ".author" } } }
          ],
          "to": []
        }
      ]
    }
  }
}
```

## How a run reads it

A spider is a graph. Its nodes are the **states** the site can be in, and its edges are blocks of **steps**.

1. The run opens `entry`, or the URL of the start state's `recognise` when there is no `entry`.
2. In each state it takes the first edge whose `when` holds, runs its steps in order, and checks that the page arrived in one of the edge's `to` states. An edge with an empty `to` ends the run.
3. Each step acts, then waits for everything in its `until`: a response, a navigation or a condition. A spider never sleeps for a fixed time.
4. **Interrupts** can fire in any state: a cookie banner to dismiss, or a CAPTCHA that stops the run.
5. A page that matches no state is drift. The run stops and reports where.

Every action step says whether it only reads or submits something (`effect`). `graphdom runtime run` asks before each `submit` step unless you pass `--yes`.

Steps act on **targets**, which are named in one place and described by several locators, best first. URLs start with an origin name such as `{site}`, a role the spider acts on, so the same spider runs against the real site, a staging copy (`--origin site=https://staging.example.com`) and the offline sandbox. Origins a page only touches, such as a CDN, are listed apart as `incidental`. Inputs and secrets come in as `{ "input": "query" }` and `{ "secret": "password" }`; a secret's value is never in the file.

## Validate a spider

To check `spider.json` in your editor, map it to the schema. In VS Code, add this to your settings:

```json
"json.schemas": [
  { "fileMatch": ["spiders/**/spider.json"], "url": "https://docs.graphdom.dev/schema/spider.json" }
]
```

The schema checks the shape. Graphdom checks the rest before every run and compile, and names the path of each error:

- every name a spider refers to exists: targets, states, inputs, outputs, origins and pages;
- ids and names use letters, digits, `_` and `-`, and step and edge ids are unique;
- every URL starts with an origin the spider defines;
- a `select` has exactly one of `label` and `value`;
- a list target (`many`) is read only by `extract`, which then needs `fields` and a list output;
- a secret is only the value of `fill` or `type`, or the answer to a prompt dialog;
- an interrupt never submits.

## Spider

A spider: a graph of the states a site can be in, whose edges are blocks of steps.

- `format` (string, required): The IR version, "graphdom-ir@1.<minor>".
- `spider` (string, required): The id of the task the spider does.
- `version` (integer, required): The spider's version, from 1, never changed once approved.
- `origins` (map of string, required): The roles: the sites the task acts on, by name; a URL starts with "{role}", and a run binds each role to an origin.
- `incidental` (map of string): Origins the pages only touch (a CDN, analytics), by name; they appear in request patterns and are not assigned at run time.
- `inputs` (map of object, required): The values a run takes, by name.
  - `type` ("string" | "number" | "boolean" | "file", required): The type of the value.
  - `values` ((string | number)[]): The values it may take, when they are a fixed set.
  - `example` (Expr | Expr[]): A value seen in a recording, or the files a session picked.
- `secrets` (map of object): The secrets a run takes, such as a password, by name; only their labels are in the spider.
  - `label` (string, required): What the secret is, as a person is asked for it.
- `outputs` (map of object, required): The values a run returns, by name.
  - `type` ("string" | "number" | "file" | "list", required): The type of the value returned.
  - `of` (map of "string" | "number" | "string[]"): The fields of each item of the list, by name, with their types.
- `targets` (map of object, required): The elements the steps act on, by name.
  - `frames` (Locator[][]): The frames the element is in, outermost first; each hop is the ranked locators of an iframe or frame.
  - `hosts` (object[]): The shadow hosts the element is in, outermost first.
    - `css` (string, required): A CSS selector for the shadow host.
    - `closed` (true): Present when the shadow root is closed.
  - `within` (string): Another target the element is looked for in, such as a table row or a dialog.
  - `is` (object): The role or tag the element found must have.
    - `role` (string): The ARIA role the element must have.
    - `tag` (string): The HTML tag name the element must have.
  - `many` (true): Present when the target is a list of elements, which only extract reads.
  - `by` (Locator[], required): The locators that find the element, best first, at least one.
- `start` (string, required): The state the run begins in.
- `entry` (string): The URL the run goes to first; by default the URL of the start state's recogniser.
- `states` (map of object, required): The states the site can be in, by name.
  - `recognise` (object, required): What tells this state from the others: every rule given must hold.
    - `url` (string): The URL pattern the page's main frame must match.
    - `present` (string[]): Targets that must be found.
    - `absent` (string[]): Targets that must not be found.
    - `when` (Cond): Any other condition, read at once and never waited for.
    - `page` (string): The page recognised; by default "main".
  - `edges` (object[], required): The ways out of the state, in order: the first whose condition holds is taken; none ends the run here.
    - `id` (string, required): The edge's id, unique in the spider.
    - `when` (Cond): The condition under which the edge is taken; by default always.
    - `max` (integer): How many times a run may take the edge; by default 1.
    - `steps` (BlockStep[], required): The steps the edge runs, in order.
    - `to` (string[], required): The states the run may arrive in; none ends the run after the steps.
  - `note` (string): A note for the people reading the spider.
- `interrupts` (object[]): Detours that may leave from any state, such as a cookie banner, or end the run, such as a CAPTCHA.
  - `id` (string, required): The interrupt's id, unique in the spider.
  - `when` (Cond, required): The condition that sets it off, in any state.
  - `steps` (BlockStep[]): The steps it runs, none of them a submit; then the run goes on from the state it was in.
  - `stop` (string): The reason the run ends instead, such as a CAPTCHA or new terms.
  - `max` (integer): How many times a run may take it.

## Step

One step of a block: an action on the page, an extract, or closing a page.

Fields shared by several variants:

- `id` (string, required): The step's id, unique in the spider.
- `note` (string): A note for the people reading the spider.
- `effect` ("read" | "submit", required): Whether the step only reads or submits something; required, never defaulted.
- `until` (Until[]): What the step waits for after acting, every one of them.
- `timeoutMs` (integer): How long the step waits, in milliseconds.
- `opens` (object): The page or the download the step opens.
  - `page` (string): The name the page opened goes by in later steps.
  - `download` (string): The file output the download is saved as.
- `dialog` (object): The dialog the step opens, and how it is answered.
  - `kind` ("alert" | "confirm" | "prompt" | "beforeunload", required): The kind of dialog.
  - `message` (string): The message the dialog shows.
  - `answer` ("accept" | "dismiss", required): Whether the dialog is accepted or dismissed.
  - `text` (Expr): The text typed into a prompt before accepting it.
- `cadenceMs` (number, required): The time between keys, in milliseconds.

One of:

- **do: "goto"**
  - `url` (string, required): The URL to go to.
  - `page` (string): The page the step acts on; by default "main".
  - Also takes `id`, `note`, `effect`, `until`, `timeoutMs`, `opens`, `dialog`.
- **do: "click"**
  - `target` (string, required): The target the step acts on.
  - `offset` (object): Where to click in the element; by default its centre.
    - `x` (number, required): Pixels from the left edge.
    - `y` (number, required): Pixels from the top edge.
  - `button` ("left" | "middle" | "right"): The mouse button; by default left.
  - `modifiers` (("Alt" | "Control" | "Meta" | "Shift")[]): The keys held down while clicking.
  - `count` (1 | 2): One click or a double click.
  - `page` (string): The page the step acts on; by default "main".
  - Also takes `id`, `note`, `effect`, `until`, `timeoutMs`, `opens`, `dialog`.
- **do: "hover"**
  - `target` (string, required): The target the step acts on.
  - `page` (string): The page the step acts on; by default "main".
  - Also takes `id`, `note`, `effect`, `until`, `timeoutMs`, `opens`, `dialog`.
- **do: "fill"**
  - `target` (string, required): The target the step acts on.
  - `value` (Expr, required): The value put in the field.
  - `page` (string): The page the step acts on; by default "main".
  - Also takes `id`, `note`, `effect`, `until`, `timeoutMs`, `opens`, `dialog`.
- **do: "type"**
  - `target` (string, required): The target the step acts on.
  - `value` (Expr, required): The value put in the field.
  - `page` (string): The page the step acts on; by default "main".
  - Also takes `id`, `cadenceMs`, `note`, `effect`, `until`, `timeoutMs`, `opens`, `dialog`.
- **do: "press"**
  - `target` (string): The target focused before pressing; by default the element in focus.
  - `keys` (string[], required): The keys pressed in turn, such as "Enter" or "Control+a".
  - `page` (string): The page the step acts on; by default "main".
  - Also takes `id`, `note`, `effect`, `until`, `timeoutMs`, `opens`, `dialog`.
- **do: "select"**
  - `target` (string, required): The target the step acts on.
  - `label` (Expr): The label of the option selected.
  - `value` (Expr): The value of the option selected.
  - `page` (string): The page the step acts on; by default "main".
  - Also takes `id`, `note`, `effect`, `until`, `timeoutMs`, `opens`, `dialog`.
- **do: "check"**
  - `target` (string, required): The target the step acts on.
  - `checked` (Expr, required): Whether the box ends up checked.
  - `page` (string): The page the step acts on; by default "main".
  - Also takes `id`, `note`, `effect`, `until`, `timeoutMs`, `opens`, `dialog`.
- **do: "choose"**
  - `target` (string, required): The target the step acts on.
  - `query` (Expr, required): The text typed into the field to show the suggestions.
  - `option` (string, required): The target of the option to commit.
  - `commit` ("click" | "keys", required): How the option is committed: a click, or arrow keys then Enter.
  - `match` (object): How the option is picked among the suggestions shown.
    - `value` (Expr, required): The value the option chosen must match.
    - `rule` ("text" | "address-it" | "comune-it", required): How texts are compared: as text, an Italian address, or an Italian comune.
    - `check` (string): The target whose value must match too after the commit; by default the field.
  - `page` (string): The page the step acts on; by default "main".
  - Also takes `id`, `cadenceMs`, `note`, `effect`, `until`, `timeoutMs`, `opens`, `dialog`.
- **do: "upload"**
  - `target` (string, required): The target the step acts on.
  - `files` (Expr[], required): The files uploaded.
  - `page` (string): The page the step acts on; by default "main".
  - Also takes `id`, `note`, `effect`, `until`, `timeoutMs`, `opens`, `dialog`.
- **do: "scroll"**
  - `target` (string): The target to scroll to, or to scroll within to its end.
  - `to` ("end" | "target", required): Where to scroll: to the end of the page or to the target.
  - `page` (string): The page the step acts on; by default "main".
  - Also takes `id`, `note`, `effect`, `until`, `timeoutMs`, `opens`, `dialog`.
- **do: "extract"**
  - `target` (string, required): The target the step acts on.
  - `into` (string, required): The output the value read goes into.
  - `page` (string): The page read; by default "main".
  - `read` (Read): How the element is read; by default its text.
  - `fields` (map of object): The fields read from each element of a list target, by output field name.
    - `css` (string, required): A CSS selector for the field within each element.
    - `read` (Read): How the field is read; by default its text.
    - `all` (true): Present when the field is a list of every match.
  - Also takes `id`, `note`.
- **do: "close"**
  - `page` (string, required): The page closed.
  - Also takes `id`, `note`.

## Locator

One way to find an element; a target ranks several, best first.

One of:

- **testId**
  - `testId` (string, required): The element's test id attribute.
- **field**
  - `field` (string, required): The name of the form field.
- **role**
  - `role` (string, required): The element's ARIA role.
  - `name` (Expr): Its accessible name, exact after collapsing whitespace.
- **label**
  - `label` (Expr, required): The text of the form field's label.
- **id**
  - `id` (string, required): The element's id attribute.
- **css**
  - `css` (string, required): A CSS selector.
  - `text` (Expr): The whole text the element selected must have, whitespace collapsed.
  - `contains` (Expr): A part of the text the element selected must have, whitespace collapsed; not with text.
- **text**
  - `text` (Expr, required): The element's visible text.
- **attr**
  - `attr` (map of Expr, required): Attribute values the element must have, by attribute name.
- **frameUrl**
  - `frameUrl` (string, required): The URL of the document a frame committed, only in a frame hop.

## Cond

A condition on the page or the inputs, read when it is checked.

Fields shared by several variants:

- `eq` (Expr): Holds when the value equals this one.
- `ne` (Expr): Holds when the value differs from this one.
- `lt` (number): Holds when the value is a number less than this one.
- `le` (number): Holds when the value is a number at most this one.
- `gt` (number): Holds when the value is a number greater than this one.
- `ge` (number): Holds when the value is a number at least this one.
- `in` (Expr[]): Holds when the value is one of these.
- `contains` (Expr): Holds when the value contains this text.
- `matches` (string): Holds when the value matches this regular expression.
- `nonEmpty` (true): Holds when the value is not empty.

One of:

- **input**
  - `input` (string, required): The name of the input compared.
  - Also takes `eq`, `ne`, `lt`, `le`, `gt`, `ge`, `in`, `contains`, `matches`, `nonEmpty`.
- **value**
  - `value` (string, required): The target whose form value is compared.
  - Also takes `eq`, `ne`, `lt`, `le`, `gt`, `ge`, `in`, `contains`, `matches`, `nonEmpty`.
- **text**
  - `text` (string, required): The target whose text is compared.
  - Also takes `eq`, `ne`, `lt`, `le`, `gt`, `ge`, `in`, `contains`, `matches`, `nonEmpty`.
- **url**
  - `url` (string, required): Holds when the main frame's URL matches this pattern.
- **present**
  - `present` (string, required): Holds when this target is found on the page.
- **visible**
  - `visible` (string, required): Holds when this target is found and visible.
- **all**
  - `all` (Cond[], required): Holds when every one of these conditions holds.
- **any**
  - `any` (Cond[], required): Holds when at least one of these conditions holds.
- **not**
  - `not` (Cond, required): Holds when this condition does not.

## Until

What a step waits for after acting: a condition, a response, or a navigation.

One of:

- `Cond`
- **response**
  - `response` (string, required): A request, "METHOD {origin}/path", whose response arrives.
  - `status` (integer): The HTTP status the response must have.
- **navigation**
  - `navigation` (string, required): A request, "METHOD {origin}/path", whose document the main frame commits and loads.

## Expr

A value: a literal, or the value of an input or a secret in this run.

One of:

- `string`
- `number`
- `boolean`
- **input**
  - `input` (string, required): The name of an input: its value in this run.
- **secret**
  - `secret` (string, required): The name of a secret: its value in this run, only where a secret may go.

## Read

How an element is read: its text, its form value, or one of its attributes.

One of:

- `"text" | "value"`
- **attr**
  - `attr` (string, required): The name of the attribute to read.