Spider IR · Graphdom docs

The format of spider.json, the file a spider is. Every field, generated from its JSON Schema.

A spider is one JSON file, spider.json. graphdom build writes it to spiders/<task>/v<n>/, beside the Playwright script it compiles to. The runtime, the compiler and Studio all read this file, and the model that repairs a spider edits this file and nothing else. You can edit it by hand too; check the result with graphdom runtime test.

The format is graphdom-ir@1.0. Its JSON Schema is at /schema/spider.json, and the reference below is generated from it.

Example

A spider that searches a quotes site and returns every quote on the results page.

{
  "format": "graphdom-ir@1.0",
  "spider": "search_quotes",
  "version": 1,
  "origins": { "site": "https://quotes.example.com" },
  "inputs": { "query": { "type": "string", "example": "love" } },
  "outputs": { "quotes": { "type": "list", "of": { "text": "string", "author": "string" } } },
  "targets": {
    "search": { "is": { "role": "searchbox" }, "by": [{ "role": "searchbox", "name": "Search" }, { "css": "#q" }] },
    "results": { "many": true, "by": [{ "css": ".quote" }] }
  },
  "start": "home",
  "states": {
    "home": {
      "recognise": { "url": "{site}/", "present": ["search"] },
      "edges": [
        {
          "id": "search",
          "steps": [
            { "id": "s1", "do": "fill", "target": "search", "value": { "input": "query" }, "effect": "read" },
            { "id": "s2", "do": "press", "target": "search", "keys": ["Enter"], "effect": "read",
              "until": [{ "navigation": "GET {site}/search" }], "timeoutMs": 5000 }
          ],
          "to": ["results"]
        }
      ]
    },
    "results": {
      "recognise": { "url": "{site}/search?q=*" },
      "edges": [
        {
          "id": "read",
          "steps": [
            { "id": "s3", "do": "extract", "target": "results", "into": "quotes",
              "fields": { "text": { "css": ".text" }, "author": { "css": ".author" } } }
          ],
          "to": []
        }
      ]
    }
  }
}

How a run reads it

A spider is a graph. Its nodes are the states the site can be in, and its edges are blocks of steps.

The run opens entry, or the URL of the start state’s recognise when there is no entry.
In each state it takes the first edge whose when holds, runs its steps in order, and checks that the page arrived in one of the edge’s to states. An edge with an empty to ends the run.
Each step acts, then waits for everything in its until: a response, a navigation or a condition. A spider never sleeps for a fixed time.
Interrupts can fire in any state: a cookie banner to dismiss, or a CAPTCHA that stops the run.
A page that matches no state is drift. The run stops and reports where.

Every action step says whether it only reads or submits something (effect). graphdom runtime run asks before each submit step unless you pass --yes.

Steps act on targets, which are named in one place and described by several locators, best first. URLs start with an origin name such as {site}, a role the spider acts on, so the same spider runs against the real site, a staging copy (--origin site=https://staging.example.com) and the offline sandbox. Origins a page only touches, such as a CDN, are listed apart as incidental. Inputs and secrets come in as { "input": "query" } and { "secret": "password" }; a secret’s value is never in the file.

Validate a spider

To check spider.json in your editor, map it to the schema. In VS Code, add this to your settings:

"json.schemas": [
  { "fileMatch": ["spiders/**/spider.json"], "url": "https://docs.graphdom.dev/schema/spider.json" }
]

The schema checks the shape. Graphdom checks the rest before every run and compile, and names the path of each error:

every name a spider refers to exists: targets, states, inputs, outputs, origins and pages;
ids and names use letters, digits, _ and -, and step and edge ids are unique;
every URL starts with an origin the spider defines;
a select has exactly one of label and value;
a list target (many) is read only by extract, which then needs fields and a list output;
a secret is only the value of fill or type, or the answer to a prompt dialog;
an interrupt never submits.

Spider

A spider: a graph of the states a site can be in, whose edges are blocks of steps.

formatstringrequired
The IR version, "graphdom-ir@1.<minor>".
spiderstringrequired
The id of the task the spider does.
versionintegerrequired
The spider's version, from 1, never changed once approved.
originsmap of stringrequired
The roles: the sites the task acts on, by name; a URL starts with "{role}", and a run binds each role to an origin.
incidentalmap of string
Origins the pages only touch (a CDN, analytics), by name; they appear in request patterns and are not assigned at run time.
inputsmap of objectrequired
The values a run takes, by name.
3 child fields
- type"string" | "number" | "boolean" | "file"required
  The type of the value.
- values(string | number)[]
  The values it may take, when they are a fixed set.
- exampleExpr | Expr[]
  A value seen in a recording, or the files a session picked.
secretsmap of object
The secrets a run takes, such as a password, by name; only their labels are in the spider.
1 child field
- labelstringrequired
  What the secret is, as a person is asked for it.
outputsmap of objectrequired
The values a run returns, by name.
2 child fields
- type"string" | "number" | "file" | "list"required
  The type of the value returned.
- ofmap of "string" | "number" | "string[]"
  The fields of each item of the list, by name, with their types.
targetsmap of objectrequired
The elements the steps act on, by name.
6 child fields
- framesLocator[][]
  The frames the element is in, outermost first; each hop is the ranked locators of an iframe or frame.
- hostsobject[]
  The shadow hosts the element is in, outermost first.
  2 child fields
  - cssstringrequired
    A CSS selector for the shadow host.
  - closedtrue
    Present when the shadow root is closed.
- withinstring
  Another target the element is looked for in, such as a table row or a dialog.
- isobject
  The role or tag the element found must have.
  2 child fields
  - rolestring
    The ARIA role the element must have.
  - tagstring
    The HTML tag name the element must have.
- manytrue
  Present when the target is a list of elements, which only extract reads.
- byLocator[]required
  The locators that find the element, best first, at least one.
startstringrequired
The state the run begins in.
entrystring
The URL the run goes to first; by default the URL of the start state's recogniser.
statesmap of objectrequired
The states the site can be in, by name.
3 child fields
- recogniseobjectrequired
  What tells this state from the others: every rule given must hold.
  5 child fields
  - urlstring
    The URL pattern the page's main frame must match.
  - presentstring[]
    Targets that must be found.
  - absentstring[]
    Targets that must not be found.
  - whenCond
    Any other condition, read at once and never waited for.
  - pagestring
    The page recognised; by default "main".
- edgesobject[]required
  The ways out of the state, in order: the first whose condition holds is taken; none ends the run here.
  5 child fields
  - idstringrequired
    The edge's id, unique in the spider.
  - whenCond
    The condition under which the edge is taken; by default always.
  - maxinteger
    How many times a run may take the edge; by default 1.
  - stepsBlockStep[]required
    The steps the edge runs, in order.
  - tostring[]required
    The states the run may arrive in; none ends the run after the steps.
- notestring
  A note for the people reading the spider.
interruptsobject[]
Detours that may leave from any state, such as a cookie banner, or end the run, such as a CAPTCHA.
5 child fields
- idstringrequired
  The interrupt's id, unique in the spider.
- whenCondrequired
  The condition that sets it off, in any state.
- stepsBlockStep[]
  The steps it runs, none of them a submit; then the run goes on from the state it was in.
- stopstring
  The reason the run ends instead, such as a CAPTCHA or new terms.
- maxinteger
  How many times a run may take it.

Step

One step of a block: an action on the page, an extract, or closing a page.

Shared fields

idstringrequired
The step's id, unique in the spider.
notestring
A note for the people reading the spider.
effect"read" | "submit"required
Whether the step only reads or submits something; required, never defaulted.
untilUntil[]
What the step waits for after acting, every one of them.
timeoutMsinteger
How long the step waits, in milliseconds.
opensobject
The page or the download the step opens.
2 child fields
- pagestring
  The name the page opened goes by in later steps.
- downloadstring
  The file output the download is saved as.
dialogobject
The dialog the step opens, and how it is answered.
4 child fields
- kind"alert" | "confirm" | "prompt" | "beforeunload"required
  The kind of dialog.
- messagestring
  The message the dialog shows.
- answer"accept" | "dismiss"required
  Whether the dialog is accepted or dismissed.
- textExpr
  The text typed into a prompt before accepting it.
cadenceMsnumberrequired
The time between keys, in milliseconds.

One of

do: "goto"
- urlstringrequired
  The URL to go to.
- pagestring
  The page the step acts on; by default "main".
Also takes id, note, effect, until, timeoutMs, opens, dialog.
do: "click"
- targetstringrequired
  The target the step acts on.
- offsetobject
  Where to click in the element; by default its centre.
  2 child fields
  - xnumberrequired
    Pixels from the left edge.
  - ynumberrequired
    Pixels from the top edge.
- button"left" | "middle" | "right"
  The mouse button; by default left.
- modifiers("Alt" | "Control" | "Meta" | "Shift")[]
  The keys held down while clicking.
- count1 | 2
  One click or a double click.
- pagestring
  The page the step acts on; by default "main".
Also takes id, note, effect, until, timeoutMs, opens, dialog.
do: "hover"
- targetstringrequired
  The target the step acts on.
- pagestring
  The page the step acts on; by default "main".
Also takes id, note, effect, until, timeoutMs, opens, dialog.
do: "fill"
- targetstringrequired
  The target the step acts on.
- valueExprrequired
  The value put in the field.
- pagestring
  The page the step acts on; by default "main".
Also takes id, note, effect, until, timeoutMs, opens, dialog.
do: "type"
- targetstringrequired
  The target the step acts on.
- valueExprrequired
  The value put in the field.
- pagestring
  The page the step acts on; by default "main".
Also takes id, cadenceMs, note, effect, until, timeoutMs, opens, dialog.
do: "press"
- targetstring
  The target focused before pressing; by default the element in focus.
- keysstring[]required
  The keys pressed in turn, such as "Enter" or "Control+a".
- pagestring
  The page the step acts on; by default "main".
Also takes id, note, effect, until, timeoutMs, opens, dialog.
do: "select"
- targetstringrequired
  The target the step acts on.
- labelExpr
  The label of the option selected.
- valueExpr
  The value of the option selected.
- pagestring
  The page the step acts on; by default "main".
Also takes id, note, effect, until, timeoutMs, opens, dialog.
do: "check"
- targetstringrequired
  The target the step acts on.
- checkedExprrequired
  Whether the box ends up checked.
- pagestring
  The page the step acts on; by default "main".
Also takes id, note, effect, until, timeoutMs, opens, dialog.
do: "choose"
- targetstringrequired
  The target the step acts on.
- queryExprrequired
  The text typed into the field to show the suggestions.
- optionstringrequired
  The target of the option to commit.
- commit"click" | "keys"required
  How the option is committed: a click, or arrow keys then Enter.
- matchobject
  How the option is picked among the suggestions shown.
  3 child fields
  - valueExprrequired
    The value the option chosen must match.
  - rule"text" | "address-it" | "comune-it"required
    How texts are compared: as text, an Italian address, or an Italian comune.
  - checkstring
    The target whose value must match too after the commit; by default the field.
- pagestring
  The page the step acts on; by default "main".
Also takes id, cadenceMs, note, effect, until, timeoutMs, opens, dialog.
do: "upload"
- targetstringrequired
  The target the step acts on.
- filesExpr[]required
  The files uploaded.
- pagestring
  The page the step acts on; by default "main".
Also takes id, note, effect, until, timeoutMs, opens, dialog.
do: "scroll"
- targetstring
  The target to scroll to, or to scroll within to its end.
- to"end" | "target"required
  Where to scroll: to the end of the page or to the target.
- pagestring
  The page the step acts on; by default "main".
Also takes id, note, effect, until, timeoutMs, opens, dialog.
do: "extract"
- targetstringrequired
  The target the step acts on.
- intostringrequired
  The output the value read goes into.
- pagestring
  The page read; by default "main".
- readRead
  How the element is read; by default its text.
- fieldsmap of object
  The fields read from each element of a list target, by output field name.
  3 child fields
  - cssstringrequired
    A CSS selector for the field within each element.
  - readRead
    How the field is read; by default its text.
  - alltrue
    Present when the field is a list of every match.
Also takes id, note.
do: "close"
- pagestringrequired
  The page closed.
Also takes id, note.

Locator

One way to find an element; a target ranks several, best first.

One of

testId
- testIdstringrequired
  The element's test id attribute.
field
- fieldstringrequired
  The name of the form field.
role
- rolestringrequired
  The element's ARIA role.
- nameExpr
  Its accessible name, exact after collapsing whitespace.
label
- labelExprrequired
  The text of the form field's label.
id
- idstringrequired
  The element's id attribute.
css
- cssstringrequired
  A CSS selector.
- textExpr
  The whole text the element selected must have, whitespace collapsed.
- containsExpr
  A part of the text the element selected must have, whitespace collapsed; not with text.
text
- textExprrequired
  The element's visible text.
attr
- attrmap of Exprrequired
  Attribute values the element must have, by attribute name.
frameUrl
- frameUrlstringrequired
  The URL of the document a frame committed, only in a frame hop.

Cond

A condition on the page or the inputs, read when it is checked.

Shared fields

eqExpr
Holds when the value equals this one.
neExpr
Holds when the value differs from this one.
ltnumber
Holds when the value is a number less than this one.
lenumber
Holds when the value is a number at most this one.
gtnumber
Holds when the value is a number greater than this one.
genumber
Holds when the value is a number at least this one.
inExpr[]
Holds when the value is one of these.
containsExpr
Holds when the value contains this text.
matchesstring
Holds when the value matches this regular expression.
nonEmptytrue
Holds when the value is not empty.

One of

input
- inputstringrequired
  The name of the input compared.
Also takes eq, ne, lt, le, gt, ge, in, contains, matches, nonEmpty.
value
- valuestringrequired
  The target whose form value is compared.
Also takes eq, ne, lt, le, gt, ge, in, contains, matches, nonEmpty.
text
- textstringrequired
  The target whose text is compared.
Also takes eq, ne, lt, le, gt, ge, in, contains, matches, nonEmpty.
url
- urlstringrequired
  Holds when the main frame's URL matches this pattern.
present
- presentstringrequired
  Holds when this target is found on the page.
visible
- visiblestringrequired
  Holds when this target is found and visible.
all
- allCond[]required
  Holds when every one of these conditions holds.
any
- anyCond[]required
  Holds when at least one of these conditions holds.
not
- notCondrequired
  Holds when this condition does not.

Until

What a step waits for after acting: a condition, a response, or a navigation.

One of

Cond
response
- responsestringrequired
  A request, "METHOD {origin}/path", whose response arrives.
- statusinteger
  The HTTP status the response must have.
navigation
- navigationstringrequired
  A request, "METHOD {origin}/path", whose document the main frame commits and loads.

Expr

A value: a literal, or the value of an input or a secret in this run.

One of

string
number
boolean
input
- inputstringrequired
  The name of an input: its value in this run.
secret
- secretstringrequired
  The name of a secret: its value in this run, only where a secret may go.

Read

How an element is read: its text, its form value, or one of its attributes.

One of

"text" | "value"
attr
- attrstringrequired
  The name of the attribute to read.