observe

Observes the current session page

Observation Response:

• a list of actions that can be taken on the page (e.g. click on a button, scroll, etc.)
• a screenshot of the page (base64 encoded)
• some metadata about the page (title, url, etc.)

# Observe the page
obs = session.observe()
# Select an action from the list of interactible elements on the page
actions = obs.space.interaction_actions
# display the action space as a string to be able to visualize it
print(obs.space.description)
# get the screenshot
screenshot = obs.screenshot.bytes()

Once you have selected an action (either manually or using an LLM), you can execute it with:

session.execute(action)

Note that by default, a very simple page perception is used to generate the action space (i.e perception_type='fast') to make the query fast. If you want a more powerful and LLM-ready action space, you can use:

obs = session.observe(perception_type='deep')
print(obs.space.description)

At the cost of a slower query since this uses an LLM call to format the interactive elements.

Additionally, you can use the instructions parameter to narrow down the action space to a specific intent on a website. This is useful if you want to quickly create a workflow using natural language:

_ = session.execute(type="goto", url="https://console.notte.cc")
actions = session.observe(instructions="Fill the email input")
print(actions[0].model_dump())

Parameters

min_nb_actions · int | None

The minimum number of actions to list before stopping. If not provided, the listing will continue until the maximum number of actions is reached.

max_nb_actions · int

The maximum number of actions to list after which the listing will stop. Used when min_nb_actions is not provided.

instructions · str | None

Additional instructions to use for the observation.

perception_type · typing.Optional[typing.Literal['fast', 'deep']]

Returns

UnionType[Observation, list[Annotated[UnionType[ClickAction, FillAction, MultiFactorFillAction, FallbackFillAction, CheckAction, SelectDropdownOptionAction, UploadFileAction, DownloadFileAction], annotation=NoneType required=True discriminator='type']]]: The formatted observation result from the API response when no instructions provided. list[InteractionActionUnion]: The filtered list of actions when instructions is provided.