Skip to content

Intro

Info

Editor Agents are API endpoints triggered on individual tasks within the Label Editor. They differ from Task Agents, which are Workflow components that activate on all tasks passing through the Agent stage.

Editor Agents enable you to integrate your own API endpoint, such as your own model hosted on a server (or in a cloud function), with Encord to enhance your annotation processes. Agents can be called while annotating in the Label Editor. Some common use-cases are:

  • Validate the current state of the annotations within a frame, video, or image group. You might, for example, want to give the labelers an option to annotate the current state of the labels before submitting.
  • Do custom conversions or modifications of labels before they are submitted. For example, you could be simplifying polygons with an RDP algorithm.
  • Employ custom prompting models like DINOv or T-Rex2 to speed up annotations.
  • Trigger notifications internally related to the given task.

Think of these agents as agents that your annotators can trigger at will while they are labeling.

The general concepts

Editor agents work in the following way:

sequenceDiagram
    participant A as Annotator
    participant B as Encord Label Editor #5F47FE
    participant C as Editor Agent [custom API]
    participant D as Encord SDK #5F47FE

    A->>B: Trigger Agent
    B->>C: Call with project, data, and frame num
    C->>D: Perform edits to the underlying label row
    D->>C: confirmation
    C->>B: 200 response
    B->>A: Label editor refresh

What you can use encord-agents for is filling in the logic that's supposed to happen in the "Editor Agent [custom API]" part of the diagram. That is, you will be responsible for programatically defining what to do, when your custom api is hit with a project_hash, data_hash, and potentially a frame number.

We help with two different ways of building such Custom APIs:

  1. Using Google run functions which is Google's way of building cloud functions.
  2. Using FastAPI which is a flexible (self-hosted) python library for building custom APIs.

Tip

Actually, the encord-agents take a lot of inspiration from FastAPI. Specifically, we have adopted the idea of dependency injections from that library. While our injection scheme is not as sophisticated, it should feel familiar.

Google run functions are good for more light-weight operations like acting as proxies to other model inference APIs or tweaking labels. FastAPI apps are better suited for actually hosting your own models.

In the next section, you will find a GCP example. If you need to build a FastAPI application, feel free to skip it.