Core Reference¶
encord_agents.core.data_model
¶
EditorAgentResponse
¶
Bases: BaseModel
A base class for all return types of editor agent functions.
Source code in encord_agents/core/data_model.py
message
class-attribute
instance-attribute
¶
A message to be displayed to the user.
Frame
dataclass
¶
A dataclass to hold the content of one frame in a video.
Source code in encord_agents/core/data_model.py
b64_encoding
¶
b64_encoding(image_format: Literal['.jpeg', '.jpg', '.png'] = '.jpeg', output_format: Literal['url', 'openai', 'anthropic', 'raw'] = 'url') -> str | dict[str, str | dict[str, str]]
Get a base64 representation of the image content.
This method allows you to convert the content into a base64 representation based on various different image encodings. This is useful, e.g., for prompting LLMs with image content.
Please see details for formats below.
Parameters:
-
image_format
(Literal['.jpeg', '.jpg', '.png']
, default:'.jpeg'
) –Which type of image encoding to use.
-
output_format
(Literal['url', 'openai', 'anthropic', 'raw']
, default:'url'
) –Different common formats. -
raw
: the image content as a raw b64 string -url
: url encoded image content. Compatible with, e.g.,<img src="<the_encoding>" />
-openai
: a dict withtype
andimage_url
keys _anthropic
: a dict withmedia_type
,type
, anddata
keys.
Returns: a dict or string depending on output_format
.
Source code in encord_agents/core/data_model.py
FrameData
¶
Bases: BaseModel
Holds the data sent from the Encord Label Editor at the time of triggering the agent.
Source code in encord_agents/core/data_model.py
data_hash
class-attribute
instance-attribute
¶
The identifier of the given data asset.
frame
class-attribute
instance-attribute
¶
The frame number. If single image, it's default 0.
object_hashes
class-attribute
instance-attribute
¶
Object hashes if the request was made on particular objects from the App
project_hash
class-attribute
instance-attribute
¶
The identifier of the given project.
InstanceCrop
dataclass
¶
Bases: Frame
A dataclass to hold the frame content of one object instance in a video or image.
Source code in encord_agents/core/data_model.py
LabelRowInitialiseLabelsArgs
¶
Bases: BaseModel
Arguments used to specify how to initialise labels via the SDK.
The arguments are passed to LabelRowV2.initialise_labels
.
Source code in encord_agents/core/data_model.py
LabelRowMetadataIncludeArgs
¶
Bases: BaseModel
Warning, including metadata via label rows is good for reading metadata not for writing to the metadata.
If you need to write to metadata, use the dep_storage_item
dependencies instead.
Source code in encord_agents/core/data_model.py
encord_agents.core.dependencies
¶
serverless
¶
This module defines dependencies available for injection within serverless Editor Agents. These dependencies can be used independently, even when reliant on other dependencies.
Note: The injection mechanism necessitates the presence of type annotations for the following parameters to ensure proper resolution.
from encord.project import Project
from encord.objects.ontology_labels_impl import LabelRowV2
from encord_agents import FrameData
...
@editor_agent()
def my_agent(
frame_data: FrameData,
project: Project,
label_row: LabelRowV2,
):
...
FrameData
is automatically injected via the api request body.Project
is automatically loaded based on the frame data.label_row_v2
is automatically loaded based on the frame data.
DAssetPath
module-attribute
¶
Get a local file path to data asset temporarily stored till end of agent execution.
DEncordClient
module-attribute
¶
Get an authenticated user client.
DObjectCrops
module-attribute
¶
Get all object crops that the agent was triggered on. The instance crop contains the object instance, the frame content (pixel values), and the frame.
DObjectsInstances
module-attribute
¶
Get all object instances that the agent was triggered on. No pixels, just the annotation.
DSingleFrame
module-attribute
¶
Get the single frame that the agent was triggered on.
DStorageItem
module-attribute
¶
Get the storage item associated with the underlying agent task to, for example, read/write client metadata or read data properties.
DVideoIterator
module-attribute
¶
Get a video frame iterator for doing things over many frames.
dep_asset
¶
Returns a local file path to the data asset, temporarily stored for the duration of the agent's execution.
This dependency fetches the underlying data asset using a signed URL.
The asset is temporarily stored on disk for the duration of the task and is automatically removed once the task completes.
Example:
from encord_agents.gcp import editor_agent
from encord_agents.gcp.dependencies import dep_asset
...
runner = Runner(project_hash="<project_hash_a>")
@editor_agent()
def my_agent(
asset: Annotated[Path, Depends(dep_asset)]
) -> None:
asset.stat() # read file stats
...
Returns:
-
None
–The path to the asset.
Raises:
-
ValueError
–if the underlying assets are not videos, images, or audio.
-
EncordException
–if data type not supported by SDK yet.
Source code in encord_agents/core/dependencies/serverless.py
dep_client
¶
Dependency to provide an authenticated user client.
Example:
from encord.user_client import EncordUserClient
from encord_agents.gcp import editor_agent
from encord_agents.gcp.dependencies import dep_client
...
@editor_agent()
def (
client: Annotated[EncordUserClient, Depends(dep_client)]
):
# Client will authenticated and ready to use.
client.get_dataset("")
Source code in encord_agents/core/dependencies/serverless.py
dep_data_lookup
¶
Returns a lookup for easily retrieving data rows and storage items associated with the given task.
Deprecated
dep_data_lookup
is deprecated and will be removed in version 0.2.10.
Use dep_storage_item
instead for accessing storage items.
Migration Guide:
# Old way (deprecated)
from encord_agents.core.dependencies.serverless import dep_data_lookup, DataLookup
@editor_agent()
def my_agent(
frame_data: FrameData,
lookup: Annotated[DataLookup, Depends(dep_data_lookup)]
):
storage_item = lookup.get_storage_item(frame_data.data_hash)
...
# New way (recommended)
from encord_agents.gcp.dependencies import dep_storage_item
# or from encord_agents.aws.dependencies import dep_storage_item
# or from encord_agents.fastapi.dependencies import dep_storage_item
@editor_agent()
def my_agent(
frame_data: FrameData,
storage_item: Annotated[StorageItem, Depends(dep_storage_item)]
):
# storage_item is directly available
print(storage_item.client_metadata)
...
Parameters:
-
lookup
(Annotated[DataLookup, Depends(sharable)]
) –The object that you can use to lookup data rows and storage items. Automatically injected.
Returns:
-
DataLookup
–The (shared) lookup object.
Source code in encord_agents/core/dependencies/serverless.py
dep_object_crops
¶
dep_object_crops(filter_ontology_objects: list[Object | str] | None = None) -> Callable[[FrameData, LabelRowV2, NDArray[np.uint8]], list[InstanceCrop]]
Returns a list of object instances and frame crops associated with each object.
One example use-case is to run each crop against a model.
Example:
@editor_agent
def my_agent(crops: Annotated[list[InstanceCrop], Depends[dep_object_crops(filter_ontology_objects=["eBw/75bg"])]]):
for crop in crops:
crop.content # <- this is raw numpy rgb values
crop.frame # <- this is the frame number in video
crop.instance # <- this is the object instance from the label row
crop.b64_encoding() # <- a base64 encoding of the image content
...
Parameters:
-
filter_ontology_objects
(list[Object | str] | None
, default:None
) –Specify a list of ontology objects to include. If provided, only instances of these object types are included. Strings are matched against
feature_node_hashes
.
Returns: The dependency to be injected into the cloud function.
Source code in encord_agents/core/dependencies/serverless.py
dep_single_frame
¶
Dependency to inject the first frame of the underlying asset.
The downloaded asset will be named lr.data_hash.{suffix}
.
When the function has finished running, the downloaded file is removed from the file system.
Example:
from encord_agents import FrameData
from encord_agents.gcp import editor_agent
from encord_agents.gcp.dependencies import dep_single_frame
...
@editor_agent()
def my_agent(
frame: Annotated[NDArray[np.uint8], Depends(dep_single_frame)]
):
assert frame.ndim == 3, "Will work"
Parameters:
-
storage_item
(StorageItem
) –The Storage item. Automatically injected (see example above).
Returns:
-
NDArray[uint8]
–Numpy array of shape [h, w, 3] RGB colors.
Source code in encord_agents/core/dependencies/serverless.py
dep_storage_item
¶
Get the storage item associated with the underlying agent task.
The StorageItem
is useful for multiple things like
- Updating client metadata
- Reading file properties like storage location, fps, duration, DICOM tags, etc.
Example
from typing_extensions import Annotated
from encord.storage import StorageItem
from encord_agents.gcp import editor_agent, Depends
from encord_agents.gcp.dependencies import dep_storage_item
@editor_agent()
def my_agent(storage_item: Annotated[StorageItem, Depends(dep_storage_item)]):
print("uuid", storage_item.uuid)
print("client_metadata", storage_item.client_metadata)
...
Source code in encord_agents/core/dependencies/serverless.py
dep_video_iterator
¶
Dependency to inject a video frame iterator for performing operations over many frames.
Example:
from encord_agents import FrameData
from encord_agents.gcp import editor_agent
from encord_agents.gcp.dependencies import dep_video_iterator
...
@editor_agent()
def my_agent(
video_frames: Annotated[Iterator[Frame], Depends(dep_video_iterator)]
):
for frame in video_frames:
print(frame.frame, frame.content.shape)
Parameters:
-
storage_item
(StorageItem
) –Automatically injected storage item dependency.
Raises:
-
NotImplementedError
–Fails for data types other than video.
Yields:
-
Iterator[Frame]
–An iterator.
Source code in encord_agents/core/dependencies/serverless.py
shares
¶
DataLookup
¶
Deprecated
DataLookup
is deprecated and will be removed in version 0.2.10.
Migration Guide:
- For accessing storage items, use
dep_storage_item
instead:# Old way (deprecated) from encord_agents.core.dependencies.shares import DataLookup lookup: Annotated[DataLookup, Depends(dep_data_lookup)] storage_item = lookup.get_storage_item(data_hash) # New way (recommended) from encord_agents.tasks.dependencies import dep_storage_item # or from encord_agents.aws.dependencies import dep_storage_item # or from encord_agents.gcp.dependencies import dep_storage_item # or from encord_agents.fastapi.dependencies import dep_storage_item storage_item: Annotated[StorageItem, Depends(dep_storage_item)]
Source code in encord_agents/core/dependencies/shares.py
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 |
|
backing_item_uuids
property
¶
Get all backing item uuids for all data rows in the data lookup.
Deprecated
This property is deprecated and will be removed in version 0.2.10. Use the EncordUserClient directly to access backing item UUIDs from label rows.
get_storage_item
¶
get_storage_item(data_hash: str | UUID, dataset_hash: str | UUID | None = None, sign_url: bool = False) -> StorageItem
Deprecated
This method is deprecated and will be removed in version 0.2.10.
Use dep_storage_item
dependency instead.
Parameters:
-
data_hash
(str | UUID
) –Data hash for the asset for which you need the underlying storage item.
-
dataset_hash
(str | UUID | None
, default:None
) –If you didn't provide the associated dataset hash in the constructor, this is your last chance.
-
sign_url
(bool
, default:False
) –If
True
, pre-fetch a signed URLs for the items (otherwise the URLs will be signed on demand).
Raises:
-
ValueError
–Mainly if underlying data row cannot be found.
Returns:
-
StorageItem
–The underlying storage item from which, e.g., client metadata can be updated.
Source code in encord_agents/core/dependencies/shares.py
get_storage_items
¶
get_storage_items(data_hashes: list[str | UUID], dataset_hash: str | UUID | None = None, sign_urls: bool = False) -> list[StorageItem]
Deprecated
This method is deprecated and will be removed in version 0.2.10. Use the EncordUserClient directly for bulk storage item access.
Parameters:
-
data_hashes
(list[str | UUID]
) –Data hashes for the assets for which you need the underlying storage items.
-
dataset_hash
(str | UUID | None
, default:None
) –If you didn't provided the associated dataset hash in the constructor, this is your last chance.
-
sign_urls
(bool
, default:False
) –If
True
, pre-fetch a signed URLs for the items (otherwise the URLs will be signed on demand).
Raises:
-
ValueError
–Mainly if underlying data row cannot be found.
Returns:
-
list[StorageItem]
–list of underlying storage items from which, e.g., client metadata can be updated.
Source code in encord_agents/core/dependencies/shares.py
encord_agents.core.ontology
¶
FieldType
module-attribute
¶
Field from pydantic can be anything so hard to type. This is supposed to indicate that you should use the
pydantic.Field
function to construct this var.
GenericFieldModel
¶
Bases: BaseModel
Source code in encord_agents/core/ontology.py
set_answer
¶
This function will be called from the parsing loop to allow the model to set it self as answer on the classification instance.
OntologyDataModel
¶
Bases: Generic[OntologyType]
Class to create a pydantic model equivalent to an arbitrary classification ontology.
The model can be used to form a json schema based on the ontology. This is useful if you are, e.g., trying to get a structured response from an LLM.
Example:
from pydantic import ValidationError
classifications = project.ontology_structure.classifications
objects = project.ontology_structure.classifications
data_model = OntologyDataModel([objects])
# or
data_model = OntologyDataModel([classifications])
# Get a json schema for the ontology
print(data_model.model_json_schema_str)
# Parse json following the schema into label instances
json_str = my_favourite_llm(
f"what is this? pls follow {schema}", img
)
try:
instances = data_model(json_str)
except ValidationError:
# invalid json
...
for ins in instances:
label_row.add_classification_instance(ins)
label_row.save()
For a concrete example, please see
Attributes:
-
ontology
– -
DataModel
(BaseModel
) –
Source code in encord_agents/core/ontology.py
298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 |
|
__call__
¶
Validate a json response in accordance to the pydantic model.
This function allows you to convert from a json object (e.g., coming from an llm) back to the encord "instance format".
Parameters:
-
answer
(str
) –The json object as a raw string.
a list of classification / object instances that you will then
-
list[ClassificationInstance] | ObjectInstance
–have to add to a label row.
Source code in encord_agents/core/ontology.py
validate_json
¶
Validate a json response in accordance to the pydantic model.
This function allows you to convert from a json object (e.g., coming from an llm) back to the encord "instance format".
Parameters:
-
answer_str
(str
) –The json object as a raw string.
a list of classification / object instances that you will then
-
list[ClassificationInstance] | ObjectInstance
–have to add to a label row.
Source code in encord_agents/core/ontology.py
encord_agents.core.rich_columns
¶
TaskSpeedColumn
¶
Bases: ProgressColumn
Renders human readable transfer speed.
Source code in encord_agents/core/rich_columns.py
render
¶
Show data transfer speed.
Source code in encord_agents/core/rich_columns.py
encord_agents.core.settings
¶
Settings used throughout the module.
Note that central settings will be read via environment variables.
Settings
¶
Bases: BaseSettings
Source code in encord_agents/core/settings.py
ssh_key_content
class-attribute
instance-attribute
¶
The content of the private ssh key file to authenticate with Encord.
Either this or the ENCORD_SSH_KEY
needs to be set for most use-cases.
To setup a key with Encord, please see
the platform docs.
ssh_key_file
class-attribute
instance-attribute
¶
The path to the private ssh key file to authenticate with Encord.
Either this or the ENCORD_SSH_KEY
needs to be set for most use-cases.
To setup a key with Encord, please see
the platform docs.
encord_agents.core.utils
¶
batch_iterator
¶
Yield batches of items from an iterator.
Parameters:
-
iterator
(Iterable[T]
) –The source iterator
-
batch_size
(int
) –Size of each batch > 0
Returns:
-
Iterable[List[T]]
–Iterable of lists, each containing up to batch_size items
Source code in encord_agents/core/utils.py
download_asset
¶
Download the asset associated to a label row to disk.
This function is a context manager. Data will be cleaned up when the context is left.
Example usage:
with download_asset(storage_item, 10) as asset_path:
# In here the file exists
pixel_values = np.asarray(Image.open(asset_path))
# outside, it will be cleaned up
Parameters:
-
storage_item
(StorageItem
) –The Storage item for which you want to download the associated asset.
-
frame
(int | None
, default:None
) –The frame that you need. If frame is none for a video, you will get the video path.
Raises:
-
NotImplementedError
–If you try to get all frames of an image group.
-
ValueError
–If you try to download an unsupported data type (e.g., DICOM).
Yields:
-
Path
–The file path for the requested asset.
Source code in encord_agents/core/utils.py
get_frame_count
¶
Get the number of frames in a video.
Source code in encord_agents/core/utils.py
get_initialised_label_row
¶
get_initialised_label_row(frame_data: FrameData, include_args: LabelRowMetadataIncludeArgs | None = None, init_args: LabelRowInitialiseLabelsArgs | None = None) -> LabelRowV2
Get an initialised label row from the frame_data information.
Parameters:
-
frame_data
(FrameData
) –The data pointing to the data asset.
Raises:
-
Exception
–If the
frame_data
cannot be matched to a label row
Returns:
-
LabelRowV2
–The initialized label row.
Source code in encord_agents/core/utils.py
get_user_client
¶
Generate an user client to access Encord.
Returns:
-
EncordUserClient
–An EncordUserClient authenticated with the credentials from the encord_agents.core.settings.Settings.
Source code in encord_agents/core/utils.py
encord_agents.core.video
¶
get_frame
¶
Extract an exact frame from a video.
Parameters:
-
video_path
(Path
) –The file path to where the video is stored.
-
desired_frame
(int
) –The frame to extract
Raises:
-
Exception
–If the video cannot be opened properly or the requested frame could not be retrieved from the video.
Returns:
-
NDArray[uint8]
–Numpy array of shape [h, w, c] where channels are BGR.
Source code in encord_agents/core/video.py
iter_video
¶
Iterate video frame by frame.
Parameters:
-
video_path
(Path
) –The file path to the video you wish to iterate.
Raises:
-
Exception
–If the video file could not be opened properly.
Yields:
-
Frame
–Frames from the video.
Source code in encord_agents/core/video.py
iter_video_with_indices
¶
Iterate video frame by frame with specified frame indices.
Parameters:
-
video_path
(Path
) –The file path to the video you wish to iterate.
-
frame_indices
(Iterable[int]
) –The frame indices to iterate over.
Yields:
-
Frame
–Frames from the video.
Source code in encord_agents/core/video.py
write_frame
¶
Write a frame to a file.
Parameters:
-
frame_path
(Path
) –The file path to write the frame to.
-
frame
(NDArray[uint8]
) –The frame to write.
encord_agents.core.vision
¶
b64_encode_image
¶
Encode an image to a base64 string.
Parameters:
-
img
(NDArray[uint8]
) –The image to encode. Expects [RGB] channels
-
format
(Base64Formats
, default:'.jpg'
) –The format of the image.
Returns:
-
str
–The base64 encoded image.