\n",
" Expand to see ontology JSON
\n",
"\n",
"{\n",
" \"objects\": [],\n",
" \"classifications\": [\n",
" {\n",
" \"id\": \"1\",\n",
" \"featureNodeHash\": \"GCH8VHIK\",\n",
" \"attributes\": [\n",
" {\n",
" \"id\": \"1.1\",\n",
" \"name\": \"Caption\",\n",
" \"type\": \"text\",\n",
" \"required\": false,\n",
" \"featureNodeHash\": \"Yg7xXEfC\"\n",
" }\n",
" ]\n",
" },\n",
" {\n",
" \"id\": \"2\",\n",
" \"featureNodeHash\": \"PwQAwYid\",\n",
" \"attributes\": [\n",
" {\n",
" \"id\": \"2.1\",\n",
" \"name\": \"Caption Rephrased 1\",\n",
" \"type\": \"text\",\n",
" \"required\": false,\n",
" \"featureNodeHash\": \"aQdXJwbG\"\n",
" }\n",
" ]\n",
" },\n",
" {\n",
" \"id\": \"3\",\n",
" \"featureNodeHash\": \"3a/aSnHO\",\n",
" \"attributes\": [\n",
" {\n",
" \"id\": \"3.1\",\n",
" \"name\": \"Caption Rephrased 2\",\n",
" \"type\": \"text\",\n",
" \"required\": false,\n",
" \"featureNodeHash\": \"8zY6H62x\"\n",
" }\n",
" ]\n",
" },\n",
" {\n",
" \"id\": \"4\",\n",
" \"featureNodeHash\": \"FNjXp5TU\",\n",
" \"attributes\": [\n",
" {\n",
" \"id\": \"4.1\",\n",
" \"name\": \"Caption Rephrased 3\",\n",
" \"type\": \"text\",\n",
" \"required\": false,\n",
" \"featureNodeHash\": \"sKg1Kq/m\"\n",
" }\n",
" ]\n",
" }\n",
" ]\n",
"}\n",
"\n",
" \n",
"\n",
" Code for generating ontology
\n",
" \n",
"import json\n",
"from encord.objects.ontology_structure import OntologyStructure\n",
"from encord.objects.attributes import TextAttribute\n",
"\n",
"structure = OntologyStructure()\n",
"caption = structure.add_classification()\n",
"caption.add_attribute(TextAttribute, \"Caption\")\n",
"re1 = structure.add_classification()\n",
"re1.add_attribute(TextAttribute, \"Recaption 1\")\n",
"re2 = structure.add_classification()\n",
"re2.add_attribute(TextAttribute, \"Recaption 2\")\n",
"re3 = structure.add_classification()\n",
"re3.add_attribute(TextAttribute, \"Recaption 3\")\n",
"\n",
"print(json.dumps(structure.to_dict(), indent=2))\n",
"\n",
"create_ontology = False\n",
"if create_ontology:\n",
" from encord.user_client import EncordUserClient\n",
" client = EncordUserClient.create_with_ssh_private_key() # Look in auth section for authentication\n",
" client.create_ontology(\"title\", \"description\", structure)\n",
" \n",
" \n",
"\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How It Works\n",
"\n",
"### Workflow Requirements\n",
"\n",
"For this notebook to work, your project should have:\n",
"\n",
"1. An ontology with four text classifications:\n",
" - A text classification for a human-created caption\n",
" - Three text classifications for the LLM to fill in\n",
"\n",
"2. A workflow with at least one agent stage named \"Recaptioning\" and at least two pathways:\n",
" - \"Completed\" - for tasks that successfully generated captions\n",
" - \"No Human Caption\" - for tasks that had no human caption to work with\n",
"\n",
"### Process Flow\n",
"\n",
"1. The Runner periodically checks for tasks in the \"Recaption Video\" stage\n",
"2. For each task, it loads the label row and checks if there's a human-created caption\n",
"3. If a caption exists, it sends the caption and the first frame to GPT-4o-mini\n",
"4. The LLM generates three alternative captions\n",
"5. The agent adds these captions to the label row and saves it\n",
"6. The task moves to the \"Completed\" pathway\n",
"\n",
"### Agent Customization\n",
"\n",
"You can customize this agent by:\n",
"- Changing the system prompt to get different types of rephrases\n",
"- Using a different LLM model\n",
"- Adjusting the temperature for more or less creative outputs\n",
"- Adding more classification fields for additional types of captions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"This notebook will walk through:\n",
"- Setting up the required dependencies\n",
"- Integrating GPT-4o-mini\n",
"- Implementing the task agent using the Runner\n",
"- Running the agent on your project"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"First, let's install the required dependencies:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install encord-agents langchain-openai openai"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Authentication\n",
"\n",
"**Encord**\n",
"Encord uses ssh-keys for authentication. The following is a code cell for setting the `ENCORD_SSH_KEY` environment variable. It contains the raw content of your private ssh key file.\n",
"\n",
"If you have not setup an ssh key, see our [documentation](https://agents-docs.encord.com/authentication/).\n",
"\n",
"**OpenAI**\n",
"\n",
"We will also need an OpenAI api key to be able to interact with gptp-4o-mini.\n",
"\n",
"\n",
"> 💡 In colab, you can set the key once in the secrets in the left sidebar and load it in new notebooks. IF YOU ARE NOT RUNNING THE CODE IN THE COLLAB NOTEBOOK, you must set the environment variable directly.\n",
"> ```python\n",
"> os.environ[\"ENCORD_SSH_KEY\"] = \"\"\"paste-private-key-here\"\"\"\n",
"> os.environ[\"OPENAI_API_KEY\"] = \"\"\"openai-api-key\"\"\"\n",
"> ```\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"from google.colab import userdata\n",
"\n",
"os.environ[\"ENCORD_SSH_KEY\"] = userdata.get(\"ENCORD_SSH_KEY\")\n",
"# or you can set a path to a file\n",
"# os.environ[\"ENCORD_SSH_KEY_FILE\"] = \"/path/to/your/private/key\"\n",
"\n",
"os.environ[\"OPENAI_API_KEY\"] = userdata.get(\"OPENAI_API_KEY\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### [Alternative] Temporary Key\n",
"There's also the option of generating a temporary (fresh) ssh key pair via the code cell below.\n",
"Please follow the instructions printed when executing the code."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# ⚠️ Safe to skip if you have authenticated already\n",
"import os\n",
"\n",
"from encord_agents.utils.colab import generate_public_private_key_pair_with_instructions\n",
"\n",
"private_key_path, public_key_path = generate_public_private_key_pair_with_instructions()\n",
"os.environ[\"ENCORD_SSH_KEY_FILE\"] = private_key_path.as_posix()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Imports\n",
"Now let's import the necessary libraries:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from typing import Annotated\n",
"\n",
"import numpy as np\n",
"from encord.objects.ontology_labels_impl import LabelRowV2\n",
"from langchain_openai import ChatOpenAI\n",
"from pydantic import BaseModel\n",
"\n",
"from encord_agents.tasks import Runner\n",
"from encord_agents.tasks.dependencies import Frame, dep_single_frame"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## LLM Setup\n",
"\n",
"Let's define the response model for the GPT-4o-mini to follow.\n",
"This is a convenient method for ensuring that we work with typed and well-defined python objects.\n",
"\n",
"The class will govern what we want the LLM to return.\n",
"In this case, we want three different rephrasings of the given human caption."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class AgentCaptionResponse(BaseModel):\n",
" rephrase_1: str\n",
" rephrase_2: str\n",
" rephrase_3: str"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's set up the LLM with the system prompt.\n",
"\n",
"> 💡 You probably want to adjust this to your particular needs."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# System prompt for the LLM to follow\n",
"SYSTEM_PROMPT = \"\"\"\n",
"You are a helpful assistant that rephrases captions.\n",
"\n",
"I will provide you with a video caption and an image of the scene of the video.\n",
"\n",
"The captions follow this format:\n",
"\n",
"\"The droid picks up