Humanloop API (0.0.1)
Download OpenAPI specification:Download
The Humanloop REST API allows you to interact with Humanloop AI models from your product or service. Getting predictions from your models, creating tasks for your annotators, providing feedback to improve your models, and more.
The API has predictable resource-oriented URLs, accepts JSON request bodies, returns JSON-encoded responses, and uses standard HTTP response codes, authentication, and verbs.
We are here to help you with your human-in-the-loop AI deployment.
Individual and Team Support
On the Individual and Team plans, please write us on in-app chat. We will reply as promptly as we can. Alternatively, email hello@humanloop.com with any questions.
Enterprise Support
If you are an enterprise customer you will have your own slack channel with direct access to an account manager as well as a dedicated support email address and phone number.
APIKeyHeader
Each project has its own API key available on the integrations tab of that projects page. Each user also has their own API key that provides API access across all of the projects they own and is available on your users profile page
Security Scheme Type | API Key |
---|---|
Header parameter name: | X-API-KEY |
An oAuth2 Bearer token. You'll need one of these to access the full Humanloop API!
Login For Access Token
Validate a user exists with a valid password and return an access token used to authenticate with other subsequent API calls in a session.
Request Body schema: application/x-www-form-urlencoded
grant_type | string (Grant Type) password |
username required | string (Username) |
password required | string (Password) |
scope | string (Scope) Default: "" |
client_id | string (Client Id) |
client_secret | string (Client Secret) |
Responses
Response samples
- 200
- 422
null
A person using Humanloop - depending on the role value, either as an annotator, owner, or admin
Read User
Validate user exists with valid password and return access token
Authorizations:
Responses
Response samples
- 200
{- "id": 0,
- "username": "string",
- "email_address": "string",
- "role": "owner",
- "full_name": "string",
- "auth_key": "string",
- "active": true,
- "verified": true,
- "tier_id": 0,
- "tier": {
- "name": "string",
- "limits": {
- "property1": 0,
- "property2": 0
}
}, - "intent": null,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "feature_flags": {
- "property1": true,
- "property2": true
}
}
Update Me
Update the profile of your current user
Authorizations:
Request Body schema: application/json
username | string (Username) |
email_address | string (Email Address) |
password | string (Password) |
full_name | string (Full Name) |
intent | Array of strings (Intent) |
Responses
Request samples
- Payload
{- "username": "string",
- "email_address": "string",
- "password": "string",
- "full_name": "string",
- "intent": [
- "string"
]
}
Response samples
- 200
- 422
{- "id": 0,
- "username": "string",
- "email_address": "string",
- "role": "owner",
- "full_name": "string",
- "auth_key": "string",
- "active": true,
- "verified": true,
- "tier_id": 0,
- "tier": {
- "name": "string",
- "limits": {
- "property1": 0,
- "property2": 0
}
}, - "intent": null,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "feature_flags": {
- "property1": true,
- "property2": true
}
}
Create User
Create a new user.
Request Body schema: application/json
username required | string (Username) |
email_address required | string (Email Address) |
password required | string (Password) |
role | string Default: "owner" Enum: "owner" "annotator" "admin" An enumeration. |
full_name | string (Full Name) |
intent | Array of strings (Intent) |
Responses
Request samples
- Payload
{- "username": "string",
- "email_address": "string",
- "password": "string",
- "role": "owner",
- "full_name": "string",
- "intent": [
- "string"
]
}
Response samples
- 200
- 422
null
Invite User
Invite a user to your project using their email address. They will receive an email invite with a link to signup to Humanloop. When they log in they will then see the project they have been invited to.
Authorizations:
query Parameters
id | integer (Id) |
Request Body schema: application/json
email_address required | string (Email Address) |
project_id required | integer (Project Id) |
role | string Default: "annotator" Enum: "owner" "annotator" "admin" An enumeration. |
full_name | string (Full Name) |
Responses
Request samples
- Payload
{- "email_address": "string",
- "project_id": 0,
- "role": "annotator",
- "full_name": "string"
}
Response samples
- 200
- 422
null
Get Datasets
Get dataset details and associated field meta-data for any data-sets associated to your user.
Authorizations:
Responses
Response samples
- 200
[- {
- "id": 0,
- "name": "string",
- "count": 0,
- "data": [
- {
- "id": 0,
- "dataset_id": 0,
- "values": { },
- "source": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "processing_status": "processing"
}
], - "description": "string",
- "public": true,
- "fields": [
- {
- "id": 0,
- "dataset_id": 0,
- "name": "string",
- "display_name": "string",
- "data_type": "text",
- "required": true,
- "meta": { },
- "count": 0,
- "null_count": 0,
- "value_counts": {
- "property1": 0,
- "property2": 0
}, - "latest_counts_increase": {
- "property1": 0,
- "property2": 0
}, - "distinct_values": [
- "string"
], - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
], - "field_names": [
- "string"
], - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "status_counts": {
- "property1": 0,
- "property2": 0
}, - "active": true
}
]
Create Dataset
Upload a dataset that you can use in your Humanloop projects to train models.
Authorizations:
Request Body schema: application/json
name required | string (Name) |
required | Array of objects (Data) [ items ] |
required | Array of objects (Fields) [ items ] |
description | string (Description) |
Responses
Request samples
- Payload
{- "name": "string",
- "data": [
- {
- "property1": "string",
- "property2": "string"
}
], - "fields": [
- {
- "name": "string",
- "data_type": "text",
- "display_name": "string",
- "required": true,
- "default": "string",
- "distinct_values": [
- "string"
], - "meta": { }
}
], - "description": "string"
}
Response samples
- 200
- 422
{- "id": 0,
- "attempted": 0,
- "succeeded": 0,
- "processing": 0,
- "msg": "string",
- "fields": [
- {
- "id": 0,
- "dataset_id": 0,
- "name": "string",
- "display_name": "string",
- "data_type": "text",
- "required": true,
- "meta": { },
- "count": 0,
- "null_count": 0,
- "value_counts": {
- "property1": 0,
- "property2": 0
}, - "latest_counts_increase": {
- "property1": 0,
- "property2": 0
}, - "distinct_values": [
- "string"
], - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
]
}
Get Dataset
Get dataset details and associated field meta-data for a specific data-set associated to your user.
Authorizations:
path Parameters
id required | integer (Id) |
query Parameters
num_records | integer (Num Records) Default: 0 |
Responses
Response samples
- 200
- 422
{- "id": 0,
- "name": "string",
- "count": 0,
- "data": [
- {
- "id": 0,
- "dataset_id": 0,
- "values": { },
- "source": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "processing_status": "processing"
}
], - "description": "string",
- "public": true,
- "fields": [
- {
- "id": 0,
- "dataset_id": 0,
- "name": "string",
- "display_name": "string",
- "data_type": "text",
- "required": true,
- "meta": { },
- "count": 0,
- "null_count": 0,
- "value_counts": {
- "property1": 0,
- "property2": 0
}, - "latest_counts_increase": {
- "property1": 0,
- "property2": 0
}, - "distinct_values": [
- "string"
], - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
], - "field_names": [
- "string"
], - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "status_counts": {
- "property1": 0,
- "property2": 0
}, - "active": true
}
Update Dataset
Update an existing dataset by adding additional data points.
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
required | Array of objects (Data) [ items ] | ||||
Array
|
Responses
Request samples
- Payload
{- "data": [
- {
- "property1": "string",
- "property2": "string"
}
]
}
Response samples
- 200
- 422
{- "id": 0,
- "attempted": 0,
- "succeeded": 0,
- "processing": 0,
- "msg": "string",
- "fields": [
- {
- "id": 0,
- "dataset_id": 0,
- "name": "string",
- "display_name": "string",
- "data_type": "text",
- "required": true,
- "meta": { },
- "count": 0,
- "null_count": 0,
- "value_counts": {
- "property1": 0,
- "property2": 0
}, - "latest_counts_increase": {
- "property1": 0,
- "property2": 0
}, - "distinct_values": [
- "string"
], - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
]
}
Update Dataset Details
Update the details of your dataset such as its name and description.
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
name | string (Name) |
description | string (Description) |
Responses
Request samples
- Payload
{- "name": "string",
- "description": "string"
}
Response samples
- 200
- 422
{- "id": 0,
- "name": "string",
- "count": 0,
- "data": [
- {
- "id": 0,
- "dataset_id": 0,
- "values": { },
- "source": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "processing_status": "processing"
}
], - "description": "string",
- "public": true,
- "fields": [
- {
- "id": 0,
- "dataset_id": 0,
- "name": "string",
- "display_name": "string",
- "data_type": "text",
- "required": true,
- "meta": { },
- "count": 0,
- "null_count": 0,
- "value_counts": {
- "property1": 0,
- "property2": 0
}, - "latest_counts_increase": {
- "property1": 0,
- "property2": 0
}, - "distinct_values": [
- "string"
], - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
], - "field_names": [
- "string"
], - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "status_counts": {
- "property1": 0,
- "property2": 0
}, - "active": true
}
Get Dataset Data File
Get file for a data point.
Implementation assumes each dataset can only have one file field.
Authorizations:
path Parameters
id required | integer (Id) |
data_id required | integer (Data Id) |
Responses
Response samples
- 422
{- "detail": [
- {
- "loc": [
- "string"
], - "msg": "string",
- "type": "string"
}
]
}
Each project links users to a model and supporting dataset, specifies the expected project inputs and outputs, and manages tasks for users to collect more data and improve their model.
Create Project
Create a new project on the Humanloop platform by defining inputs and outputs and associated users.
Authorizations:
Request Body schema: application/json
name required | string (Name) |
required | Array of objects (Outputs) [ items ] |
required | Array of objects (Inputs) [ items ] |
instructions | string (Instructions) |
guidelines | string (Guidelines) |
description | string (Description) |
users required | Array of strings (Users) |
learner_config | object (Learner Config) |
object (Policy) A policy for how how many annotations to get on some proportion of the data. Providing multiple annotations for the same data-points provides consensus metrics and can help improve the quality of the final annotations provided to the model | |
task_allocation_strategy | string (TaskAllocationStrategy) Enum: "automatic" "manual" Strategy for automatic task allocation Controls whether new tasks are automatically allocated when a user completes their tasks. |
task_allocation_batch_size | integer (Task Allocation Batch Size) |
review_existing_annotations | boolean (If your dataset fields contain existing annotations then these can be used to warm start your project by specifying their field id as output data sources. You can have your annotator users first review these annotations before they are used to train your model by setting this 'review_existing_annotations' bool to True.) Default: false |
Responses
Request samples
- Payload
{- "name": "string",
- "outputs": [
- {
- "name": "string",
- "data_sources": [
- {
- "field_id": 0
}
], - "display_name": "string",
- "description": "string",
- "instructions": "string",
- "data_type": "text",
- "task_type": "classification",
- "labels": [
- {
- "id": "string",
- "name": "string",
- "display_name": "string",
- "description": "string"
}
], - "meta": { },
- "input": "string"
}
], - "inputs": [
- {
- "name": "string",
- "data_sources": [
- {
- "field_id": 0
}
], - "data_type": "text",
- "description": "string",
- "display_only": false,
- "meta": { }
}
], - "instructions": "string",
- "guidelines": "string",
- "description": "string",
- "users": [
- "string"
], - "learner_config": { },
- "policy": {
- "num_annotators": 1,
- "data_proportion": 1,
- "object": "policy",
- "id": "string"
}, - "task_allocation_strategy": "automatic",
- "task_allocation_batch_size": 0,
- "review_existing_annotations": false
}
Response samples
- 200
- 422
{- "id": 0,
- "name": "string",
- "shared_id": "string",
- "external_id": "string",
- "active": true,
- "auth_key": "string",
- "pred_count": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "default_dataset_id": 0,
- "description": "string",
- "guidelines": "string",
- "policy": { },
- "instructions": "string",
- "status": "Maintenance",
- "updated_at": "2019-08-24T14:15:22Z",
- "inputs": [
- {
- "data_type": "text",
- "description": "string",
- "data_sources": [
- {
- "id": 0,
- "name": "string"
}
], - "display_only": true,
- "id": 0,
- "meta": { },
- "name": "string",
- "project_id": 0,
- "updated_at": "2019-08-24T14:15:22Z",
- "created_at": "2019-08-24T14:15:22Z"
}
], - "outputs": [
- {
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "data_type": "text",
- "description": "string",
- "data_sources": [
- {
- "id": 0,
- "name": "string"
}
], - "display_name": "string",
- "id": 0,
- "input": "string",
- "instructions": "string",
- "labels": [
- {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
], - "meta": { },
- "name": "string",
- "project_id": 0,
- "task_type": "classification"
}
], - "users": [
- {
- "active": true,
- "email_address": "string",
- "full_name": "string",
- "id": 0,
- "intent": null,
- "role": "owner",
- "tier_id": 0,
- "tier": {
- "updated_at": "2019-08-24T14:15:22Z",
- "created_at": "2019-08-24T14:15:22Z",
- "id": 0,
- "name": "string",
- "limits": {
- "projects": 0,
- "annotators": 0,
- "dataset_size": 0,
- "predictions": 0
}
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "username": "string",
- "verified": true,
- "complete_tasks": 0,
- "incomplete_tasks": 0,
- "complete_and_not_flagged_tasks": 0,
- "incomplete_and_not_flagged_tasks": 0
}
], - "learner": {
- "id": 0,
- "current_status": "Untrained",
- "evaluations": [
- {
- "id": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "eval_metrics": { }
}
], - "latest_evaluation": {
- "id": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "eval_metrics": { }
}
}, - "datasets": [
- {
- "id": 0,
- "name": "string",
- "count": 0,
- "description": "string",
- "fields": [
- {
- "id": 0,
- "name": "string",
- "data_type": "text",
- "display_name": "string"
}
]
}
], - "data_count": 0,
- "data_without_tasks_count": 0,
- "tasks_count": 0,
- "latest_completed_tasks_batch_size": 0,
- "completed_tasks_count": 0,
- "incomplete_tasks_count": 0,
- "flagged_count": 0,
- "task_allocation_strategy": "automatic",
- "task_allocation_batch_size": 0,
- "dashboard_configuration": { }
}
Get Project
Get details of a specific project.
Authorizations:
path Parameters
id required | integer (Id) |
Responses
Response samples
- 200
- 422
{- "id": 0,
- "name": "string",
- "shared_id": "string",
- "external_id": "string",
- "active": true,
- "auth_key": "string",
- "pred_count": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "default_dataset_id": 0,
- "description": "string",
- "guidelines": "string",
- "policy": { },
- "instructions": "string",
- "status": "Maintenance",
- "updated_at": "2019-08-24T14:15:22Z",
- "inputs": [
- {
- "data_type": "text",
- "description": "string",
- "data_sources": [
- {
- "id": 0,
- "name": "string"
}
], - "display_only": true,
- "id": 0,
- "meta": { },
- "name": "string",
- "project_id": 0,
- "updated_at": "2019-08-24T14:15:22Z",
- "created_at": "2019-08-24T14:15:22Z"
}
], - "outputs": [
- {
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "data_type": "text",
- "description": "string",
- "data_sources": [
- {
- "id": 0,
- "name": "string"
}
], - "display_name": "string",
- "id": 0,
- "input": "string",
- "instructions": "string",
- "labels": [
- {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
], - "meta": { },
- "name": "string",
- "project_id": 0,
- "task_type": "classification"
}
], - "users": [
- {
- "active": true,
- "email_address": "string",
- "full_name": "string",
- "id": 0,
- "intent": null,
- "role": "owner",
- "tier_id": 0,
- "tier": {
- "updated_at": "2019-08-24T14:15:22Z",
- "created_at": "2019-08-24T14:15:22Z",
- "id": 0,
- "name": "string",
- "limits": {
- "projects": 0,
- "annotators": 0,
- "dataset_size": 0,
- "predictions": 0
}
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "username": "string",
- "verified": true,
- "complete_tasks": 0,
- "incomplete_tasks": 0,
- "complete_and_not_flagged_tasks": 0,
- "incomplete_and_not_flagged_tasks": 0
}
], - "learner": {
- "id": 0,
- "current_status": "Untrained",
- "evaluations": [
- {
- "id": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "eval_metrics": { }
}
], - "latest_evaluation": {
- "id": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "eval_metrics": { }
}
}, - "datasets": [
- {
- "id": 0,
- "name": "string",
- "count": 0,
- "description": "string",
- "fields": [
- {
- "id": 0,
- "name": "string",
- "data_type": "text",
- "display_name": "string"
}
]
}
], - "data_count": 0,
- "data_without_tasks_count": 0,
- "tasks_count": 0,
- "latest_completed_tasks_batch_size": 0,
- "completed_tasks_count": 0,
- "incomplete_tasks_count": 0,
- "flagged_count": 0,
- "task_allocation_strategy": "automatic",
- "task_allocation_batch_size": 0,
- "dashboard_configuration": { }
}
Update Project
Update the helper details of your project such as guidelines and descriptions.
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
name | string (Name) |
instructions | string (Instructions) |
guidelines | string (Guidelines) |
description | string (Description) |
learner_config | object (Learner Config) |
object (Policy) A policy for how how many annotations to get on some proportion of the data. Providing multiple annotations for the same data-points provides consensus metrics and can help improve the quality of the final annotations provided to the model | |
task_allocation_strategy | string (TaskAllocationStrategy) Enum: "automatic" "manual" Strategy for automatic task allocation Controls whether new tasks are automatically allocated when a user completes their tasks. |
task_allocation_batch_size | integer (Task Allocation Batch Size) |
Responses
Request samples
- Payload
{- "name": "string",
- "instructions": "string",
- "guidelines": "string",
- "description": "string",
- "learner_config": { },
- "policy": {
- "num_annotators": 1,
- "data_proportion": 1,
- "object": "policy",
- "id": "string"
}, - "task_allocation_strategy": "automatic",
- "task_allocation_batch_size": 0
}
Response samples
- 200
- 422
{- "id": 0,
- "name": "string",
- "shared_id": "string",
- "external_id": "string",
- "active": true,
- "auth_key": "string",
- "pred_count": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "default_dataset_id": 0,
- "description": "string",
- "guidelines": "string",
- "policy": { },
- "instructions": "string",
- "status": "Maintenance",
- "updated_at": "2019-08-24T14:15:22Z",
- "inputs": [
- {
- "data_type": "text",
- "description": "string",
- "data_sources": [
- {
- "id": 0,
- "name": "string"
}
], - "display_only": true,
- "id": 0,
- "meta": { },
- "name": "string",
- "project_id": 0,
- "updated_at": "2019-08-24T14:15:22Z",
- "created_at": "2019-08-24T14:15:22Z"
}
], - "outputs": [
- {
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "data_type": "text",
- "description": "string",
- "data_sources": [
- {
- "id": 0,
- "name": "string"
}
], - "display_name": "string",
- "id": 0,
- "input": "string",
- "instructions": "string",
- "labels": [
- {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
], - "meta": { },
- "name": "string",
- "project_id": 0,
- "task_type": "classification"
}
], - "users": [
- {
- "active": true,
- "email_address": "string",
- "full_name": "string",
- "id": 0,
- "intent": null,
- "role": "owner",
- "tier_id": 0,
- "tier": {
- "updated_at": "2019-08-24T14:15:22Z",
- "created_at": "2019-08-24T14:15:22Z",
- "id": 0,
- "name": "string",
- "limits": {
- "projects": 0,
- "annotators": 0,
- "dataset_size": 0,
- "predictions": 0
}
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "username": "string",
- "verified": true,
- "complete_tasks": 0,
- "incomplete_tasks": 0,
- "complete_and_not_flagged_tasks": 0,
- "incomplete_and_not_flagged_tasks": 0
}
], - "learner": {
- "id": 0,
- "current_status": "Untrained",
- "evaluations": [
- {
- "id": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "eval_metrics": { }
}
], - "latest_evaluation": {
- "id": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "eval_metrics": { }
}
}, - "datasets": [
- {
- "id": 0,
- "name": "string",
- "count": 0,
- "description": "string",
- "fields": [
- {
- "id": 0,
- "name": "string",
- "data_type": "text",
- "display_name": "string"
}
]
}
], - "data_count": 0,
- "data_without_tasks_count": 0,
- "tasks_count": 0,
- "latest_completed_tasks_batch_size": 0,
- "completed_tasks_count": 0,
- "incomplete_tasks_count": 0,
- "flagged_count": 0,
- "task_allocation_strategy": "automatic",
- "task_allocation_batch_size": 0,
- "dashboard_configuration": { }
}
Clone Project
Clone an existing project on the Humanloop platform with updated config.
Currently this is aimed at the ability to quickly experiment with different configurations of a project. It does not yet support editing core aspects of the project such as input, output definitions and project data.
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
name | string (Name) |
instructions | string (Instructions) |
guidelines | string (Guidelines) |
description | string (Description) |
learner_config | object (Learner Config) |
object (Policy) A policy for how how many annotations to get on some proportion of the data. Providing multiple annotations for the same data-points provides consensus metrics and can help improve the quality of the final annotations provided to the model | |
task_allocation_strategy | string (TaskAllocationStrategy) Enum: "automatic" "manual" Strategy for automatic task allocation Controls whether new tasks are automatically allocated when a user completes their tasks. |
task_allocation_batch_size | integer (Task Allocation Batch Size) |
Responses
Request samples
- Payload
{- "name": "string",
- "instructions": "string",
- "guidelines": "string",
- "description": "string",
- "learner_config": { },
- "policy": {
- "num_annotators": 1,
- "data_proportion": 1,
- "object": "policy",
- "id": "string"
}, - "task_allocation_strategy": "automatic",
- "task_allocation_batch_size": 0
}
Response samples
- 200
- 422
{- "id": 0,
- "name": "string",
- "shared_id": "string",
- "external_id": "string",
- "active": true,
- "auth_key": "string",
- "pred_count": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "default_dataset_id": 0,
- "description": "string",
- "guidelines": "string",
- "policy": { },
- "instructions": "string",
- "status": "Maintenance",
- "updated_at": "2019-08-24T14:15:22Z",
- "inputs": [
- {
- "data_type": "text",
- "description": "string",
- "data_sources": [
- {
- "id": 0,
- "name": "string"
}
], - "display_only": true,
- "id": 0,
- "meta": { },
- "name": "string",
- "project_id": 0,
- "updated_at": "2019-08-24T14:15:22Z",
- "created_at": "2019-08-24T14:15:22Z"
}
], - "outputs": [
- {
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "data_type": "text",
- "description": "string",
- "data_sources": [
- {
- "id": 0,
- "name": "string"
}
], - "display_name": "string",
- "id": 0,
- "input": "string",
- "instructions": "string",
- "labels": [
- {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
], - "meta": { },
- "name": "string",
- "project_id": 0,
- "task_type": "classification"
}
], - "users": [
- {
- "active": true,
- "email_address": "string",
- "full_name": "string",
- "id": 0,
- "intent": null,
- "role": "owner",
- "tier_id": 0,
- "tier": {
- "updated_at": "2019-08-24T14:15:22Z",
- "created_at": "2019-08-24T14:15:22Z",
- "id": 0,
- "name": "string",
- "limits": {
- "projects": 0,
- "annotators": 0,
- "dataset_size": 0,
- "predictions": 0
}
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "username": "string",
- "verified": true,
- "complete_tasks": 0,
- "incomplete_tasks": 0,
- "complete_and_not_flagged_tasks": 0,
- "incomplete_and_not_flagged_tasks": 0
}
], - "learner": {
- "id": 0,
- "current_status": "Untrained",
- "evaluations": [
- {
- "id": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "eval_metrics": { }
}
], - "latest_evaluation": {
- "id": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "eval_metrics": { }
}
}, - "datasets": [
- {
- "id": 0,
- "name": "string",
- "count": 0,
- "description": "string",
- "fields": [
- {
- "id": 0,
- "name": "string",
- "data_type": "text",
- "display_name": "string"
}
]
}
], - "data_count": 0,
- "data_without_tasks_count": 0,
- "tasks_count": 0,
- "latest_completed_tasks_batch_size": 0,
- "completed_tasks_count": 0,
- "incomplete_tasks_count": 0,
- "flagged_count": 0,
- "task_allocation_strategy": "automatic",
- "task_allocation_batch_size": 0,
- "dashboard_configuration": { }
}
Add Dataset
Add a dataset to the given project.
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
dataset_id required | integer (ID of the dataset to add) |
required | Array of objects (Inputs) [ items ] |
required | Array of objects (Outputs) [ items ] |
Responses
Request samples
- Payload
{- "dataset_id": 0,
- "inputs": [
- {
- "name": "string",
- "data_sources": [
- {
- "field_id": 0
}
], - "data_type": "text",
- "description": "string",
- "display_only": false,
- "meta": { }
}
], - "outputs": [
- {
- "name": "string",
- "data_sources": [
- {
- "field_id": 0
}
], - "display_name": "string",
- "description": "string",
- "instructions": "string",
- "data_type": "text",
- "task_type": "classification",
- "labels": [
- {
- "id": "string",
- "name": "string",
- "display_name": "string",
- "description": "string"
}
], - "meta": { },
- "input": "string"
}
]
}
Response samples
- 200
- 422
{- "id": 0,
- "name": "string",
- "shared_id": "string",
- "external_id": "string",
- "active": true,
- "auth_key": "string",
- "pred_count": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "default_dataset_id": 0,
- "description": "string",
- "guidelines": "string",
- "policy": { },
- "instructions": "string",
- "status": "Maintenance",
- "updated_at": "2019-08-24T14:15:22Z",
- "inputs": [
- {
- "data_type": "text",
- "description": "string",
- "data_sources": [
- {
- "id": 0,
- "name": "string"
}
], - "display_only": true,
- "id": 0,
- "meta": { },
- "name": "string",
- "project_id": 0,
- "updated_at": "2019-08-24T14:15:22Z",
- "created_at": "2019-08-24T14:15:22Z"
}
], - "outputs": [
- {
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "data_type": "text",
- "description": "string",
- "data_sources": [
- {
- "id": 0,
- "name": "string"
}
], - "display_name": "string",
- "id": 0,
- "input": "string",
- "instructions": "string",
- "labels": [
- {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
], - "meta": { },
- "name": "string",
- "project_id": 0,
- "task_type": "classification"
}
], - "users": [
- {
- "active": true,
- "email_address": "string",
- "full_name": "string",
- "id": 0,
- "intent": null,
- "role": "owner",
- "tier_id": 0,
- "tier": {
- "updated_at": "2019-08-24T14:15:22Z",
- "created_at": "2019-08-24T14:15:22Z",
- "id": 0,
- "name": "string",
- "limits": {
- "projects": 0,
- "annotators": 0,
- "dataset_size": 0,
- "predictions": 0
}
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "username": "string",
- "verified": true,
- "complete_tasks": 0,
- "incomplete_tasks": 0,
- "complete_and_not_flagged_tasks": 0,
- "incomplete_and_not_flagged_tasks": 0
}
], - "learner": {
- "id": 0,
- "current_status": "Untrained",
- "evaluations": [
- {
- "id": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "eval_metrics": { }
}
], - "latest_evaluation": {
- "id": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "eval_metrics": { }
}
}, - "datasets": [
- {
- "id": 0,
- "name": "string",
- "count": 0,
- "description": "string",
- "fields": [
- {
- "id": 0,
- "name": "string",
- "data_type": "text",
- "display_name": "string"
}
]
}
], - "data_count": 0,
- "data_without_tasks_count": 0,
- "tasks_count": 0,
- "latest_completed_tasks_batch_size": 0,
- "completed_tasks_count": 0,
- "incomplete_tasks_count": 0,
- "flagged_count": 0,
- "task_allocation_strategy": "automatic",
- "task_allocation_batch_size": 0,
- "dashboard_configuration": { }
}
Remove Dataset
Remove the given dataset from the given project.
Authorizations:
path Parameters
project_id required | integer (Project Id) |
dataset_id required | integer (Dataset Id) |
Responses
Response samples
- 422
{- "detail": [
- {
- "loc": [
- "string"
], - "msg": "string",
- "type": "string"
}
]
}
Create Prediction
Use this endpoint to make a prediction on new data with your model.
Provide a batch of data points that include the input fields required by your project to generate a batch of predictions with confidence scores from the latest version of your Humanloop AI model.
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
required | Array of objects (An array of data containing the input fields defined by your project and any additional key-value metadata you wish to record for a prediction such as your document ids.) [ items ] |
n_best | integer (Controls the number of most confident predictions from the model to return for each data point.) Default: 1 |
save | boolean (Whether or not the Humanloop platform will log the request and response data from a prediction) Default: true |
Responses
Request samples
- Payload
{- "data": [
- {
- "text": "Your document data...",
- "external_id": "123"
}
], - "n_best": 1,
- "save": true
}
Response samples
- 200
- 422
[- {
- "id": 0,
- "data": {
- "text": "Your document data...",
- "external_id": "123"
}, - "results": [
- {
- "id": "string",
- "confidence": 0,
- "start": 0,
- "end": 0,
- "text": "string",
- "label": {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}, - "result_type": "span"
}
], - "usage": "testing"
}
]
Create Prediction With Fallback
Use this endpoint to make a prediction on new or existing data with your latest Humanloop model.
For existing data, provide a batch of data_ids.
For new data, provide a batch of data points that include the input fields required by your project.
To support fallback to a human for difficult examples you can set a confidence threshold. If the model's confidence is less than this threshold, a task will be routed to one of your project team members. To be notified of when the task has been completed, you must first have created a subscription to our task_completed event (see /subscriptions endpoint for more details).
An array of predictions with confidence scores and the corresponding task details if relevant will be returned.
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
required | Array of objects (An array of datapoints containing the input fields defined by your project and any additional key-value metadata you wish to record for a prediction such as your document ids. A new datapoint will automatically be created for this data.) [ items ] |
data_ids | Array of integers (Humanloop ids of existing datapoints that you wish to generate a new prediction with. This should be specified instead of the data field where appropriate) [ items ] |
n_best | integer (Controls the number of most confident predictions from the model to return for each data point.) Default: 1 |
batch_id | string (A user provided id for grouping batches of predictions.) |
confidence_threshold | number (A value between 0 and 1. If the model confidence (for the highest confident value returned) is below this threshold, a task will be routed to a user to complete) Default: 0 |
Responses
Request samples
- Payload
{- "data": [
- {
- "text": "Your document data...",
- "external_id": "123"
}
], - "data_ids": [
- 0
], - "n_best": 1,
- "batch_id": "string",
- "confidence_threshold": 0
}
Response samples
- 200
- 422
[- {
- "prediction_id": 0,
- "batch_id": "string",
- "data": {
- "text": "Your document data...",
- "external_id": "123"
}, - "data_id": 0,
- "results": [
- {
- "id": "string",
- "confidence": 0,
- "start": 0,
- "end": 0,
- "text": "string",
- "label": {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}, - "result_type": "span"
}
], - "confidence_threshold": 0,
- "created_at": "2019-08-24T14:15:22Z",
- "task_id": 0,
- "user": "string",
- "task_url": "string"
}
]
Get Project Job Details
Retrieve details of the long running/batched async jobs that have been triggered against this project historically.
Use this endpoint to check the status of batch jobs for example, when waiting:
- for batch predictions to complete across all existing data associated to a project.
- for a model to re-train
- for a model to re-rank the unlabelled data using active learning
Authorizations:
path Parameters
id required | integer (Id) |
query Parameters
job_type | string Default: "update_learner" Enum: "update_learner" "batch_scoring" What jobs do we support to submit to a worker |
Responses
Response samples
- 200
- 422
null
Get Labelled Data
Download all the completed task data for your project - including all your user assignments and their annotations and details of which data points were used for training, validation and testing purposes.
Authorizations:
path Parameters
id required | integer (Id) |
query Parameters
format | string (Format) Default: "json" |
Responses
Response samples
- 200
- 422
null
Get Paginated Project Data
Get a page of project data for a given project with corresponding task data.
You must provide page
and size
parameters. Up to size
records
will be returned, starting from page
* size
.
Authorizations:
path Parameters
project_id required | integer (Project Id) |
query Parameters
size | integer (Size) Default: 50 |
page | integer (Page) Default: 0 |
status | string (ProjectDataStatus) Enum: "complete" "needs_review" "pending" "in_progress" Status of a project's datapoint |
flagged | boolean (Flagged) |
sort | string (Sort) Default: "id_ASC" |
Responses
Response samples
- 200
- 422
null
Get Model Statistics
Get detailed information on the performance of your projects model, including training, validation and test evaluation loss and evaluation data over time.
Authorizations:
path Parameters
project_id required | integer (Project Id) |
Responses
Response samples
- 200
- 422
[- {
- "id": 0,
- "learner_id": 0,
- "eval_metrics": {
- "train_loss": [
- 0
], - "dev_loss": [
- 0
], - "test_loss": [
- 0
], - "train_eval": [
- {
- "main_score": 0,
- "accuracy": 0,
- "macro_f_score": 0,
- "micro_f_score": 0,
- "class_breakdown": {
- "property1": {
- "precision": 0,
- "recall": 0,
- "f1-score": 0,
- "support": 0
}, - "property2": {
- "precision": 0,
- "recall": 0,
- "f1-score": 0,
- "support": 0
}
}
}
], - "dev_eval": [
- {
- "main_score": 0,
- "accuracy": 0,
- "macro_f_score": 0,
- "micro_f_score": 0,
- "class_breakdown": {
- "property1": {
- "precision": 0,
- "recall": 0,
- "f1-score": 0,
- "support": 0
}, - "property2": {
- "precision": 0,
- "recall": 0,
- "f1-score": 0,
- "support": 0
}
}
}
], - "test_eval": [
- {
- "main_score": 0,
- "accuracy": 0,
- "macro_f_score": 0,
- "micro_f_score": 0,
- "class_breakdown": {
- "property1": {
- "precision": 0,
- "recall": 0,
- "f1-score": 0,
- "support": 0
}, - "property2": {
- "precision": 0,
- "recall": 0,
- "f1-score": 0,
- "support": 0
}
}
}
], - "learning_rate": [
- 0
], - "test_score": 0,
- "num_labels": 0
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
]
Get Task Summary Statistics
Get detailed task summary statistics for this project, including the number of flagged datapoints, the datapoint statuses, and a summary of overall datapoint task completion
Authorizations:
path Parameters
project_id required | integer (Project Id) |
Responses
Response samples
- 200
- 422
null
Trigger Model Training
Trigger your model to retrain against all your existing annotated data
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
max_epochs | integer (Max Epochs) |
learning_rate | number (Learning Rate) |
reinitialise | boolean (Reinitialise) Default: false |
Responses
Request samples
- Payload
{- "max_epochs": 0,
- "learning_rate": 0,
- "reinitialise": false
}
Response samples
- 200
- 422
null
A project task is assigned to a user to provide annotations for a particular project data-point and tracks the lifecycle of the users work within the application. A project data-point may have many tasks assigned in the case where the project requires multiple users to annotate the same data-point.
Get Tasks
Get all task data for a given project.
You can provide different criteria to sort the response by as well as provide search words to filter the response by.
Authorizations:
path Parameters
id required | integer (Id) |
query Parameters
size | integer (Size) Default: 5000 |
page | integer (Page) Default: 0 |
complete | boolean (Complete) |
flagged | boolean (Flagged) |
sortby | string Default: "confusion" Enum: "confidence" "confusion" "last_updated" Different sorting modes for task data |
search_keywords | string (Search Keywords) |
Responses
Response samples
- 200
- 422
null
Create Tasks
Add new tasks to your project for your team to annotate more data to help improve your models over time. If no user is provided, we automatically select a suitable user.
Each task corresponds to a data point that requires annotation by a user. One data point can have more than one task associated to it in the case where multiple annotations from different users are required.
If the data points provided when creating tasks do not yet exist in your associated dataset, we add them to the dataset automatically, otherwise we reference the existing data points.
See our /subscriptions webhook docs for information setting up notifications for when tasks are created and completed.
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
required | Array of objects (An array of data containing the input fields defined by your project and any additional key-value meta-data you wish to record for a task such as your document ids.) [ items ] |
user | string (The email address of the user to assign the new task to.) |
usage | string (Specify if the associated datapoint and annotations for this task should be used for model training, validation or testing. If this is notspecified Humanloop will determine the usage automatically.) Enum: "testing" "validation" "training" How a data point has been used when updating an AI model for this project |
complete | boolean (By default if the datapoint associated to a task has a valid annotation defined on creation the task will automatically be marked as complete so it used for model training. This field allows you to mark tasks with annotations defined as incomplete so that it is added to the queue for review before being used for model training.) |
Responses
Request samples
- Payload
{- "data": [
- {
- "text": "Your document data...",
- "external_id": "123"
}
], - "user": "string",
- "usage": "testing",
- "complete": true
}
Response samples
- 200
- 422
null
Get User Tasks
Get all task data for a given user from a project.
You can provide different criteria to sort the response by as well as provide search words to filter the response on.
Authorizations:
path Parameters
id required | integer (Id) |
user_id required | integer (User Id) |
query Parameters
size | integer (Size) Default: 50 |
page | integer (Page) Default: 0 |
complete | boolean (Complete) |
flagged | boolean (Flagged) |
sortby | string Default: "confusion" Enum: "confidence" "confusion" "last_updated" Different sorting modes for task data |
search_keywords | string (Search Keywords) |
Responses
Response samples
- 200
- 422
null
Get Task
Get details of a task.
Authorizations:
path Parameters
id required | integer (Id) |
task_id required | integer (Task Id) |
Responses
Response samples
- 200
- 422
{- "id": 0,
- "user_id": 0,
- "full_name": "string",
- "email_address": "string",
- "project_id": 0,
- "project_data_id": 0,
- "data_id": 0,
- "inputs": {
- "text": "Your document data...",
- "external_id": "123"
}, - "complete": true,
- "flagged": true,
- "score": 0,
- "usage": "testing",
- "url": "string",
- "comment": "string",
- "annotations": [
- {
- "id": "string",
- "result_type": "span",
- "start": 0,
- "end": 0,
- "text": "string",
- "label_id": "string",
- "label_name": "string",
- "label_display_name": "string",
- "label_description": "string"
}
], - "predictions": [
- {
- "id": "string",
- "result_id": "string",
- "result_type": "span",
- "start": 0,
- "end": 0,
- "text": "string",
- "confidence": 0,
- "label_id": "string",
- "label_name": "string",
- "label_display_name": "string",
- "label_description": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
]
}
Update Task
Update a given task.
If you provide annotations when updating a task and set the task as complete, these will be used as feedback to improve the performance of your model automatically.
You can also update the user assignment of a task by providing the email_address of a user from your project.
Authorizations:
path Parameters
task_id required | integer (Task Id) |
id required | integer (Id) |
Request Body schema: application/json
Classification (object) or Array of Span (objects) or Array of Classification (objects) (The user-provided annotations) | |
email_address | string (The email address of the user to be assigned to the task to provide annotations.) |
flagged | boolean (Whether the task has been flagged by a user - usually indicating it needs to be revisited at a later time.) |
complete | boolean (Whether or not the task has been completed meaning the annotations will be used to improve your AI model) |
comment | string (A comment on the associated data-point,generally used to explain why data has been flagged.) |
usage | string (Specify if the associated datapoint and annotations for this task should be used for model training, validation or testing. If this is notspecified Humanloop will determine the usage automatically.) Enum: "testing" "validation" "training" How a data point has been used when updating an AI model for this project |
Responses
Request samples
- Payload
{- "annotations": {
- "label": "string",
- "confidence": 0,
- "strength": 0,
- "id": "string",
- "object": "classification"
}, - "email_address": "string",
- "flagged": true,
- "complete": true,
- "comment": "string",
- "usage": "testing"
}
Response samples
- 200
- 422
{- "id": 0,
- "user_id": 0,
- "full_name": "string",
- "email_address": "string",
- "project_id": 0,
- "project_data_id": 0,
- "data_id": 0,
- "inputs": {
- "text": "Your document data...",
- "external_id": "123"
}, - "complete": true,
- "flagged": true,
- "score": 0,
- "usage": "testing",
- "url": "string",
- "comment": "string",
- "annotations": [
- {
- "id": "string",
- "result_type": "span",
- "start": 0,
- "end": 0,
- "text": "string",
- "label_id": "string",
- "label_name": "string",
- "label_display_name": "string",
- "label_description": "string"
}
], - "predictions": [
- {
- "id": "string",
- "result_id": "string",
- "result_type": "span",
- "start": 0,
- "end": 0,
- "text": "string",
- "confidence": 0,
- "label_id": "string",
- "label_name": "string",
- "label_display_name": "string",
- "label_description": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
]
}
Create Tasks For User
Add new tasks for datapoints without any tasks for the specified user to annotate.
Tasks are created preferentially for datapoints that will help the model the most (i.e. those with the highest active learning scores).
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
user required | string (The email address of the user to assign the new tasks to. Note that other users may also receive tasks to satisfy the project's annotation policy.) |
datapoints required | integer (Number of datapoints to create tasks for.) |
Responses
Request samples
- Payload
{- "user": "string",
- "datapoints": 0
}
Response samples
- 200
- 422
{- "created_tasks": 0
}
Bulk Create Tasks
Add new annotation tasks to your project to help improve your models over time.
A more performant endpoint for Task creation suitable for batches of new data.
This is a lighter weight version of the /projects/{id}/tasks endpoint, with the following differences in behaviour:
- Assumes data does not exist and so does not perform de-dupe checks
- Does not trigger the model to re-train
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
required | Array of objects (An array of data containing the input fields and optionally the output fields defined by your project. Include the output fields if you have existing annotations available.You can also include any additional key-value meta-data you wish to record for a task such as your document ids.) [ items ] |
user | string (The email address of the user to assign the batch of tasks to.) |
usage | string (Specify if the associated datapoint and annotations for this task should be used for model training, validation or testing. If this is notspecified Humanloop will determine the usage automatically.) Enum: "testing" "validation" "training" How a data point has been used when updating an AI model for this project |
complete | boolean (By default if the datapoint associated to a task has a valid annotation defined on creation the task will automatically be marked as complete so it used for model training. This field allows you to mark tasks with annotations defined as incomplete so that it is added to the queue for review by an annotator user before being used for model training.) |
Responses
Request samples
- Payload
{- "data": [
- {
- "text": "Your document data...",
- "external_id": "123"
}
], - "user": "string",
- "usage": "testing",
- "complete": true
}
Response samples
- 200
- 422
null
Reassign Tasks
Reassign a number of tasks from a source user to a target user.
Care is taken to ensure that the target user does not receive new tasks for datapoints they have already annotated (or have incomplete tasks for).
Returns the number of tasks successfully transferred, and the source and target users' task counts.
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
source_user required | string (The email address of the user to take tasks from.) |
target_user required | string (The email address of the user to give tasks to.) |
tasks required | integer (The number of tasks to attempt to reassign) |
Responses
Request samples
- Payload
{- "source_user": "string",
- "target_user": "string",
- "tasks": 0
}
Response samples
- 200
- 422
{- "reassigned_tasks": 0
}
The label taxonomy definition for your projects outputs. Used to define your annotations and your model predictions.
Get Labels
Get all labels for an existing project output definition
Authorizations:
path Parameters
id required | integer (Id) |
Responses
Response samples
- 200
- 422
[- {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
]
Update Labels
Update the data on an array of labels based on id
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
id | string (Id) |
name | string (Name) |
display_name | string (Display Name) |
description | string (Description) |
Responses
Request samples
- Payload
[- {
- "id": "string",
- "name": "string",
- "display_name": "string",
- "description": "string"
}
]
Response samples
- 200
- 422
[- {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
]
Add Labels
Add a label or list of labels to an existing project's output definition
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
name required | string (Name) |
display_name | string (Display Name) |
description | string (Description) |
Responses
Request samples
- Payload
{- "name": "string",
- "display_name": "string",
- "description": "string"
}
Response samples
- 200
- 422
[- {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
]
Delete Labels
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
id | string (Id) |
name | string (Name) |
display_name | string (Display Name) |
description | string (Description) |
Responses
Request samples
- Payload
[- {
- "id": "string",
- "name": "string",
- "display_name": "string",
- "description": "string"
}
]
Response samples
- 200
- 422
[- {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
]
Update Label
Update any of the data for an existing label with label_id
Authorizations:
path Parameters
id required | integer (Id) |
label_id required | string (Label Id) |
Request Body schema: application/json
id | string (Id) |
name | string (Name) |
display_name | string (Display Name) |
description | string (Description) |
Responses
Request samples
- Payload
{- "id": "string",
- "name": "string",
- "display_name": "string",
- "description": "string"
}
Response samples
- 200
- 422
[- {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
]
Delete Label
Authorizations:
path Parameters
id required | integer (Id) |
label_id required | any (Label Id) |
Responses
Response samples
- 200
- 422
[- {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
]
Get Annotations
Get a page of annotations data for a given project.
This endpoint works like /tasks
, but you must provide
page
and size
parameters. Up to size
records
will be returned, starting from page
* size
.
user relates to user.full_name.
Authorizations:
query Parameters
project_id required | integer (Project Id) |
size | integer (Size) Default: 50 |
page | integer (Page) Default: 0 |
complete | boolean (Complete) |
flagged | boolean (Flagged) |
reviewed | boolean (Reviewed) |
user | string (User) |
keywords | string (Keywords) |
labels | Array of strings (Labels) |
task_id | integer (Task Id) |
Responses
Response samples
- 200
- 422
null
Create Annotation
Create an annotation
Authorizations:
Request Body schema: application/json
task_id required | integer (ID of the annotation's task) |
external_id | string (ID of the annotation to create) |
label_id | string (ID of the annotation's label) |
start | integer (Start offset of the annotation (span only)) |
end | integer (End offset of the annotation (span only)) |
text | string (The text the annotation covers (span only)) |
strength | number (strength of the association with the annotation's label (ordinal regression only)) |
Responses
Request samples
- Payload
{- "task_id": 0,
- "external_id": "string",
- "label_id": "string",
- "start": 0,
- "end": 0,
- "text": "string",
- "strength": 0
}
Response samples
- 200
- 422
null
Upsert Annotation
Create or update an annotation.
This endpoint is only used for single-label classification.
Authorizations:
path Parameters
annotation_id required | string (Annotation Id) |
Request Body schema: application/json
task_id required | integer (ID of the annotation's task) |
label_id | string (ID of the annotation's label) |
start | integer (Start offset of the annotation (span only)) |
end | integer (End offset of the annotation (span only)) |
text | string (The text the annotation covers (span only)) |
strength | number (strength of the association with the annotation's label (ordinal regression only)) |
Responses
Request samples
- Payload
{- "task_id": 0,
- "label_id": "string",
- "start": 0,
- "end": 0,
- "text": "string",
- "strength": 0
}
Response samples
- 200
- 422
null
Update Annotation
Update a given annotation.
Authorizations:
path Parameters
annotation_id required | string (Annotation Id) |
Request Body schema: application/json
reviewed | boolean (Whether this annotation has been reviewed) |
start | integer (Start offset of the annotation (span only)) |
end | integer (End offset of the annotation (span only)) |
label_id | string (ID of the annotation's label) |
strength | number (strength of the association with the annotation's label) |
text | string (The text the annotation covers) |
Responses
Request samples
- Payload
{- "reviewed": true,
- "start": 0,
- "end": 0,
- "label_id": "string",
- "strength": 0,
- "text": "string"
}
Response samples
- 200
- 422
{- "id": "string",
- "external_id": "string",
- "confidence": 0,
- "user": {
- "id": 0,
- "username": "string",
- "full_name": "string",
- "email_address": "string"
}, - "task": {
- "id": 0,
- "url": "string"
}, - "start": 0,
- "end": 0,
- "text": "string",
- "strength": 0,
- "label_id": "string",
- "label": {
- "name": "string",
- "id": "string",
- "display_name": "string",
- "description": "string",
- "count": 0,
- "object": "label",
- "output_id": 0,
- "output": {
- "id": 0,
- "name": "string",
- "display_name": "string",
- "description": "string",
- "task_type": "classification",
- "data_key": "string",
- "data_type": "text",
- "meta": { },
- "project_id": 0,
- "object": "output"
}, - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}, - "result_type": "string",
- "inputs": { },
- "reviewed": true
}
Manage subscriptions using our RESTHook interface to get notified of particular events from your Humanloop projects. For example, it may be helpful for your system to be notified when annotation tasks have been completed.
Create Subscription
Add a subscription to a project to get notifications for events of interest such as when a task has been completed.
Authorizations:
path Parameters
id required | integer (Id) |
Request Body schema: application/json
event | string (An event one can subscribe to for a project.) Default: "task.completed" Enum: "task.completed" "task.created" Events users of Humanloop can subscribe to |
callback_url | string (The callback URL for Humanloop to post notifications to for this subscription.) |
callback_headers | object (A dictionary of valid HTTP headers that Humanloop should include in posts to the callback_url.) |
email_address | string (The email_address for humanloop to send notifications to or this subscription.) |
active | boolean (Whether or not the subscription is active, meaning notifications will be sent for the specified events.) Default: true |
Responses
Request samples
- Payload
{- "event": "task.completed",
- "callback_url": "string",
- "callback_headers": { },
- "email_address": "string",
- "active": true
}
Response samples
- 200
- 422
null
Update Subscription
Update an existing subscription. This is useful if you want to change callback details.
Authorizations:
path Parameters
id required | integer (Id) |
sub_id required | string (Sub Id) |
Request Body schema: application/json
event | string (An event one can subscribe to for a project.) Default: "task.completed" Enum: "task.completed" "task.created" Events users of Humanloop can subscribe to |
callback_url | string (The callback URL for Humanloop to post notifications to for this subscription.) |
callback_headers | object (A dictionary of valid HTTP headers that Humanloop should include in posts to the callback_url.) |
email_address | string (The email_address for humanloop to send notifications to or this subscription.) |
active | boolean (Whether or not the subscription is active, meaning notifications will be sent for the specified events.) Default: true |
Responses
Request samples
- Payload
{- "event": "task.completed",
- "callback_url": "string",
- "callback_headers": { },
- "email_address": "string",
- "active": true
}
Response samples
- 200
- 422
null
Deactivate Subscription
Deactivate an existing subscription, which will result in you no longer receiving notifications from the events associated to this subscription.
Authorizations:
path Parameters
id required | integer (Id) |
sub_id required | string (Sub Id) |
Responses
Response samples
- 200
- 422
null
Test Subscription
Test a subscription.
Posting to this endpoint using the test_message query param will trigger a test notification from the Humanloop platform to the subscriptions callback url.
Authorizations:
path Parameters
id required | integer (Id) |
sub_id required | string (Sub Id) |
query Parameters
test_message required | string (Test Message) |
Responses
Response samples
- 200
- 422
null
Get Logs
Retrieve paginated logs from the server.
Sorting and filtering are supported through query params.
Sorting and filtering are mainly supported for the categorical columns, and text search is supported for the model inputs and output.
Sorting is supported for the source
, model
, timestamp
, and feedback-{output_name}
columns.
Specify sorting with the sort
query param, with values {column}.{ordering}
.
E.g. ?sort=source.asc&sort=model.desc will yield a multi-column sort. First by source then by model.
Filtering is supported for the source
, model
, and feedback-{output_name}
columns.
Specify filtering with the source_filter
, model_filter
, and feedback-{output.name}_filter
query params.
E.g. ?source_filter=AI&source_filter=user_1234&feedback-explicit_filter=good
will only show rows where the source is "AI" or "user_1234", and where the latest feedback for the "explicit" output
group is "good".
Searching is supported for the model inputs and output.
Specify a search term with the search
query param.
E.g. ?search=hello%20there will cause a case-insensitive search across model inputs and output.
Authorizations:
query Parameters
project_id required | integer (Project Id) |
search | string (Search) |
size | integer (Size) Default: 50 |
page | integer (Page) Default: 0 |
Responses
Response samples
- 200
- 422
{- "page": 0,
- "size": 0,
- "records": [
- {
- "id": "string",
- "source": "string",
- "model_config": {
- "id": "string",
- "display_name": "string",
- "model_name": "string",
- "prompt_template": "string",
- "parameters": { },
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "description": "string",
- "last_used": "2019-08-24T14:15:22Z",
- "feedback_stats": [
- {
- "feedback_group_id": 0,
- "feedback_group_name": "string",
- "feedback_label": "string",
- "feedback_count": 0
}
], - "num_datapoints": 0
}, - "inputs": {
- "property1": "string",
- "property2": "string"
}, - "output": "string",
- "metadata": {
- "property1": "string",
- "property2": "string"
}, - "feedback": [
- {
- "id": "string",
- "log_id": "string",
- "group": "string",
- "label": "string",
- "text": "string",
- "source": "string",
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
], - "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z"
}
], - "total": 0
}
Log
Log a datapoint or array of datapoints to your Humanloop project.
Authorizations:
Request Body schema: application/json
project required | string (Project name) Unique project name. If it does not exist, a new project will be created. |
required | object (Model input data) List of name, value pairs for the inputs used by your prompt template, or directly by your model. |
output required | string (Model output) Generated output from your model for the provided inputs. |
model | string (Type of model used) What model type was used for the generation? e.g. text-davinci-002. This can be null if the generation is user provided |
prompt_template | string (Prompt template) Prompt template that incorporated your specified inputs to form your final request to the model. |
source required | string (Source of generation) What was used to generate the output? e.g. gpt-3. |
object (Model parameters) The hyper-parameter settings that along with your model source and prompt template (if provided) will uniquely determine a model configuration on Humanloop. For example the temperature setting. | |
object (Metadata) Any additional metadata that you would like to log for reference. | |
Feedback (object) or Array of Feedback (objects) (Feedback labels) Optional parameter to provide feedback with your logged datapoint. | |
created_at | string <date-time> (Created at) Timestamp for when the log was created. If not provided, the time the log call was made will be used as a timestamp. |
Responses
Request samples
- Payload
[ ]
Response samples
- 200
- 422
[ ]
Feedback
Submit a list of feedback for existing log_ids
Authorizations:
Request Body schema: application/json
group required | string (Group) |
label | string (Label) A categorical label to characterize the type of feedback.Only one of label or correction should be provided for each feedback. |
text | string (Text) Text feedback. Can be used to record model corrections from your users. Only one of label or text should be provided for each feedback. |
log_id | string (Log ID) id to associate the feedback to a previously logged datapoint.When providing instant feedback as part of the hl.log(...) call you don't need to provide a log_id. |
source | string (Source) A unique to who/what provided the feedback. For example a unique user reference. |
created_at | string <date-time> (Created at) Timestamp for when the feedback was created. If not provided, the time the call was made will be used as a timestamp. |
Responses
Request samples
- Payload
{- "group": "string",
- "label": "string",
- "text": "string",
- "log_id": "string",
- "source": "string",
- "created_at": "2019-08-24T14:15:22Z"
}
Response samples
- 200
- 422
[ ]
Get Project Model Configs Deprecated
Get an array of model_configs associated to your project
Authorizations:
query Parameters
project_id required | integer (Project Id) |
Responses
Response samples
- 200
- 422
[- {
- "id": "string",
- "display_name": "string",
- "model_name": "string",
- "prompt_template": "string",
- "parameters": { },
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "description": "string",
- "last_used": "2019-08-24T14:15:22Z",
- "feedback_stats": [
- {
- "feedback_group_id": 0,
- "feedback_group_name": "string",
- "feedback_label": "string",
- "feedback_count": 0
}
], - "num_datapoints": 0
}
]
Register Project Model Config
Register a model config to a project and optionally tag it with an experiment. If the project does not exist, a new project will be added automatically.
Authorizations:
Request Body schema: application/json
name | string (Config name) A friendly display name for config. If not provided a name will be generated. |
project required | string (Project name) Unique project name. If it does not exist, a new project will be created. |
experiment | string (Experiment tag) Include a model configuration in an experiment group. Experiment groups are used for A/B testing and optimizing hyper-parameters. |
model required | string (Type of model used) What model type was used for the generation? e.g. text-davinci-002. |
prompt_template | string (Prompt template) Prompt template that incorporated your specified inputs to form your final request to the model. |
object (Model parameters) The hyper-parameter settings that along with your model source and prompt template (if provided) will uniquely determine a model configuration on Humanloop. For example the temperature setting. |
Responses
Request samples
- Payload
{- "name": "string",
- "project": "string",
- "experiment": "string",
- "model": "string",
- "prompt_template": "string",
- "parameters": {
- "property1": "string",
- "property2": "string"
}
}
Response samples
- 200
- 422
{- "id": "string",
- "display_name": "string",
- "model_name": "string",
- "prompt_template": "string",
- "parameters": { },
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "description": "string",
- "last_used": "2019-08-24T14:15:22Z",
- "feedback_stats": [
- {
- "feedback_group_id": 0,
- "feedback_group_name": "string",
- "feedback_label": "string",
- "feedback_count": 0
}
], - "num_datapoints": 0
}
Get Project Model Configs
Get an array of model_configs associated to your project
Authorizations:
path Parameters
project_id required | integer (Project Id) |
Responses
Response samples
- 200
- 422
[- {
- "id": "string",
- "display_name": "string",
- "model_name": "string",
- "prompt_template": "string",
- "parameters": { },
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "description": "string",
- "last_used": "2019-08-24T14:15:22Z",
- "feedback_stats": [
- {
- "feedback_group_id": 0,
- "feedback_group_name": "string",
- "feedback_label": "string",
- "feedback_count": 0
}
], - "num_datapoints": 0
}
]
Update Project Model Config
Update a model configuration within your project
Authorizations:
path Parameters
project_id required | integer (Project Id) |
model_config_id required | string (Model Config Id) |
Request Body schema: application/json
display_name | string (Display Name) |
description | string (Description) |
Responses
Request samples
- Payload
{- "display_name": "string",
- "description": "string"
}
Response samples
- 200
- 422
{- "id": "string",
- "display_name": "string",
- "model_name": "string",
- "prompt_template": "string",
- "parameters": { },
- "created_at": "2019-08-24T14:15:22Z",
- "updated_at": "2019-08-24T14:15:22Z",
- "description": "string",
- "last_used": "2019-08-24T14:15:22Z",
- "feedback_stats": [
- {
- "feedback_group_id": 0,
- "feedback_group_name": "string",
- "feedback_label": "string",
- "feedback_count": 0
}
], - "num_datapoints": 0
}
Export Fine Tune
Retrieve filtered logs in the format for fine-tuning.
Authorizations:
query Parameters
project_id required | integer (Project Id) |
search | string (Search) |
Request Body schema: application/json
separator | string (Separator) Default: " ->" |
completion_initial_whitespace | string (Completion Initial Whitespace) Default: " " |
stop_sequence | string (Stop Sequence) Default: "##" |
Responses
Request samples
- Payload
{- "separator": " ->",
- "completion_initial_whitespace": " ",
- "stop_sequence": "##"
}
Response samples
- 200
- 422
null