Humanloop API ( 0.0.1 )

Responses

Response samples

200

Content type

application/json

{"id": 0,
"username": "string",
"email_address": "string",
"role": "owner",
"full_name": "string",
"auth_key": "string",
"active": true,
"verified": true,
"tier_id": 0,
"tier": {"name": "string",
"limits": {"property1": 0,
"property2": 0
}
},
"intent": null,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"feature_flags": {"property1": true,
"property2": true
}
}

Update Me

Update the profile of your current user

Authorizations:

Request Body schema: application/json

username	string (Username)
email_address	string (Email Address)
password	string (Password)
full_name	string (Full Name)
intent	Array of strings (Intent)

Responses

Request samples

Payload

Content type

application/json

{"username": "string",
"email_address": "string",
"password": "string",
"full_name": "string",
"intent": ["string"
]
}

Response samples

200
422

Content type

application/json

{"id": 0,
"username": "string",
"email_address": "string",
"role": "owner",
"full_name": "string",
"auth_key": "string",
"active": true,
"verified": true,
"tier_id": 0,
"tier": {"name": "string",
"limits": {"property1": 0,
"property2": 0
}
},
"intent": null,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"feature_flags": {"property1": true,
"property2": true
}
}

Create User

Create a new user.

Request Body schema: application/json

username required	string (Username)
email_address required	string (Email Address)
password required	string (Password)
role	string Default: "owner" Enum: "owner" "annotator" "admin" An enumeration.
full_name	string (Full Name)
intent	Array of strings (Intent)

Responses

Request samples

Payload

Content type

application/json

{"username": "string",
"email_address": "string",
"password": "string",
"role": "owner",
"full_name": "string",
"intent": ["string"
]
}

Response samples

200
422

Content type

application/json

null

Invite User

Invite a user to your project using their email address. They will receive an email invite with a link to signup to Humanloop. When they log in they will then see the project they have been invited to.

Authorizations:

query Parameters

id	integer (Id)

Request Body schema: application/json

email_address required	string (Email Address)
project_id required	integer (Project Id)
role	string Default: "annotator" Enum: "owner" "annotator" "admin" An enumeration.
full_name	string (Full Name)

Responses

Request samples

Payload

Content type

application/json

{"email_address": "string",
"project_id": 0,
"role": "annotator",
"full_name": "string"
}

Response samples

200
422

Content type

application/json

null

Recover Password

Trigger a password recovery email to be sent.

path Parameters

email

required

string (Email)

Responses

Response samples

200
422

Content type

application/json

null

datasets

A dataset combines multiple data points and their schema definitions

Get Datasets

Get dataset details and associated field meta-data for any data-sets associated to your user.

Authorizations:

Responses

Response samples

200

Content type

application/json

[{"id": 0,
"name": "string",
"count": 0,
"data": [{"id": 0,
"dataset_id": 0,
"values": { },
"source": "string",
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"processing_status": "processing"
}
],
"description": "string",
"public": true,
"fields": [{"id": 0,
"dataset_id": 0,
"name": "string",
"display_name": "string",
"data_type": "text",
"required": true,
"meta": { },
"count": 0,
"null_count": 0,
"value_counts": {"property1": 0,
"property2": 0
},
"latest_counts_increase": {"property1": 0,
"property2": 0
},
"distinct_values": ["string"
],
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
],
"field_names": ["string"
],
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"status_counts": {"property1": 0,
"property2": 0
},
"active": true
}
]

Create Dataset

Upload a dataset that you can use in your Humanloop projects to train models.

Authorizations:

Request Body schema: application/json

name required	string (Name)
required	Array of objects (Data) [ items ]
required	Array of objects (Fields) [ items ]
description	string (Description)

Responses

Request samples

Payload

Content type

application/json

{"name": "string",
"data": [{"property1": "string",
"property2": "string"
}
],
"fields": [{"name": "string",
"data_type": "text",
"display_name": "string",
"required": true,
"default": "string",
"distinct_values": ["string"
],
"meta": { }
}
],
"description": "string"
}

Response samples

200
422

Content type

application/json

{"id": 0,
"attempted": 0,
"succeeded": 0,
"processing": 0,
"msg": "string",
"fields": [{"id": 0,
"dataset_id": 0,
"name": "string",
"display_name": "string",
"data_type": "text",
"required": true,
"meta": { },
"count": 0,
"null_count": 0,
"value_counts": {"property1": 0,
"property2": 0
},
"latest_counts_increase": {"property1": 0,
"property2": 0
},
"distinct_values": ["string"
],
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
]
}

Get Dataset

Get dataset details and associated field meta-data for a specific data-set associated to your user.

Authorizations:

path Parameters

id

required

integer (Id)

query Parameters

num_records

integer (Num Records)

Default: 0

Responses

Response samples

200
422

Content type

application/json

{"id": 0,
"name": "string",
"count": 0,
"data": [{"id": 0,
"dataset_id": 0,
"values": { },
"source": "string",
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"processing_status": "processing"
}
],
"description": "string",
"public": true,
"fields": [{"id": 0,
"dataset_id": 0,
"name": "string",
"display_name": "string",
"data_type": "text",
"required": true,
"meta": { },
"count": 0,
"null_count": 0,
"value_counts": {"property1": 0,
"property2": 0
},
"latest_counts_increase": {"property1": 0,
"property2": 0
},
"distinct_values": ["string"
],
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
],
"field_names": ["string"
],
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"status_counts": {"property1": 0,
"property2": 0
},
"active": true
}

Update Dataset

Update an existing dataset by adding additional data points.

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

required

Array of objects (Data) [ items ]

Array

	string or number or Array of strings or Array of Span (objects) or Array of Classification (objects)
Any of string

Responses

Request samples

Payload

Content type

application/json

{"data": [{"property1": "string",
"property2": "string"
}
]
}

Response samples

200
422

Content type

application/json

{"id": 0,
"attempted": 0,
"succeeded": 0,
"processing": 0,
"msg": "string",
"fields": [{"id": 0,
"dataset_id": 0,
"name": "string",
"display_name": "string",
"data_type": "text",
"required": true,
"meta": { },
"count": 0,
"null_count": 0,
"value_counts": {"property1": 0,
"property2": 0
},
"latest_counts_increase": {"property1": 0,
"property2": 0
},
"distinct_values": ["string"
],
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
]
}

Delete Dataset

Delete your dataset if it is no longer needed.

Authorizations:

path Parameters

id

required

integer (Id)

Responses

Response samples

422

Content type

application/json

{"detail": [{"loc": ["string"
],
"msg": "string",
"type": "string"
}
]
}

Update Dataset Details

Update the details of your dataset such as its name and description.

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

name	string (Name)
description	string (Description)

Responses

Request samples

Payload

Content type

application/json

{"name": "string",
"description": "string"
}

Response samples

200
422

Content type

application/json

{"id": 0,
"name": "string",
"count": 0,
"data": [{"id": 0,
"dataset_id": 0,
"values": { },
"source": "string",
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"processing_status": "processing"
}
],
"description": "string",
"public": true,
"fields": [{"id": 0,
"dataset_id": 0,
"name": "string",
"display_name": "string",
"data_type": "text",
"required": true,
"meta": { },
"count": 0,
"null_count": 0,
"value_counts": {"property1": 0,
"property2": 0
},
"latest_counts_increase": {"property1": 0,
"property2": 0
},
"distinct_values": ["string"
],
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
],
"field_names": ["string"
],
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"status_counts": {"property1": 0,
"property2": 0
},
"active": true
}

Get Dataset Data File

Get file for a data point.

Implementation assumes each dataset can only have one file field.

Authorizations:

path Parameters

id required	integer (Id)
data_id required	integer (Data Id)

Responses

Response samples

422

Content type

application/json

{"detail": [{"loc": ["string"
],
"msg": "string",
"type": "string"
}
]
}

projects

Each project links users to a model and supporting dataset, specifies the expected project inputs and outputs, and manages tasks for users to collect more data and improve their model.

Get Projects

Get details of all the projects associated to your user.

Authorizations:

query Parameters

page	integer (Page) Default: 0
size	integer (Size) Default: 10
filter	string (Filter)

Responses

Response samples

200
422

Content type

application/json

null

Create Project

Create a new project on the Humanloop platform by defining inputs and outputs and associated users.

Authorizations:

Request Body schema: application/json

name required	string (Name)
required	Array of objects (Outputs) [ items ]
required	Array of objects (Inputs) [ items ]
instructions	string (Instructions)
guidelines	string (Guidelines)
description	string (Description)
users required	Array of strings (Users)
learner_config	object (Learner Config)
	object (Policy) A policy for how how many annotations to get on some proportion of the data. Providing multiple annotations for the same data-points provides consensus metrics and can help improve the quality of the final annotations provided to the model
task_allocation_strategy	string (TaskAllocationStrategy) Enum: "automatic" "manual" Strategy for automatic task allocation Controls whether new tasks are automatically allocated when a user completes their tasks.
task_allocation_batch_size	integer (Task Allocation Batch Size)
review_existing_annotations	boolean (If your dataset fields contain existing annotations then these can be used to warm start your project by specifying their field id as output data sources. You can have your annotator users first review these annotations before they are used to train your model by setting this 'review_existing_annotations' bool to True.) Default: false

Responses

Request samples

Payload

Content type

application/json

{"name": "string",
"outputs": [{"name": "string",
"data_sources": [{"field_id": 0
}
],
"display_name": "string",
"description": "string",
"instructions": "string",
"data_type": "text",
"task_type": "classification",
"labels": [{"id": "string",
"name": "string",
"display_name": "string",
"description": "string"
}
],
"meta": { },
"input": "string"
}
],
"inputs": [{"name": "string",
"data_sources": [{"field_id": 0
}
],
"data_type": "text",
"description": "string",
"display_only": false,
"meta": { }
}
],
"instructions": "string",
"guidelines": "string",
"description": "string",
"users": ["string"
],
"learner_config": { },
"policy": {"num_annotators": 1,
"data_proportion": 1,
"object": "policy",
"id": "string"
},
"task_allocation_strategy": "automatic",
"task_allocation_batch_size": 0,
"review_existing_annotations": false
}

Response samples

200
422

Content type

application/json

{"id": 0,
"name": "string",
"shared_id": "string",
"external_id": "string",
"active": true,
"auth_key": "string",
"pred_count": 0,
"created_at": "2019-08-24T14:15:22Z",
"default_dataset_id": 0,
"description": "string",
"guidelines": "string",
"policy": { },
"instructions": "string",
"status": "Maintenance",
"updated_at": "2019-08-24T14:15:22Z",
"inputs": [{"data_type": "text",
"description": "string",
"data_sources": [{"id": 0,
"name": "string"
}
],
"display_only": true,
"id": 0,
"meta": { },
"name": "string",
"project_id": 0,
"updated_at": "2019-08-24T14:15:22Z",
"created_at": "2019-08-24T14:15:22Z"
}
],
"outputs": [{"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"data_type": "text",
"description": "string",
"data_sources": [{"id": 0,
"name": "string"
}
],
"display_name": "string",
"id": 0,
"input": "string",
"instructions": "string",
"labels": [{"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
],
"meta": { },
"name": "string",
"project_id": 0,
"task_type": "classification"
}
],
"users": [{"active": true,
"email_address": "string",
"full_name": "string",
"id": 0,
"intent": null,
"role": "owner",
"tier_id": 0,
"tier": {"updated_at": "2019-08-24T14:15:22Z",
"created_at": "2019-08-24T14:15:22Z",
"id": 0,
"name": "string",
"limits": {"projects": 0,
"annotators": 0,
"dataset_size": 0,
"predictions": 0
}
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"username": "string",
"verified": true,
"complete_tasks": 0,
"incomplete_tasks": 0,
"complete_and_not_flagged_tasks": 0,
"incomplete_and_not_flagged_tasks": 0
}
],
"learner": {"id": 0,
"current_status": "Untrained",
"evaluations": [{"id": 0,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"eval_metrics": { }
}
],
"latest_evaluation": {"id": 0,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"eval_metrics": { }
}
},
"datasets": [{"id": 0,
"name": "string",
"count": 0,
"description": "string",
"fields": [{"id": 0,
"name": "string",
"data_type": "text",
"display_name": "string"
}
]
}
],
"data_count": 0,
"data_without_tasks_count": 0,
"tasks_count": 0,
"latest_completed_tasks_batch_size": 0,
"completed_tasks_count": 0,
"incomplete_tasks_count": 0,
"flagged_count": 0,
"task_allocation_strategy": "automatic",
"task_allocation_batch_size": 0,
"dashboard_configuration": { }
}

Get Project

Get details of a specific project.

Authorizations:

path Parameters

id

required

integer (Id)

Responses

Response samples

200
422

Content type

application/json

{"id": 0,
"name": "string",
"shared_id": "string",
"external_id": "string",
"active": true,
"auth_key": "string",
"pred_count": 0,
"created_at": "2019-08-24T14:15:22Z",
"default_dataset_id": 0,
"description": "string",
"guidelines": "string",
"policy": { },
"instructions": "string",
"status": "Maintenance",
"updated_at": "2019-08-24T14:15:22Z",
"inputs": [{"data_type": "text",
"description": "string",
"data_sources": [{"id": 0,
"name": "string"
}
],
"display_only": true,
"id": 0,
"meta": { },
"name": "string",
"project_id": 0,
"updated_at": "2019-08-24T14:15:22Z",
"created_at": "2019-08-24T14:15:22Z"
}
],
"outputs": [{"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"data_type": "text",
"description": "string",
"data_sources": [{"id": 0,
"name": "string"
}
],
"display_name": "string",
"id": 0,
"input": "string",
"instructions": "string",
"labels": [{"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
],
"meta": { },
"name": "string",
"project_id": 0,
"task_type": "classification"
}
],
"users": [{"active": true,
"email_address": "string",
"full_name": "string",
"id": 0,
"intent": null,
"role": "owner",
"tier_id": 0,
"tier": {"updated_at": "2019-08-24T14:15:22Z",
"created_at": "2019-08-24T14:15:22Z",
"id": 0,
"name": "string",
"limits": {"projects": 0,
"annotators": 0,
"dataset_size": 0,
"predictions": 0
}
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"username": "string",
"verified": true,
"complete_tasks": 0,
"incomplete_tasks": 0,
"complete_and_not_flagged_tasks": 0,
"incomplete_and_not_flagged_tasks": 0
}
],
"learner": {"id": 0,
"current_status": "Untrained",
"evaluations": [{"id": 0,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"eval_metrics": { }
}
],
"latest_evaluation": {"id": 0,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"eval_metrics": { }
}
},
"datasets": [{"id": 0,
"name": "string",
"count": 0,
"description": "string",
"fields": [{"id": 0,
"name": "string",
"data_type": "text",
"display_name": "string"
}
]
}
],
"data_count": 0,
"data_without_tasks_count": 0,
"tasks_count": 0,
"latest_completed_tasks_batch_size": 0,
"completed_tasks_count": 0,
"incomplete_tasks_count": 0,
"flagged_count": 0,
"task_allocation_strategy": "automatic",
"task_allocation_batch_size": 0,
"dashboard_configuration": { }
}

Update Project

Update the helper details of your project such as guidelines and descriptions.

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

name	string (Name)
instructions	string (Instructions)
guidelines	string (Guidelines)
description	string (Description)
learner_config	object (Learner Config)
	object (Policy) A policy for how how many annotations to get on some proportion of the data. Providing multiple annotations for the same data-points provides consensus metrics and can help improve the quality of the final annotations provided to the model
task_allocation_strategy	string (TaskAllocationStrategy) Enum: "automatic" "manual" Strategy for automatic task allocation Controls whether new tasks are automatically allocated when a user completes their tasks.
task_allocation_batch_size	integer (Task Allocation Batch Size)

Responses

Request samples

Payload

Content type

application/json

{"name": "string",
"instructions": "string",
"guidelines": "string",
"description": "string",
"learner_config": { },
"policy": {"num_annotators": 1,
"data_proportion": 1,
"object": "policy",
"id": "string"
},
"task_allocation_strategy": "automatic",
"task_allocation_batch_size": 0
}

Response samples

200
422

Content type

application/json

{"id": 0,
"name": "string",
"shared_id": "string",
"external_id": "string",
"active": true,
"auth_key": "string",
"pred_count": 0,
"created_at": "2019-08-24T14:15:22Z",
"default_dataset_id": 0,
"description": "string",
"guidelines": "string",
"policy": { },
"instructions": "string",
"status": "Maintenance",
"updated_at": "2019-08-24T14:15:22Z",
"inputs": [{"data_type": "text",
"description": "string",
"data_sources": [{"id": 0,
"name": "string"
}
],
"display_only": true,
"id": 0,
"meta": { },
"name": "string",
"project_id": 0,
"updated_at": "2019-08-24T14:15:22Z",
"created_at": "2019-08-24T14:15:22Z"
}
],
"outputs": [{"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"data_type": "text",
"description": "string",
"data_sources": [{"id": 0,
"name": "string"
}
],
"display_name": "string",
"id": 0,
"input": "string",
"instructions": "string",
"labels": [{"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
],
"meta": { },
"name": "string",
"project_id": 0,
"task_type": "classification"
}
],
"users": [{"active": true,
"email_address": "string",
"full_name": "string",
"id": 0,
"intent": null,
"role": "owner",
"tier_id": 0,
"tier": {"updated_at": "2019-08-24T14:15:22Z",
"created_at": "2019-08-24T14:15:22Z",
"id": 0,
"name": "string",
"limits": {"projects": 0,
"annotators": 0,
"dataset_size": 0,
"predictions": 0
}
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"username": "string",
"verified": true,
"complete_tasks": 0,
"incomplete_tasks": 0,
"complete_and_not_flagged_tasks": 0,
"incomplete_and_not_flagged_tasks": 0
}
],
"learner": {"id": 0,
"current_status": "Untrained",
"evaluations": [{"id": 0,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"eval_metrics": { }
}
],
"latest_evaluation": {"id": 0,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"eval_metrics": { }
}
},
"datasets": [{"id": 0,
"name": "string",
"count": 0,
"description": "string",
"fields": [{"id": 0,
"name": "string",
"data_type": "text",
"display_name": "string"
}
]
}
],
"data_count": 0,
"data_without_tasks_count": 0,
"tasks_count": 0,
"latest_completed_tasks_batch_size": 0,
"completed_tasks_count": 0,
"incomplete_tasks_count": 0,
"flagged_count": 0,
"task_allocation_strategy": "automatic",
"task_allocation_batch_size": 0,
"dashboard_configuration": { }
}

Delete Project

Delete your project if it is no longer needed.

Authorizations:

path Parameters

id

required

integer (Id)

Responses

Response samples

200
422

Content type

application/json

null

Clone Project

Clone an existing project on the Humanloop platform with updated config.

Currently this is aimed at the ability to quickly experiment with different configurations of a project. It does not yet support editing core aspects of the project such as input, output definitions and project data.

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

name	string (Name)
instructions	string (Instructions)
guidelines	string (Guidelines)
description	string (Description)
learner_config	object (Learner Config)
	object (Policy) A policy for how how many annotations to get on some proportion of the data. Providing multiple annotations for the same data-points provides consensus metrics and can help improve the quality of the final annotations provided to the model
task_allocation_strategy	string (TaskAllocationStrategy) Enum: "automatic" "manual" Strategy for automatic task allocation Controls whether new tasks are automatically allocated when a user completes their tasks.
task_allocation_batch_size	integer (Task Allocation Batch Size)

Responses

Request samples

Payload

Content type

application/json

{"name": "string",
"instructions": "string",
"guidelines": "string",
"description": "string",
"learner_config": { },
"policy": {"num_annotators": 1,
"data_proportion": 1,
"object": "policy",
"id": "string"
},
"task_allocation_strategy": "automatic",
"task_allocation_batch_size": 0
}

Response samples

200
422

Content type

application/json

{"id": 0,
"name": "string",
"shared_id": "string",
"external_id": "string",
"active": true,
"auth_key": "string",
"pred_count": 0,
"created_at": "2019-08-24T14:15:22Z",
"default_dataset_id": 0,
"description": "string",
"guidelines": "string",
"policy": { },
"instructions": "string",
"status": "Maintenance",
"updated_at": "2019-08-24T14:15:22Z",
"inputs": [{"data_type": "text",
"description": "string",
"data_sources": [{"id": 0,
"name": "string"
}
],
"display_only": true,
"id": 0,
"meta": { },
"name": "string",
"project_id": 0,
"updated_at": "2019-08-24T14:15:22Z",
"created_at": "2019-08-24T14:15:22Z"
}
],
"outputs": [{"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"data_type": "text",
"description": "string",
"data_sources": [{"id": 0,
"name": "string"
}
],
"display_name": "string",
"id": 0,
"input": "string",
"instructions": "string",
"labels": [{"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
],
"meta": { },
"name": "string",
"project_id": 0,
"task_type": "classification"
}
],
"users": [{"active": true,
"email_address": "string",
"full_name": "string",
"id": 0,
"intent": null,
"role": "owner",
"tier_id": 0,
"tier": {"updated_at": "2019-08-24T14:15:22Z",
"created_at": "2019-08-24T14:15:22Z",
"id": 0,
"name": "string",
"limits": {"projects": 0,
"annotators": 0,
"dataset_size": 0,
"predictions": 0
}
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"username": "string",
"verified": true,
"complete_tasks": 0,
"incomplete_tasks": 0,
"complete_and_not_flagged_tasks": 0,
"incomplete_and_not_flagged_tasks": 0
}
],
"learner": {"id": 0,
"current_status": "Untrained",
"evaluations": [{"id": 0,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"eval_metrics": { }
}
],
"latest_evaluation": {"id": 0,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"eval_metrics": { }
}
},
"datasets": [{"id": 0,
"name": "string",
"count": 0,
"description": "string",
"fields": [{"id": 0,
"name": "string",
"data_type": "text",
"display_name": "string"
}
]
}
],
"data_count": 0,
"data_without_tasks_count": 0,
"tasks_count": 0,
"latest_completed_tasks_batch_size": 0,
"completed_tasks_count": 0,
"incomplete_tasks_count": 0,
"flagged_count": 0,
"task_allocation_strategy": "automatic",
"task_allocation_batch_size": 0,
"dashboard_configuration": { }
}

Add Dataset

Add a dataset to the given project.

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

dataset_id required	integer (ID of the dataset to add)
required	Array of objects (Inputs) [ items ]
required	Array of objects (Outputs) [ items ]

Responses

Request samples

Payload

Content type

application/json

{"dataset_id": 0,
"inputs": [{"name": "string",
"data_sources": [{"field_id": 0
}
],
"data_type": "text",
"description": "string",
"display_only": false,
"meta": { }
}
],
"outputs": [{"name": "string",
"data_sources": [{"field_id": 0
}
],
"display_name": "string",
"description": "string",
"instructions": "string",
"data_type": "text",
"task_type": "classification",
"labels": [{"id": "string",
"name": "string",
"display_name": "string",
"description": "string"
}
],
"meta": { },
"input": "string"
}
]
}

Response samples

200
422

Content type

application/json

{"id": 0,
"name": "string",
"shared_id": "string",
"external_id": "string",
"active": true,
"auth_key": "string",
"pred_count": 0,
"created_at": "2019-08-24T14:15:22Z",
"default_dataset_id": 0,
"description": "string",
"guidelines": "string",
"policy": { },
"instructions": "string",
"status": "Maintenance",
"updated_at": "2019-08-24T14:15:22Z",
"inputs": [{"data_type": "text",
"description": "string",
"data_sources": [{"id": 0,
"name": "string"
}
],
"display_only": true,
"id": 0,
"meta": { },
"name": "string",
"project_id": 0,
"updated_at": "2019-08-24T14:15:22Z",
"created_at": "2019-08-24T14:15:22Z"
}
],
"outputs": [{"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"data_type": "text",
"description": "string",
"data_sources": [{"id": 0,
"name": "string"
}
],
"display_name": "string",
"id": 0,
"input": "string",
"instructions": "string",
"labels": [{"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
],
"meta": { },
"name": "string",
"project_id": 0,
"task_type": "classification"
}
],
"users": [{"active": true,
"email_address": "string",
"full_name": "string",
"id": 0,
"intent": null,
"role": "owner",
"tier_id": 0,
"tier": {"updated_at": "2019-08-24T14:15:22Z",
"created_at": "2019-08-24T14:15:22Z",
"id": 0,
"name": "string",
"limits": {"projects": 0,
"annotators": 0,
"dataset_size": 0,
"predictions": 0
}
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"username": "string",
"verified": true,
"complete_tasks": 0,
"incomplete_tasks": 0,
"complete_and_not_flagged_tasks": 0,
"incomplete_and_not_flagged_tasks": 0
}
],
"learner": {"id": 0,
"current_status": "Untrained",
"evaluations": [{"id": 0,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"eval_metrics": { }
}
],
"latest_evaluation": {"id": 0,
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"eval_metrics": { }
}
},
"datasets": [{"id": 0,
"name": "string",
"count": 0,
"description": "string",
"fields": [{"id": 0,
"name": "string",
"data_type": "text",
"display_name": "string"
}
]
}
],
"data_count": 0,
"data_without_tasks_count": 0,
"tasks_count": 0,
"latest_completed_tasks_batch_size": 0,
"completed_tasks_count": 0,
"incomplete_tasks_count": 0,
"flagged_count": 0,
"task_allocation_strategy": "automatic",
"task_allocation_batch_size": 0,
"dashboard_configuration": { }
}

Remove Dataset

Remove the given dataset from the given project.

Authorizations:

path Parameters

project_id required	integer (Project Id)
dataset_id required	integer (Dataset Id)

Responses

Response samples

422

Content type

application/json

{"detail": [{"loc": ["string"
],
"msg": "string",
"type": "string"
}
]
}

Create Prediction

Use this endpoint to make a prediction on new data with your model.

Provide a batch of data points that include the input fields required by your project to generate a batch of predictions with confidence scores from the latest version of your Humanloop AI model.

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

required	Array of objects (An array of data containing the input fields defined by your project and any additional key-value metadata you wish to record for a prediction such as your document ids.) [ items ]
n_best	integer (Controls the number of most confident predictions from the model to return for each data point.) Default: 1
save	boolean (Whether or not the Humanloop platform will log the request and response data from a prediction) Default: true

Responses

Request samples

Payload

Content type

application/json

{"data": [{"text": "Your document data...",
"external_id": "123"
}
],
"n_best": 1,
"save": true
}

Response samples

200
422

Content type

application/json

[{"id": 0,
"data": {"text": "Your document data...",
"external_id": "123"
},
"results": [{"id": "string",
"confidence": 0,
"start": 0,
"end": 0,
"text": "string",
"label": {"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
},
"result_type": "span"
}
],
"usage": "testing"
}
]

Create Prediction With Fallback

Use this endpoint to make a prediction on new or existing data with your latest Humanloop model.

For existing data, provide a batch of data_ids.

For new data, provide a batch of data points that include the input fields required by your project.

To support fallback to a human for difficult examples you can set a confidence threshold. If the model's confidence is less than this threshold, a task will be routed to one of your project team members. To be notified of when the task has been completed, you must first have created a subscription to our task_completed event (see /subscriptions endpoint for more details).

An array of predictions with confidence scores and the corresponding task details if relevant will be returned.

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

required	Array of objects (An array of datapoints containing the input fields defined by your project and any additional key-value metadata you wish to record for a prediction such as your document ids. A new datapoint will automatically be created for this data.) [ items ]
data_ids	Array of integers (Humanloop ids of existing datapoints that you wish to generate a new prediction with. This should be specified instead of the data field where appropriate) [ items ]
n_best	integer (Controls the number of most confident predictions from the model to return for each data point.) Default: 1
batch_id	string (A user provided id for grouping batches of predictions.)
confidence_threshold	number (A value between 0 and 1. If the model confidence (for the highest confident value returned) is below this threshold, a task will be routed to a user to complete) Default: 0

Responses

Request samples

Payload

Content type

application/json

{"data": [{"text": "Your document data...",
"external_id": "123"
}
],
"data_ids": [0
],
"n_best": 1,
"batch_id": "string",
"confidence_threshold": 0
}

Response samples

200
422

Content type

application/json

[{"prediction_id": 0,
"batch_id": "string",
"data": {"text": "Your document data...",
"external_id": "123"
},
"data_id": 0,
"results": [{"id": "string",
"confidence": 0,
"start": 0,
"end": 0,
"text": "string",
"label": {"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
},
"result_type": "span"
}
],
"confidence_threshold": 0,
"created_at": "2019-08-24T14:15:22Z",
"task_id": 0,
"user": "string",
"task_url": "string"
}
]

Get Project Job Details

Retrieve details of the long running/batched async jobs that have been triggered against this project historically.

Use this endpoint to check the status of batch jobs for example, when waiting:

for batch predictions to complete across all existing data associated to a project.
for a model to re-train
for a model to re-rank the unlabelled data using active learning

Authorizations:

path Parameters

id

required

integer (Id)

query Parameters

job_type

string

Default: "update_learner"

Enum: "update_learner" "batch_scoring"

What jobs do we support to submit to a worker

Responses

Response samples

200
422

Content type

application/json

null

Get Labelled Data

Download all the completed task data for your project - including all your user assignments and their annotations and details of which data points were used for training, validation and testing purposes.

Authorizations:

path Parameters

id

required

integer (Id)

query Parameters

format

string (Format)

Default: "json"

Responses

Response samples

200
422

Content type

application/json

null

Get Paginated Project Data

Get a page of project data for a given project with corresponding task data.

You must provide page and size parameters. Up to size records will be returned, starting from page * size.

Authorizations:

path Parameters

project_id

required

integer (Project Id)

query Parameters

size	integer (Size) Default: 50
page	integer (Page) Default: 0
status	string (ProjectDataStatus) Enum: "complete" "needs_review" "pending" "in_progress" Status of a project's datapoint
flagged	boolean (Flagged)
sort	string (Sort) Default: "id_ASC"

Responses

Response samples

200
422

Content type

application/json

null

Get Model Statistics

Get detailed information on the performance of your projects model, including training, validation and test evaluation loss and evaluation data over time.

Authorizations:

path Parameters

project_id

required

integer (Project Id)

Responses

Response samples

200
422

Content type

application/json

[{"id": 0,
"learner_id": 0,
"eval_metrics": {"train_loss": [0
],
"dev_loss": [0
],
"test_loss": [0
],
"train_eval": [{"main_score": 0,
"accuracy": 0,
"macro_f_score": 0,
"micro_f_score": 0,
"class_breakdown": {"property1": {"precision": 0,
"recall": 0,
"f1-score": 0,
"support": 0
},
"property2": {"precision": 0,
"recall": 0,
"f1-score": 0,
"support": 0
}
}
}
],
"dev_eval": [{"main_score": 0,
"accuracy": 0,
"macro_f_score": 0,
"micro_f_score": 0,
"class_breakdown": {"property1": {"precision": 0,
"recall": 0,
"f1-score": 0,
"support": 0
},
"property2": {"precision": 0,
"recall": 0,
"f1-score": 0,
"support": 0
}
}
}
],
"test_eval": [{"main_score": 0,
"accuracy": 0,
"macro_f_score": 0,
"micro_f_score": 0,
"class_breakdown": {"property1": {"precision": 0,
"recall": 0,
"f1-score": 0,
"support": 0
},
"property2": {"precision": 0,
"recall": 0,
"f1-score": 0,
"support": 0
}
}
}
],
"learning_rate": [0
],
"test_score": 0,
"num_labels": 0
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
]

Get Task Summary Statistics

Get detailed task summary statistics for this project, including the number of flagged datapoints, the datapoint statuses, and a summary of overall datapoint task completion

Authorizations:

path Parameters

project_id

required

integer (Project Id)

Responses

Response samples

200
422

Content type

application/json

null

Trigger Model Training

Trigger your model to retrain against all your existing annotated data

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

max_epochs	integer (Max Epochs)
learning_rate	number (Learning Rate)
reinitialise	boolean (Reinitialise) Default: false

Responses

Request samples

Payload

Content type

application/json

{"max_epochs": 0,
"learning_rate": 0,
"reinitialise": false
}

Response samples

200
422

Content type

application/json

null

tasks

A project task is assigned to a user to provide annotations for a particular project data-point and tracks the lifecycle of the users work within the application. A project data-point may have many tasks assigned in the case where the project requires multiple users to annotate the same data-point.

Get Tasks

Get all task data for a given project.

You can provide different criteria to sort the response by as well as provide search words to filter the response by.

Authorizations:

path Parameters

id

required

integer (Id)

query Parameters

size	integer (Size) Default: 5000
page	integer (Page) Default: 0
complete	boolean (Complete)
flagged	boolean (Flagged)
sortby	string Default: "confusion" Enum: "confidence" "confusion" "last_updated" Different sorting modes for task data
search_keywords	string (Search Keywords)

Responses

Response samples

200
422

Content type

application/json

null

Create Tasks

Add new tasks to your project for your team to annotate more data to help improve your models over time. If no user is provided, we automatically select a suitable user.

Each task corresponds to a data point that requires annotation by a user. One data point can have more than one task associated to it in the case where multiple annotations from different users are required.

If the data points provided when creating tasks do not yet exist in your associated dataset, we add them to the dataset automatically, otherwise we reference the existing data points.

See our /subscriptions webhook docs for information setting up notifications for when tasks are created and completed.

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

required	Array of objects (An array of data containing the input fields defined by your project and any additional key-value meta-data you wish to record for a task such as your document ids.) [ items ]
user	string (The email address of the user to assign the new task to.)
usage	string (Specify if the associated datapoint and annotations for this task should be used for model training, validation or testing. If this is notspecified Humanloop will determine the usage automatically.) Enum: "testing" "validation" "training" How a data point has been used when updating an AI model for this project
complete	boolean (By default if the datapoint associated to a task has a valid annotation defined on creation the task will automatically be marked as complete so it used for model training. This field allows you to mark tasks with annotations defined as incomplete so that it is added to the queue for review before being used for model training.)

Responses

Request samples

Payload

Content type

application/json

{"data": [{"text": "Your document data...",
"external_id": "123"
}
],
"user": "string",
"usage": "testing",
"complete": true
}

Response samples

200
422

Content type

application/json

null

Get User Tasks

Get all task data for a given user from a project.

You can provide different criteria to sort the response by as well as provide search words to filter the response on.

Authorizations:

path Parameters

id required	integer (Id)
user_id required	integer (User Id)

query Parameters

size	integer (Size) Default: 50
page	integer (Page) Default: 0
complete	boolean (Complete)
flagged	boolean (Flagged)
sortby	string Default: "confusion" Enum: "confidence" "confusion" "last_updated" Different sorting modes for task data
search_keywords	string (Search Keywords)

Responses

Response samples

200
422

Content type

application/json

null

Get Task

Get details of a task.

Authorizations:

path Parameters

id required	integer (Id)
task_id required	integer (Task Id)

Responses

Response samples

200
422

Content type

application/json

{"id": 0,
"user_id": 0,
"full_name": "string",
"email_address": "string",
"project_id": 0,
"project_data_id": 0,
"data_id": 0,
"inputs": {"text": "Your document data...",
"external_id": "123"
},
"complete": true,
"flagged": true,
"score": 0,
"usage": "testing",
"url": "string",
"comment": "string",
"annotations": [{"id": "string",
"result_type": "span",
"start": 0,
"end": 0,
"text": "string",
"label_id": "string",
"label_name": "string",
"label_display_name": "string",
"label_description": "string"
}
],
"predictions": [{"id": "string",
"result_id": "string",
"result_type": "span",
"start": 0,
"end": 0,
"text": "string",
"confidence": 0,
"label_id": "string",
"label_name": "string",
"label_display_name": "string",
"label_description": "string",
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
]
}

Update Task

Update a given task.

If you provide annotations when updating a task and set the task as complete, these will be used as feedback to improve the performance of your model automatically.

You can also update the user assignment of a task by providing the email_address of a user from your project.

Authorizations:

path Parameters

task_id required	integer (Task Id)
id required	integer (Id)

Request Body schema: application/json

	Classification (object) or Array of Span (objects) or Array of Classification (objects) (The user-provided annotations)
email_address	string (The email address of the user to be assigned to the task to provide annotations.)
flagged	boolean (Whether the task has been flagged by a user - usually indicating it needs to be revisited at a later time.)
complete	boolean (Whether or not the task has been completed meaning the annotations will be used to improve your AI model)
comment	string (A comment on the associated data-point,generally used to explain why data has been flagged.)
usage	string (Specify if the associated datapoint and annotations for this task should be used for model training, validation or testing. If this is notspecified Humanloop will determine the usage automatically.) Enum: "testing" "validation" "training" How a data point has been used when updating an AI model for this project

Responses

Request samples

Payload

Content type

application/json

{"annotations": {"label": "string",
"confidence": 0,
"strength": 0,
"id": "string",
"object": "classification"
},
"email_address": "string",
"flagged": true,
"complete": true,
"comment": "string",
"usage": "testing"
}

Response samples

200
422

Content type

application/json

{"id": 0,
"user_id": 0,
"full_name": "string",
"email_address": "string",
"project_id": 0,
"project_data_id": 0,
"data_id": 0,
"inputs": {"text": "Your document data...",
"external_id": "123"
},
"complete": true,
"flagged": true,
"score": 0,
"usage": "testing",
"url": "string",
"comment": "string",
"annotations": [{"id": "string",
"result_type": "span",
"start": 0,
"end": 0,
"text": "string",
"label_id": "string",
"label_name": "string",
"label_display_name": "string",
"label_description": "string"
}
],
"predictions": [{"id": "string",
"result_id": "string",
"result_type": "span",
"start": 0,
"end": 0,
"text": "string",
"confidence": 0,
"label_id": "string",
"label_name": "string",
"label_display_name": "string",
"label_description": "string",
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
]
}

Delete Task

Delete a specific task.

Authorizations:

path Parameters

id required	integer (Id)
task_id required	integer (Task Id)

Responses

Response samples

200
422

Content type

application/json

null

Create Tasks For User

Add new tasks for datapoints without any tasks for the specified user to annotate.

Tasks are created preferentially for datapoints that will help the model the most (i.e. those with the highest active learning scores).

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

user required	string (The email address of the user to assign the new tasks to. Note that other users may also receive tasks to satisfy the project's annotation policy.)
datapoints required	integer (Number of datapoints to create tasks for.)

Responses

Request samples

Payload

Content type

application/json

{"user": "string",
"datapoints": 0
}

Response samples

200
422

Content type

application/json

{"created_tasks": 0
}

Bulk Create Tasks

Add new annotation tasks to your project to help improve your models over time.

A more performant endpoint for Task creation suitable for batches of new data.

This is a lighter weight version of the /projects/{id}/tasks endpoint, with the following differences in behaviour:

Assumes data does not exist and so does not perform de-dupe checks
Does not trigger the model to re-train

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

required	Array of objects (An array of data containing the input fields and optionally the output fields defined by your project. Include the output fields if you have existing annotations available.You can also include any additional key-value meta-data you wish to record for a task such as your document ids.) [ items ]
user	string (The email address of the user to assign the batch of tasks to.)
usage	string (Specify if the associated datapoint and annotations for this task should be used for model training, validation or testing. If this is notspecified Humanloop will determine the usage automatically.) Enum: "testing" "validation" "training" How a data point has been used when updating an AI model for this project
complete	boolean (By default if the datapoint associated to a task has a valid annotation defined on creation the task will automatically be marked as complete so it used for model training. This field allows you to mark tasks with annotations defined as incomplete so that it is added to the queue for review by an annotator user before being used for model training.)

Responses

Request samples

Payload

Content type

application/json

{"data": [{"text": "Your document data...",
"external_id": "123"
}
],
"user": "string",
"usage": "testing",
"complete": true
}

Response samples

200
422

Content type

application/json

null

Reassign Tasks

Reassign a number of tasks from a source user to a target user.

Care is taken to ensure that the target user does not receive new tasks for datapoints they have already annotated (or have incomplete tasks for).

Returns the number of tasks successfully transferred, and the source and target users' task counts.

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

source_user required	string (The email address of the user to take tasks from.)
target_user required	string (The email address of the user to give tasks to.)
tasks required	integer (The number of tasks to attempt to reassign)

Responses

Request samples

Payload

Content type

application/json

{"source_user": "string",
"target_user": "string",
"tasks": 0
}

Response samples

200
422

Content type

application/json

{"reassigned_tasks": 0
}

labels

The label taxonomy definition for your projects outputs. Used to define your annotations and your model predictions.

Get Labels

Get all labels for an existing project output definition

Authorizations:

path Parameters

id

required

integer (Id)

Responses

Response samples

200
422

Content type

application/json

[{"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
]

Update Labels

Update the data on an array of labels based on id

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

Array

id	string (Id)
name	string (Name)
display_name	string (Display Name)
description	string (Description)

Responses

Request samples

Payload

Content type

application/json

[{"id": "string",
"name": "string",
"display_name": "string",
"description": "string"
}
]

Response samples

200
422

Content type

application/json

[{"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
]

Add Labels

Add a label or list of labels to an existing project's output definition

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

Any of

Array

name required	string (Name)
display_name	string (Display Name)
description	string (Description)

Responses

Request samples

Payload

Content type

application/json

{"name": "string",
"display_name": "string",
"description": "string"
}

Response samples

200
422

Content type

application/json

[{"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
]

Delete Labels

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

Array

id	string (Id)
name	string (Name)
display_name	string (Display Name)
description	string (Description)

Responses

Request samples

Payload

Content type

application/json

[{"id": "string",
"name": "string",
"display_name": "string",
"description": "string"
}
]

Response samples

200
422

Content type

application/json

[{"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
]

Update Label

Update any of the data for an existing label with label_id

Authorizations:

path Parameters

id required	integer (Id)
label_id required	string (Label Id)

Request Body schema: application/json

id	string (Id)
name	string (Name)
display_name	string (Display Name)
description	string (Description)

Responses

Request samples

Payload

Content type

application/json

{"id": "string",
"name": "string",
"display_name": "string",
"description": "string"
}

Response samples

200
422

Content type

application/json

[{"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
]

Delete Label

Authorizations:

path Parameters

id required	integer (Id)
label_id required	any (Label Id)

Responses

Response samples

200
422

Content type

application/json

[{"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
]

annotations

The annotations associated to your project's data - used to train your model.

Get Annotations

Get a page of annotations data for a given project.

This endpoint works like /tasks, but you must provide page and size parameters. Up to size records will be returned, starting from page * size.

user relates to user.full_name.

Authorizations:

query Parameters

project_id required	integer (Project Id)
size	integer (Size) Default: 50
page	integer (Page) Default: 0
complete	boolean (Complete)
flagged	boolean (Flagged)
reviewed	boolean (Reviewed)
user	string (User)
keywords	string (Keywords)
labels	Array of strings (Labels)
task_id	integer (Task Id)

Responses

Response samples

200
422

Content type

application/json

null

Create Annotation

Create an annotation

Authorizations:

Request Body schema: application/json

task_id required	integer (ID of the annotation's task)
external_id	string (ID of the annotation to create)
label_id	string (ID of the annotation's label)
start	integer (Start offset of the annotation (span only))
end	integer (End offset of the annotation (span only))
text	string (The text the annotation covers (span only))
strength	number (strength of the association with the annotation's label (ordinal regression only))

Responses

Request samples

Payload

Content type

application/json

{"task_id": 0,
"external_id": "string",
"label_id": "string",
"start": 0,
"end": 0,
"text": "string",
"strength": 0
}

Response samples

200
422

Content type

application/json

null

Upsert Annotation

Create or update an annotation.

This endpoint is only used for single-label classification.

Authorizations:

path Parameters

annotation_id

required

string (Annotation Id)

Request Body schema: application/json

task_id required	integer (ID of the annotation's task)
label_id	string (ID of the annotation's label)
start	integer (Start offset of the annotation (span only))
end	integer (End offset of the annotation (span only))
text	string (The text the annotation covers (span only))
strength	number (strength of the association with the annotation's label (ordinal regression only))

Responses

Request samples

Payload

Content type

application/json

{"task_id": 0,
"label_id": "string",
"start": 0,
"end": 0,
"text": "string",
"strength": 0
}

Response samples

200
422

Content type

application/json

null

Delete Annotation

Delete a given annotation

Authorizations:

path Parameters

annotation_id

required

string (Annotation Id)

Responses

Response samples

200
422

Content type

application/json

null

Update Annotation

Update a given annotation.

Authorizations:

path Parameters

annotation_id

required

string (Annotation Id)

Request Body schema: application/json

reviewed	boolean (Whether this annotation has been reviewed)
start	integer (Start offset of the annotation (span only))
end	integer (End offset of the annotation (span only))
label_id	string (ID of the annotation's label)
strength	number (strength of the association with the annotation's label)
text	string (The text the annotation covers)

Responses

Request samples

Payload

Content type

application/json

{"reviewed": true,
"start": 0,
"end": 0,
"label_id": "string",
"strength": 0,
"text": "string"
}

Response samples

200
422

Content type

application/json

{"id": "string",
"external_id": "string",
"confidence": 0,
"user": {"id": 0,
"username": "string",
"full_name": "string",
"email_address": "string"
},
"task": {"id": 0,
"url": "string"
},
"start": 0,
"end": 0,
"text": "string",
"strength": 0,
"label_id": "string",
"label": {"name": "string",
"id": "string",
"display_name": "string",
"description": "string",
"count": 0,
"object": "label",
"output_id": 0,
"output": {"id": 0,
"name": "string",
"display_name": "string",
"description": "string",
"task_type": "classification",
"data_key": "string",
"data_type": "text",
"meta": { },
"project_id": 0,
"object": "output"
},
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
},
"result_type": "string",
"inputs": { },
"reviewed": true
}

subscriptions

Manage subscriptions using our RESTHook interface to get notified of particular events from your Humanloop projects. For example, it may be helpful for your system to be notified when annotation tasks have been completed.

Get Subscriptions

Get a list of all subscriptions for a given project.

Authorizations:

path Parameters

id

required

integer (Id)

Responses

Response samples

200
422

Content type

application/json

null

Create Subscription

Add a subscription to a project to get notifications for events of interest such as when a task has been completed.

Authorizations:

path Parameters

id

required

integer (Id)

Request Body schema: application/json

event	string (An event one can subscribe to for a project.) Default: "task.completed" Enum: "task.completed" "task.created" Events users of Humanloop can subscribe to
callback_url	string (The callback URL for Humanloop to post notifications to for this subscription.)
callback_headers	object (A dictionary of valid HTTP headers that Humanloop should include in posts to the callback_url.)
email_address	string (The email_address for humanloop to send notifications to or this subscription.)
active	boolean (Whether or not the subscription is active, meaning notifications will be sent for the specified events.) Default: true

Responses

Request samples

Payload

Content type

application/json

{"event": "task.completed",
"callback_url": "string",
"callback_headers": { },
"email_address": "string",
"active": true
}

Response samples

200
422

Content type

application/json

null

Update Subscription

Update an existing subscription. This is useful if you want to change callback details.

Authorizations:

path Parameters

id required	integer (Id)
sub_id required	string (Sub Id)

Request Body schema: application/json

event	string (An event one can subscribe to for a project.) Default: "task.completed" Enum: "task.completed" "task.created" Events users of Humanloop can subscribe to
callback_url	string (The callback URL for Humanloop to post notifications to for this subscription.)
callback_headers	object (A dictionary of valid HTTP headers that Humanloop should include in posts to the callback_url.)
email_address	string (The email_address for humanloop to send notifications to or this subscription.)
active	boolean (Whether or not the subscription is active, meaning notifications will be sent for the specified events.) Default: true

Responses

Request samples

Payload

Content type

application/json

{"event": "task.completed",
"callback_url": "string",
"callback_headers": { },
"email_address": "string",
"active": true
}

Response samples

200
422

Content type

application/json

null

Deactivate Subscription

Deactivate an existing subscription, which will result in you no longer receiving notifications from the events associated to this subscription.

Authorizations:

path Parameters

id required	integer (Id)
sub_id required	string (Sub Id)

Responses

Response samples

200
422

Content type

application/json

null

Test Subscription

Test a subscription.

Posting to this endpoint using the test_message query param will trigger a test notification from the Humanloop platform to the subscriptions callback url.

Authorizations:

path Parameters

id required	integer (Id)
sub_id required	string (Sub Id)

query Parameters

test_message

required

string (Test Message)

Responses

Response samples

200
422

Content type

application/json

null

public

Publicly accessible project and prediction endpoints

monitoring

Get Logs

Retrieve paginated logs from the server.

Sorting and filtering are supported through query params.

Sorting and filtering are mainly supported for the categorical columns, and text search is supported for the model inputs and output.

Sorting is supported for the source, model, timestamp, and feedback-{output_name} columns. Specify sorting with the sort query param, with values {column}.{ordering}. E.g. ?sort=source.asc&sort=model.desc will yield a multi-column sort. First by source then by model.

Filtering is supported for the source, model, and feedback-{output_name} columns. Specify filtering with the source_filter, model_filter, and feedback-{output.name}_filter query params. E.g. ?source_filter=AI&source_filter=user_1234&feedback-explicit_filter=good will only show rows where the source is "AI" or "user_1234", and where the latest feedback for the "explicit" output group is "good".

Searching is supported for the model inputs and output. Specify a search term with the search query param. E.g. ?search=hello%20there will cause a case-insensitive search across model inputs and output.

Authorizations:

query Parameters

project_id required	integer (Project Id)
search	string (Search)
size	integer (Size) Default: 50
page	integer (Page) Default: 0

Responses

Response samples

200
422

Content type

application/json

{"page": 0,
"size": 0,
"records": [{"id": "string",
"source": "string",
"model_config": {"id": "string",
"display_name": "string",
"model_name": "string",
"prompt_template": "string",
"parameters": { },
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"description": "string",
"last_used": "2019-08-24T14:15:22Z",
"feedback_stats": [{"feedback_group_id": 0,
"feedback_group_name": "string",
"feedback_label": "string",
"feedback_count": 0
}
],
"num_datapoints": 0
},
"inputs": {"property1": "string",
"property2": "string"
},
"output": "string",
"metadata": {"property1": "string",
"property2": "string"
},
"feedback": [{"id": "string",
"log_id": "string",
"group": "string",
"label": "string",
"text": "string",
"source": "string",
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
],
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z"
}
],
"total": 0
}

Log

Log a datapoint or array of datapoints to your Humanloop project.

Authorizations:

Request Body schema: application/json

Any of

project required	string (Project name) Unique project name. If it does not exist, a new project will be created.
required	object (Model input data) List of name, value pairs for the inputs used by your prompt template, or directly by your model.
output required	string (Model output) Generated output from your model for the provided inputs.
model	string (Type of model used) What model type was used for the generation? e.g. text-davinci-002. This can be null if the generation is user provided
prompt_template	string (Prompt template) Prompt template that incorporated your specified inputs to form your final request to the model.
source required	string (Source of generation) What was used to generate the output? e.g. gpt-3.
	object (Model parameters) The hyper-parameter settings that along with your model source and prompt template (if provided) will uniquely determine a model configuration on Humanloop. For example the temperature setting.
	object (Metadata) Any additional metadata that you would like to log for reference.
	Feedback (object) or Array of Feedback (objects) (Feedback labels) Optional parameter to provide feedback with your logged datapoint.
created_at	string <date-time> (Created at) Timestamp for when the log was created. If not provided, the time the log call was made will be used as a timestamp.

Responses

Request samples

Payload

Content type

application/json

[ ]

Response samples

200
422

Content type

application/json

[ ]

Feedback

Submit a list of feedback for existing log_ids

Authorizations:

Request Body schema: application/json

Any of

Array

group required	string (Group)
label	string (Label) A categorical label to characterize the type of feedback.Only one of label or correction should be provided for each feedback.
text	string (Text) Text feedback. Can be used to record model corrections from your users. Only one of label or text should be provided for each feedback.
log_id	string (Log ID) id to associate the feedback to a previously logged datapoint.When providing instant feedback as part of the hl.log(...) call you don't need to provide a log_id.
source	string (Source) A unique to who/what provided the feedback. For example a unique user reference.
created_at	string <date-time> (Created at) Timestamp for when the feedback was created. If not provided, the time the call was made will be used as a timestamp.

Responses

Request samples

Payload

Content type

application/json

{"group": "string",
"label": "string",
"text": "string",
"log_id": "string",
"source": "string",
"created_at": "2019-08-24T14:15:22Z"
}

Response samples

200
422

Content type

application/json

[ ]

Get Project Model Configs Deprecated

Get an array of model_configs associated to your project

Authorizations:

query Parameters

project_id

required

integer (Project Id)

Responses

Response samples

200
422

Content type

application/json

[{"id": "string",
"display_name": "string",
"model_name": "string",
"prompt_template": "string",
"parameters": { },
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"description": "string",
"last_used": "2019-08-24T14:15:22Z",
"feedback_stats": [{"feedback_group_id": 0,
"feedback_group_name": "string",
"feedback_label": "string",
"feedback_count": 0
}
],
"num_datapoints": 0
}
]

Register Project Model Config

Register a model config to a project and optionally tag it with an experiment. If the project does not exist, a new project will be added automatically.

Authorizations:

Request Body schema: application/json

name	string (Config name) A friendly display name for config. If not provided a name will be generated.
project required	string (Project name) Unique project name. If it does not exist, a new project will be created.
experiment	string (Experiment tag) Include a model configuration in an experiment group. Experiment groups are used for A/B testing and optimizing hyper-parameters.
model required	string (Type of model used) What model type was used for the generation? e.g. text-davinci-002.
prompt_template	string (Prompt template) Prompt template that incorporated your specified inputs to form your final request to the model.
	object (Model parameters) The hyper-parameter settings that along with your model source and prompt template (if provided) will uniquely determine a model configuration on Humanloop. For example the temperature setting.

Responses

Request samples

Payload

Content type

application/json

{"name": "string",
"project": "string",
"experiment": "string",
"model": "string",
"prompt_template": "string",
"parameters": {"property1": "string",
"property2": "string"
}
}

Response samples

200
422

Content type

application/json

{"id": "string",
"display_name": "string",
"model_name": "string",
"prompt_template": "string",
"parameters": { },
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"description": "string",
"last_used": "2019-08-24T14:15:22Z",
"feedback_stats": [{"feedback_group_id": 0,
"feedback_group_name": "string",
"feedback_label": "string",
"feedback_count": 0
}
],
"num_datapoints": 0
}

Get Project Model Configs

Get an array of model_configs associated to your project

Authorizations:

path Parameters

project_id

required

integer (Project Id)

Responses

Response samples

200
422

Content type

application/json

[{"id": "string",
"display_name": "string",
"model_name": "string",
"prompt_template": "string",
"parameters": { },
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"description": "string",
"last_used": "2019-08-24T14:15:22Z",
"feedback_stats": [{"feedback_group_id": 0,
"feedback_group_name": "string",
"feedback_label": "string",
"feedback_count": 0
}
],
"num_datapoints": 0
}
]

Update Project Model Config

Update a model configuration within your project

Authorizations:

path Parameters

project_id required	integer (Project Id)
model_config_id required	string (Model Config Id)

Request Body schema: application/json

display_name	string (Display Name)
description	string (Description)

Responses

Request samples

Payload

Content type

application/json

{"display_name": "string",
"description": "string"
}

Response samples

200
422

Content type

application/json

{"id": "string",
"display_name": "string",
"model_name": "string",
"prompt_template": "string",
"parameters": { },
"created_at": "2019-08-24T14:15:22Z",
"updated_at": "2019-08-24T14:15:22Z",
"description": "string",
"last_used": "2019-08-24T14:15:22Z",
"feedback_stats": [{"feedback_group_id": 0,
"feedback_group_name": "string",
"feedback_label": "string",
"feedback_count": 0
}
],
"num_datapoints": 0
}

Export Raw

Retrieve filtered logs in a normalized CSV.

Authorizations:

query Parameters

project_id required	integer (Project Id)
search	string (Search)

Responses

Response samples

200
422

Content type

application/json

null

Export Fine Tune

Retrieve filtered logs in the format for fine-tuning.

Authorizations: