Using your own model/provider

The humanloop.generate() call encapsulates the LLM provider calls (for example openai.Completions.create()), the model-config selection and logging steps in a single unified interface. There may be scenarios that you wish to manage the LLM provider calls directly in your own code instead of relying on Humanloop.

For example, you may be using an LLM provider that currently is not directly supported by Humanloop such as Hugging Face.

To support using your own model provider, we provide additional log() and get_model_config() methods in the SDK.

In this guide, we walk through how to use these SDK methods to log data to Humanloop and run experiments.

Prerequisites

1. You already have a project created - if not, please pause and first follow our project creation guides.

Install and Initialize the SDK

First you need to install and initialize the SDK. If you have already done this, skip to the next section. Otherwise, open up your terminal and follow these steps:

1. Install the Humanloop Python SDK:
   Shell

   $ pip install humanloop
   

2. Start a Python interpreter:
   Shell

   $ python
   

3. Initialize the SDK with your Humanloop API key (get your API key from your User Settings page)
   Python

   >>> import humanloop as hl
   >>> hl.init(api_key="sk_abcdef1234567")
   

Log data to your project

1. Set up your code to first call your LLM provider to get a generation and then log this to Humanloop, explicitly passing details of the model config alongside the inputs and output:

   Python

   import humanloop as hl
   import openai
   
   # Initialize Humanloop with your API key
   hl.init(api_key="<YOUR Humanloop API KEY>")
   
   # Make a generation using the OpenAI SDK.
   response = openai.Completion.create(
       prompt="Answer the following question like Paul Graham from YCombinator:\n"
       "How should I think about competition for my startup?",
       model="text-davinci-002",
       temperature=0.2,
   )
   
   # Parse the output from the OpenAI response.
   output = response.choices[0].text
   
   # Log the inputs, outputs and model config to your project.
   log_response = hl.log(
       project="<YOUR UNIQUE PROJECT NAME>",
       inputs={"question": "How should I think about competition for my startup?"},
       output=output,
       model_config={
           "model": "text-davinci-002",
           "prompt_template": "Answer the following question like Paul Graham from YCombinator:\n{{question}}",
           "temperature": 0.2,
       },
   )
   
   # Use this ID to associate feedback received later to this datapoint.
   data_id = log_response.id
   

2. The process of capturing feedback then uses the returned log_id as before.
   See our guide to recording feedback with the SDK.

3. You can also log immediate feedback alongside the input and outputs:

   # Log the inputs, outputs and model config to your project.
   log_response = hl.log(
       project="<YOUR UNIQUE PROJECT NAME>",
       inputs={"question": "How should I think about competition for my startup?"},
       output=output,
       model_config={
           "model": "text-davinci-002",
           "prompt_template": "Answer the following question like Paul Graham from YCombinator:\n{{question}}",
           "temperature": 0.2,
       },
       feedback={"type": "rating", "value": "good"}
   )
   

📘

Hugging Face example

Note that you can also use a similar pattern for non-OpenAI LLM providers. For example, to use Hugging Face’s Inference API:

Python

import requests
import humanloop as hl

# Initialize the SDK with your Humanloop API key
hl.init(api_key="<YOUR Humanloop API KEY>")

# Make a generation using the Hugging Face Inference API.
response = requests.post(
    "https://api-inference.huggingface.co/models/gpt2",
    headers={"Authorization": f"Bearer {<YOUR HUGGING FACE API TOKEN>}"},
    json={
        "inputs": "Answer the following question like Paul Graham from YCombinator:\n"
        "How should I think about competition for my startup?",
        "parameters": {
            "temperature": 0.2,
            "return_full_text": False,  # Otherwise, Hugging Face will return the prompt as part of the generation.
        },
    },
).json()

# Parse the output from the Hugging Face response.
output = response[0]["generated_text"]

# Log the inputs, outputs and model config to your project.
log_response = hl.log(
    project="<YOUR UNIQUE PROJECT NAME>",
    inputs={"question": "How should I think about competition for my startup?"},
    output=output,
    model_config={
        "model": "gpt2",
        "prompt_template": "Answer the following question like Paul Graham from YCombinator:\n{{question}}",
        "temperature": 0.2,
    },
)

Use experiments to compare model configs

📘

Prerequisites

You already have an experiment set up for your project - if not, please pause and first follow the guide to setting up an experiment.

In order to log data for your experiment without using hl.generate(), you must first determine which model config to use for your LLM provider calls. This is where the hl.get_model_config() endpoint comes in.

1. First navigate to the Experiments tab of your project and select your Experiment card.
2. Copy the experiment_id from the experiment summary:

3. Alter your logging code from the previous section to now first sample a model_config from your experiment to use when making your call to OpenAI:

   Python

   import humanloop as hl
   import openai 
   
   # Initialize the SDK with your Humanloop API key
   hl.init(api_key="<YOUR Humanloop API KEY>")
   
   # Sample a model_config from your experiment.
   model_config = hl.get_model_config(experiment_id=experiment_id)
   
   # Make a generation using OpenAI using the parameters from the sampled model_config.
   response = openai.Completion.create(
       prompt="Answer the following question like Paul Graham from YCombinator:\n"
       "How should I think about competition for my startup?",
       model=model_config.model,
       temperature=model_config.temperature,
   )
   
   # Parse the output from the OpenAI response.
   output = response.choices[0].text
   
   # Log the inputs and outputs to the experiment trial associated to the sampled model_config.
   log_response = hl.log(
       project="<YOUR UNIQUE PROJECT NAME>",
       inputs={"question": "How should I think about competition for my startup?"},
       output=output,
       trial_id=model_config.trial_id,
   )
   
   # Use this ID to associate feedback received later to this datapoint.
   data_id = log_response.id