Custom Models

While Tune Studio supports the most popular open source and commercial LLMs out of the box, you might want to add a custom model that is not natively supported.

To do this, navigate to Models -> Deploy Model -> Custom Model.

Supply the endpoint to where your custom model is hosted and if necessary any custom headers required for authentication.

Test the model by clicking on the Test button and then click Add model. It will now appear in your list of models on Tune Chat.

​

Understanding context length

Context length refers to the maximum number of tokens (words or characters) that the model can consider in a single interaction. Setting the correct context length ensures the model functions optimally.

Setting context length too high: If you set a context length greater than what your custom model supports, the chat will not trim the content correctly, leading to broken API calls. This means the chat might stop unexpectedly, and other settings like maximum tokens could be miscalculated.

Setting context length too low: Setting a shorter context length than supported is less harmful but still suboptimal. The chat will have less context to work with, potentially resulting in less coherent responses and underutilization of your custom model.

​

Guidance on Setting Context Length:

• Default Values: For known models, we provide default context lengths to ensure optimal performance.
• Custom models: When configuring a custom model, check the documentation of your model provider to find the maximum supported context length.

​

Steps to set the correct context length

• Identify the Model: Determine the LLM you are using for the custom model.
• Refer to Documentation: Check the official documentation for the maximum context length.
• Set the Value: Enter the context length in our chat app settings based on the model’s capability.
• Test the Configuration: Run a few test interactions to ensure the chat behaves as expected without errors.

​

Example walkthrough for setting context length

Imagine you are setting up a custom model using an LLM from Provider X. Here’s how you would set the context length correctly:

1. Identify the Model:

   • You are using Model Y from Provider X.

2. Refer to Documentation:

   • According to Provider X’s documentation, Model Y supports a maximum context length of 2048 tokens.

3. Set the Value:

   • In our chat app, go to the custom model settings.
   • Enter 2048 as the context length.

4. Test the Configuration:

   • Start a conversation and observe the responses.
   • Ensure there are no truncation errors and the interactions are smooth.