Configure GitLab to access self-hosted models

DETAILS: Tier: For a limited time, Ultimate. On October 17, 2024, Ultimate with GitLab Duo Enterprise. Offering: Self-managed Status: Beta

• Introduced in GitLab 17.1 with a flag named ai_custom_model. Disabled by default.

FLAG: The availability of this feature is controlled by a feature flag. For more information, see the history.

To configure your GitLab instance to access the available self-hosted models in your infrastructure:

1. Configure the self-hosted model.
2. Configure the GitLab Duo features to use your self-hosted model.

Configure the self-hosted model

Prerequisites:

• You must be an administrator.

To configure a self-hosted model:

1. On the left sidebar, at the bottom, select Admin.
2. Select AI-powered features.
   • If the AI-powered features menu item is not available, synchronize your subscription after purchase:
     1. On the left sidebar, select Subscription.
     2. In Subscription details, to the right of Last sync, select synchronize subscription ({retry}).
3. Select Models.
4. Select New self-hosted model.
5. Complete the fields:
   • Enter the model name, for example, Mistral.
   • From the Model family dropdown list, select the model. Only GitLab-approved models are in this list.
   • For Endpoint, select the self-hosted model endpoint. For example, the server hosting the model.
   • Optional. For API token, add an API key if you need one to access the model.
   • Optional. For Model identifier, enter the cloud provider where the model is hosted, and the name the cloud provider uses for that model (for example: anthropic/claude-3-5-sonnet-20240620).
6. Select Create model.

Configure GitLab Duo features to use self-hosted models

Prerequisites:

• You must be an administrator.

View configured features

1. On the left sidebar, at the bottom, select Admin.
2. Select AI-powered features.
   • If the AI-powered features menu item is not available, synchronize your subscription after purchase:
     1. On the left sidebar, select Subscription.
     2. In Subscription details, to the right of Last sync, select synchronize subscription ({retry}).
3. Select Features.

Configure the feature to use a self-hosted model

Configure the GitLab Duo feature to send queries to the configured self-hosted model:

1. In Features, for the feature you want to set, select Edit. For example, Code Generation.
2. Select the model provider for the feature:
   • From the list, select Self-Hosted Model.
   • Choose the self-hosted model you want to use, for example, Mistral.
3. Select Save Changes.

Configure the features to use GitLab AI Vendor models

You can choose a GitLab AI vendor to be the GitLab Duo feature's model provider. The feature then uses the GitLab-hosted model through the GitLab Cloud Connector:

1. In Features, for the feature you want to set, select Edit.
2. In the list of model providers for the feature, select AI Vendor.
3. Select Save Changes.