Parallel Works
View as Markdown
Share:

Custom OpenAI-Compatible Provider

A custom provider connects ACTIVATE AI Chat to any endpoint that implements the OpenAI chat completions API. This gives you flexibility to use self-hosted models, third-party inference services, or any other compatible backend.

Endpoint Requirements

Your endpoint must support the following:

  • POST /v1/chat/completions — The standard OpenAI chat completions endpoint. ACTIVATE sends all chat messages to this route.
  • GET /v1/models (optional) — If your endpoint exposes this route, ACTIVATE can list available models dynamically so users can switch models without reconfiguring the provider.

The endpoint must accept an API key via the Authorization: Bearer <key> header.

Prerequisites

Before creating a custom provider, ensure that:

  • Your inference endpoint is running and reachable from the ACTIVATE platform
  • You have the endpoint URL and a valid API key

Creating a Provider

  1. Navigate to AI Chat Providers in the ACTIVATE sidebar.
  2. Click Add Provider.
  3. Select Custom as the cloud service provider.
  4. Fill in the required fields:
    • Name — A descriptive name for the provider (for example, "vLLM Llama 3").
    • Endpoint — The full URL to your chat completions endpoint (for example, https://my-server.example.com/v1/chat/completions).
    • API Key — The authentication key for your endpoint.
  5. Optionally add a Description and Tags.
  6. Click Create.

Custom providers are available immediately after creation — no provisioning step is required.

Verifying the Provider

  1. Open AI Chat.
  2. Select your new custom provider from the provider dropdown.
  3. Send a test message to confirm the endpoint responds correctly.

ACTIVATE automatically discovers available models by querying your endpoint's /v1/models route. If the model list dropdown is populated, your endpoint supports this route and ACTIVATE has successfully queried it.

Updating a Provider

You can update the endpoint and API key for a custom provider at any time:

  1. Navigate to AI Chat Providers.
  2. Select the provider you want to update.
  3. Modify the fields as needed.
  4. Save your changes.

This is useful for rotating API keys or pointing the provider at a different endpoint without creating a new provider.

Notes for Common Inference Servers

vLLM — Start vLLM with the --api-key flag to enable API key authentication. The default port is 8000. Your endpoint URL will be http://<host>:8000/v1/chat/completions.

Ollama — Ollama exposes an OpenAI-compatible API on port 11434. Set the endpoint to http://<host>:11434/v1/chat/completions. The model name should match the Ollama model tag (for example, "llama3").

Text Generation Inference (TGI) — TGI provides an OpenAI-compatible Messages API. Set the endpoint to http://<host>:8080/v1/chat/completions.

Deleting a Provider

  1. Navigate to AI Chat Providers.
  2. Find the provider you want to remove.
  3. Click the delete action for that provider.
  4. Confirm the deletion.

Deleting a custom provider only removes it from ACTIVATE. It does not affect the external inference endpoint.

Related Documentation