Groq Integration

Official Groq API Documentation

Groq offers fast inference for large language models and other generative AI tasks. Use KeyPool to call Groq with your team token and a stable base URL.

KeyPool Endpoint for Groq

To interact with Groq via KeyPool, configure your SDK with the following base URL:

{YOUR_KEYPOOL_BASE_URL}/v1/groq

This applies to any KeyPool host, including custom domains such as:

https://your-keypool.example.com/v1/groq

Important Note: Do not include /openai/v1 in the base_url when using KeyPool. The Groq SDK already appends Groq's OpenAI-compatible paths (for example /openai/v1/chat/completions).

Authentication

Use your KeyPool team token as a bearer token when making requests to the KeyPool Groq endpoint.

Do not send a personal Groq API key when using KeyPool.

Published model limits and free-plan verification

Groq's public rate-limit table currently labels the values below as base Developer-plan limits; it does not provide a separate static Free-plan table. Limits are model-specific and enforced at the organization level. The Groq Console Limits page and the upstream x-ratelimit-* response headers are authoritative for the current account.

Model family	RPM	RPD	TPM	TPD
openai/gpt-oss-120b, openai/gpt-oss-20b, openai/gpt-oss-safeguard-20b	30	1,000	8,000	200,000
qwen/qwen3.6-27b, qwen/qwen3.8-27b	30	1,000	8,000	200,000
groq/compound, groq/compound-mini	30	250	70,000	Not listed
meta-llama/llama-prompt-guard-2-22m, meta-llama/llama-prompt-guard-2-86m	30	14,400	15,000	500,000
canopylabs/orpheus-arabic-saudi, canopylabs/orpheus-v1-english	10	100	1,200	3,600
whisper-large-v3, whisper-large-v3-turbo	20	2,000	—	—

The live production KeyPool model list currently also includes allam-2-7b. Groq documents a 4,096-token context window and 4,096 maximum output for that model, but does not publish an RPM/TPM row. KeyPool therefore uses the conservative unknown-model fallback for its scheduler. Model availability and limits are account-specific, so client.models.list() and the live response headers remain authoritative.

For Whisper, the quota is audio seconds rather than tokens: 7,200 audio seconds/hour and 28,800 audio seconds/day. Groq may also apply separate input-token-per-minute and output-token-per-minute limits. Cached tokens do not count. A 131,072-token context window on GPT-OSS does not bypass the published 8,000-TPM and 200,000-TPD baseline.

KeyPool's Groq scheduler now uses these published model-specific baselines for text inference and keeps a conservative fallback for unknown model IDs. If multiple credentials belong to the same Groq organization, assign the same metadata.groq_pool_id; otherwise KeyPool treats their local budgets as independent while Groq still enforces one organization quota.

Groq traffic is best suited for demo, development, and prototype usage. For replayable JSON chat-completion and Responses API calls, KeyPool transparently retries an upstream 429 against another eligible credential pool, up to five total attempts. The X-KeyPool-Retry-Count response header reports how many alternate-pool attempts were made. KeyPool returns 429 only when its scheduler predicts that every eligible pool is unavailable or the bounded upstream attempts are exhausted; the response includes a useful Retry-After value.

If you receive a 429 from KeyPool:

• Check the Retry-After response header before retrying.
• Reduce prompt size or max_tokens / max_completion_tokens for lower-limit GPT-OSS requests.
• Prefer a model returned by client.models.list(); model availability changes over time.
• Treat very large coding or document-manipulation prompts as best-effort on lower-limit Groq access; use a higher-limit provider or a paid Groq tier for sustained large-context work.

Before selecting a model in an application, call client.models.list() and choose an ID currently available to your workspace. See Groq's supported models and rate limits for the current catalog and published quotas.

Key Features and Usage

KeyPool supports Groq chat completions, audio transcriptions, and model listing.

1. Chat Completions

Perform text generation using Groq's powerful language models.

Basic Completion

Python Example:

import os
from groq import Groq

KEYPOOL_BASE_URL = os.environ.get("KEYPOOL_BASE_URL")
KEYPOOL_TOKEN = os.environ.get("KEYPOOL_TOKEN")

client = Groq(
    api_key=KEYPOOL_TOKEN,
    base_url=f"{KEYPOOL_BASE_URL}/v1/groq",
)

chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Explain the concept of quantum entanglement in a short paragraph.",
        }
    ],
    model="openai/gpt-oss-20b",
)

print(chat_completion.choices[0].message.content)

TypeScript Example:

import Groq from "groq-sdk";

const KEYPOOL_BASE_URL = process.env.KEYPOOL_BASE_URL;
const KEYPOOL_TOKEN = process.env.KEYPOOL_TOKEN;

const client = new Groq({
  apiKey: KEYPOOL_TOKEN,
  baseURL: `${KEYPOOL_BASE_URL}/v1/groq`,
});

async function basicCompletion() {
  const chatCompletion = await client.chat.completions.create({
    messages: [
      {
        role: "user",
        content: "Explain the concept of quantum entanglement in a short paragraph.",
      },
    ],
    model: "openai/gpt-oss-20b",
  });

  console.log(chatCompletion.choices[0].message.content);
}

basicCompletion();

Streaming Completions

Receive responses in real-time as they are generated.

Python Example:

import os
from groq import Groq

KEYPOOL_BASE_URL = os.environ.get("KEYPOOL_BASE_URL")
KEYPOOL_TOKEN = os.environ.get("KEYPOOL_TOKEN")

client = Groq(
    api_key=KEYPOOL_TOKEN,
    base_url=f"{KEYPOOL_BASE_URL}/v1/groq",
)

stream = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Count to 10, one number per response.",
        }
    ],
    model="openai/gpt-oss-20b",
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
print()

TypeScript Example:

import Groq from "groq-sdk";

const KEYPOOL_BASE_URL = process.env.KEYPOOL_BASE_URL;
const KEYPOOL_TOKEN = process.env.KEYPOOL_TOKEN;

const client = new Groq({
  apiKey: KEYPOOL_TOKEN,
  baseURL: `${KEYPOOL_BASE_URL}/v1/groq`,
});

async function streamingCompletion() {
  const stream = await client.chat.completions.create({
    messages: [
      {
        role: "user",
        content: "Count to 10, one number per response.",
      },
    ],
    model: "openai/gpt-oss-20b",
    stream: true,
  });

  for await (const chunk of stream) {
    process.stdout.write(chunk.choices[0]?.delta?.content || "");
  }
  process.stdout.write("\n");
}

streamingCompletion();

Tool Use / Function Calling

Enable the model to call external functions or tools.

Python Example:

import os
from groq import Groq

KEYPOOL_BASE_URL = os.environ.get("KEYPOOL_BASE_URL")
KEYPOOL_TOKEN = os.environ.get("KEYPOOL_TOKEN")

client = Groq(
    api_key=KEYPOOL_TOKEN,
    base_url=f"{KEYPOOL_BASE_URL}/v1/groq",
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    }
                },
                "required": ["location"],
            },
        },
    }
]

chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What's the weather like in Boston?",
        }
    ],
    model="openai/gpt-oss-20b",
    tools=tools,
    tool_choice="auto",
)

choice = chat_completion.choices[0]
if choice.message.tool_calls:
    tool_call = choice.message.tool_calls[0]
    print(f"Model wants to call tool: {tool_call.function.name} with arguments: {tool_call.function.arguments}")
elif choice.message.content:
    print(choice.message.content)

TypeScript Example:

import Groq from "groq-sdk";

const KEYPOOL_BASE_URL = process.env.KEYPOOL_BASE_URL;
const KEYPOOL_TOKEN = process.env.KEYPOOL_TOKEN;

const client = new Groq({
  apiKey: KEYPOOL_TOKEN,
  baseURL: `${KEYPOOL_BASE_URL}/v1/groq`,
});

async function chatToolUse() {
  const tools = [
    {
      type: "function" as const,
      function: {
        name: "get_current_weather",
        description: "Get the current weather in a given location",
        parameters: {
          type: "object",
          properties: {
            location: {
              type: "string",
              description: "The city and state, e.g. San Francisco, CA",
            },
          },
          required: ["location"],
        },
      },
    },
  ];

  const chatCompletion = await client.chat.completions.create({
    messages: [
      {
        role: "user",
        content: "What's the weather like in Boston?",
      },
    ],
    model: "openai/gpt-oss-20b",
    tools: tools,
    tool_choice: "auto",
  });

  const choice = chatCompletion.choices[0];
  if (choice.message.tool_calls) {
    const toolCall = choice.message.tool_calls[0];
    console.log(`Model wants to call tool: ${toolCall.function.name} with arguments: ${toolCall.function.arguments}`);
  } else if (choice.message.content) {
    console.log(choice.message.content);
  }
}

chatToolUse();

JSON Mode

Force the model to respond with valid JSON.

Python Example:

import os
import json
from groq import Groq

KEYPOOL_BASE_URL = os.environ.get("KEYPOOL_BASE_URL")
KEYPOOL_TOKEN = os.environ.get("KEYPOOL_TOKEN")

client = Groq(
    api_key=KEYPOOL_TOKEN,
    base_url=f"{KEYPOOL_BASE_URL}/v1/groq",
)

chat_completion = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "Respond in JSON only."},
        {"role": "user", "content": 'Generate a JSON object with "item" and "price" keys.'},
    ],
    model="openai/gpt-oss-20b",
    response_format={"type": "json_object"},
)

parsed_response = json.loads(chat_completion.choices[0].message.content)
print(parsed_response)
assert "item" in parsed_response
assert "price" in parsed_response

TypeScript Example:

import Groq from "groq-sdk";

const KEYPOOL_BASE_URL = process.env.KEYPOOL_BASE_URL;
const KEYPOOL_TOKEN = process.env.KEYPOOL_TOKEN;

const client = new Groq({
  apiKey: KEYPOOL_TOKEN,
  baseURL: `${KEYPOOL_BASE_URL}/v1/groq`,
});

async function chatJsonMode() {
  const chatCompletion = await client.chat.completions.create({
    messages: [
      { role: "system", content: "Respond in JSON only." },
      { role: "user", content: 'Generate a JSON object with "item" and "price" keys.' },
    ],
    model: "openai/gpt-oss-20b",
    response_format: { type: "json_object" },
  });

  const parsedResponse = JSON.parse(chatCompletion.choices[0].message.content!);
  console.log(parsedResponse);
  // Further assertions could be made on the structure of parsedResponse
}

chatJsonMode();

2. Audio Transcription (Whisper)

Transcribe audio files using Groq's Whisper integration.

Python Example:

import os
import httpx
from groq import Groq
import tempfile
from pathlib import Path

SAMPLE_AUDIO_URL = "https://cdn.openai.com/API/docs/audio/alloy.wav"

KEYPOOL_BASE_URL = os.environ.get("KEYPOOL_BASE_URL")
KEYPOOL_TOKEN = os.environ.get("KEYPOOL_TOKEN")

client = Groq(
    api_key=KEYPOOL_TOKEN,
    base_url=f"{KEYPOOL_BASE_URL}/v1/groq",
)

# Download sample audio
with tempfile.TemporaryDirectory() as tmpdir:
    audio_path = Path(tmpdir) / "sample.wav"
    response = httpx.get(SAMPLE_AUDIO_URL)
    audio_path.write_bytes(response.content)

    with open(audio_path, "rb") as f:
        transcription = client.audio.transcriptions.create(
            file=("sample.wav", f.read()),
            model="whisper-large-v3-turbo",
            response_format="json",
        )
    print(f"Transcription: {transcription.text}")

TypeScript Example:

import Groq from "groq-sdk";

const SAMPLE_AUDIO_URL = "https://cdn.openai.com/API/docs/audio/alloy.wav";

const KEYPOOL_BASE_URL = process.env.KEYPOOL_BASE_URL;
const KEYPOOL_TOKEN = process.env.KEYPOOL_TOKEN;

const client = new Groq({
  apiKey: KEYPOOL_TOKEN,
  baseURL: `${KEYPOOL_BASE_URL}/v1/groq`,
});

async function audioTranscription() {
  // Fetch the audio file
  const audioResp = await fetch(SAMPLE_AUDIO_URL);
  const audioBuffer = await audioResp.arrayBuffer();
  const audioFile = new File([audioBuffer], "sample.wav", {
    type: "audio/wav",
  });

  const transcription = await client.audio.transcriptions.create({
    file: audioFile,
    model: "whisper-large-v3-turbo",
    response_format: "json",
  });
  console.log(`Transcription: ${transcription.text}`);
}

audioTranscription();

3. List Available Models

Retrieve a list of models available through the Groq API.

Python Example:

import os
from groq import Groq

KEYPOOL_BASE_URL = os.environ.get("KEYPOOL_BASE_URL")
KEYPOOL_TOKEN = os.environ.get("KEYPOOL_TOKEN")

client = Groq(
    api_key=KEYPOOL_TOKEN,
    base_url=f"{KEYPOOL_BASE_URL}/v1/groq",
)

models = client.models.list()
print("Available Groq Models:")
for model in models.data:
    print(f"- {model.id}")

TypeScript Example:

import Groq from "groq-sdk";

const KEYPOOL_BASE_URL = process.env.KEYPOOL_BASE_URL;
const KEYPOOL_TOKEN = process.env.KEYPOOL_TOKEN;

const client = new Groq({
  apiKey: KEYPOOL_TOKEN,
  baseURL: `${KEYPOOL_BASE_URL}/v1/groq`,
});

async function listModels() {
  const models = await client.models.list();
  console.log("Available Groq Models:");
  models.data.forEach((model) => {
    console.log(`- ${model.id}`);
  });
}

listModels();

Troubleshooting Blocked Requests

If Groq returns a non-rate-limit 403 or a response that says the request was blocked, check the User-Agent sent by your client. Some TypeScript SDKs use an OpenAI-compatible default user agent, such as OpenAI/JS, which can be treated differently by provider-side request filters.

For server-side Node.js, Bun, or Worker-style runtimes where your client can set request headers, use an application-specific User-Agent:

import Groq from "groq-sdk";

const client = new Groq({
  apiKey: process.env.KEYPOOL_TOKEN,
  baseURL: `${process.env.KEYPOOL_BASE_URL}/v1/groq`,
  defaultHeaders: {
    "User-Agent": "my-app-groq-client/1.0",
  },
});

For raw fetch calls, set the same header explicitly:

const response = await fetch(`${process.env.KEYPOOL_BASE_URL}/v1/groq/openai/v1/chat/completions`, {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.KEYPOOL_TOKEN}`,
    "Content-Type": "application/json",
    "User-Agent": "my-app-groq-client/1.0",
  },
  body: JSON.stringify({
    model: "openai/gpt-oss-20b",
    messages: [{ role: "user", content: "Say hello." }],
  }),
});

Browsers generally do not allow JavaScript to set User-Agent. If this problem appears in a browser-only integration, make the Groq call from a backend route or server-side function where you control request headers.

Direct Groq 403 From Server-Side Runtimes

Groq's direct API can return HTTP 403 or Cloudflare 1010 from some server-side runtimes when request headers are rejected before the API request is processed.

When using KeyPool, send requests to your KeyPool Groq endpoint instead of calling Groq directly. If you still see a blocked response, check the troubleshooting section above and set an application-specific User-Agent where your runtime allows it.

Error Handling

KeyPool handles various error conditions gracefully, providing informative responses. Common error scenarios include:

• 401 Unauthorized: Invalid or missing KeyPool Team Token.
• 403 Forbidden: Your Team Token does not have permission to access the Groq service.
• 429 Quota Exceeded: Your team's rate limit for Groq has been reached.
• 503 No Available Keys: Groq is temporarily unavailable through your KeyPool workspace.
• 502 Upstream Failed: The request to the Groq API failed (e.g., network error, Groq API down).

Interactive API Reference

For all Groq endpoints and in-browser testing, see API Reference → Groq.