Tokenizer

The Tokenizer is a free and a useful tool provided by the iQ Suite Platform that allows you to calculate the number of tokens in a given piece of text. This is particularly useful for estimating token expenditure when interacting with language models, as many models charge based on token usage.

Rate Limiting

The Tokenizer method is rate-limited to 50 requests per minute to ensure fair usage and maintain system performance. Exceeding this limit will result in throttled requests.

HTTP Request

POST /tokenizer

Authentication

All requests must include an Authorization header with a valid API key.

Headers

Request Body

• text: The input text for which you want to calculate the token count.

JSON

Request Example

try:
# Call the tokenizer method with the input text
response = client.tokenizer(text="hello world")
 
# Print the token count from the response
print(f"Token Count: {response.tokens_count}")
except APIError as e:
# Handle any API-related errors
print(f"Error tokenizing text: {e}")

Response

Success

Status Code: 200 OK

Body:

{
  "tokens_count": 2
}

Fields