Token Limit Scanner

(Input scanner)

It ensures that prompts do not exceed a predetermined token count, helping prevent resource-intensive operations and potential denial of service attacks on large language models (LLMs).

How it works

The scanner works by calculating the number of tokens in the provided prompt using tiktoken library. If the token count exceeds the configured limit, the prompt is flagged as being too long.

One token usually equates to approximately 4 characters in common English text. Roughly speaking, 100 tokens are equivalent to about 75 words.

Token Limit Detection Policy for AI Chatbot

Create a new policy as same as shown in LLM Guardrails Policy, for Token Limit detection select scanner Token Limit (Set the token limit based on the number of words or sentences in the token to determine how many tokens are in a sentence).

For an in-depth understanding, refer to:

• OpenAI Tokenizer Guide

Optionally, perform a test to ensure the policy is functioning as intended. Check that Token Limit is detected and blocked as specified.