Dev Tools · 2h ago
Stop Using tiktoken for Claude Token Counting
Using OpenAI's tiktoken to count tokens for Claude models can undercount by 15–20%, leading to inaccurate cost estimates. The correct approach is to use Claude's dedicated count_tokens endpoint, which returns model-specific token counts. For example, a prompt estimated at 8,000 tokens by tiktoken may actually be 9,500 tokens for Claude.
Meridian48 take
This is a practical reminder that tokenizers are model-specific, and borrowing tools across platforms introduces systematic errors in cost and context budgeting.
token-countingclaude-api