I Check Token Counts Before Sending Another File

Start with the units your model actually reads

When I watch my AI spending, I start with tokens rather than message length. A token is a piece of text produced by a tokenizer. It might represent a word, part of a word, punctuation, or whitespace. The model processes these pieces, and API pricing commonly charges separately for input and output tokens.

This means two messages that look equally long can cost different amounts. Chinese prose, English prose, and source code have different patterns, and tokenizers handle those patterns differently. The useful habit is not memorizing one conversion rule. It is measuring the material you actually send.

Build three small samples before estimating a project

Choose one short English paragraph, one Chinese paragraph with approximately the same meaning, and one small source code excerpt from your work. Keep their tasks comparable: ask for a brief summary of each, with the same requested answer length.

Equal meaning does not guarantee equal token counts. A Chinese translation can occupy less visual space while consuming more tokens than its English counterpart on some tokenizers. Other tokenizers encode common Chinese text more efficiently. Source code introduces another pattern through identifiers, indentation, punctuation, and repeated syntax.

Keep each sample unchanged while comparing it. If you shorten one prompt or request a longer answer from another, you are measuring more than tokenization.

Treat English shortcuts as rough estimates

You may have heard that one English token is roughly four characters, or that a token is about three quarters of a word. Those shortcuts can help estimate ordinary English prose, but they are not dependable limits.

Common words may fit into single tokens, while unusual names and specialized vocabulary may split into several. Numbers, punctuation, and spacing also affect the result. A paragraph from a textbook and a paragraph full of product identifiers can tokenize differently despite having similar character counts.

Use a provider's tokenizer or token-counting tool when available. Make sure it matches the model you plan to use. A count from an unrelated tokenizer is an estimate, not a billing prediction.

Check Chinese without assuming one character means one token

For Chinese, avoid replacing the English shortcut with another rigid rule. Depending on the tokenizer, a character may correspond to one token, multiple tokens, or part of a token that represents a common sequence.

Mixed-language text deserves particular attention. Chinese notes containing English technical terms, numbers, and punctuation can produce counts that are difficult to predict by sight. Test a representative passage rather than an unusually simple sentence.

Do not translate everything into English merely to chase a lower count. Translation can lose nuance and adds work. If Chinese is the clearest language for your task, first reduce irrelevant context and unnecessary repetition.

Inspect code for volume, not just visible lines

Source code is not priced by line count. A short line containing a long identifier or dense data can use more tokens than several simple statements. Comments, repeated declarations, generated content, and embedded strings all contribute.

Before sending a file, identify what the model needs to answer your question. For a bug, that might be the relevant function, its caller, and the error message. Include necessary dependencies and constraints, but leave out unrelated modules.

Do not blindly remove whitespace or compress code. Savings are tokenizer-dependent, and unreadable input can lead to worse answers and extra requests. Removing irrelevant content is usually a better first move.

Compare recorded usage before scaling up

Run your three samples through the same model and record input tokens, output tokens, and cost separately. Input usage may also include conversation history, system instructions, or tool definitions. Output length is a separate variable, so request concise answers when that suits the task.

MyRelay lets you see each request's tokens and cost, which makes this small experiment easier to review. You can create a MyRelay account to compare your own samples before committing a larger budget.

Finally, check the applicable input, output, and caching prices rather than comparing token counts alone. Repeat the test when you change models or workloads. A few measured requests provide a better spending forecast than guessing from characters, words, or lines.