StressingLLMs Token Results

Hello, here are the results of the token evaluation side of my StressingLLMs project; it is titled The cost of unfamiliar text. The results consisted of evaluating how large language models handle different type of character encodings. My early assumption for token inflation was that repeated characters would stress the tokenizer but this was incorrect. What stresses the tokenizer is rare Unicode characters. Here is the TL;DR

Rare high-plane Unicode can multiply an LLM request's token count and cost, creating a context and billing amplification risk for systems that process untrusted text.

 A fair amount of the text from the link was generated by AI but also edited by myself. The page is more of a way for me to explore the data and learn the underlying concepts. I hope it does the same for others. Cheers.