Token Visualiser
What you'll learn
- Understand how AI breaks text into smaller pieces called tokens
- See why there are limits on how much text AI can process
- Learn that tokens aren't always whole words
What are Tokens?
AI doesn't read text like we do. Instead, it breaks text into smaller pieces called tokens. A token might be a whole word, part of a word, or even just punctuation.
AI systems have a maximum number of tokens they can handle at once (their "context window"). This visualiser shows you approximately how text gets tokenised.
Try It Out
Try these: