🌐日本語EnglishBahasa IndonesiaDeutschFrançaisSvenskaРусский中文日本語EntityQ2438971· pop 5トークン化Also known as tokenisationbreaking a stream of text up into chunks for analysis or further processingAvailable in 4 languagesDeutschРусскийBahasa Indonesiavia Wikidata sitelinks · CC0Connectionslexical analysisEntity