Lexical analysis

computing process of parsing a sequence of characters into a sequence of tokens

standard UNIX utility

free software alternative to lex

in lexical analysis, atomic string with an assigned and thus identified meaning; can be either word, bi-gram or sentence