The entrance point into Lexer API is
{@link org.netbeans.api.lexer.TokenHierarchy} class with
its static methods that provide its instance for the given input source.
{@link org.netbeans.api.lexer.TokenHierarchy#tokenSequence()}
allows to walk through the tokens that were created for the given input.
{@link org.netbeans.api.lexer.Token} models a single token recognized in the input.
It carries a token identification {@link org.netbeans.api.lexer.TokenId} returned by
{@link org.netbeans.api.lexer.Token#id()}
and token text represented as {@link java.lang.CharSequence}
returned by {@link org.netbeans.api.lexer.Token#text()}.
{@link org.netbeans.api.lexer.Language} represents
all the token ids for a single language.
Each token id may belong to one or more token categories that allow to better operate
tokens of the same type (e.g. keywords or operators).