Contents

lexer

Lexes Scintilla documents with Lua and LPeg.

Overview

Lexers are the mechanism for highlighting the syntax of source code. Scintilla (the editing component behind Textadept and SciTE) traditionally uses static, compiled C++ lexers which are notoriously difficult to create and/or extend. On the other hand, lexers written with Lua make it easy to rapidly create new lexers, extend existing ones, and embed lexers within one another. They tend to be more readable than C++ lexers too.

Lexers are written using Parsing Expression Grammars, or PEGs, with the Lua LPeg library. The following table is taken from the LPeg documentation and summarizes all you need to know about constructing basic LPeg patterns. This module provides convenience functions for creating and working with other more advanced patterns and concepts.

Operator Description
lpeg.P(string) Matches string literally.
lpeg.P(n) Matches exactly n characters.
lpeg.S(string) Matches any character in string (Set).
lpeg.R("xy") Matches any character between x and y (Range).
patt^n Matches at least n repetitions of patt.
patt^-n Matches at most n repetitions of patt.
patt1 * patt2 Matches patt1 followed by patt2.
patt1 + patt2 Matches patt1 or patt2 (ordered choice).
patt1 - patt2 Matches patt1 if patt2 does not match.
-patt Equivalent to ("" - patt).
#patt Matches patt but consumes no input.

The first part of this document deals with rapidly constructing a simple lexer. The next part deals with more advanced techniques, such as custom coloring and embedding lexers within one another. Following that is a discussion about code folding, or being able to tell Scintilla what code blocks can be “folded” (hidden temporarily from view). After that, instructions on how to use LPeg lexers with the aforementioned Textadept and SciTE editors is listed. Finally, considerations on performance and limitations are given.

Lexer Basics

All lexers are contained in the lexers/ directory. Your new lexer will also be included in this directory. Before attempting to write one from scratch though, first determine if your programming language is similar to any of the 80+ languages supported. If so, you may be able to copy and modify that lexer, saving some time and effort. The filename of your lexer should be the name of your programming language in lower case followed by a .lua extension. For example, a new Lua lexer would have the name lua.lua.

Note: It is not recommended to use one-character language names like “b”, “c”, or “d”. These lexers happen to be named b_lang.lua, cpp.lua, and dmd.lua respectively, for example.

New Lexer Template

There is a lexers/template.txt file that contains a simple template for a new lexer. Feel free to use it, replacing the ‘?’s with the name of your lexer:

-- ? LPeg lexer.

local l = lexer
local token, word_match = l.token, l.word_match
local style, color = l.style, l.color
local P, R, S = lpeg.P, lpeg.R, lpeg.S

local M = {_NAME = '?'}

-- Whitespace.
local ws = token(l.WHITESPACE, l.space^1)

M._rules = {
  {'whitespace', ws},
  {'any_char', l.any_char},
}

M._tokenstyles = {

}

return M

The first 4 lines of code are simply defining convenience variables you will be using often. The 5th and last lines define and return the lexer object used by Scintilla; they are very important and must be part of every lexer. The sixth line defines what is called a “token”, an essential building block of a lexer. Tokens will be discussed shortly. The rest of the code defines a set of grammar rules and token styles. Those will be discussed later. Note, however, the M. prefix in front of _rules and _tokenstyles: not only do these tables belong to their respective lexers, but any non-local variables should be prefixed by M. so-as not to affect Lua’s global environment. All in all, this is a minimal, working lexer that can be built on.

Tokens

Take a moment to think about your programming language’s structure. What kind of key elements does it have? In the template shown earlier, one predefined element all languages have is whitespace. Your language probably also has elements like comments, strings, and keywords. These elements are called “tokens”. They are the so-called “building blocks” of lexers. Source code is broken down into tokens and subsequently colored, resulting in the syntax highlighting you are familiar with. It is up to you how specific you would like your lexer to be when it comes to tokens. Perhaps you would like to only distinguish between keywords and identifiers, or maybe you would like to also recognize constants and built-in functions, methods, or libraries. The Lua lexer, for example, defines 11 tokens: whitespace, comments, strings, numbers, keywords, built-in functions, constants, built-in libraries, identifiers, labels, and operators. Even though constants, built-in functions, and built-in libraries are a subset of identifiers, it is helpful to Lua programmers for the lexer to distinguish between them all. It would have otherwise been perfectly acceptable to just recognize keywords and identifiers.

In a lexer, tokens are composed of a token name and an LPeg pattern that matches a sequence of characters recognized to be an instance of that token. Tokens are created using the token() function. Let us examine the “whitespace” token defined in the template shown earlier:

local ws = token(l.WHITESPACE, l.space^1)

At first glance, the first argument does not appear to be a string name and the second argument does not appear to be an LPeg pattern. Perhaps you were expecting something like:

local ws = token('whitespace', S('\t\v\f\n\r ')^1)

The lexer (l) module actually provides a convenient list of common token names and common LPeg patterns for you to use. Token names include DEFAULT, WHITESPACE, COMMENT, STRING, NUMBER, KEYWORD, IDENTIFIER, OPERATOR, ERROR, PREPROCESSOR, CONSTANT, VARIABLE, FUNCTION, CLASS, TYPE, LABEL, and REGEX. Patterns include any, ascii, extend, alpha, digit, alnum, lower, upper, xdigit, cntrl, graph, print, punct, space, newline, nonnewline, nonnewline_esc, dec_num, hex_num, oct_num, integer, float, and word. However, you are not limited to the token names and LPeg patterns listed. You can do whatever you like. However, the advantage of using predefined token names is that your lexer’s tokens will inherit the universal syntax highlighting color theme used by your text editor.

Example Tokens

So, how might other tokens like comments, strings, and keywords be defined? Here are some examples.

Comments

Line-style comments with a prefix character(s) are easy to express with LPeg:

local shell_comment = token(l.COMMENT, '#' * l.nonnewline^0)
local c_line_comment = token(l.COMMENT, '//' * l.nonnewline_esc^0)

The comments above start with a ‘#’ or “//” and go to the end of the line. The second comment recognizes the next line also as a comment if the current line ends with a ‘\’ escape character.

C-style “block” comments with a start and end delimiter are also easy to express:

local c_comment = token(l.COMMENT, '/*' * (l.any - '*/')^0 * P('*/')^-1)

This comment starts with a “/*” sequence and can contain anything up to, and including, an ending “*/” sequence. The ending “*/” is defined to be optional so that an unfinished comment is still matched as a comment and highlighted as you would expect.

Strings

It may be tempting to think that a string is not much different from the block comment shown above in that both have start and end delimiters:

local dq_str = '"' * (l.any - '"')^0 * P('"')^-1
local sq_str = "'" * (l.any - "'")^0 * P("'")^-1
local simple_string = token(l.STRING, dq_str + sq_str)

However, most programming languages allow escape sequences in strings such that a sequence like “\"” in a double-quoted string indicates that the ‘"’ is not the end of the string. The above token would incorrectly match such a string. Instead, a convenient function is provided for you: delimited_range().

local dq_str = l.delimited_range('"', '\\', true)
local sq_str = l.delimited_range("'", '\\', true)
local string = token(l.STRING, dq_str + sq_str)

In this case, ‘\’ is treated as an escape character in a string sequence. The true argument is analogous to P('"')^-1 in that non-terminated strings are highlighted as expected.

Keywords

Instead of matching n keywords with n P('keyword_n') ordered choices, another convenience function, word_match(), is provided. It is much easier and more efficient to write word matches like:

local keyword = token(l.KEYWORD, l.word_match{
  'keyword_1', 'keyword_2', ..., 'keyword_n'
})

local case_insensitive_keyword = token(l.KEYWORD, l.word_match({
  'KEYWORD_1', 'keyword_2', ..., 'KEYword_n'
}, nil, true))

local hyphened_keyword = token(l.KEYWORD, l.word_match({
  'keyword-1', 'keyword-2', ..., 'keyword-n'
}, '-'))

By default, characters considered to be in keywords are in the set of alphanumeric characters and underscores. The last token demonstrates how to allow ‘-’ (hyphen) characters to be in keywords as well.

Numbers

Most programming languages have the same format for integer and float tokens, so it might be as simple as using a couple of predefined LPeg patterns:

local number = token(l.NUMBER, l.float + l.integer)

However, some languages allow postfix characters on integers.

local integer = P('-')^-1 * (l.dec_num * S('lL')^-1)
local number = token(l.NUMBER, l.float + l.hex_num + integer)

Your language may have other tweaks that may be necessary, but it is up to you how fine-grained you want your highlighting to be. After all, it is not like you are writing a compiler or interpreter!

Rules

Programming languages have grammars, which specify how their tokens may be used structurally. For example, comments usually cannot appear within a string. Grammars are broken down into rules, which are simply combinations of tokens. Recall from the lexer template the _rules table, which defines all the rules used by the lexer grammar:

M._rules = {
  {'whitespace', ws},
  {'any_char', l.any_char},
}

Each entry in a lexer’s _rules table is composed of a rule name and its associated pattern. Rule names are completely arbitrary and serve only to identify and distinguish between different rules. Rule order is important: if text does not match the first rule, the second rule is tried, and so on. This simple grammar says to match whitespace tokens under a rule named “whitespace” and anything else under a rule named “any_char”.

To illustrate why rule order is important, here is an example of a simplified Lua grammar:

M._rules = {
  {'whitespace', ws},
  {'keyword', keyword},
  {'identifier', identifier},
  {'string', string},
  {'comment', comment},
  {'number', number},
  {'label', label},
  {'operator', operator},
  {'any_char', l.any_char},
}

Note how identifiers come after keywords. In Lua, as with most programming languages, the characters allowed in keywords and identifiers are in the same set (alphanumerics plus underscores). If the “identifier” rule was listed before the “keyword” rule, all keywords would match identifiers and thus would be incorrectly highlighted as identifiers instead of keywords. The same idea applies to function, constant, etc. tokens that you may want to distinguish between: their rules should come before identifiers.

Now, you may be wondering what l.any_char is and why the “any_char” rule exists. l.any_char is a special, predefined token that matches one single character as a DEFAULT token. The “any_char” rule should appear in every lexer because there may be some text that does not match any of the rules you defined. How is that possible? Well in Lua, for example, the ‘!’ character is meaningless outside a string or comment. Therefore, if the lexer encounters a ‘!’ in such a circumstance, it would not match any existing rules other than “any_char”. With “any_char”, the lexer can “skip” over the “error” and continue highlighting the rest of the source file correctly. Without “any_char”, the lexer would fail to continue. Perhaps you instead want your language to highlight such “syntax errors”. You would replace the “any_char” rule such that the grammar looks like:

M._rules = {
  {'whitespace', ws},
  {'error', token(l.ERROR, l.any)},
}

This would identify and highlight any character not matched by an existing rule as an ERROR token.

Even though the rules defined in the examples above contain a single token, rules can consist of multiple tokens. For example, a rule for an HTML tag could be composed of a tag token followed by an arbitrary number of attribute tokens, allowing all tokens to be highlighted separately. The rule might look something like this:

{'tag', tag_start * (ws * attributes)^0 * tag_end^-1}

Summary

Lexers are primarily composed of tokens and grammar rules. A number of convenience patterns and functions are available for rapidly creating a lexer. If you choose to use predefined token names for your tokens, you do not have to define how tokens are highlighted. They will inherit the default syntax highlighting color theme your editor uses.

Advanced Techniques

Styles and Styling

The most basic form of syntax highlighting is assigning different colors to different tokens. Instead of highlighting with just colors, Scintilla allows for more rich highlighting, or “styling”, with different fonts, font sizes, font attributes, and foreground and background colors, just to name a few. The unit of this rich highlighting is called a “style”. Styles are created using the style() function. By default, predefined token names like WHITESPACE, COMMENT, STRING, etc. are associated with a particular style as part of a universal color theme. These predefined styles include style_nothing, style_class, style_comment, style_constant, style_definition, style_error, style_function, style_keyword, style_label, style_number, style_operator, style_regex, style_string, style_preproc, style_tag, style_type, style_variable, style_whitespace, style_embedded, and style_identifier. Like with predefined token names and LPeg patterns, you are not limited to these predefined styles. At their core, styles are just Lua tables, so you can create new ones and/or modify existing ones. Each style consists of a set of attributes:

Attribute Description
font The name of the font the style uses.
size The size of the font the style uses.
bold Whether or not the font face is bold.
italic Whether or not the font face is italic.
underline Whether or not the font face is underlined.
fore The foreground color of the font face.
back The background color of the font face.
eolfilled Does the background color extend to the end of the line?
case The case of the font (1 = upper, 2 = lower, 0 = normal).
visible Whether or not the text is visible.
changeable Whether the text is changeable or read-only.
hotspot Whether or not the text is clickable.

Font colors are defined using the color() function. Like with token names, LPeg patterns, and styles, there is a set of predefined colors in the l.colors table, but the color names depend on the current theme being used. It is generally not a good idea to manually define colors within styles in your lexer because they might not fit into a user’s chosen color theme. It is not even recommended to use a predefined color in a style because that color may be theme-specific. Instead, the best practice is to either use predefined styles or derive new color-agnostic styles from predefined ones. For example, Lua “longstring” tokens use the existing style_string style instead of defining a new one.

Example Styles

Defining styles is pretty straightforward. An empty style that inherits the default theme settings is defined like this:

local style_nothing = l.style{}

A similar style but with a bold font face is defined like this:

local style_bold = l.style{bold = true}

If you wanted the same style, but also with an italic font face, you can define the new style in terms of the old one:

local style_bold_italic = style_bold..{italic = true}

This allows you to derive new styles from predefined ones without having to rewrite them. This operation leaves the old style unchanged. Thus if you had a “static variable” token whose style you wanted to base off of style_variable, it would probably look like:

local style_static_var = l.style_variable..{italic = true}

More examples of style definitions are in the color theme files in the lexers/themes/ folder.

Token Styles

Tokens are assigned to a particular style with the lexer’s _tokenstyles table. Recall the token definition and _tokenstyles table from the lexer template:

local ws = token(l.WHITESPACE, l.space^1)

...

M._tokenstyles = {

}

Why is a style not assigned to the WHITESPACE token? As mentioned earlier, tokens that use predefined token names are automatically associated with a particular style. Only tokens with custom token names need manual style associations. As an example, consider a custom whitespace token:

local ws = token('custom_whitespace', l.space^1)

Assigning a style to this token looks like:

M._tokenstyles = {
  {'custom_whitespace', l.style_whitespace}
}

Each entry in a lexer’s _tokenstyles table is composed of a token’s name and its associated style. Unlike with _rules, the ordering in _tokenstyles does not matter since entries are just associations. Do not confuse token names with rule names. They are completely different entities. In the example above, the “custom_whitespace” token is just being assigned the existing style for WHITESPACE tokens. If instead you wanted to color the background of whitespace a shade of grey, it might look like:

local style = l.style_whitespace..{back = l.colors.grey}
M._tokenstyles = {
  {'custom_whitespace', style}
}

Remember it is generally not recommended to assign specific colors in styles, but in this case, the color grey likely exists in all user color themes.

Line Lexers

By default, lexers match the arbitrary chunks of text passed to them by Scintilla. These chunks may be a full document, only the visible part of a document, or even just portions of lines. Some lexers need to match whole lines. For example, a lexer for the output of a file “diff” needs to know if the line started with a ‘+’ or ‘-’ and then style the entire line accordingly. To indicate that your lexer matches by line, use the _LEXBYLINE field:

M._LEXBYLINE = true

Now the input text for the lexer is a single line at a time. Keep in mind that line lexers do not have the ability to look ahead at subsequent lines.

Embedded Lexers

Lexers can be embedded within one another very easily, requiring minimal effort. In the following sections, the lexer being embedded is called the “child” lexer and the lexer a child is being embedded in is called the “parent”. For example, consider an HTML lexer and a CSS lexer. Either lexer can stand alone for styling their respective HTML and CSS files. However, CSS can be embedded inside HTML. In this specific case, the CSS lexer is referred to as the “child” lexer with the HTML lexer being the “parent”. Now consider an HTML lexer and a PHP lexer. This sounds a lot like the case with CSS, but there is a subtle difference: PHP embeds itself into HTML while CSS is embedded in HTML. This fundamental difference results in two types of embedded lexers: a parent lexer that embeds other child lexers in it (like HTML embedding CSS), and a child lexer that embeds itself within a parent lexer (like PHP embedding itself in HTML).

Parent Lexer

Before you can embed a child lexer into a parent lexer, the child lexer needs to be loaded inside the parent. This is done with the load() function. For example, loading the CSS lexer within the HTML lexer looks like:

local css = l.load('css')

The next part of the embedding process is telling the parent lexer when to switch over to the child lexer and when to switch back. These indications are called the “start rule” and “end rule”, respectively, and are just LPeg patterns. Continuing with the HTML/CSS example, the transition from HTML to CSS is when a “style” tag with a “type” attribute whose value is “text/css” is encountered:

local css_tag = P('<style') * P(function(input, index)
  if input:find('^[^>]+type="text/css"', index) then
    return index
  end
end)

This pattern looks for the beginning of a “style” tag and searches its attribute list for the text “type="text/css"”. (In this simplified example, the Lua pattern does not consider whitespace between the ‘=’ nor does it consider that single quotes can be used instead of double quotes.) If there is a match, the functional pattern returns a value instead of nil. In this case, the value returned does not matter because we ultimately want the “style” tag to be styled as an HTML tag, so the actual start rule looks like this:

local css_start_rule = #css_tag * tag

Now that the parent knows when to switch to the child, it needs to know when to switch back. In the case of HTML/CSS, the switch back occurs when an ending “style” tag is encountered, but the tag should still be styled as an HTML tag:

local css_end_rule = #P('</style>') * tag

Once the child lexer is loaded and its start and end rules are defined, you can embed it in the parent using the embed_lexer() function:

l.embed_lexer(M, css, css_start_rule, css_end_rule)

The first parameter is the parent lexer object to embed the child in, which in this case is M. The other three parameters are the child lexer object loaded earlier followed by its start and end rules.

Child Lexer

The process for instructing a child lexer to embed itself into a parent is very similar to embedding a child into a parent: first, load the parent lexer into the child lexer with the load() function and then create start and end rules for the child lexer. However, in this case, swap the lexer object arguments to embed_lexer() and indicate through a _lexer field in the child lexer that the parent should be used as the primary lexer. For example, in the PHP lexer:

local html = l.load('hypertext')
local php_start_rule = token('php_tag', '<?php ')
local php_end_rule = token('php_tag', '?>')
l.embed_lexer(html, M, php_start_rule, php_end_rule)
M._lexer = html

The last line is very important. Without it, the PHP lexer’s rules would be used instead of the HTML lexer’s rules.

Code Folding

When reading source code, it is occasionally helpful to temporarily hide blocks of code like functions, classes, comments, etc. This concept is called “folding”. In the Textadept and SciTE editors for example, little indicators in the editor margins appear next to code that can be folded at places called “fold points”. When an indicator is clicked, the code associated with it is visually hidden until the indicator is clicked again. A lexer can specify these fold points and what code exactly to fold.

The fold points for most languages occur on keywords or character sequences. Examples of fold keywords are “if” and “end” in Lua and examples of fold character sequences are ‘{’, ‘}’, “/*”, and “*/” in C for code block and comment delimiters, respectively. However, these fold points cannot occur just anywhere. For example, fold keywords that appear within strings or comments should not be recognized as fold points. Your lexer can conveniently define fold points with such granularity in a _foldsymbols table. For example, consider C:

M._foldsymbols = {
  [l.OPERATOR] = {['{'] = 1, ['}'] = -1},
  [l.COMMENT] = {['/*'] = 1, ['*/'] = -1},
  _patterns = {'[{}]', '/%*', '%*/'}
}

The first assignment states that any ‘{’ or ‘}’ that the lexer recognized as an OPERATOR token is a fold point. The integer 1 indicates the match is a beginning fold point and -1 indicates the match is an ending fold point. Likewise, the second assignment states that any “/*” or “*/” that the lexer recognizes as part of a COMMENT token is a fold point. Any occurences of these characters outside their defined tokens (such as in a string) would not be considered a fold point. Finally, every _foldsymbols table must have a _patterns field that contains a list of Lua patterns that match fold points. If the lexer encounters text that matches one of those patterns, the matched text is looked up in its token’s table to determine whether or not it is a fold point. In the example above, the first Lua pattern matches any ‘{’ or ‘}’ characters. When the lexer comes across one of those characters, it checks if the match is an OPERATOR token. If so, the match is identified as a fold point. It is the same idea for the other patterns. (The ‘%’ is in the other patterns because ‘*’ is a special character in Lua patterns and it must be escaped.) How are fold keywords specified? Here is an example for Lua:

M._foldsymbols = {
  [l.KEYWORD] = {
    ['if'] = 1, ['do'] = 1, ['function'] = 1,
    ['end'] = -1, ['repeat'] = 1, ['until'] = -1
  },
  _patterns = {'%l+'},
}

Any time the lexer encounters a lower case word, if that word is a KEYWORD token and in the associated list of fold points, it is identified as a fold point.

If your lexer needs to do some additional processing to determine if a fold point has occurred on a match, you can assign a function that returns an integer. Returning 1 or -1 indicates the match is a fold point. Returning 0 indicates it is not. For example:

local function fold_strange_token(text, pos, line, s, match)
  if ... then
    return 1 -- beginning fold point
  elseif ... then
    return -1 -- ending fold point
  end
  return 0
end

M._foldsymbols = {
  ['strange_token'] = {['|'] = fold_strange_token},
  _patterns = {'|'}
}

Any time the lexer encounters a ‘|’ that is a “strange_token”, it calls the fold_strange_token function to determine if ‘|’ is a fold point. These kinds of functions are called with the following arguments: the text to fold, the position of the start of the current line in the text to fold, the text of the current line, the position in the current line the matched text starts at, and the matched text itself.

Using Lexers

Textadept

Put your lexer in your ~/.textadept/lexers/ directory so it will not be overwritten when upgrading Textadept. Also, lexers in this directory override default lexers. Thus, a user lua lexer would be loaded instead of the default lua lexer. This is convenient if you wish to tweak a default lexer to your liking. Then add a mime-type for your lexer if necessary.

SciTE

Create a .properties file for your lexer and import it in either your SciTEUser.properties or SciTEGlobal.properties. The contents of the .properties file should contain:

file.patterns.[lexer_name]=[file_patterns]
lexer.$(file.patterns.[lexer_name])=[lexer_name]

where [lexer_name] is the name of your lexer (minus the .lua extension) and [file_patterns] is a set of file extensions matched to your lexer.

Please note any styling information in .properties files is ignored. Styling information for Lua lexers is contained in your theme file in the lexers/themes/ directory.

Considerations

Performance

There might be some slight overhead when initializing a lexer, but loading a file from disk into Scintilla is usually more expensive. On modern computer systems, I see no difference in speed between LPeg lexers and Scintilla’s C++ ones. Lexers can usually be optimized for speed by re-arranging rules in the _rules table so that the most common rules are matched first. Do keep in mind the fact that order matters for similar rules.

Limitations

Embedded preprocessor languages like PHP are not completely embedded in their parent languages in that the parent’s tokens do not support start and end rules. This mostly goes unnoticed, but code like

<div id="<?php echo $id; ?>">

or

<div <?php if ($odd) { echo 'class="odd"'; } ?>>

will not style correctly.

Troubleshooting

Errors in lexers can be tricky to debug. Lua errors are printed to io.stderr and _G.print() statements in lexers are printed to io.stdout. Running your editor from a terminal is the easiest way to see errors as they occur.

Risks

Poorly written lexers have the ability to crash Scintilla (and thus its containing application), so unsaved data might be lost. However, these crashes have only been observed in early lexer development, when syntax errors or pattern errors are present. Once the lexer actually starts styling text (either correctly or incorrectly, it does not matter), no crashes have been observed.

Acknowledgements

Thanks to Peter Odding for his lexer post on the Lua mailing list that inspired me, and thanks to Roberto Ierusalimschy for LPeg.


Fields


CLASS (string)

The token name for class tokens.


COMMENT (string)

The token name for comment tokens.


CONSTANT (string)

The token name for constant tokens.


DEFAULT (string)

The token name for default tokens.


ERROR (string)

The token name for error tokens.


FOLD_BASE (number)

The initial (root) fold level.


FOLD_BLANK (number)

Flag indicating that the line is blank.


FOLD_HEADER (number)

Flag indicating the line is fold point.


FUNCTION (string)

The token name for function tokens.


IDENTIFIER (string)

The token name for identifier tokens.


KEYWORD (string)

The token name for keyword tokens.


LABEL (string)

The token name for label tokens.


NUMBER (string)

The token name for number tokens.


OPERATOR (string)

The token name for operator tokens.


PREPROCESSOR (string)

The token name for preprocessor tokens.


REGEX (string)

The token name for regex tokens.


STRING (string)

The token name for string tokens.


TYPE (string)

The token name for type tokens.


VARIABLE (string)

The token name for variable tokens.


WHITESPACE (string)

The token name for whitespace tokens.


alnum (pattern)

A pattern matching any alphanumeric character (A-Z, a-z, 0-9).


alpha (pattern)

A pattern matching any alphabetic character (A-Z, a-z).


any (pattern)

A pattern matching any single character.


any_char (pattern)

A DEFAULT token matching any single character, useful in a fallback rule for a grammar.


ascii (pattern)

A pattern matching any ASCII character (0..127).


cntrl (pattern)

A pattern matching any control character (0..31).


dec_num (pattern)

A pattern matching a decimal number.


digit (pattern)

A pattern matching any digit (0-9).


extend (pattern)

A pattern matching any ASCII extended character (0..255).


float (pattern)

A pattern matching a floating point number.


graph (pattern)

A pattern matching any graphical character (! to ~).


hex_num (pattern)

A pattern matching a hexadecimal number.


integer (pattern)

A pattern matching a decimal, hexadecimal, or octal number.


lower (pattern)

A pattern matching any lower case character (a-z).


newline (pattern)

A pattern matching any newline characters.


nonnewline (pattern)

A pattern matching any non-newline character.


nonnewline_esc (pattern)

A pattern matching any non-newline character excluding newlines escaped with ‘\’.


oct_num (pattern)

A pattern matching an octal number.


print (pattern)

A pattern matching any printable character (space to ~).


punct (pattern)

A pattern matching any punctuation character not alphanumeric (! to /, : to @, [ to ', { to ~).


space (pattern)

A pattern matching any whitespace character (\t, \v, \f, \n, \r, space).


style_bracebad (table)

The style used for unmatched brace characters.


style_bracelight (table)

The style used for highlighted brace characters.


style_calltip (table)

The style used by call tips if buffer.call_tip_use_style is set. Only the font name, size, and color attributes are used.


style_class (table)

The style typically used for class definitions.


style_comment (table)

The style typically used for code comments.


style_constant (table)

The style typically used for constants.


style_controlchar (table)

The style used for control characters. Color attributes are ignored.


style_default (table)

The style all styles are based off of.


style_definition (table)

The style typically used for definitions.


style_embedded (table)

The style typically used for embedded code.


style_error (table)

The style typically used for erroneous syntax.


style_function (table)

The style typically used for function definitions.


style_identifier (table)

The style typically used for identifier words.


style_indentguide (table)

The style used for indentation guides.


style_keyword (table)

The style typically used for language keywords.


style_label (table)

The style typically used for labels.


style_line_number (table)

The style used for all margins except fold margins.


style_nothing (table)

The style typically used for no styling.


style_number (table)

The style typically used for numbers.


style_operator (table)

The style typically used for operators.


style_preproc (table)

The style typically used for preprocessor statements.


style_regex (table)

The style typically used for regular expression strings.


style_string (table)

The style typically used for strings.


style_tag (table)

The style typically used for markup tags.


style_type (table)

The style typically used for static types.


style_variable (table)

The style typically used for variables.


style_whitespace (table)

The style typically used for whitespace.


upper (pattern)

A pattern matching any upper case character (A-Z).


word (pattern)

A pattern matching a typical word starting with a letter or underscore and then any alphanumeric or underscore characters.


xdigit (pattern)

A pattern matching any hexadecimal digit (0-9, A-F, a-f).


Functions


color(r, g, b)

Creates and returns a Scintilla color from r, g, and b string hexadecimal color components.

Parameters:

Usage:

Return:


delimited_range(chars, escape, end_optional, balanced, forbidden)

Creates and returns a pattern that matches a range of text bounded by chars characters. This is a convenience function for matching more complicated delimited ranges like strings with escape characters and balanced parentheses. escape specifies the escape characters a range can have, end_optional indicates whether or not unterminated ranges match, balanced indicates whether or not to handle balanced ranges like parentheses and requires chars to be composed of two characters, and forbidden is a set of characters disallowed in ranges such as newlines.

Parameters:

Usage:

Return:

See also:


embed_lexer(parent, child, start_rule, end_rule)

Embeds child lexer in parent with start_rule and end_rule, patterns that signal the beginning and end of the embedded lexer, respectively.

Parameters:

Usage:


fold(text, start_pos, start_line, start_level)

Folds text, a chunk of text starting at position start_pos on line number start_line with a beginning fold level of start_level in the buffer. Called by the Scintilla lexer; do not call from Lua. If the current lexer has a _fold function or a _foldsymbols table, it is used to perform folding. Otherwise, if a fold.by.indentation property is set, folding by indentation is done.

Parameters:

Return:


fold_line_comments(prefix)

Returns a fold function, to be used within the lexer’s _foldsymbols table, that folds consecutive line comments beginning with string prefix.

Parameters:

Usage:


get_fold_level(line_number)

Returns the fold level for line number line_number. This level already has SC_FOLDLEVELBASE added to it, so you do not need to add it yourself.

Parameters:

Return:


get_indent_amount(line_number)

Returns the amount of indentation the text on line number line_number has.

Parameters:

Return:


get_property(key, default)

Returns the integer property value associated with string property key, or default.

Parameters:

Return:


get_style_at(pos)

Returns the string style name and style number at position pos in the buffer.

Parameters:

Return:


last_char_includes(s)

Creates and returns a pattern that matches any previous non-whitespace character in s and consumes no input.

Parameters:

Usage:

Return:


lex(text, init_style)

Lexes a chunk of text text with an initial style number of init_style. Called by the Scintilla lexer; do not call from Lua. If the lexer has a _LEXBYLINE flag set, the text is lexed one line at a time. Otherwise the text is lexed as a whole.

Parameters:

Return:


load(lexer_name)

Initializes or loads lexer lexer_name and returns the lexer object. Scintilla calls this function to load a lexer. Parent lexers also call this function to load child lexers and vice-versa.

Parameters:

Return:


nested_pair(start_chars, end_chars, end_optional)

Similar to delimited_range(), but allows for multi-character, nested delimiters start_chars and end_chars. end_optional indicates whether or not unterminated ranges match. With single-character delimiters, this function is identical to delimited_range(start_chars..end_chars, nil, end_optional, true).

Parameters:

Usage:

Return:

See also:


starts_line(patt)

Creates and returns a pattern that matches pattern patt only at the beginning of a line.

Parameters:

Usage:

Return:


style(style_table)

Creates and returns a Scintilla style from the given table of style properties.

Parameters:

Usage:

Return:

See also:


token(name, patt)

Creates and returns a token pattern with the name name and pattern patt. If name is not a predefined token name, its style must be defined in the lexer’s _tokenstyles table.

Parameters:

Usage:

Return:


word_match(words, word_chars, case_insensitive)

Creates and returns a pattern that matches any word in the set words case-sensitively, unless case_insensitive is true, with the set of word characters being alphanumerics, underscores, and all of the characters in word_chars. This is a convenience function for simplifying a set of ordered choice word patterns.

Parameters:

Usage:

Return:


Tables


colors

Table of common colors for a theme. This table should be redefined in each theme.


lexer

Individual lexer fields.

Fields: