LuaTeXWiki - User contributions [en]

Documentation and help

2013-12-06T10:03:47Z

Paul:

=Documentation=

* The definitive source of information on LuaTeX is the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX Reference]; the link points to the latest version, documenting latest builds, but can be used for older versions too.

* The previous document reduced to a cheatsheet is also available with the [http://www.luatex.org/svn/trunk/manual/functionref.pdf Short Reference].

* Besides general information, the [http://www.luatex.org/ LuaTeX site] also hosts [http://www.luatex.org/documentation.html some presentations] by members of the team.

* [http://tug.org/TUGboat/ TUGboat] (communications of the TeX User Group) has featured regular articles on LuaTeX for some years now.

* More can be found in publications by local user groups and conferences.

=Help=

* The [http://tug.org/mailman/listinfo/luatex LuaTeX mailing list] is frequented by knowledgeable people and questions rarely remain unanswered.

* There is also a [http://www.ntg.nl/mailman/listinfo/dev-luatex list for developers] for very technical things.

* Questions about LuaTeX can also be asked in other TeX-related lists or sites: [http://tug.org/mailman/listinfo/texhax texhax], [http://groups.google.com/group/comp.text.tex comp.text.tex] (and similar groups for different languages), or [http://tex.stackexchange.com/ TeX StackExchange].

=On Lua=

* The main source on the Lua language is the [http://www.lua.org/manual/5.2/manual.html Lua reference manual]; the link here documents version 5.2, currently used by LuaTeX, but Lua 5.1 is what LuaTeX used up to v0.74; see the [http://www.lua.org/manual/5.1/manual.html Lua 5.1 reference manual] if necessary. See also the [http://www.lua.org/manual/5.2/readme.html#changes changes] and [http://www.lua.org/manual/5.2/manual.html#8 incompatibilities].

* [http://www.lua.org/pil/index.html Programming in Lua], by Lua's chief architect Roberto Ierusalimschy, is available online for version 5.0, quite usable for later versions.

* There is also a [http://www.lua.org/lua-l.html mailing list] and a [http://forum.luahub.com/ forum].

* All this and more can be found on the [http://www.lua.org/ Lua website].

Show error hook

2012-02-28T09:24:14Z

Paul: The API is not deprecated in LuaTeX itself, it's luatexbase that deprecates it; it's better here to document LuaTeX, not luatexbase, because it's not used by everybody. See the page on callbacks.

=The show_error_hook callback=

This [[Callbacks|callback]] doesn't replace any internal code and isn't supposed to do anything meaningful. It is called when an error message occurs, precisely right after the first line of the error message (beginning with an exclamation mark) has been printed, and before the rest of the message is given.

No argument is passed to the function registered in the callback, and return values, if any, are ignored. Its best use is probably to report additional information besides the error message. For instance, the input text <tt>a_1</tt> will produce the following error message (and another one quite similar at the end of the paragraph):

<pre>
! Missing $ inserted.
<inserted text>
$
<to be read again>
_
l.1 a_
1
I've inserted a begin-math/end-math symbol since I think
you left one out. Proceed, with fingers crossed.
</pre>

This message can be annoying for two reasons: first, if one has a document made of several files, one doesn't know which file is concerned here (one can read the log file upward to spot the latest input file, but that isn't very convenient); second, the line number takes some time to arrive after the message's first line. The lines 2 to 4 aren't very informative, and it could be useful if the line number was available at once.

The following code inserts the input file's name and line number just after the first line of the message; thus the rest of the message can be ignored (as many users do, the first line, along with where the error happened, being often informative enough):

<pre>
callback.register("show_error_hook",
function ()
texio.write_nl("[FILE: " .. status.filename .. ", LINE: " .. status.linenumber .. "]")
end)
</pre>

Some information from the <tt>status</tt> library is put to use here: <tt>status.filename</tt> returns a string with the name of the current input file, <tt>status.linenumber</tt> returns a number (automatically converted to a string here) indicating the line TeX is currently reading.

Now the error message looks like this, assuming that <tt>a_1</tt> appeared in file called <tt>test_input.tex</tt> in the same directory as the main file:

<pre>
! Missing $ inserted.
[FILE: ./test_input.tex, LINE: 1]
<inserted text>
$
<to be read again>
_
l.1 a_
1
I've inserted a begin-math/end-math symbol since I think
you left one out. Proceed, with fingers crossed.
</pre>

If one wants a full path instead of a relative path, one can use [http://keplerproject.github.com/luafilesystem/ the LuaFileSystem library], shipped with LuaTeX (the use of <tt>%</tt> implies that this code isn't simply passed to <tt>\directlua</tt>; see [[Writing Lua in TeX]]):

<pre>
callback.register("show_error_hook",
function ()
local f = status.filename
f = string.gsub(f, "^%.", lfs.currentdir())
texio.write_nl("[FILE: " .. f .. ", LINE: " .. status.linenumber .. "]")
end)
</pre>

This replaces <tt>./</tt> at the beginning of the filename (if any) with the string returned by <tt>lfs.currentdir()</tt>, whose meaning should be obvious. (See [http://www.lua.org/manual/5.1/manual.html#pdf-string.gsub the Lua reference manual] for information on <tt>string.gsib()</tt>.)

Callbacks

2012-02-01T07:46:20Z

Paul: Modified remark on the distinction between nil and false.

Callbacks allow the user to interact with TeX's internal operations; those can be enhanced by additional ones, but also replaced altogether. All callbacks are written in Lua.

Since functions registered in callbacks occur in a sequence of actions that makes up TeX's processing, many of them will receive arguments (and thus should be defined to handle them) and, most importantly, many are supposed to return values. What arguments are passed, if any, and what should be return, vary from callback to callback.

= Registering a callback =

To install a function in a callback one must use the <tt>callback.register()</tt> function, whose syntax is:

<pre>
callback.register(callback_name, function)
</pre>

where <tt>callback_name</tt> is a string representing a predefined callback name (see below), and <tt>function</tt> is either a function variable or an anonymous function. The former is just the name of a function previously defined, for instance:

<pre>
local my_function = function (an_arg)
do_this_or_that(an_arg)
end
callback.register("some_callback", my_function)
</pre>

An anonymous function is a function definition that is not assigned to a variable, e.g.:

<pre>
callback.register("some_callback", function (an_arg) do_this_or_that(an_arg) end)
</pre>

Even in the case of a function variable, what LuaTeX registers is the function itself, and any further assignment to the variable has no effect on the callback.

If registering is successful, <tt>callback.register()</tt> returns a number, which is the internal representation of the callback; if registering fails (for instance because a callback doesn't exist, or what is registered isn't a function), the function returns <tt>nil</tt> and a string representing an error message.

Besides functions, <tt>callback.register()</tt> can take <tt>nil</tt> or <tt>false</tt>. The first value clears the callback, and returns to the default behavior (i.e. either nothing or TeX's default processing); the second value prevents anything from happening at this point, even TeX's default operations, and should be used with great care (the benefits are minor speed gain).

== Several functions in a callback ==

Each call to <tt>callback.register()</tt> erases any function previously registered. I.e. after

<pre>
callback.register("some_callback", function_one)
callback.register("some_callback", function_two)
</pre>

only <tt>function_two</tt> is registered in <tt>some_callback</tt>. Hence, if one wants several successive functions to occur in a callback, <tt>callback.register()</tt> is clearly insufficient. Instead, one must register a metafunction which calls the functions one after the other. This is taken care of in ConTeXt, and there exists <tt>luatexbase-mcb</tt> for plain TeX and LaTeX, but here's how to do it oneself. We'll take the <tt>process_input_buffer</tt> callback has an example; it is called whenever TeX reads an input line, it receives a string (the line) and should return one. A rough approach would be:

<pre>
local function_table = { }

local function metafunction (str)
for _, fct in ipairs(function_table) do
str = fct(str)
end
return str
end

function add_function (fct)
table.insert(function_table, fct)
end

callback.register("process_input_buffer", metafunction)
</pre>

Then one can call <tt>add_function(my_function)</tt> to add <tt>my_function</tt> to the list of functions registered in <tt>process_input_buffer</tt>, and <tt>metafunction</tt> will pass the callback's argument (an input line) to the first function, set that argument to what the function returns, then repeat the process for all functions and finally return the resulting string.

If one also wants to be able to remove functions (and not the entire metafunction), a subtler approach is required, where names are associated with functions and can be used as handles.

= Information about callbacks =

LuaTeX has two functions to get some information about callbacks. The first is <tt>callback.list()</tt>, called without arguments and returning a table with the callback names as keys and booleans as values: if the value is <tt>true</tt>, it means that something is registered in the callback, or that it is set to <tt>false</tt> (see above). The other function is <tt>callback.find()</tt>, called with a callback name. If a function is registered, it is returned; if the callback is set to <tt>false</tt>, a boolean with that value is returned; finally, if nothing is registered, <tt>nil</tt> is returned (note that if <tt>ret</tt> is the returned value, then <tt>if ret</tt> is false if <tt>ret</tt> is either <tt>false</tt> or <tt>nil</tt>; you must use <tt>if ret == nil </tt> to distinguish between the two).

= List of callbacks =

For suggestions on how to document callbacks, see [[Talk:Callbacks|the discussion page]].

== File discovery callbacks ==

These are called whenever an input file is needed. Each receives the <tt>asked_name</tt>, a string representing what file name was called by the user (for instance, <tt>my_file.tex</tt> in <tt>\input my_file.tex</tt>) and should return the name of a file to be read (or written to). The <tt>id_number</tt> in the first two callbacks is the stream's number plus one, because 0 denotes <tt>log</tt> and <tt>\input</tt> files.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[find_read_file]] ||id_number, asked_name ||actual_name ||File called with <tt>\read</tt> or <tt>\input</tt>
|-
|[[find_write_file]] ||id_number, asked_name ||actual_name ||<tt>log</tt> file or file called with <tt>\write</tt>
|-
|[[find_font_file]] ||asked_name ||actual_name
|-
|[[find_output_file]] ||asked_name ||actual_name
|-
|[[find_format_file]] ||asked_name ||actual_name
|-
|[[find_vf_file]] ||asked_name ||actual_name
|-
|[[find_map_file]] ||asked_name ||actual_name
|-
|[[find_enc_file]] ||asked_name ||actual_name
|-
|[[find_sfd_file]] ||asked_name ||actual_name
|-
|[[find_pk_file]] ||asked_name ||actual_name
|-
|[[find_data_file]] ||asked_name ||actual_name
|-
|[[find_opentype_file]] ||asked_name ||actual_name
|-
|[[find_truetype_file]] ||asked_name ||actual_name
|-
|[[find_type_file]] ||asked_name ||actual_name
|-
|[[find_image_file]] ||asked_name ||actual_name
|}

== File reading callbacks ==

The previous callbacks were called when a file was asked for; the following are triggered when the file is read. In all, <tt>filename</tt> is the string returned by the file discovery callbacks. The first callback reads <tt>\input</tt> or <tt>\read</tt> files; it must return a table, whose first cell is a function that will be called whenever a line from the input file is readed; the second cell is an optional function executed when the input file terminates.

The other callbacks are used for binary files: <tt>success</tt> is a boolean, <tt>data</tt> is a string and <tt>data_size</tt> is a number (the length of <tt>data</tt> in bytes).

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[open_read_file]] ||filename ||{reader, close}
|-
|[[read_font_file]] ||filename ||success, data, data_size
|-
|[[read_vf_file]] ||filename ||success, data, data_size
|-
|[[read_map_file]] ||filename ||success, data, data_size
|-
|[[read_enc_file]] ||filename ||success, data, data_size
|-
|[[read_sfd_file]] ||filename ||success, data, data_size
|-
|[[read_pk_file]] ||filename ||success, data, data_size
|-
|[[read_data_file]] ||filename ||success, data, data_size
|-
|[[read_truetype_file]] ||filename ||success, data, data_size
|-
|[[read_type1_file]] ||filename ||success, data, data_size
|-
|[[read_opentype_file]] ||filename ||success, data, data_size
|}

== Data processing callbacks ==

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[process_input_buffer]] ||line ||modified_line ||Called for each input line (after <tt>reader</tt> in <tt>open_read_file</tt> if TeX is not reading the main file).
|-
|[[process_output_buffer]] ||line ||modified_line ||Called for each line TeX writes to an external file; this excludes the <tt>log</tt> file, the terminal and <tt>\write18</tt> calls.
|-
|[[process_jobname]] ||string ||modified_string ||Processes the jobname when queried with <tt>\jobname</tt> or <tt>tex.jobname</tt>. The internal job name (used for the output and log files) remains untouched. (This callback appeared in v.0.71.)
|-
|[[token_filter]] || ||token ||Called when TeX needs token; no argument is passed to the callback, the next token must be fetched with <tt>token.get_next()</tt>; what is returned is immediately processed by TeX.
|}

== Node list processing callbacks ==

These callbacks deal with node manipulations and occur at various stages of TeX's typesetting processes.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[buildpage_filter]] ||information || ||Called when TeX moves material to the main vertical list. (This callback is an endangered species.)
|-
|[[hyphenate]] ||head, tail || || Inserts discretionary nodes in the node list; called in restricted and unrestricted horizontal mode (i.e. <tt>\hbox</tt>'es and paragraph building). A function doing what TeX does here by default is <tt>lang.hyphenate()</tt>.
|-
|[[ligaturing]] ||head, tail || || Same as <tt>hyphenate</tt>, but applying ligatures (and adding some discretionaries too). An associated function is <tt>node.ligaturing()</tt>.
|-
|[[kerning]] ||head, tail || || Same as above, with font kerns. The associated function is <tt>node.kerning()</tt>.
|-
|[[pre_linebreak_filter]] ||head, context ||boolean or newhead ||Called in unrestricted horizontal mode, after <tt>kerning</tt>, and before calling the paragraph builder; it receives the list of nodes that is the contents of the paragraph (plus the <tt>\parfillskip</tt> glue) and must return something similar.
|-
|[[linebreak_filter]] ||head, is_display ||newhead ||The paragraph builder. It receives what the previous callback has returned and is in charge of making a paragraph out of it; it must return what it has built. An associated function is <tt>tex.linebreak()</tt>.
|-
|[[post_linebreak_filter]] ||head, groupcode, ||boolean or newhead ||Receives what has been returned by <tt>linebreak_filter</tt> and must return something similar.
|-
|[[hpack_filter]] ||head, context, size, packtype[, dir] ||boolean or newhead ||Called when TeX is going to build an <tt>\hbox</tt>.
|-
|[[vpack_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Called when TeX is going to build a <tt>\vbox</tt>.
|-
|[[pre_output_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Same as <tt>vpack_filter</tt>, but when TeX packs box 255 (or any other value of <tt>\outputbox</tt>) and goes to the output routine.
|-
|[[mlist_to_hlist]] ||head, display_type, need_penalties ||newhead ||Called when TeX turns a math list into a horizontal list (as described in Appendix G of the TeXbook).
|}

== Information reporting callbacks ==

These callbacks are called in various places when TeX has something to say. None of them takes an argument nor returns anything.

{| cellpadding="5"
!Name !! Short description
|-
|[[pre_dump]] ||Called before dumping a format.
|-
|[[start_run]] ||LuaTeX's banner (`This is LuaTeX...'); must be changed in the initialization script, otherwise the banner has already been printed.
|-
|[[stop_run]] ||TeX's final words (statistics and `Output written to...').
|-
|[[start_page_number]] ||The opening bracket and the page number printed when a page is shipped out.
|-
|[[stop_page_number]] ||The closing bracket printed after the previous callback.
|-
|[[show_error_hook]] || Called at the beginning of each error message.
|}

== Miscellaneous ==
{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
||[[finish_pdffile]] || || ||Called when all pages have been written to the PDF file.
|-
|[[define_font]] ||name, size, id ||font_table ||Called when the <tt>\font</tt> command is executed; <tt>name</tt> is the filename asked for, <tt>size</tt> is the <tt>at</tt> size, <tt>id</tt> is the number that will be assigned as an internal representation to the font thus loaded. The callback should return a table representing a font.
|}

Main Page

2011-12-29T06:04:55Z

Paul: Reverted edits by Lethuphuong89 (talk) to last revision by 217.235.241.143

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Some articles ==

* [[Documentation and help]] points to other online resources (manuals, mailing list, etc.) related to LuaTeX.
* [[Writing Lua in TeX]] explains how to write Lua code in a TeX document (and back).
* [[Attributes]] introduces LuaTeX's thrilling new concept.
* Pages on callbacks:
** [[Callbacks]] introduces callbacks and how to use them.
** There is a page on the [[Post linebreak filter|<tt>post_linebreak_filter</tt>]] callback, explaining and illustrating it with a couple of examples. The [[Show the hyphenation points]] article is another example of use.
** The page on [[process input buffer|the <tt>process_input_buffer</tt> callback]] illustrates how to read documents with non-UTF-8 encoding, and how to write TeX with lightweight markup instead of the usual commands.
** Another callback is [[show error hook|<tt>show_error_hook</tt>]], which lets you enliven your error messages!
** You can fake XeTeX's interchar tokens with the [[token filter|<tt>token_filter</tt>]].
* [[TeX without TeX]] is about using TeX's functionality (typesetting, pdf writing) only using Lua code (no <tt>\TeX</tt> macros).
* [[fontsampler|Create a fontsampler]] using plain LuaTeX and <tt>luaotfload</tt>.

== From the old bluwiki.com ==

(Mostly by and thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

Use a TrueType font

2011-10-02T08:39:19Z

Paul: Corrected truetype/opentype.

A TrueType (or Openype) font can be used without going through a TFM or an OFM file, like in XeTeX.

The best way to do so is to use the package luaotfload, on the CTAN, and developped here: http://github.com/mpg/luaotfload/

The code shows how we can use the table of a TrueType font to produce an internal TeX font table:

<pre>
\pdfoutput1
\directlua{
callback.register("define_font",
function(name, size)
local fonttype, f
local fonttype = string.match(string.lower(name), "otf$") and "opentype"
or string.match(string.lower(name), "ttf$") and "truetype"

if fonttype then
filename = kpse.find_file(name, "opentype fonts") or kpse.find_file(name, "truetype fonts")
if size < 0 then
size = (- 655.36) * size
end
ttffont = fontloader.to_table(fontloader.open(filename))
if ttffont then
f = { }
f.name = ttffont.fontname
f.fullname = ttffont.names[1].names.fullname
f.parameters = { }
f.designsize = size
f.size = size
f.direction = 0
f.parameters.slant = 0
f.parameters.space = size * 0.25
f.parameters.space_stretch = 0.3 * size
f.parameters.space_shrink = 0.1 * size
f.parameters.x_height = 0.4 * size
f.parameters.quad = 1.0 * size
f.parameters.extra_space = 0
f.characters = { }
local mag = size / ttffont.units_per_em

local names_of_char = { }
for char, glyph in pairs(ttffont.map.map) do
names_of_char[ttffont.glyphs[glyph].name] = ttffont.map.backmap[glyph]
end

for char, glyph in pairs(ttffont.map.map) do
local glyph_table = ttffont.glyphs[glyph]

f.characters[char] = {
index = glyph,
width = glyph_table.width * mag,
name = glyph_table.name }
if glyph_table.boundingbox[4] then
f.characters[char].height = glyph_table.boundingbox[4] * mag
end
if glyph_table.boundingbox[2] then
f.characters[char].depth = -glyph_table.boundingbox[2] * mag
end

if glyph_table.kerns then
local kerns = { }
for _, kern in pairs(glyph_table.kerns) do
kerns[names_of_char[kern.char]] = kern.off * mag
end
f.characters[char].kerns = kerns
end
end

f.filename = filename
f.type = "real"
f.format = fonttype
f.embedding = "subset"
f.cidinfo = {
registry = "Adobe",
ordering = "Identity",
supplement = 0,
version = 1 }
end
else
f = font.read_tfm(name, size)
end
return f
end)
}
</pre>

The index of the <tt>f.characters</tt> table is the Unicode value, among the properties of each entry of that table we have <tt>index</tt> which is the glyph index. Using this property we get the mapping between Unicode characters and glyphs. Kerning is handled via TeX font LKP.

What is missing in the code above is better calculation of standard TeX font parameters and OpenType features.

Writing Lua in TeX

2011-08-30T06:40:52Z

Paul: /* debug note */ Correction and clean-up.

= Embedding Lua code in a TeX document =

Although it is simpler to put Lua code in Lua files, from time to time one may want or need to go Lua in the middle of a document. To this end, LuaTeX has two commands: <tt>\directlua</tt> and <tt>\latelua</tt>. They work the same, except <tt>\latelua</tt> is processed when the page where it appears is shipped out, whereas <tt>\directlua</tt> is processed at once; the distinction is immaterial here, and what is said of <tt>\directlua</tt> also applies to <tt>\latelua</tt>.

<tt>\directlua</tt> can be called in three ways:

<pre>
\directlua {<lua code>}
\directlua name {<name>} {<lua code>}
\directlua <number> {<lua code>}
</pre>

Those three ways are equivalent when it comes to process <tt><lua code></tt>, but in the second case processing will occur in a chunk named <tt><name></tt>, and in the third it will occur in a chunk whose name is the entry <tt><number></tt> in the table <tt>lua.name</tt>. The difference manifests itself only when errors occur, in which case the name of the chunk, if any, is reported.

Each call to <tt>\directlua</tt>, named or not, is processed in a separate chunk. That means that any <tt>local</tt> variable is defined for this call only and is lost afterward. Hence:

<pre>
\directlua{
one = 1
local two = 2
}
\directlua{
texio.write_nl(type(one))
texio.write_nl(type(two))
}
</pre>

will report <tt>number</tt> and <tt>nil</tt> (<tt>texio.write_nl</tt> writes to the log file). On the other hand, Lua code is completely insensitive to TeX's grouping mechanism. In other words, calling <tt>\directlua</tt> between <tt>\bgroup</tt> and <tt>\egroup</tt> doesn't affect the code to be processed.

= TeX catcodes in Lua =

By default, the code passed to <tt>\directlua</tt> is treated as normal TeX input and only then sent to the Lua interpreter. This may lead to unwanted results and must be acknowledged.

== Expansion ==

As with any other special, the code in <tt>\directlua</tt> (and the <tt><name></tt>, if specifed) is fully expanded. This means that macros can be safely passed to <tt>\directlua</tt> if one wants their values, but that they should also be properly escaped when needed. For instance:

<pre>
\def\macro{1}
\directlua{
myvar = \macro
}
</pre>

defines <tt>myvar</tt> as the number <tt>1</tt>. To store the control sequence <tt>\macro</tt> instead, another course of action is needed: see the [[#Backslash|section on backslash]] below.

It should be noted that <tt>\par</tt> tokens are removed from <tt>\directlua</tt> if they are unexpandable (i.e., most likely, if they have their original meaning). Hence, empty lines can be used in Lua code.

== Line ends ==

When TeX reads a file, it normally turns line ends into spaces. That means that what looks like several lines is actually fed to the Lua interpreter as one big line. For instance:

<pre>
\directlua{
myvar = 1
anothervar = 2
onelastvar = 3
}
</pre>

amounts to the following, if it were written in a separate Lua file:

<pre>
myvar = 1 anothervar = 2 onelastvar = 3
</pre>

That is perfectly legitimate, but strange things might happen. First, TeX macros gobble spaces as usual. Hence:

<pre>
\def\macro{1}
\directlua{
myvar = \macro
anothervar = 2
}
</pre>

will be fed to the interpreter as

<pre>
myvar = 1anothervar = 2
</pre>

which is not legitimate at all. Second, the Lua comment <tt>--</tt> will affect everything to the end of the <tt>\directlua</tt> call. That is:

<pre>
\directlua{
myvar = 1
-- anothervar = 2
onelastvar = 3
}
</pre>

will be processed as

<pre>
myvar = 1 -- anothervar = 2 onelastvar = 3
</pre>

which works but only defines <tt>myvar</tt>. Third, when reporting error, the Lua interpreter will always mention the line number as 1, since it processes one big line only; that isn't extremely useful when the code is large.

The solution is to set <tt>\endlinechar=10</tt> or <tt>\catcode`\^^M=12</tt>. In both cases, line ends will be preserved and the code will be processed as it is input.

== Special characters ==

In TeX, some characters have a special behavior. That must be taken into account when writing Lua code: one must change their catcodes beforehand if one wants to handle them as Lua would, as has just been done for line ends. That means that <tt>\directlua</tt>, as such, is clearly insufficient to write any extended chunk of code. It is thus better to devise a special macro that sets the catcodes to the appropriate values, reads the Lua code, feeds it to <tt>\directlua</tt>, and restores the catcodes. The following code does the job:

<pre>
\def\luacode{%
\bgroup
\catcode`\{=12
\catcode`\}=12
\catcode`\^^M=12
\catcode`\#=12
\catcode`\~=12
\catcode`\%=12
\doluacode
}

\bgroup
\catcode`\^^M=12 %
\long\gdef\doluacode#1^^M#2\endluacode{\directlua{#2}\egroup}%
\egroup
</pre>

Note that not all special characters are set to normal (catcode 12) characters; that is explained for each below. Note also that <tt>\doluacode</tt>, internally called by <tt>\luacode</tt>, is defined to get rid of anything up to the line end, and then pass anything up to <tt>\endluacode</tt> to <tt>\directlua</tt>. Discarding what follows <tt>\luacode</tt> is important, otherwise a simple code as

<pre>
\luacode
myvar = 1
\endluacode
</pre>

would actually create two lines, the first being empty; it is annoying because errors are then reported with the wrong line number (i.e. any error in this one-line code would be reported to happen on line 2).

However, the rest of the line after <tt>\luacode</tt> could also be processed, instead of discarded, to manage special effects (e.g. specifying a chunk's name, storing the code in a control sequence, or even setting which catcodes should be changed or not).

=== Backslash ===

The backslash in TeX is used to form control sequences. In the definition of <tt>\luacode</tt> above, it isn't changed and thus behaves as usual. It allows commands to be passed and expanded to the Lua code. Anyway a backslash in Lua is also an escape character in strings. Hence, if one wants to store the name of a macro in Lua code, the following won't work:

<pre>
\luacode
myvar = "\noexpand\macro"
\endluacode
</pre>

because to the Lua interpreter the string is made of <tt>\m</tt> followed by <tt>acro</tt>; since <tt>\m</tt> is not defined in Lua, the string is read as <tt>macro</tt>, but in other circumstances strange things might happen: for instance, <tt>\n</tt> is a newline. The proper way to pass a macro verbatim is:

<pre>
\luacode
myvar = "\noexpand\\macro"
\endluacode
</pre>

which Lua will correctly read as

<pre>
myvar = "\\macro"
</pre>

with the backslash escaped to represent itself. Another solution is:

<pre>
myvar = [[\noexpand\macro]]
</pre>

because the double brackets signals a string in Lua where no escape sequence occurs (and the string can also run on several lines). Note however that in the second case <tt>myvar</tt> will be defined with a trailing space, i.e. as <tt>"\macro "</tt>, because of TeX's habit to append a trailing space to unexpanded (or unexpandable) control sequences.

If one wants the backslash to be treated as any other character (not creating control sequences), then one has to rewrite <tt>\luacode</tt> as follows:

<pre>
\def\luacode{%
\bgroup
\catcode`\\=12
\catcode`\{=12
\catcode`\}=12
\catcode`\^^M=12
\catcode`\#=12
\catcode`\~=12
\catcode`\%=12
\doluacode
}

\bgroup
\catcode`\|=0
\catcode`\^^M=12 %
\catcode`\\=12 %
|long|gdef|doluacode#1^^M#2\endluacode{|directlua{#2}|egroup}%
|egroup
</pre>

Backslashes remain escape characters in Lua, though.

=== Braces ===

One may want to define a string in Lua which contains unbalanced braces, i.e.:

<pre>
\luacode
myvar = "{"
\endluacode
</pre>

If the braces' catcodes hadn't been changed beforehand, that would be impossible. Note, however, that this means that one can't feed arguments to commands in the usual way. I.e. the following will produce nothing good:

<pre>
\luacode
myvar = "\dosomething{\macro}"
\endluacode
</pre>

<tt>\dosomething</tt> will be expanded with the left brace (devoid of its usual delimiter-ness) as its argument, and the rest of the line might produce chaos. Thus, one may also choose not to change the catcodes of braces, depending on how <tt>\luacode</tt> is most likely to be used. Note that strings with unbalanced braces can still be defined, even if braces have their usual catcodes, thanks to the following trick:

<pre>
\luacode
myvar = "{" -- }
\endluacode
</pre>

When the code is passed to <tt>\directlua</tt>, braces are balanced because the Lua comment means nothing to TeX; when passed to the Lua interpreter, on the other hand, the right brace is ignored.

=== Hash and comment ===

The hash sign <tt>#</tt> in Lua is the length operator: prefixed to a string or table variable, it returns its length. If its catcode weren't taken care of, LuaTeX would pass to <tt>\directlua</tt> a double hash for each hash, i.e. each <tt>#</tt> would be turned into <tt>##</tt>. That is normal TeX behavior, but unwanted here.

As for the commen sign <tt>%</tt>, it is useful in Lua when manipulating strings. If it weren't escaped it would discard parts of the code when TeX reads it, and a mutilated version of the input would be passed to the Lua interpreter. In turn, discarding a line by commenting it in <tt>\luacode</tt> should be done with the Lua comment <tt>--</tt>.

=== Active characters ===

The <tt>~</tt> character is generally active and used as a no-break space in TeX. It it were passed as is to <tt>\directlua</tt>, it would expand to uninterpretable control sequences, whereas in Lua it is used to form the unequal operator <tt>~=</tt>.

Other possible active characters should be taken care of, but which characters are active is unpredictable; punctuation marks might be so to accommodate special spacing, as with LaTeX's ''babel'' package, but such tricks are unlikely to survive in LuaTeX (cleaner methods exist that add a space before punctuation marks when necessary).

=== Other characters ===

When processing verbatim text in TeX, one generally also changes the catcodes of <tt>$</tt>, <tt>&</tt>, <tt>^</tt>, <tt>_</tt> and the space character, because they too are special. When passed to the Lua interpreter, though, their usual catcodes won't do any harm, that is why they are left unmodified here.

== \luaescapestring ==

Although it can't do all of what's been explained, the <tt>\luaescapestring</tt> command might be useful in some cases: it expands its argument (which must be enclosed in ''real'' braces) fully, then modify it so that dangerous characters are escaped: backslashes, hashes, quotes and line ends. For instance:

<pre>
\def\macro{"\noexpand\foo"}
\luacode
myvar = "\luaescapestring{\macro}"
\endluacode
</pre>

will be passed to Lua as

<pre>
myvar = "\"\\foo \""
</pre>

so that <tt>myvar</tt> is defined as <tt>"\foo "</tt>, with the quotes as parts of it. Note that the trailing space after <tt>\foo</tt> still happens.

= From Lua to TeX =

Inside Lua code, one can pass strings to be processed by TeX with the functions <tt>tex.print()</tt>, <tt>tex.sprint()</tt> and <tt>tex.tprint()</tt>. All such calls are processed at the end of a <tt>\directlua</tt> call, even though they might happen in the middle of the code. This behavior is worth noting because it might be surprising in some cases, although it is generally harmless.

== tex.print() ==

This function receives as its argument(s) either one or more strings or an array of strings. Each string is processed as an input line: an end-of-line character is appended (except to the last string), and TeX is in state <tt>newline</tt> when processing it (i.e. leading spaces are skipped). Hence the two equivalent calls:

<pre>
tex.print("a", "b")
tex.print({"a", "b"})
</pre>

are both interpreted by TeX as would the following two lines:

<pre>
a
b
</pre>

Thus `a b' is produced, since line ends normally produce a space.

The function can also take an optional number as its first argument; it is interpreted as referring to a catcode table (as defined by <tt>\initcatcodetable</tt> and <tt>\savecatcodetable</tt>), and each line is processed by TeX with that catcode regime. For instance (note that with such a minimal catcode table, braces don't even have their usual values):

<pre>
\bgroup
\initcatcodetable1
\catcode`\_=0
\savecatcodetable1
\egroup

\directlua{tex.print(1, "_TeX")}
</pre>

The string will be read with <tt>_</tt> as an escape character, and thus interpreted as the command commonly known as <tt>\TeX</tt>. The catcode regime holds only for the strings passed to <tt>tex.print()</tt> and the rest of the document isn't affected.

If the optional number is <tt>-1</tt>, or points to an invalid (i.e. undefined) catcode table, then the strings are processed with the current catcodes, as if there was no optional argument. If it is <tt>-2</tt>, then the strings are read as if the result of <tt>\detokenize</tt>: all characters have catcode 12 (i.e. `other', characters that have no function beside representing themselves), except space, which has catcode 10 (as usual).

== tex.sprint() ==

Like <tt>tex.print()</tt>, this function can receive either one or more strings or an array of strings, with an optional number as its first argument pointing to a catcode table. Unlike <tt>tex.print()</tt>, however, each string is processed as if TeX were in the middle of a line and not at the beginning of a new one: spaces aren't skipped, no end-of-line character is added and trailing spaces aren't ignored. Thus:

<pre>
tex.sprint("a", "b")
</pre>

is interpreted by TeX as

<pre>
ab
</pre>

without any space inbetween.

== tex.tprint() ==

This function takes an unlimited number of tables as its arguments; each table must be an array of strings, with the first entry optionally being a number pointing to a catcode table. Then each table is processed as if passed to <tt>tex.sprint()</tt>. Thus:

<pre>
tex.tprint({1, "a", "b"}, {"c", "d"})
</pre>

is equivalent to

<pre>
tex.sprint(1, "a", "b")
tex.sprint("c", "d")
</pre>

=== debug note ===

Note that there are some tables that Lua inside LuaTeX has; one reference is in [http://www.luatex.org/svn/trunk/manual/functionref.pdf functionref]: ''`<tt>luatex</tt> is a typesetter; <tt>texlua</tt> and <tt>luatex --luaonly</tt> are lua interpreters. In lua interpreter mode, the lua tables <tt>tex</tt>, <tt>token</tt>, <tt>node</tt>, and <tt>pdf</tt> are unavailable.'''

For utilizing lua-specific debug techniques, see
* [http://www.lua.org/pil/23.html 23. The Debug Library - Lua],
* [http://www.lua.org/pil/23.1.html 23.1 - Introspective Facilities],
* [http://lua-users.org/wiki/DebugLibraryTutorial lua -users wiki: Debug Library Tutorial]

You can use commands like these in your .tex file:
<pre>
\directlua{
for k,v in pairs(tex) do print(k, v) end
print"==="
for k,v in pairs(lua) do print(k,v) end
print"==="
print(debug.traceback(1))
}
</pre>
... and the output of lua's <tt>print</tt> will be sent to stdout; the three <tt>=</tt> signs will serve simply as `delimiters' in the output, which will look something like:

<pre style="height:250px;overflow:scroll;">
tprint function: 0x89086e8
box table: 0x895a0a0
getsfcode function: 0x895ae10
pdffontsize function: 0x895b048
shipout function: 0x895b2d8
getmath function: 0x895b380
setuccode function: 0x895ae48
getskip function: 0x8908860
pdfxformname function: 0x895b170
setlist function: 0x895ab38
setattribute function: 0x8908898
count table: 0x895b600
getuccode function: 0x895ae80
getbox function: 0x895ab00
uniformdeviate function: 0x895b080
fontname function: 0x895af58
primitives function: 0x895b220
badness function: 0x895b310
setmath function: 0x895b348
setskip function: 0x8908828
getmathcode function: 0x895ada0
getcount function: 0x8908950
toks table: 0x895b6b0
setsfcode function: 0x895add8
run function: 0x8908638
sfcode table: 0x895a130
pdfpageref function: 0x895b138
setcount function: 0x8908918
setdimen function: 0x89087b8
print function: 0x89086d0
setlccode function: 0x895acf8
fontidentifier function: 0x895af90
getlccode function: 0x895ad30
setmathcode function: 0x895ad68
round function: 0x895aeb8
getdimen function: 0x89087f0
write function: 0x89086b8
set function: 0x8908770
definefont function: 0x895b1b0
dimen table: 0x895b550
get function: 0x89087a0
number function: 0x895b0c0
nest table: 0x895a600
lists table: 0x895a550
setcatcode function: 0x895ac18
delcode table: 0x895a4a0
setnest function: 0x895aba8
mathcode table: 0x895a3f0
getdelcode function: 0x895acc0
catcode table: 0x895a340
pdffontname function: 0x895afd0
uccode table: 0x895a290
setdelcode function: 0x895ac88
lccode table: 0x895a1e0
sp function: 0x895af28
attribute table: 0x895b410
getcatcode function: 0x895ac50
extraprimitives function: 0x895b258
linebreak function: 0x895b3b8
setbox function: 0x895aac8
skip table: 0x895b4c0
pdffontobjnum function: 0x895b008
settoks function: 0x8908988
gettoks function: 0x895aa90
enableprimitives function: 0x895b298
getlist function: 0x895ab70
getnest function: 0x895abe0
getattribute function: 0x89088d8
romannumeral function: 0x895b0f8
scale function: 0x895aef0
hashtokens function: 0x895b1e8
error function: 0x8908720
finish function: 0x8908668

====
setluaname function: 0x895efb8
setbytecode function: 0x895f028
name table: 0x895ef50
getluaname function: 0x895ef80
getbytecode function: 0x895eff0
version Lua 5.1
bytecode table: 0x895f080
====
1
stack traceback:
<\directlua >:1: in main chunk
</pre>

Note that the string returned by <tt>debug.traceback</tt> contains <tt>\directlua</tt> and thus should be handled with care if passed to TeX, which might try to execute it.

= The expansion of \directlua =

A call to <tt>\directlua</tt> is fully expandable; i.e. it can be used in contexts where full expansion is required, as in:

<pre>
\csname\directlua{tex.print("TeX")}\endcsname
</pre>

which is a somewhat convoluted way of saying <tt>\TeX</tt>. Besides, since Lua code is processed at once, things that were previously unthinkable can now be done easily. For instance, it is impossible to perform an assignment in an <tt>\edef</tt> by TeX's traditional means. I.e. the following:

<pre>
\edef\macro{\count1=5}
</pre>

defines <tt>\macro</tt> as <tt>\count1=5</tt> but doesn't perform the assignment (the <tt>\edef</tt> does nothing more than a simple <tt>\def</tt>). After the definition, the value of <tt>\count1</tt> hasn't changed. The same is not true, though, if such an assigment is made with Lua code. The following:

<pre>
\edef\macro{\directlua{tex.count[1] = 5}}
</pre>

defines <tt>\macro</tt> emptily (since nothing remains after <tt>\directlua</tt> has been processed) ''and'' sets count 1 to 5. Since such a behavior is totally unexpected in normal TeX, one should be wary when using <tt>\directlua</tt> in such contexts.

Main Page

2011-07-01T05:05:29Z

Paul: Added link to token_filter.

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Some articles ==

* [[Documentation and help]] points to other online resources (manuals, mailing list, etc.) related to LuaTeX.
* [[Writing Lua in TeX]] explains how to write Lua code in a TeX document (and back).
* [[Attributes]] introduces LuaTeX's thrilling new concept.
* Pages on callbacks:
** [[Callbacks]] introduces callbacks and how to use them.
** There is a page on the [[Post linebreak filter|<tt>post_linebreak_filter</tt>]] callback, explaining and illustrating it with a couple of examples. The [[Show the hyphenation points]] article is another example of use.
** The page on [[process input buffer|the <tt>process_input_buffer</tt> callback]] illustrates how to read documents with non-UTF-8 encoding, and how to write TeX with lightweight markup instead of the usual commands.
** Another callback is [[show error hook|<tt>show_error_hook</tt>]], which lets you enliven your error messages!
** You can fake XeTeX's interchar tokens with the [[token filter|<tt>token_filter</tt>]].

== From the old bluwiki.com ==

(Mostly by and thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

Token filter

2011-07-01T04:59:10Z

Paul: Corrections.

= Syntax =

This [[Callbacks|callback]] is called whenever TeX needs a new token. No arguments are passed to the callback, and the return value should be either a Lua table representing a to-be-processed token, a table consisting of a list of such tokens, or something else like <tt>nil</tt> or an empty table. Since no argument is passed, if you want to get the next token in the document, you should use the <tt>token.get_next()</tt> function. Your lua function should therefore look like this:

<pre>
function()
return <table> token
end
</pre>

If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by LuaTeX immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level. If what is passed to TeX is expandable, then the result of expansion is inserted into input and <tt>token.get_next()</tt> will grab it.

= Examples =

== Faking \XeTeXinterchartoks ==

Lua code:

<pre>
% luatexinterchartoks.lua

charclasses = charclasses or {}

function setcharclass (a,b)
charclasses[a] = b
end

local i = 0
while i < 65536 do
charclasses[i] = 0
i = i + 1
end

interchartoks = interchartoks or {}

function setinterchartoks (a,b,c)
interchartoks[a] = interchartoks[a] or {}
interchartoks[a][b] = c
end

local nc, oc
oc = 255

function do_intertoks ()
local tok = token.get_next()
if tex.count['XeTeXinterchartokenstate'] == 1 then
if tok[1] == 11 or tok[1] == 12 then
nc = charclasses[tok[2]]
newchar = tok[2]
else
nc = 255
newchar = ''
end
local insert = ''
if interchartoks[oc] and interchartoks[oc][nc] then
insert = interchartoks[oc][nc]
local newtok = tok
if insert<100 then
local dec = math.floor(insert / 10) + 48;
local unit = math.floor(insert % 10) + 48;
newtok = {
-- \XeTeXinterchartokenstate=0 \the\toks<n> \XeTeXinterchartokenstate=1
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('0'),12),
token.create(string.byte(' '),10),
token.create('the'),
token.create('toks'),
token.create(dec,12),
token.create(unit,12),
token.create(string.byte(' '),10),
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('1'),12),
token.create(string.byte(' '),10),
{tok[1], tok[2], tok[3]}}
end
tok = newtok
end
oc = nc
end
return tok
end

callback.register ('token_filter', do_intertoks)
</pre>

LaTeX package file:

<pre>
% luatexinterchartoks.sty
\newcount\XeTeXinterchartokenstate

\newcount\charclasses
\def\newXeTeXintercharclass#1%
{\global\advance\charclasses1\relax
\newcount#1
\global#1=\the\charclasses
}

\newcount\cchone
\newcount\cchtwo

\def\dodoXeTeXcharclass
{\directlua{setcharclass(\the\cchone,\the\cchtwo)}}

\def\doXeTeXcharclass%
{\afterassignment\dodoXeTeXcharclass\cchtwo }

\def\XeTeXcharclass%
{\afterassignment\doXeTeXcharclass\cchone }

\protected\def\XeTeXdointerchartoks%
{\directlua{setinterchartoks(\the\cchone,\the\cchtwo,\the\allocationnumber)}}

\protected\def\dodoXeTeXinterchartoks%
{\newtoks\mytoks\afterassignment\XeTeXdointerchartoks\global\mytoks }

\protected\def\doXeTeXinterchartoks%
{\afterassignment\dodoXeTeXinterchartoks\cchtwo }

\def\XeTeXinterchartoks%
{\afterassignment\doXeTeXinterchartoks\cchone }

\luatexdirectlua{require('luatexinterchartoks.lua')}

\endinput
</pre>

Test document:

<pre>
\documentclass{article}

\usepackage{luatexinterchartoks}
\usepackage{color}

\begin{document}

\newXeTeXintercharclass \mycharclassa
\newXeTeXintercharclass \mycharclassA
\newXeTeXintercharclass \mycharclassB
\XeTeXcharclass `\a \mycharclassa
\XeTeXcharclass `\A \mycharclassA
\XeTeXcharclass `\B \mycharclassB
% between "a" and "A":
\XeTeXinterchartoks \mycharclassa \mycharclassA = {[\itshape}
\XeTeXinterchartoks \mycharclassA \mycharclassa = {\upshape]}
% between " " and "B":
\XeTeXinterchartoks 255 \mycharclassB = {\bgroup\color{blue}}
\XeTeXinterchartoks \mycharclassB 255 = {\egroup}
% between "B" and "B":
\XeTeXinterchartoks \mycharclassB \mycharclassB = {.}

\begingroup
\XeTeXinterchartokenstate = 1
aAa A a B aBa BB
\endgroup

\end{document}

</pre>

Callbacks

2011-05-29T09:49:30Z

Paul: Added process_jobname.

Callbacks allow the user to interact with TeX's internal operations; those can be enhanced by additional ones, but also replaced altogether. All callbacks are written in Lua.

Since functions registered in callbacks occur in a sequence of actions that makes up TeX's processing, many of them will receive arguments (and thus should be defined to handle them) and, most importantly, many are supposed to return values. What arguments are passed, if any, and what should be return, vary from callback to callback.

= Registering a callback =

To install a function in a callback one must use the <tt>callback.register()</tt> function, whose syntax is:

<pre>
callback.register(callback_name, function)
</pre>

where <tt>callback_name</tt> is a string representing a predefined callback name (see below), and <tt>function</tt> is either a function variable or an anonymous function. The former is just the name of a function previously defined, for instance:

<pre>
local my_function = function (an_arg)
do_this_or_that(an_arg)
end
callback.register("some_callback", my_function)
</pre>

An anonymous function is a function definition that is not assigned to a variable, e.g.:

<pre>
callback.register("some_callback", function (an_arg) do_this_or_that(an_arg) end)
</pre>

Even in the case of a function variable, what LuaTeX registers is the function itself, and any further assignment to the variable has no effect on the callback.

If registering is successful, <tt>callback.register()</tt> returns a number, which is the internal representation of the callback; if registering fails (for instance because a callback doesn't exist, or what is registered isn't a function), the function returns <tt>nil</tt> and a string representing an error message.

Besides functions, <tt>callback.register()</tt> can take <tt>nil</tt> or <tt>false</tt>. The first value clears the callback, and returns to the default behavior (i.e. either nothing or TeX's default processing); the second value prevents anything from happening at this point, even TeX's default operations, and should be used with great care (the benefits are minor speed gain).

== Several functions in a callback ==

Each call to <tt>callback.register()</tt> erases any function previously registered. I.e. after

<pre>
callback.register("some_callback", function_one)
callback.register("some_callback", function_two)
</pre>

only <tt>function_two</tt> is registered in <tt>some_callback</tt>. Hence, if one wants several successive functions to occur in a callback, <tt>callback.register()</tt> is clearly insufficient. Instead, one must register a metafunction which calls the functions one after the other. This is taken care of in ConTeXt, and there exists <tt>luatexbase-mcb</tt> for plain TeX and LaTeX, but here's how to do it oneself. We'll take the <tt>process_input_buffer</tt> callback has an example; it is called whenever TeX reads an input line, it receives a string (the line) and should return one. A rough approach would be:

<pre>
local function_table = { }

local function metafunction (str)
for _, fct in ipairs(function_table) do
str = fct(str)
end
return str
end

function add_function (fct)
table.insert(function_table, fct)
end

callback.register("process_input_buffer", metafunction)
</pre>

Then one can call <tt>add_function(my_function)</tt> to add <tt>my_function</tt> to the list of functions registered in <tt>process_input_buffer</tt>, and <tt>metafunction</tt> will pass the callback's argument (an input line) to the first function, set that argument to what the function returns, then repeat the process for all functions and finally return the resulting string.

If one also wants to be able to remove functions (and not the entire metafunction), a subtler approach is required, where names are associated with functions and can be used as handles.

= Information about callbacks =

LuaTeX has two functions to get some information about callbacks. The first is <tt>callback.list()</tt>, called without arguments and returning a table with the callback names as keys and booleans as values: if the value is <tt>true</tt>, it means that something is registered in the callback, or that it is set to <tt>false</tt> (see above). The other function is <tt>callback.find()</tt>, called with a callback name. If a function is registered, it is returned; if the callback is set to <tt>false</tt>, a boolean with that value is returned; finally, if nothing is registered, <tt>nil</tt> is returned (note that to distinguish between a boolean set to <tt>false</tt> and <tt>nil</tt>, you must use <tt>type()</tt>).

= List of callbacks =

For suggestions on how to document callbacks, see [[Talk:Callbacks|the discussion page]].

== File discovery callbacks ==

These are called whenever an input file is needed. Each receives the <tt>asked_name</tt>, a string representing what file name was called by the user (for instance, <tt>my_file.tex</tt> in <tt>\input my_file.tex</tt>) and should return the name of a file to be read (or written to). The <tt>id_number</tt> in the first two callbacks is the stream's number plus one, because 0 denotes <tt>log</tt> and <tt>\input</tt> files.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[find_read_file]] ||id_number, asked_name ||actual_name ||File called with <tt>\read</tt> or <tt>\input</tt>
|-
|[[find_write_file]] ||id_number, asked_name ||actual_name ||<tt>log</tt> file or file called with <tt>\write</tt>
|-
|[[find_font_file]] ||asked_name ||actual_name
|-
|[[find_output_file]] ||asked_name ||actual_name
|-
|[[find_format_file]] ||asked_name ||actual_name
|-
|[[find_vf_file]] ||asked_name ||actual_name
|-
|[[find_map_file]] ||asked_name ||actual_name
|-
|[[find_enc_file]] ||asked_name ||actual_name
|-
|[[find_sfd_file]] ||asked_name ||actual_name
|-
|[[find_pk_file]] ||asked_name ||actual_name
|-
|[[find_data_file]] ||asked_name ||actual_name
|-
|[[find_opentype_file]] ||asked_name ||actual_name
|-
|[[find_truetype_file]] ||asked_name ||actual_name
|-
|[[find_type_file]] ||asked_name ||actual_name
|-
|[[find_image_file]] ||asked_name ||actual_name
|}

== File reading callbacks ==

The previous callbacks were called when a file was asked for; the following are triggered when the file is read. In all, <tt>filename</tt> is the string returned by the file discovery callbacks. The first callback reads <tt>\input</tt> or <tt>\read</tt> files; it must return a table, whose first cell is a function that will be called whenever a line from the input file is readed; the second cell is an optional function executed when the input file terminates.

The other callbacks are used for binary files: <tt>success</tt> is a boolean, <tt>data</tt> is a string and <tt>data_size</tt> is a number (the length of <tt>data</tt> in bytes).

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[open_read_file]] ||filename ||{reader, close}
|-
|[[read_font_file]] ||filename ||success, data, data_size
|-
|[[read_vf_file]] ||filename ||success, data, data_size
|-
|[[read_map_file]] ||filename ||success, data, data_size
|-
|[[read_enc_file]] ||filename ||success, data, data_size
|-
|[[read_sfd_file]] ||filename ||success, data, data_size
|-
|[[read_pk_file]] ||filename ||success, data, data_size
|-
|[[read_data_file]] ||filename ||success, data, data_size
|-
|[[read_truetype_file]] ||filename ||success, data, data_size
|-
|[[read_type1_file]] ||filename ||success, data, data_size
|-
|[[read_opentype_file]] ||filename ||success, data, data_size
|}

== Data processing callbacks ==

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[process_input_buffer]] ||line ||modified_line ||Called for each input line (after <tt>reader</tt> in <tt>open_read_file</tt> if TeX is not reading the main file).
|-
|[[process_output_buffer]] ||line ||modified_line ||Called for each line TeX writes to an external file; this excludes the <tt>log</tt> file, the terminal and <tt>\write18</tt> calls.
|-
|[[process_jobname]] ||string ||modified_string ||Processes the jobname when queried with <tt>\jobname</tt> or <tt>tex.jobname</tt>. The internal job name (used for the output and log files) remains untouched. (This callback appeared in v.0.71.)
|-
|[[token_filter]] || ||token ||Called when TeX needs token; no argument is passed to the callback, the next token must be fetched with <tt>token.get_next()</tt>; what is returned is immediately processed by TeX.
|}

== Node list processing callbacks ==

These callbacks deal with node manipulations and occur at various stages of TeX's typesetting processes.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[buildpage_filter]] ||information || ||Called when TeX moves material to the main vertical list. (This callback is an endangered species.)
|-
|[[hyphenate]] ||head, tail || || Inserts discretionary nodes in the node list; called in restricted and unrestricted horizontal mode (i.e. <tt>\hbox</tt>'es and paragraph building). A function doing what TeX does here by default is <tt>lang.hyphenate()</tt>.
|-
|[[ligaturing]] ||head, tail || || Same as <tt>hyphenate</tt>, but applying ligatures (and adding some discretionaries too). An associated function is <tt>node.ligaturing()</tt>.
|-
|[[kerning]] ||head, tail || || Same as above, with font kerns. The associated function is <tt>node.kerning()</tt>.
|-
|[[pre_linebreak_filter]] ||head, context ||boolean or newhead ||Called in unrestricted horizontal mode, after <tt>kerning</tt>, and before calling the paragraph builder; it receives the list of nodes that is the contents of the paragraph (plus the <tt>\parfillskip</tt> glue) and must return something similar.
|-
|[[linebreak_filter]] ||head, is_display ||newhead ||The paragraph builder. It receives what the previous callback has returned and is in charge of making a paragraph out of it; it must return what it has built. An associated function is <tt>tex.linebreak()</tt>.
|-
|[[post_linebreak_filter]] ||head, groupcode, ||boolean or newhead ||Receives what has been returned by <tt>linebreak_filter</tt> and must return something similar.
|-
|[[hpack_filter]] ||head, context, size, packtype[, dir] ||boolean or newhead ||Called when TeX is going to build an <tt>\hbox</tt>.
|-
|[[vpack_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Called when TeX is going to build a <tt>\vbox</tt>.
|-
|[[pre_output_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Same as <tt>vpack_filter</tt>, but when TeX packs box 255 (or any other value of <tt>\outputbox</tt>) and goes to the output routine.
|-
|[[mlist_to_hlist]] ||head, display_type, need_penalties ||newhead ||Called when TeX turns a math list into a horizontal list (as described in Appendix G of the TeXbook).
|}

== Information reporting callbacks ==

These callbacks are called in various places when TeX has something to say. None of them takes an argument nor returns anything.

{| cellpadding="5"
!Name !! Short description
|-
|[[pre_dump]] ||Called before dumping a format.
|-
|[[start_run]] ||LuaTeX's banner (`This is LuaTeX...'); must be changed in the initialization script, otherwise the banner has already been printed.
|-
|[[stop_run]] ||TeX's final words (statistics and `Output written to...').
|-
|[[start_page_number]] ||The opening bracket and the page number printed when a page is shipped out.
|-
|[[stop_page_number]] ||The closing bracket printed after the previous callback.
|-
|[[show_error_hook]] || Called at the beginning of each error message.
|}

== Miscellaneous ==
{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
||[[finish_pdffile]] || || ||Called when all pages have been written to the PDF file.
|-
|[[define_font]] ||name, size, id ||font_table ||Called when the <tt>\font</tt> command is executed; <tt>name</tt> is the filename asked for, <tt>size</tt> is the <tt>at</tt> size, <tt>id</tt> is the number that will be assigned as an internal representation to the font thus loaded. The callback should return a table representing a font.
|}

Process input buffer

2011-05-11T10:47:51Z

Paul: Reorganised the sections.

= Syntax =

This [[Callbacks|callback]] is called whenever TeX needs a new input line from the file it is currently reading. The argument passed to the function registered there is a string representing the original input line; the function should return another string, to be processed by TeX, or <tt>nil</tt>; in the latter case, the original string is processed. So the function should be defined according to the following blueprint:

<pre>
function (<string> original_line)
return <string> adjusted_line or nil
end
</pre>

In case TeX isn't reading the main document but a file that is <tt>\input</tt> in it, the callback is called after the <tt>reader</tt> function possibly defined in the <tt>[[open_read_file]]</tt> callback.

The following examples aren't mutually exclusive; however, if one wanted to register them all in the same callback, special care would be needed, as explained [[Callbacks#Several_functions_in_a_callback|in the main page on callbacks]]. Here each call to <tt>callback.register()</tt> removes whatever there might have been in the callback beforehand.

= Examples =

== Encoding ==

LuaTeX understands UTF-8 and UTF-8 only (ASCII is a part of it, however). A line containing invalid characters will produce an error message: String contains an invalid utf-8 sequence. Thus documents with different encoding must be converted, and this can be done in <tt>process_input_buffer</tt>.

Note that such a conversion might not produce anything meaningful when the document is compiled, because the fonts used must have the proper glyphs in the right place; for instance, the Computer Modern fonts don't have characters with diacritics, and Latin Modern should be used instead. An alternative solution (quite archaic with LuaTeX) is to define the special characters as active, and to make them create a diacritic. For instance:

<pre>
\catcode`\é = 13
\def é{\'e}
</pre>

=== Latin-1 ===

Reading a document written in Latin-1 (ISO-8859-1) is relatively straightforward, because the <tt>slnunicode</tt> library embedded in LuaTeX does the conversion at once.

<pre>
local utf8_char, byte, gsub = unicode.utf8.char, string.byte, string.gsub

local function convert_char (char)
return utf8_char(byte(char))
end

local function convert_line (line)
return gsub(line, ".", convert_char)
end

callback.register("process_input_buffer", convert_line)
</pre>

The code reads as follows (bottom up): the <tt>convert_line</tt> function is registered in the <tt>process_input_buffer</tt> callback; it is passed the original input line as its sole argument, and returns another string, namely that line processed with the <tt>string.gsub()</tt> function. The latter replaces in a string (the first argument) all the occurrences of a pattern (the second argument) with the outcome of its third argument (for further information on <tt>string.gsub()</tt> see [http://www.lua.org/manual/5.1/manual.html#pdf-string.gsub the Lua reference manual]).

Here the third argument is a function, <tt>convert_char()</tt> to which is passed whatever matches the pattern. This pattern matches any character, the dot being a magic character. In <tt>convert_char()</tt>, the character is first converted to a numerical code (the codepoint in Latin-1, see the [http://www.lua.org/manual/5.1/manual.html#pdf-string.byte description of <tt>string.byte()</tt>]); the code is passed to <tt>unicode.utf8.char</tt> which, given a number, returns the associated character in UTF-8, the number being interpreted as a code point in Unicode; since the characters in Latin-1 have the same code points as in Unicode, the conversion is automatic here.

The use of <tt>local</tt> variables ensures speed, and above all that those variables aren't defined outside the current chunk, for instance, the current <tt>\directlua</tt> call or the current Lua file; actually, the code could even be embedded between <tt>do</tt> and <tt>end</tt> and leave absolutely no trace whatsoever.

=== Other 8-bit encodings ===

When using other 8-bit encoding, the previous code won't work, because it defaults to Latin-1 only. Then one must convert each character one by one by setting up a table matching each input character with the Unicode value; that value can be passed to unicode.utf8.char to yield the desired character.

For instance, here's the code needed to process a document encoded in Latin/Greek (ISO-8859-7):

<pre>
local utf8_char, byte = unicode.utf8.char, string.byte

local LatinGreek_table = {
[0] = 0x0000, 0x0001, 0x0002, 0x0003, 0x0004,
-- ... 240+ other values...
0x03C9, 0x03CA, 0x03CB, 0x03CC, 0x03CD, 0x03CE
}

local function convert_char (char)
return utf8_char(LatinGreek_table[byte(char)])
end

local function convert_line (line)
return gsub(line, ".", convert_char)
end

callback.register("process_input_buffer", convert_line)
</pre>

The <tt>convert_line()</tt> function and call to <tt>callback.register()</tt> are the same as above. What has changed is <tt>convert_char()</tt>; the numerical value of the original character is now used as an index in the <tt>LatinGreek_table</tt>, and the value returned is the corresponding Unicode code point; that number is passed to <tt>unicode.utf8.char</tt> to produce the character in UTF-8.

The values in the <tt>LatinGreek_table</tt> are assigned to the right indices because they are declared in a row (here in hexadecimal form by prefixing them with <tt>0x</tt>). The only index that needs to be specified is 0, because indexing of tables in Lua begins at 1 by default. The table length is 254 plus the value at index 0 (not 255, because there is no such character in Latin/Greek). Each index is a code point in Latin/Greek, and the value is the code point of the same character in Unicode. For instance, lowercase omega in Latin/Greek is 249, at which index one finds 0x03C9, lowercase omega in Unicode.

== TeX as a lightweight markup language ==

The <tt>process_input_buffer</tt> can be put to an entirely different use, namely to preprocess input strings using some kind of lightweight markup and turn them into proper TeX.

Let's suppose one has two control sequences, <tt>\italic</tt> and <tt>\bold</tt>, taking a single argument; let's suppose furthermore that one wants to write a sentence like

<pre>
This is in /italic/ and this is in *bold*.
</pre>

to be processed by TeX as

<pre>
This is in \italic{italic} and this is in \bold{bold}.
</pre>

so as to produce: `This is in italic and this is in bold.' The following code does exactly that (the use of the percent sing and of <tt>\\</tt> without <tt>\noexpand</tt> means that this code is either in a Lua file or in the second version of <tt>\luacode</tt> as defined in [[Writing Lua in TeX#Backslash|Writing Lua in TeX]]; also, this code illustrates the registering of an anonymous function instead of a function variable as in the previous examples):

<pre>
local gsub = string.gsub

callback.register("process_input_buffer",
function (str)
str = gsub(str, "/(.-)/", "\\italic{%1}")
str = gsub(str, "%*(.-)%*", "\\bold{%1}")
return str
end)
</pre>

What happens is that the original string is replaced with successive call to <tt>string.gsub</tt> (see [http://www.lua.org/manual/5.1/manual.html#pdf-string.gsub the Lua reference manual]), in which the captures in the patterns are replaced with themselves as arguments to the TeX function (the non-capture parts of the patterns are discarded). For instance, <tt>/a word/</tt> yields <tt>\italic{a word}</tt>. Note that with <tt>\bold</tt>, the asterisks in the pattern must be escaped with <tt>%</tt>, otherwise they would be interpreted as magic character. The line can then be processed by TeX as usual.

Only pairs of slashes or asterisks in the same line will be interpreted as markup, because lines are processed one by one and nothing is remembered from one line to the next (that can be implemented, but is a bit more complicated and dangerous). Hence, nothing will be in italics in the following example:

<pre>
This will /not
be/ in italic.
</pre>

Instead, slashes will be read.

One can add an unlimited number of replacements of the input line. For instance, here's a way to mark sections by rows of <tt>#</tt>'s:

<pre>
str = gsub(str, "^%s*#%s*(.*)", "\\section{%1}")
str = gsub(str, "^%s*##%s*(.*)", "\\subsection{%1}")
str = gsub(str, "^%s*###%s*(.*)", "\\subsubsection{%1}")
</pre>

What is recognized as a section header is any line beginning with one, two or three hashes, ignoring space before and after. So one can write:

<pre>
# A section
## A subsection
### A subsubsection
</pre>

More complicated things are possible. However, using the [[open read file]] callback (only with an <tt>\input</tt> file) is preferable if subtlety is required, because it can read an entire file (or any set of lines) before changing anything. Also, the powerful [http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html LPeg library] (embedded in LuaTeX) can be used instead of the default operations.

Finally, one must note that such manipulations can be dangerous or lead to unwanted results. For instance, a line such as

<pre>
... the file should be contained in user/directory/myfiles...
</pre>

will be processed without slashes and with directory in italics. Thus, such markup should either be used with files specially tailored for it, or contain ways to be overriden or to protect some text.

Callbacks

2011-05-11T10:28:43Z

Paul: Link to the post_linebreak_filter page.

Callbacks allow the user to interact with TeX's internal operations; those can be enhanced by additional ones, but also replaced altogether. All callbacks are written in Lua.

Since functions registered in callbacks occur in a sequence of actions that makes up TeX's processing, many of them will receive arguments (and thus should be defined to handle them) and, most importantly, many are supposed to return values. What arguments are passed, if any, and what should be return, vary from callback to callback.

= Registering a callback =

To install a function in a callback one must use the <tt>callback.register()</tt> function, whose syntax is:

<pre>
callback.register(callback_name, function)
</pre>

where <tt>callback_name</tt> is a string representing a predefined callback name (see below), and <tt>function</tt> is either a function variable or an anonymous function. The former is just the name of a function previously defined, for instance:

<pre>
local my_function = function (an_arg)
do_this_or_that(an_arg)
end
callback.register("some_callback", my_function)
</pre>

An anonymous function is a function definition that is not assigned to a variable, e.g.:

<pre>
callback.register("some_callback", function (an_arg) do_this_or_that(an_arg) end)
</pre>

Even in the case of a function variable, what LuaTeX registers is the function itself, and any further assignment to the variable has no effect on the callback.

If registering is successful, <tt>callback.register()</tt> returns a number, which is the internal representation of the callback; if registering fails (for instance because a callback doesn't exist, or what is registered isn't a function), the function returns <tt>nil</tt> and a string representing an error message.

Besides functions, <tt>callback.register()</tt> can take <tt>nil</tt> or <tt>false</tt>. The first value clears the callback, and returns to the default behavior (i.e. either nothing or TeX's default processing); the second value prevents anything from happening at this point, even TeX's default operations, and should be used with great care (the benefits are minor speed gain).

== Several functions in a callback ==

Each call to <tt>callback.register()</tt> erases any function previously registered. I.e. after

<pre>
callback.register("some_callback", function_one)
callback.register("some_callback", function_two)
</pre>

only <tt>function_two</tt> is registered in <tt>some_callback</tt>. Hence, if one wants several successive functions to occur in a callback, <tt>callback.register()</tt> is clearly insufficient. Instead, one must register a metafunction which calls the functions one after the other. This is taken care of in ConTeXt, and there exists <tt>luatexbase-mcb</tt> for plain TeX and LaTeX, but here's how to do it oneself. We'll take the <tt>process_input_buffer</tt> callback has an example; it is called whenever TeX reads an input line, it receives a string (the line) and should return one. A rough approach would be:

<pre>
local function_table = { }

local function metafunction (str)
for _, fct in ipairs(function_table) do
str = fct(str)
end
return str
end

function add_function (fct)
table.insert(function_table, fct)
end

callback.register("process_input_buffer", metafunction)
</pre>

Then one can call <tt>add_function(my_function)</tt> to add <tt>my_function</tt> to the list of functions registered in <tt>process_input_buffer</tt>, and <tt>metafunction</tt> will pass the callback's argument (an input line) to the first function, set that argument to what the function returns, then repeat the process for all functions and finally return the resulting string.

If one also wants to be able to remove functions (and not the entire metafunction), a subtler approach is required, where names are associated with functions and can be used as handles.

= Information about callbacks =

LuaTeX has two functions to get some information about callbacks. The first is <tt>callback.list()</tt>, called without arguments and returning a table with the callback names as keys and booleans as values: if the value is <tt>true</tt>, it means that something is registered in the callback, or that it is set to <tt>false</tt> (see above). The other function is <tt>callback.find()</tt>, called with a callback name. If a function is registered, it is returned; if the callback is set to <tt>false</tt>, a boolean with that value is returned; finally, if nothing is registered, <tt>nil</tt> is returned (note that to distinguish between a boolean set to <tt>false</tt> and <tt>nil</tt>, you must use <tt>type()</tt>).

= List of callbacks =

For suggestions on how to document callbacks, see [[Talk:Callbacks|the discussion page]].

== File discovery callbacks ==

These are called whenever an input file is needed. Each receives the <tt>asked_name</tt>, a string representing what file name was called by the user (for instance, <tt>my_file.tex</tt> in <tt>\input my_file.tex</tt>) and should return the name of a file to be read (or written to). The <tt>id_number</tt> in the first two callbacks is the stream's number plus one, because 0 denotes <tt>log</tt> and <tt>\input</tt> files.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[find_read_file]] ||id_number, asked_name ||actual_name ||File called with <tt>\read</tt> or <tt>\input</tt>
|-
|[[find_write_file]] ||id_number, asked_name ||actual_name ||<tt>log</tt> file or file called with <tt>\write</tt>
|-
|[[find_font_file]] ||asked_name ||actual_name
|-
|[[find_output_file]] ||asked_name ||actual_name
|-
|[[find_format_file]] ||asked_name ||actual_name
|-
|[[find_vf_file]] ||asked_name ||actual_name
|-
|[[find_map_file]] ||asked_name ||actual_name
|-
|[[find_enc_file]] ||asked_name ||actual_name
|-
|[[find_sfd_file]] ||asked_name ||actual_name
|-
|[[find_pk_file]] ||asked_name ||actual_name
|-
|[[find_data_file]] ||asked_name ||actual_name
|-
|[[find_opentype_file]] ||asked_name ||actual_name
|-
|[[find_truetype_file]] ||asked_name ||actual_name
|-
|[[find_type_file]] ||asked_name ||actual_name
|-
|[[find_image_file]] ||asked_name ||actual_name
|}

== File reading callbacks ==

The previous callbacks were called when a file was asked for; the following are triggered when the file is read. In all, <tt>filename</tt> is the string returned by the file discovery callbacks. The first callback reads <tt>\input</tt> or <tt>\read</tt> files; it must return a table, whose first cell is a function that will be called whenever a line from the input file is readed; the second cell is an optional function executed when the input file terminates.

The other callbacks are used for binary files: <tt>success</tt> is a boolean, <tt>data</tt> is a string and <tt>data_size</tt> is a number (the length of <tt>data</tt> in bytes).

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[open_read_file]] ||filename ||{reader, close}
|-
|[[read_font_file]] ||filename ||success, data, data_size
|-
|[[read_vf_file]] ||filename ||success, data, data_size
|-
|[[read_map_file]] ||filename ||success, data, data_size
|-
|[[read_enc_file]] ||filename ||success, data, data_size
|-
|[[read_sfd_file]] ||filename ||success, data, data_size
|-
|[[read_pk_file]] ||filename ||success, data, data_size
|-
|[[read_data_file]] ||filename ||success, data, data_size
|-
|[[read_truetype_file]] ||filename ||success, data, data_size
|-
|[[read_type1_file]] ||filename ||success, data, data_size
|-
|[[read_opentype_file]] ||filename ||success, data, data_size
|}

== Data processing callbacks ==

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[process_input_buffer]] ||line ||modified_line ||Called for each input line (after <tt>reader</tt> in <tt>open_read_file</tt> if TeX is not reading the main file).
|-
|[[process_output_buffer]] ||line ||modified_line ||Called for each line TeX writes to an external file; this excludes the <tt>log</tt> file, the terminal and <tt>\write18</tt> calls.
|-
|[[token_filter]] || ||token ||Called when TeX needs token; no argument is passed to the callback, the next token must be fetched with <tt>token.get_next()</tt>; what is returned is immediately processed by TeX.
|}

== Node list processing callbacks ==

These callbacks deal with node manipulations and occur at various stages of TeX's typesetting processes.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[buildpage_filter]] ||information || ||Called when TeX moves material to the main vertical list. (This callback is an endangered species.)
|-
|[[hyphenate]] ||head, tail || || Inserts discretionary nodes in the node list; called in restricted and unrestricted horizontal mode (i.e. <tt>\hbox</tt>'es and paragraph building). A function doing what TeX does here by default is <tt>lang.hyphenate()</tt>.
|-
|[[ligaturing]] ||head, tail || || Same as <tt>hyphenate</tt>, but applying ligatures (and adding some discretionaries too). An associated function is <tt>node.ligaturing()</tt>.
|-
|[[kerning]] ||head, tail || || Same as above, with font kerns. The associated function is <tt>node.kerning()</tt>.
|-
|[[pre_linebreak_filter]] ||head, context ||boolean or newhead ||Called in unrestricted horizontal mode, after <tt>kerning</tt>, and before calling the paragraph builder; it receives the list of nodes that is the contents of the paragraph (plus the <tt>\parfillskip</tt> glue) and must return something similar.
|-
|[[linebreak_filter]] ||head, is_display ||newhead ||The paragraph builder. It receives what the previous callback has returned and is in charge of making a paragraph out of it; it must return what it has built. An associated function is <tt>tex.linebreak()</tt>.
|-
|[[post_linebreak_filter]] ||head, groupcode, ||boolean or newhead ||Receives what has been returned by <tt>linebreak_filter</tt> and must return something similar.
|-
|[[hpack_filter]] ||head, context, size, packtype[, dir] ||boolean or newhead ||Called when TeX is going to build an <tt>\hbox</tt>.
|-
|[[vpack_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Called when TeX is going to build a <tt>\vbox</tt>.
|-
|[[pre_output_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Same as <tt>vpack_filter</tt>, but when TeX packs box 255 (or any other value of <tt>\outputbox</tt>) and goes to the output routine.
|-
|[[mlist_to_hlist]] ||head, display_type, need_penalties ||newhead ||Called when TeX turns a math list into a horizontal list (as described in Appendix G of the TeXbook).
|}

== Information reporting callbacks ==

These callbacks are called in various places when TeX has something to say. None of them takes an argument nor returns anything.

{| cellpadding="5"
!Name !! Short description
|-
|[[pre_dump]] ||Called before dumping a format.
|-
|[[start_run]] ||LuaTeX's banner (`This is LuaTeX...'); must be changed in the initialization script, otherwise the banner has already been printed.
|-
|[[stop_run]] ||TeX's final words (statistics and `Output written to...').
|-
|[[start_page_number]] ||The opening bracket and the page number printed when a page is shipped out.
|-
|[[stop_page_number]] ||The closing bracket printed after the previous callback.
|-
|[[show_error_hook]] || Called at the beginning of each error message.
|}

== Miscellaneous ==
{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
||[[finish_pdffile]] || || ||Called when all pages have been written to the PDF file.
|-
|[[define_font]] ||name, size, id ||font_table ||Called when the <tt>\font</tt> command is executed; <tt>name</tt> is the filename asked for, <tt>size</tt> is the <tt>at</tt> size, <tt>id</tt> is the number that will be assigned as an internal representation to the font thus loaded. The callback should return a table representing a font.
|}

Attributes

2011-05-11T10:27:14Z

Paul: Added link to example in the post_linebreak_filter article.

= Introduction =

Attributes are a new concept introduced in LuaTeX. In a way, they are a mix between count registers (they are numbers) and fonts (they are attached to the [[nodes]] created under their scope). They allow information to be carried between different stages of processing.

= Syntax =

Syntactically, attributes are no different from (and can be used as) count registers. Accordingly, primitive commands to handle attribute registers mimick those for count registers: like <tt>\count</tt>, <tt>\attribute</tt> is used to refer to a register, and like <tt>\countdef</tt>, <tt>\attributedef</tt> defines a control sequence to be a synonym for a register. The syntax is:

<pre>
\attribute <16-bit number> <optional equals> <32-bit number>
\attributedef <csname> <optional equals> <16-bit number>
</pre>

It is wise to have a \newattribute command, like \newcount (from plain TeX, used also in LaTeX and ConTeXt), for the allocation of registers. The command exists in ConTeXt, and is available in the [http://tug.ctan.org/pkg/luatexbase luatexbase] package for plain TeX and LaTeX under the name \newluatexattribute.

Whenever a count register is allowed, an attribute register is too; that is, primitives like \advance, \multiply, \divide, can be used. More generally, anywhere a number is needed, attributes will do. Of course, assignments to attribute registers obey grouping.

Attributes differ from count registers in two respects:

* They have a special value, -"7FFFFFFF in hexadecimal or -2147483647 in decimal; an attribute with that value (which is default) is said to be unset. With any larger value, the attribute is set.
* When [[nodes]] are created, they carry with them the values of all the attributes in force at creation time (except for some [[#Asynchronous attribute settings|asynchronous attribute settings]]). In other words, when inspecting a [[nodes|node]] at whatever stage of its processing, one can retrieve (and also modify) the values of the attributes when the node was created. In that respect, attributes behave like fonts (glyph [[nodes]] always carry information about their fonts).

== Box attributes ==

When creating boxes, there exists a new keyword <tt>attr</tt> besides <tt>to</tt> and <tt>spread</tt>. The node representing the box will then have the associate attribute, but not its contents. For instance, after:

<pre>
\attribute0=1
\setbox0=\hbox attr 1 = 1 {contents}
</pre>

The box (as a node) and each node of its contents have value 1 for attribute 0, but only the box node has value 1 for attribute 1; its contents has whatever value is currently assigned to that attribute (very probably the attribute is unset).

The syntax for the <tt>attr</tt> keyword is the following (all spaces are optional):

<pre>
attr <number> = <number>
</pre>

The <tt>attr</tt> declaration(s) should precede the <tt>to</tt> or <tt>spread</tt> declaration if any.

== Asynchronous attribute settings ==

Sometimes [[nodes]] are assigned attribute values depending not on the real values at creation time but on the values of the surrounding nodes. The following simplistic code illustrates that:

<pre>
{\attribute0=1 Ve}\par
</pre>

LuaTeX will insert a font kern between the two letters (if the font says so, of course). That node will have attribute 0 set to 1, even though that is no longer the real value. Indeed, the kern node is created when building the paragraph, i.e. when the <tt>\par</tt> command is encountered; at that time, attribute 0 no longer has value 1, since that value was assigned in a group and the group is closed. So the kern should normally have whatever value is in force outside the group; instead it is given the value of the nodes that prompted its creation (more precisely, the value of the first node, which can differ from the second one). Similarly, a ligature node takes the values of those nodes it is created from, as does a discretionary node.

The <tt>\leftskip</tt> and <tt>\rightskip</tt> glues, inserted when non-zero at the beginning and end of each line of a paragraph, take the value of the neighboring nodes (i.e. the first and last nodes of a line, respectively).

Those asynchronous settings are supposed to make more sense than assignments that would follow the real values. Counter-measures can easily be implemented if necessary: for instance, one can inspect all lines in a paragraph and reassign to the <tt>\leftskip</tt> and <tt>\rightskip</tt> glues the current values for the attributes.

= On the Lua side =

== Registers ==

Like all registers, attributes are stored in a Lua table in the <tt>tex</tt> library. Hence besides <tt>tex.count</tt>, <tt>tex.box</tt>, etc., there exists <tt>tex.attribute</tt>. The indices to entries are either a register number or the name of a control sequence defined with <tt>\attributedef</tt>. For instance, after

<pre>
\attributedef\myattribute=100
\myattribute=1
</pre>

both <tt>tex.attribute.myattribute</tt> and <tt>tex.attribute[100]</tt> return 1. The entry can also be assigned to, but then the assignment is necessarily local.

There are also two functions to set or retrieve attribute values, with the following syntax:

<pre>
tex.getattribute(<register>)
tex.setattribute(["global",] <register>, <number>)
</pre>

In both case <tt><register></tt> is the same as an entry in <tt>tex.attribute</tt>: either a number or a string representing a control sequence defined with <tt>\attributedef</tt>. The first function returns the current value of the attribute, while the second sets it, either locally or globally if the first optional argument is given.

== Attributes in nodes ==

The <tt>node</tt> library offers the following function to deal with the values of attributes for a given node:

<pre>
node.has_attribute(<node>, <attribute>[, <value>])
</pre>

Without the third argument, if <tt><node></tt> has <tt>attribute</tt> set, i.e. with a value different from -"7FFFFFFF, then the value is returned; otherwise <tt>nil</tt> is returned. If the third argument is given, the value of the attribute is returned if and only if it is equal to <value>, and <tt>nil</tt> otherwise.

<pre>
node.set_attribute(<node>, <attribute>, <value>)
</pre>

Sets <attribute> to <value> for <node>.

<pre>
node.unset_attribute(<node>, <attribute>[, <value>])
</pre>

Unsets <attribute> for <node> and returns the old value; if <value> is present, the attribute is unset only if it matches <value>.

== Attribute lists ==

In [[nodes]], attributes actually are implemented as nodes in a list. That is why the <tt>attr</tt> field of all nodes (except attribute nodes themselves) points to a node of type <tt>attribute_list</tt>. That node has only one field <tt>next</tt> pointing to the first attribute node in the list, if any. There might not be any such node because only set attributes are present in the list. Attribute nodes have three fields:

* <tt>number</tt>: the attribute's number;
* <tt>value</tt>: the attribute's value;
* <tt>next</tt>: the next attribute in the list, i.e. an attribute node that is set.

As an example, the following code prints <tt>0 = 5</tt> and <tt>10 = 45</tt>.

<pre>
\setbox0=\hbox attr0 = 5 attr10 = 45 {contents}

\directlua{
local attr = tex.box[0].attr.next
while attr do
texio.write_nl(attr.number .. " = " .. attr.value)
attr = attr.next
end
}
</pre>

= Examples =

The idea behind attributes is to mark material at a given time so as to spot it at a later stage of processing. The example in the page on the [[Post linebreak filter#Margin notes|<tt>post_linebreak_filter</tt>]] [[Callbacks|callback]] shows how attributes can be put to use for margin notes: the text to which a note is added is marked with an attribute, and later, when the paragraph has been built, the note is appended to the line containing that text, identified by its having a given value for the attribute.

Main Page

2011-05-11T10:22:23Z

Paul: Added link to the article on post_linebreak_filter.

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Some articles ==

* [[Documentation and help]] points to other online resources (manuals, mailing list, etc.) related to LuaTeX.
* [[Writing Lua in TeX]] explains how to write Lua code in a TeX document (and back).
* [[Attributes]] introduces LuaTeX's thrilling new concept.
* Pages on callbacks:
** [[Callbacks]] introduces callbacks and how to use them.
** There is a page on the [[Post linebreak filter|<tt>post_linebreak_filter</tt>]] callback, explaining and illustrating it with a couple of examples. The [[Show the hyphenation points]] article is another example of use.
** The page on [[process input buffer|the <tt>process_input_buffer</tt> callback]] illustrates how to read documents with non-UTF-8 encoding, and how to write TeX with lightweight markup instead of the usual commands.
** Another callback is [[show error hook|<tt>show_error_hook</tt>]], which lets you enliven your error messages!

== From the old bluwiki.com ==

(Mostly by and thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

Post linebreak filter

2011-05-11T10:14:28Z

Paul: Created page with explanation and examples.

= Syntax =

The <tt>post_linebreak_filter</tt> [[callbacks|callback]] is called after a paragraph has been built, i.e. after the <tt>[[linebreak_filter]]</tt> callback or (if the latter isn't specified) TeX's default paragraph builder. The callback receives two arguments: the first is the head of the vertical list returned by the previous operation, the second is a string specifying in which context the paragraph is being built: an empty string means the main vertical list (a simple paragraph on the page), and other values include <tt>vbox</tt>, <tt>vtop</tt>...

The return value should be one of the the following three possibilities: <tt>true</tt> signals that everything is ok and the list that was passed to the callback should be processed further; <tt>false</tt> signals that the list should be flushed from memory: the paragraph simply disappears; finally, if a node list is returned it will be processed instead of the original one. Hence the syntax:

<pre>
function (<node> head, <string> context)
return true | false | <node> newhead
end
</pre>

The callback doesn't replace any internal code and isn't expected to do anything meaningful. Its main use is to manipulate the paragraph once built; the list received as the first argument is made mostly of horizontal lists (the lines of text) interspersed with glues (to adjust baseline distances); any kind of postprocessing can be applied to those nodes (not to mention other types of nodes, if any: <tt>\vadjust</tt>ed material, inserts...).

= Examples =

== Between raggedright and justified ==

In paragraph 5.9.6 of [http://mirrors.ctan.org/info/texbytopic/TeXbyTopic.pdf TeX by Topic], Victor Eijkhout gives a code so that `lines that would stretch beyond certain limits are set with their glue at natural width' (producing a paragraph between raggedright and justified, a style popular in advertisement at the time TeX by Topic was published).

That is exactly the kind of job the <tt>post_linebreak_filter</tt> filter is designed for (the smaller the <tt>\hsize</tt>, the more the result will stand out):

<pre>
local function check_lines (head)
for line in node.traverse_id(node.id("hlist"), head) do
if line.glue_order == 0 and line.glue_sign == 1 and line.glue_set > .4 then
line.list = node.hpack(line.list)
end
end
return head
end

callback.register("post_linebreak_filter", check_lines)
</pre>

All the lines in a paragraph are inspected, thanks to the <tt>node.traverse_id()</tt> iterator, which loops over all the nodes with a given id in a list of node. For each line, one checks whether the following holds: first, the line has been justified with finite glue (it makes little sense to reset lines justified with infinite glue), i.e. the <tt>glue_order</tt> field of the line is 0 (larger values mean different orders of infinite); second, the glues have been stretched, not shrunk, i.e. <tt>glue_sign</tt> is 1, not 2; finally, the glue ratio, i.e. the amount of glue used, recorded in <tt>glue_set</tt> is above an arbitrary threshold: lines whose glues aren't stretched so much are left untouched.

If the conditions are satisfied, we simply reassign as the contents of the horizontal list that very contents processed with <tt>node.hpack()</tt>; the latter function turns a list of nodes into material suitable for a horizontal list, somehow like <tt>\hbox</tt>. Here, since no extra argument signalling that the horizontal material should be set to a certain width (as with the keyword <tt>to</tt> in <tt>\hbox</tt>), glues aren't stretched nor shrunk and the material is set to its natural width, as wanted.

At the end the head of the list is returned; returning <tt>true</tt> would have the same result, since LuaTeX is supposed to processed the same list that was received as an argument to the callback; that the list have been modified is immaterial here.

== Margin notes ==

The callback can also be used to add material to the lines of text. Margin notes are such material: they are related to something in the paragraph, but as long as the paragraph isn't built, one doesn't know where they should be placed. With <tt>post_linebreak_filter</tt>, since the paragraph can be analysed, the limitation vanishes: one can spot the lines to which a note should be appended (their contents will be marked with [[attributes]]), and add that note.

Note that the following code is meant to illustrate the use of the <tt>post_linebreak_filter</tt> and of [[attributes]], and isn't optimal for marginal notes themselves. They would be better dealt with in the output routine: there they can be moved up or down if necessary (so that they don't bleed into the bottom margin, for instance) and placed in the proper margin (left or right, depending on the page). Here all notes will be placed in the right margin.

The code works along the following lines: the <tt>\marginnote</tt> command takes two argument, the first being the text in the paragraph to which the note relates, the second the note itself. The text in the paragraph is marked with an [[attributes|attribute]], and can be identified later in the callback, each mark being a new value for the attribute. The note itself is built in a local box and assigned to a PDF Form XObject; the latter move is for practical reasons only and irrelevant to the code here (working with boxes all the way down would yield the same result). After the paragraph is built, we inspect each line of text to check whether it contains material marked with the attribute; if so, the value of that attribute points to a given Form XObject, which is appended to the line.

Here's the code for the <tt>\marginnote</tt> command:

<pre>
\def\marginnote#1#2{%
\begingroup
\setbox0=\vtop{%
\hsize10em \leftskip1em
\rightskip0pt plus 1fill
\noindent #2}%
\count0=\wd0 \count2=\ht0 \count4=\dp0
\pdfxform0
%
\directlua{%
local xform = node.new(node.id("whatsit"), node.subtype("pdf_refxform"))
xform.objnum = \the\pdflastxform;
xform.width = \the\count0;
xform.height = \the\count2;
xform.depth = \the\count4;
xforms[\the\pdflastxform] = xform
}%
%
\attribute1=\pdflastxform
#1%
\endgroup
}
</pre>

The first part typesets the note in a box (a <tt>\vtop</tt>, so the first line of the note will be aligned with the text it refers to in the paragraph), which is assigned to an XObject thanks to <tt>\pdfxform</tt>. The box itself has reduced <tt>\hsize</tt>, is typeset raggedright, and has a non-zero <tt>\leftskip</tt> meant to leave a gap between its left margin and the right margin of the paragraph. Other changes (fonts, baseline distances) would be suitable here too.

The second part of the code creates a node using that XObject: that is the kind of node <tt>\pdfrefxform\pdflastxform</tt> would have created, except we have to set the dimension by hand, which is why the dimensions of the box were recorded (the box has been emptied when assigned to <tt>\pdfxform</tt>). The node is then stored in a table (to be created permanently in the following code).

The third part simply releases the first argument, to be typeset as part of the paragraph, but marked with a special value of attribute 1 (any attribute would do, of course, and actually one should have something like <tt>\newattribute</tt> to allocate attributes properly, see [[attributes]]).

The following designs the code to handle the lines in <tt>post_linebreak_filter</tt> (and creates the <tt>xforms</tt> table beforehand):

<pre>
xforms = {}

local function find_notes (head)
for line in node.traverse_id (node.id("hlist"), head) do
for item in node.traverse (line.list) do
local attr = node.has_attribute(item, 1)
if attr and xforms[attr] then
node.insert_after(line.list, node.tail(line.list), xforms[attr])
xforms[attr] = nil
break
end
end
end
return head
end

callback.register("post_linebreak_filter", find_notes)
</pre>

In the node list representing the constructed paragraph, it inspects all lines (i.e. nodes of type <tt>hlist</tt>). It loops over the material of those lines to check whether some material is marked with attribute 1. If there is, and if the <tt>xforms</tt> table contains an XObject at the index with that value (a condition to be explained presently), the XObject is simply added at the end of the line's material with <tt>node.insert_after()</tt>. The function <tt>node.tail()</tt> returns the last node of a list, hence <tt>xform</tt> is added at the very end.

Then the entry in the table is deleted. That is necessary, because the material marked to receive a footnote can end up broken over two lines (or more); if the entry wasn't deleted, the code would try to add the note twice (once for each line). That is the reason why we check whether the entry <tt>xforms[attr]</tt> exists.

Finally, <tt>break</tt> ends the inspection of the line, under the assumption (typographic rather than logical) that there is no more than one note per line.

== Hyphenation points ==

The article [[Show the hyphenation points]] uses the callback to insert markers indicating where hyphenation might have occured in the paragraph; doing so after the paragraph has been built avoids hindering its proper construction.

Attributes

2011-05-10T15:29:46Z

Paul: Small correction.

Main Page

2011-05-10T15:08:08Z

Paul: Added link to Attributes.

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Some articles ==

* [[Documentation and help]] points to other online resources (manuals, mailing list, etc.) related to LuaTeX.
* [[Writing Lua in TeX]] explains how to write Lua code in a TeX document (and back).
* [[Attributes]] introduces LuaTeX's thrilling new concept.
* Pages on callbacks:
** [[Callbacks]] introduces callbacks and how to use them.
** [[Show the hyphenation points]] does what it says; it is also an example of use of the <tt>post_linebreak_filter</tt>.
** The page on [[process input buffer|the <tt>process_input_buffer</tt> callback]] illustrates how to read documents with non-UTF-8 encoding, and how to write TeX with lightweight markup instead of the usual commands.
** Another callback is [[show error hook|<tt>show_error_hook</tt>]], which lets you enliven your error messages!

== From the old bluwiki.com ==

(Mostly by and thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

Attributes

2011-05-10T15:06:10Z

Paul: How attributes work. There lack examples.

= Introduction =

Attributes are a new concept introduced in LuaTeX. In a way, they are a mix between count registers (they are numbers) and fonts (they are attached to the [[nodes]] created under their scope). They allow information to be carried between different stages of processing.

= Syntax =

Syntactically, attributes are no different from (and can be used as) count registers. Accordingly, primitive commands to handle attribute registers mimick those for count registers: like <tt>\count</tt>, <tt>\attribute</tt> is used to refer to a register, and like <tt>\countdef</tt>, <tt>\attributedef</tt> defines a control sequence to be a synonym for a register. The syntax is:

<pre>
\attribute <16-bit number> <optional equals> <32-bit number>
\attributedef <csname> <optional equals> <16-bit number>
</pre>

It is wise to have a \newattribute command, like \newcount (from plain TeX, used also in LaTeX and ConTeXt), for the allocation of registers. The command exists in LuaTeX, and is available in the [http://tug.ctan.org/pkg/luatexbase luatexbase] package for plain TeX and LaTeX under the name \newluatexattribute.

Whenever a count register is allowed, an attribute register too; that is, primitives like \advance, \multiply, \divide, can be used. More generally, anywhere a number is needed, attributes will do. Of course, assignments to attribute registers obey grouping.

Attributes differ from count registers in two respects:

* They have a special value, -"7FFFFFFF in hexadecimal or -2147483647 in decimal; an attribute with that value (which is default) is said to be unset. With any larger value, the attribute is set.
* When [[nodes]] are created, they carry with them the values of all the attributes in force at creation time (except for some asynchronous attribute settings). In other words, when inspecting a [[nodes|node]] at whatever stage of its processing, one can retrieve (and also modify) the values of the attributes when the node was created. In that respect, attributes behave like fonts (glyph [[nodes]] always carry information about their fonts).

== Box attributes ==

When creating boxes, there exists a new keyword <tt>attr</tt> besides <tt>to</tt> and <tt>spread</tt>. The node representing the box will then have the associate attribute, but not its contents. For instance, after:

<pre>
\attribute0=1
\setbox0=\hbox attr 1 = 1 {contents}
</pre>

The box (as a node) and each node of its contents have value 1 for attribute 0, but only the box node has value 1 for attribute 1; its contents has whatever value is currently assigned to that attribute (very probably the attribute is unset).

The syntax for the <tt>attr</tt> keyword is the following (all spaces are optional):

<pre>
attr <number> = <number>
</pre>

The <tt>attr</tt> declaration should precede the <tt>to</tt> or <tt>spread</tt> declaration if any.

== Asynchronous attribute settings ==

Sometimes [[nodes]] are assigned attribute values depending not on the real values at creation time but on the values of the surrounding nodes. The following simplistic code illustrates that:

<pre>
{\attribute0=1 Ve}\par
</pre>

LuaTeX will insert a font kern between the two letters (if the font says so, of course). That node will have attribute 0 set to 1, even though that is no longer the real value. Indeed, the kern node is created when building the paragraph, i.e. when the <tt>\par</tt> command is encountered; at that time, attribute 0 no longer has value 1, since that value was assigned in a group and the group is closed. So the kern should normally have whatever value is in force outside the group; instead it is given the value of the nodes that prompted its creation (more precisely, the value of the first node, which can differ from the second one). Similarly, a ligature node takes the values of those nodes it is created from, as does a discretionary node.

The <tt>\leftskip</tt> and <tt>\rightskip</tt> glues, inserted when non-zero at the beginning and end of each line of a paragraph, take the value of the neighboring nodes (i.e. the first and last nodes of a line, respectively).

Those asynchronous settings are supposed to make more sense than assignments that would follow the real values. Counter-measures can easily be implemented if necessary: for instance, one can inspect all lines in a paragraph and reassign to the <tt>\leftskip</tt> and <tt>\rightskip</tt> glues the current values for the attributes.

= On the Lua side =

== Registers ==

Like all registers, attributes are stored in a Lua table in the <tt>tex</tt> library. Hence besides <tt>tex.count</tt>, <tt>tex.box</tt>, etc., there exists <tt>tex.attribute</tt>. The indices to entries are either a register number or the name of a control sequence defined with <tt>\attributedef</tt>. For instance, after

<pre>
\attributedef\myattribute=100
\myattribute=1
</pre>

both <tt>tex.attribute.myattribute</tt> and <tt>tex.attribute[100]</tt> return 1. The entry can also be assigned to, but then the assignment is necessarily local.

There are also two functions to set or retrieve attribute values, with the following syntax:

<pre>
tex.getattribute(<register>)
tex.setattribute(["global",] <register>, <number>)
</pre>

In both case <tt><register></tt> is the same as an entry in <tt>tex.attribute</tt>: either a number or a string representing a control sequence defined with <tt>\attributedef</tt>. The first function returns the current value of the attribute, while the second sets it, either locally or globally if the first optional argument is given.

== Attributes in nodes ==

The <tt>node</tt> library offers the following function to deal with the values of attributes for a given node:

<pre>
node.has_attribute(<node>, <attribute>[, <value>])
</pre>

Without the third argument, if <tt><node></tt> has <tt>attribute</tt> set, i.e. with a value different from -"7FFFFFFF, then the value is returned; otherwise <tt>nil</tt> is returned. If the third argument is given, the value of the attribute is returned if and only if it is equal to <value>, and <tt>nil</tt> otherwise.

<pre>
node.set_attribute(<node>, <attribute>, <value>)
</pre>

Sets <attribute> to <value> for <node>.

<pre>
node.unset_attribute(<node>, <attribute>[, <value>])
</pre>

Unsets <attribute> for <node> and returns the old value; if <value> is present, the attribute is unset only if it matches <value>.

== Attribute lists ==

In [[nodes]], attributes actually are implemented as nodes in a list. That is why the <tt>attr</tt> field of all nodes (except attribute nodes themselves) points to a node of type <tt>attribute_list</tt>. That node has only one field <tt>next</tt> pointing to the first attribute node in the list, if any. There might not be any such node because only set attributes are present in the list. Attribute nodes have three fields:

* <tt>number</tt>: the attribute's number;
* <tt>value</tt>: the attribute's value;
* <tt>next</tt>: the next attribute in the list, i.e. an attribute node that is set.

As an example, the following code prints <tt>0 = 5</tt> and <tt>10 = 45</tt>.

<pre>
\setbox0=\hbox attr0 = 5 attr10 = 45 {contents}

\directlua{
local attr = tex.box[0].attr.next
while attr do
texio.write_nl(attr.number .. " = " .. attr.value)
attr = attr.next
end
}
</pre>

Documentation and help

2011-05-08T11:21:09Z

Paul: Corrected link.

=Documentation=

* The definitive source of information on LuaTeX is the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX Reference]; the link points to the latest version, documenting latest builds, but can be used for older versions too.

* The previous document reduced to a cheatsheet is also available with the [http://www.luatex.org/svn/trunk/manual/functionref.pdf Short Reference].

* Besides general information, the [http://www.luatex.org/ LuaTeX site] also hosts [http://www.luatex.org/documentation.html some presentations] by members of the team.

* [http://tug.org/TUGboat/ TUGboat] (communications of the TeX User Group) has featured regular articles on LuaTeX for some years now.

* More can be found in publications by local user groups and conferences.

=Help=

* The [http://tug.org/mailman/listinfo/luatex LuaTeX mailing list] is frequented by knowledgeable people and questions rarely remain unanswered.

* There is also a [http://www.ntg.nl/mailman/listinfo/dev-luatex list for developpers] for very technical things.

* Questions about LuaTeX can also be asked in other TeX-related lists or sites: [http://tug.org/mailman/listinfo/texhax texhax], [http://groups.google.com/group/comp.text.tex comp.text.tex] (and similar groups for different languages), or [http://tex.stackexchange.com/ TeX StackExchange].

=On Lua=

* The main source on the Lua language is the [http://www.lua.org/manual/5.1/manual.html Lua reference manual]; the link here documents version 5.1, currently used by LuaTeX, but version 5.2 is on its way and very probably used by LuaTeX when stable; see the [http://www.lua.org/work/doc/manual.html alpha version of the manual for Lua 5.2].

* [http://www.lua.org/pil/index.html Programming in Lua], by Lua's chief architect Roberto Ierusalimschy, is available online for version 5.0, quite usable for later versions.

* There is also a [http://www.lua.org/lua-l.html mailing list] and a [http://forum.luahub.com/ forum].

* All this and more can be found on the [http://www.lua.org/ Lua website].

Main Page

2011-05-08T11:05:30Z

Paul: Removed the section on help and added link to the article Documentation and help.

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Some articles ==

* [[Documentation and help]] points to other online resources (manuals, mailing list, etc.) related to LuaTeX.
* [[Writing Lua in TeX]] explains how to write Lua code in a TeX document (and back).
* [[Callbacks]] introduces callbacks and how to use them.
* [[Show the hyphenation points]] does what it says; it is also an example of use of the <tt>post_linebreak_filter</tt>.
* The page on [[process input buffer|the <tt>process_input_buffer</tt> callback]] illustrates how to read documents with non-UTF-8 encoding, and how to write TeX with lightweight markup instead of the usual commands.
* Another callback is [[show error hook|<tt>show_error_hook</tt>]], which lets you enliven your error messages!

== From the old bluwiki.com ==

(Mostly by and thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

Documentation and help

2011-05-08T11:03:46Z

Paul: Links to documentation and help.

=Documentation=

* The definitive source of information on LuaTeX is the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX Reference]; the link points to the latest version, documenting latest builds, but can be used for older versions too.

* The previous document reduced to a cheatsheet is also available with the [http://www.luatex.org/svn/trunk/manual/functionref.pdf Short Reference].

* Besides general information, the [http://www.luatex.org/ LuaTeX site] also hosts [http://www.luatex.org/documentation.html some presentations] by members of the team.

* [http://tug.org/TUGboat/ TUGboat] (communications of the TeX User Group) has featured regular articles on LuaTeX for some years now.

* More can be found in publications by local user groups and conferences.

=Help=

* The [http://tug.org/mailman/listinfo/luatex LuaTeX mailing list] is frequented by knowledgeable people and questions rarely remain unanswered.

* There is also a [http://www.ntg.nl/mailman/listinfo/dev-luatex list for developpers] for very technical things.

* Questions about LuaTeX can also be asked in other TeX-related lists or sites: [http://tug.org/mailman/listinfo/texhax texhax], [http://groups.google.com/group/comp.text.tex/topics comp.text.tex] (and similar groups for different languages), or [http://tex.stackexchange.com/ TeX StackExchange].

=On Lua=

* The main source on the Lua language is the [http://www.lua.org/manual/5.1/manual.html Lua reference manual]; the link here documents version 5.1, currently used by LuaTeX, but version 5.2 is on its way and very probably used by LuaTeX when stable; see the [http://www.lua.org/work/doc/manual.html alpha version of the manual for Lua 5.2].

* [http://www.lua.org/pil/index.html Programming in Lua], by Lua's chief architect Roberto Ierusalimschy, is available online for version 5.0, quite usable for later versions.

* There is also a [http://www.lua.org/lua-l.html mailing list] and a [http://forum.luahub.com/ forum].

* All this and more can be found on the [http://www.lua.org/ Lua website].

Main Page

2011-05-08T10:27:17Z

Paul: Formatting.

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Getting help ==

* There are two mailing lists related to LuaTeX, [http://tug.org/mailman/listinfo/luatex one for general discussions and questions about LuaTeX ("luatex users")] and [http://www.ntg.nl/mailman/listinfo/dev-luatex one more related to the development of LuaTeX ("dev-luatex")]
* There is also [http://www.tug.org/mailman/listinfo/lualatex-dev a LuaLaTeX (LaTeX with LuaTeX) mailing list].
* [http://tex.stackexchange.com/ tex.stackexchange.com] is a q&a site for questions related to TeX, and you can ask (and answer) LuaTeX related questions there.
* The [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf online reference manual (pdf)] describes every aspect of LuaTeX, but is rather technical and concise. There also exists [http://www.luatex.org/svn/trunk/manual/functionref.pdf a short reference (pdf)].

== Some articles ==

* [[Writing Lua in TeX]] explains how to write Lua code in a TeX document (and back).
* [[Callbacks]] introduces callbacks and how to use them.
* [[Show the hyphenation points]] does what it says; it is also an example of use of the <tt>post_linebreak_filter</tt>.
* The page on [[process input buffer|the <tt>process_input_buffer</tt> callback]] illustrates how to read documents with non-UTF-8 encoding, and how to write TeX with lightweight markup instead of the usual commands.
* Another callback is [[show error hook|<tt>show_error_hook</tt>]], which lets you enliven your error messages!

== From the old bluwiki.com ==

(Mostly by and thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

Main Page

2011-05-08T10:26:25Z

Paul: Added link to show_error_hook.

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Getting help ==

* There are two mailing lists related to LuaTeX, [http://tug.org/mailman/listinfo/luatex one for general discussions and questions about LuaTeX ("luatex users")] and [http://www.ntg.nl/mailman/listinfo/dev-luatex one more related to the development of LuaTeX ("dev-luatex")]
* There is also [http://www.tug.org/mailman/listinfo/lualatex-dev a LuaLaTeX (LaTeX with LuaTeX) mailing list].
* [http://tex.stackexchange.com/ tex.stackexchange.com] is a q&a site for questions related to TeX, and you can ask (and answer) LuaTeX related questions there.
* The [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf online reference manual (pdf)] describes every aspect of LuaTeX, but is rather technical and concise. There also exists [http://www.luatex.org/svn/trunk/manual/functionref.pdf a short reference (pdf)].

== Some articles ==

* [[Writing Lua in TeX]] explains how to write Lua code in a TeX document (and back).
* [[Callbacks]] introduces callbacks and how to use them.
* [[Show the hyphenation points]] does what it says; it is also an example of use of the <tt>post_linebreak_filter</tt>.
* The page on [[process input buffer|the <tt>process_input_buffer</tt> callback]] illustrates how to read documents with non-UTF-8 encoding, and how to write TeX with lightweight markup instead of the usual commands.
* Another callback is [[show error hook]], which lets you enliven your error messages!

== From the old bluwiki.com ==

(Mostly by and thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

Show error hook

2011-05-08T10:24:34Z

Paul: Description and example of use of the show_error_hook callback.

Process input buffer

2011-05-08T09:23:17Z

Paul: Points to main page on callbacks.

=The process_input_buffer callback=

This [[Callbacks|callback]] is called whenever TeX needs a new input line from the file it is currently reading. The argument passed to the function registered there is a string representing the original input line; the function should return another string, to be processed by TeX, or <tt>nil</tt>; in the latter case, the original string is processed. So the function should be defined according to the following blueprint:

<pre>
function (<string> original_line)
return <string> adjusted_line or nil
end
</pre>

In case TeX isn't reading the main document but a file that is <tt>\input</tt> in it, the callback is called after the <tt>reader</tt> function possibly defined in the <tt>[[open_read_file]]</tt> callback.

The following examples aren't mutually exclusive; however, if one wanted to register them all in the same callback, special care would be needed, as explained [[Callbacks#Several_functions_in_a_callback|in the main page on callbacks]]. Here each call to <tt>callback.register()</tt> removes whatever there might have been in the callback beforehand.

=Encoding=

LuaTeX understands UTF-8 and UTF-8 only (ASCII is a part of it, however). A line containing invalid characters will produce an error message: String contains an invalid utf-8 sequence. Thus documents with different encoding must be converted, and this can be done in <tt>process_input_buffer</tt>.

Note that such a conversion might not produce anything meaningful when the document is compiled, because the fonts used must have the proper glyphs in the right place; for instance, the Computer Modern fonts don't have characters with diacritics, and Latin Modern should be used instead. An alternative solution (quite archaic with LuaTeX) is to define the special characters as active, and to make them create a diacritic. For instance:

<pre>
\catcode`\é = 13
\def é{\'e}
</pre>

==Latin-1==

Reading a document written in Latin-1 (ISO-8859-1) is relatively straightforward, because the <tt>slnunicode</tt> library embedded in LuaTeX does the conversion at once.

<pre>
local utf8_char, byte, gsub = unicode.utf8.char, string.byte, string.gsub

local function convert_char (char)
return utf8_char(byte(char))
end

local function convert_line (line)
return gsub(line, ".", convert_char)
end

callback.register("process_input_buffer", convert_line)
</pre>

The code reads as follows (bottom up): the <tt>convert_line</tt> function is registered in the <tt>process_input_buffer</tt> callback; it is passed the original input line as its sole argument, and returns another string, namely that line processed with the <tt>string.gsub()</tt> function. The latter replaces in a string (the first argument) all the occurrences of a pattern (the second argument) with the outcome of its third argument (for further information on <tt>string.gsub()</tt> see [http://www.lua.org/manual/5.1/manual.html#pdf-string.gsub the Lua reference manual]).

Here the third argument is a function, <tt>convert_char()</tt> to which is passed whatever matches the pattern. This pattern matches any character, the dot being a magic character. In <tt>convert_char()</tt>, the character is first converted to a numerical code (the codepoint in Latin-1, see the [http://www.lua.org/manual/5.1/manual.html#pdf-string.byte description of <tt>string.byte()</tt>]); the code is passed to <tt>unicode.utf8.char</tt> which, given a number, returns the associated character in UTF-8, the number being interpreted as a code point in Unicode; since the characters in Latin-1 have the same code points as in Unicode, the conversion is automatic here.

The use of <tt>local</tt> variables ensures speed, and above all that those variables aren't defined outside the current chunk, for instance, the current <tt>\directlua</tt> call or the current Lua file; actually, the code could even be embedded between <tt>do</tt> and <tt>end</tt> and leave absolutely no trace whatsoever.

==Other 8-bit encodings==

When using other 8-bit encoding, the previous code won't work, because it defaults to Latin-1 only. Then one must convert each character one by one by setting up a table matching each input character with the Unicode value; that value can be passed to unicode.utf8.char to yield the desired character.

For instance, here's the code needed to process a document encoded in Latin/Greek (ISO-8859-7):

<pre>
local utf8_char, byte = unicode.utf8.char, string.byte

local LatinGreek_table = {
[0] = 0x0000, 0x0001, 0x0002, 0x0003, 0x0004,
-- ... 240+ other values...
0x03C9, 0x03CA, 0x03CB, 0x03CC, 0x03CD, 0x03CE
}

local function convert_char (char)
return utf8_char(LatinGreek_table[byte(char)])
end

local function convert_line (line)
return gsub(line, ".", convert_char)
end

callback.register("process_input_buffer", convert_line)
</pre>

The <tt>convert_line()</tt> function and call to <tt>callback.register()</tt> are the same as above. What has changed is <tt>convert_char()</tt>; the numerical value of the original character is now used as an index in the <tt>LatinGreek_table</tt>, and the value returned is the corresponding Unicode code point; that number is passed to <tt>unicode.utf8.char</tt> to produce the character in UTF-8.

The values in the <tt>LatinGreek_table</tt> are assigned to the right indices because they are declared in a row (here in hexadecimal form by prefixing them with <tt>0x</tt>). The only index that needs to be specified is 0, because indexing of tables in Lua begins at 1 by default. The table length is 254 plus the value at index 0 (not 255, because there is no such character in Latin/Greek). Each index is a code point in Latin/Greek, and the value is the code point of the same character in Unicode. For instance, lowercase omega in Latin/Greek is 249, at which index one finds 0x03C9, lowercase omega in Unicode.

=TeX as a lightweight markup language=

The <tt>process_input_buffer</tt> can be put to an entirely different use, namely to preprocess input strings using some kind of lightweight markup and turn them into proper TeX.

Let's suppose one has two control sequences, <tt>\italic</tt> and <tt>\bold</tt>, taking a single argument; let's suppose furthermore that one wants to write a sentence like

<pre>
This is in /italic/ and this is in *bold*.
</pre>

to be processed by TeX as

<pre>
This is in \italic{italic} and this is in \bold{bold}.
</pre>

so as to produce: `This is in italic and this is in bold.' The following code does exactly that (the use of the percent sing and of <tt>\\</tt> without <tt>\noexpand</tt> means that this code is either in a Lua file or in the second version of <tt>\luacode</tt> as defined in [[Writing Lua in TeX#Backslash|Writing Lua in TeX]]; also, this code illustrates the registering of an anonymous function instead of a function variable as in the previous examples):

<pre>
local gsub = string.gsub

callback.register("process_input_buffer",
function (str)
str = gsub(str, "/(.-)/", "\\italic{%1}")
str = gsub(str, "%*(.-)%*", "\\bold{%1}")
return str
end)
</pre>

What happens is that the original string is replaced with successive call to <tt>string.gsub</tt> (see [http://www.lua.org/manual/5.1/manual.html#pdf-string.gsub the Lua reference manual]), in which the captures in the patterns are replaced with themselves as arguments to the TeX function (the non-capture parts of the patterns are discarded). For instance, <tt>/a word/</tt> yields <tt>\italic{a word}</tt>. Note that with <tt>\bold</tt>, the asterisks in the pattern must be escaped with <tt>%</tt>, otherwise they would be interpreted as magic character. The line can then be processed by TeX as usual.

Only pairs of slashes or asterisks in the same line will be interpreted as markup, because lines are processed one by one and nothing is remembered from one line to the next (that can be implemented, but is a bit more complicated and dangerous). Hence, nothing will be in italics in the following example:

<pre>
This will /not
be/ in italic.
</pre>

Instead, slashes will be read.

One can add an unlimited number of replacements of the input line. For instance, here's a way to mark sections by rows of <tt>#</tt>'s:

<pre>
str = gsub(str, "^%s*#%s*(.*)", "\\section{%1}")
str = gsub(str, "^%s*##%s*(.*)", "\\subsection{%1}")
str = gsub(str, "^%s*###%s*(.*)", "\\subsubsection{%1}")
</pre>

What is recognized as a section header is any line beginning with one, two or three hashes, ignoring space before and after. So one can write:

<pre>
# A section
## A subsection
### A subsubsection
</pre>

More complicated things are possible. However, using the [[open read file]] callback (only with an <tt>\input</tt> file) is preferable if subtlety is required, because it can read an entire file (or any set of lines) before changing anything. Also, the powerful [http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html LPeg library] (embedded in LuaTeX) can be used instead of the default operations.

Finally, one must note that such manipulations can be dangerous or lead to unwanted results. For instance, a line such as

<pre>
... the file should be contained in user/directory/myfiles...
</pre>

will be processed without slashes and with directory in italics. Thus, such markup should either be used with files specially tailored for it, or contain ways to be overriden or to protect some text.

Main Page

2011-05-07T17:33:52Z

Paul: Added link to process_input_buffer.

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Getting help ==

* There are two mailing lists related to LuaTeX, [http://tug.org/mailman/listinfo/luatex one for general discussions and questions about LuaTeX ("luatex users")] and [http://www.ntg.nl/mailman/listinfo/dev-luatex one more related to the development of LuaTeX ("dev-luatex")]
* There is also [http://www.tug.org/mailman/listinfo/lualatex-dev a LuaLaTeX (LaTeX with LuaTeX) mailing list].
* [http://tex.stackexchange.com/ tex.stackexchange.com] is a q&a site for questions related to TeX, and you can ask (and answer) LuaTeX related questions there.
* The [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf online reference manual (pdf)] describes every aspect of LuaTeX, but is rather technical and concise. There also exists [http://www.luatex.org/svn/trunk/manual/functionref.pdf a short reference (pdf)].

== Some articles ==

* [[Writing Lua in TeX]] explains how to write Lua code in a TeX document (and back).
* [[Callbacks]] introduces callbacks and how to use them.
* [[Show the hyphenation points]] does what it says; it is also an example of use of the <tt>post_linebreak_filter</tt>.
* The page on [[process input buffer|the <tt>process_input_buffer</tt> callback]] illustrates how to read documents with non-UTF-8 encoding, and how to write TeX with lightweight markup instead of the usual commands.

== From the old bluwiki.com ==

(Mostly by and thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

Process input buffer

2011-05-07T17:30:42Z

Paul: Examples of use of the process_input_buffer callback.

=The process_input_buffer callback=

This callback is called whenever TeX needs a new input line from the file it is currently reading. The argument passed to the function registered there is a string representing the original input line; the function should return another string, to be processed by TeX, or <tt>nil</tt>; in the latter case, the original string is processed. So the function should be defined according to the following blueprint:

<pre>
function (<string> original_line)
return <string> adjusted_line or nil
end
</pre>

In case TeX isn't reading the main document but a file that is <tt>\input</tt> in it, the callback is called after the <tt>reader</tt> function possibly defined in the <tt>[[open_read_file]]</tt> callback.

The following examples aren't mutually exclusive; however, if one wanted to register them all in the same callback, special care would be needed, as explained [[Callbacks#Several_functions_in_a_callback|in the main page on callbacks]]. Here each call to <tt>callback.register()</tt> removes whatever there might have been in the callback beforehand.

=Encoding=

LuaTeX understands UTF-8 and UTF-8 only (ASCII is a part of it, however). A line containing invalid characters will produce an error message: String contains an invalid utf-8 sequence. Thus documents with different encoding must be converted, and this can be done in <tt>process_input_buffer</tt>.

Note that such a conversion might not produce anything meaningful when the document is compiled, because the fonts used must have the proper glyphs in the right place; for instance, the Computer Modern fonts don't have characters with diacritics, and Latin Modern should be used instead. An alternative solution (quite archaic with LuaTeX) is to define the special characters as active, and to make them create a diacritic. For instance:

<pre>
\catcode`\é = 13
\def é{\'e}
</pre>

==Latin-1==

Reading a document written in Latin-1 (ISO-8859-1) is relatively straightforward, because the <tt>slnunicode</tt> library embedded in LuaTeX does the conversion at once.

<pre>
local utf8_char, byte, gsub = unicode.utf8.char, string.byte, string.gsub

local function convert_char (char)
return utf8_char(byte(char))
end

local function convert_line (line)
return gsub(line, ".", convert_char)
end

callback.register("process_input_buffer", convert_line)
</pre>

The code reads as follows (bottom up): the <tt>convert_line</tt> function is registered in the <tt>process_input_buffer</tt> callback; it is passed the original input line as its sole argument, and returns another string, namely that line processed with the <tt>string.gsub()</tt> function. The latter replaces in a string (the first argument) all the occurrences of a pattern (the second argument) with the outcome of its third argument (for further information on <tt>string.gsub()</tt> see [http://www.lua.org/manual/5.1/manual.html#pdf-string.gsub the Lua reference manual]).

Here the third argument is a function, <tt>convert_char()</tt> to which is passed whatever matches the pattern. This pattern matches any character, the dot being a magic character. In <tt>convert_char()</tt>, the character is first converted to a numerical code (the codepoint in Latin-1, see the [http://www.lua.org/manual/5.1/manual.html#pdf-string.byte description of <tt>string.byte()</tt>]); the code is passed to <tt>unicode.utf8.char</tt> which, given a number, returns the associated character in UTF-8, the number being interpreted as a code point in Unicode; since the characters in Latin-1 have the same code points as in Unicode, the conversion is automatic here.

The use of <tt>local</tt> variables ensures speed, and above all that those variables aren't defined outside the current chunk, for instance, the current <tt>\directlua</tt> call or the current Lua file; actually, the code could even be embedded between <tt>do</tt> and <tt>end</tt> and leave absolutely no trace whatsoever.

==Other 8-bit encodings==

When using other 8-bit encoding, the previous code won't work, because it defaults to Latin-1 only. Then one must convert each character one by one by setting up a table matching each input character with the Unicode value; that value can be passed to unicode.utf8.char to yield the desired character.

For instance, here's the code needed to process a document encoded in Latin/Greek (ISO-8859-7):

<pre>
local utf8_char, byte = unicode.utf8.char, string.byte

local LatinGreek_table = {
[0] = 0x0000, 0x0001, 0x0002, 0x0003, 0x0004,
-- ... 240+ other values...
0x03C9, 0x03CA, 0x03CB, 0x03CC, 0x03CD, 0x03CE
}

local function convert_char (char)
return utf8_char(LatinGreek_table[byte(char)])
end

local function convert_line (line)
return gsub(line, ".", convert_char)
end

callback.register("process_input_buffer", convert_line)
</pre>

The <tt>convert_line()</tt> function and call to <tt>callback.register()</tt> are the same as above. What has changed is <tt>convert_char()</tt>; the numerical value of the original character is now used as an index in the <tt>LatinGreek_table</tt>, and the value returned is the corresponding Unicode code point; that number is passed to <tt>unicode.utf8.char</tt> to produce the character in UTF-8.

The values in the <tt>LatinGreek_table</tt> are assigned to the right indices because they are declared in a row (here in hexadecimal form by prefixing them with <tt>0x</tt>). The only index that needs to be specified is 0, because indexing of tables in Lua begins at 1 by default. The table length is 254 plus the value at index 0 (not 255, because there is no such character in Latin/Greek). Each index is a code point in Latin/Greek, and the value is the code point of the same character in Unicode. For instance, lowercase omega in Latin/Greek is 249, at which index one finds 0x03C9, lowercase omega in Unicode.

=TeX as a lightweight markup language=

The <tt>process_input_buffer</tt> can be put to an entirely different use, namely to preprocess input strings using some kind of lightweight markup and turn them into proper TeX.

Let's suppose one has two control sequences, <tt>\italic</tt> and <tt>\bold</tt>, taking a single argument; let's suppose furthermore that one wants to write a sentence like

<pre>
This is in /italic/ and this is in *bold*.
</pre>

to be processed by TeX as

<pre>
This is in \italic{italic} and this is in \bold{bold}.
</pre>

so as to produce: `This is in italic and this is in bold.' The following code does exactly that (the use of the percent sing and of <tt>\\</tt> without <tt>\noexpand</tt> means that this code is either in a Lua file or in the second version of <tt>\luacode</tt> as defined in [[Writing Lua in TeX#Backslash|Writing Lua in TeX]]; also, this code illustrates the registering of an anonymous function instead of a function variable as in the previous examples):

<pre>
local gsub = string.gsub

callback.register("process_input_buffer",
function (str)
str = gsub(str, "/(.-)/", "\\italic{%1}")
str = gsub(str, "%*(.-)%*", "\\bold{%1}")
return str
end)
</pre>

What happens is that the original string is replaced with successive call to <tt>string.gsub</tt> (see [http://www.lua.org/manual/5.1/manual.html#pdf-string.gsub the Lua reference manual]), in which the captures in the patterns are replaced with themselves as arguments to the TeX function (the non-capture parts of the patterns are discarded). For instance, <tt>/a word/</tt> yields <tt>\italic{a word}</tt>. Note that with <tt>\bold</tt>, the asterisks in the pattern must be escaped with <tt>%</tt>, otherwise they would be interpreted as magic character. The line can then be processed by TeX as usual.

Only pairs of slashes or asterisks in the same line will be interpreted as markup, because lines are processed one by one and nothing is remembered from one line to the next (that can be implemented, but is a bit more complicated and dangerous). Hence, nothing will be in italics in the following example:

<pre>
This will /not
be/ in italic.
</pre>

Instead, slashes will be read.

One can add an unlimited number of replacements of the input line. For instance, here's a way to mark sections by rows of <tt>#</tt>'s:

<pre>
str = gsub(str, "^%s*#%s*(.*)", "\\section{%1}")
str = gsub(str, "^%s*##%s*(.*)", "\\subsection{%1}")
str = gsub(str, "^%s*###%s*(.*)", "\\subsubsection{%1}")
</pre>

What is recognized as a section header is any line beginning with one, two or three hashes, ignoring space before and after. So one can write:

<pre>
# A section
## A subsection
### A subsubsection
</pre>

More complicated things are possible. However, using the [[open read file]] callback (only with an <tt>\input</tt> file) is preferable if subtlety is required, because it can read an entire file (or any set of lines) before changing anything. Also, the powerful [http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html LPeg library] (embedded in LuaTeX) can be used instead of the default operations.

Finally, one must note that such manipulations can be dangerous or lead to unwanted results. For instance, a line such as

<pre>
... the file should be contained in user/directory/myfiles...
</pre>

will be processed without slashes and with directory in italics. Thus, such markup should either be used with files specially tailored for it, or contain ways to be overriden or to protect some text.

Main Page

2011-05-07T14:21:33Z

Paul: Corrected link.

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Getting help ==

* There are two mailing lists related to LuaTeX, [http://tug.org/mailman/listinfo/luatex one for general discussions and questions about LuaTeX ("luatex users")] and [http://www.ntg.nl/mailman/listinfo/dev-luatex one more related to the development of LuaTeX ("dev-luatex")]
* There is also [http://www.tug.org/mailman/listinfo/lualatex-dev a LuaLaTeX (LaTeX with LuaTeX) mailing list].
* [http://tex.stackexchange.com/ tex.stackexchange.com] is a q&a site for questions related to TeX, and you can ask (and answer) LuaTeX related questions there.
* The [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf online reference manual (pdf)] describes every aspect of LuaTeX, but is rather technical and concise. There also exists [http://www.luatex.org/svn/trunk/manual/functionref.pdf a short reference (pdf)].

== Some articles ==

* [[Writing Lua in TeX]] explains how to write Lua code in a TeX document (and back).
* [[Callbacks]] introduces callbacks and how to use them.
* [[Show the hyphenation points]] does what it says; it is also an example of use of the <tt>post_linebreak_filter</tt>.

== From the old bluwiki.com ==

(Mostly by and thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

Main Page

2011-05-07T14:19:09Z

Paul: Mentionned \functionref; also introduces the articles written especially for this wiki.

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Getting help ==

* There are two mailing lists related to LuaTeX, [http://tug.org/mailman/listinfo/luatex one for general discussions and questions about LuaTeX ("luatex users")] and [http://www.ntg.nl/mailman/listinfo/dev-luatex one more related to the development of LuaTeX ("dev-luatex")]
* There is also [http://www.tug.org/mailman/listinfo/lualatex-dev a LuaLaTeX (LaTeX with LuaTeX) mailing list].
* [http://tex.stackexchange.com/ tex.stackexchange.com] is a q&a site for questions related to TeX, and you can ask (and answer) LuaTeX related questions there.
* The [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf online reference manual (pdf)] describes every aspect of LuaTeX, but is rather technical and concise. There also exists [[http://www.luatex.org/svn/trunk/manual/functionref.pdf|a short reference]].

== Some articles ==

* [[Writing Lua in TeX]] explains how to write Lua code in a TeX document (and back).
* [[Callbacks]] introduces callbacks and how to use them.
* [[Show the hyphenation points]] does what it says; it is also an example of use of the <tt>post_linebreak_filter</tt>.

== From the old bluwiki.com ==

(Mostly by and thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

Main Page

2011-05-07T14:12:01Z

Paul: Changed a sentence.

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Getting help ==

* There are two mailing lists related to LuaTeX, [http://tug.org/mailman/listinfo/luatex one for general discussions and questions about LuaTeX ("luatex users")] and [http://www.ntg.nl/mailman/listinfo/dev-luatex one more related to the development of LuaTeX ("dev-luatex")]
* There is also [http://www.tug.org/mailman/listinfo/lualatex-dev a LuaLaTeX (LaTeX with LuaTeX) mailing list].
* [http://tex.stackexchange.com/ tex.stackexchange.com] is a q&a site for questions related to TeX, and you can ask (and answer) LuaTeX related questions there.
* The [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf online reference manual (pdf)] describes every aspect of LuaTeX, but is rather technical and concise.

== Some articles ==

(Mostly from the old wiki at bluwiki.com -- thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]
* [[show the hyphenation points]]

Callbacks

2011-05-07T14:08:34Z

Paul: Corrected description of mlist_to_hlist.

Callbacks allow the user to interact with TeX's internal operations; those can be enhanced by additional ones, but also replaced altogether. All callbacks are written in Lua.

Since functions registered in callbacks occur in a sequence of actions that makes up TeX's processing, many of them will receive arguments (and thus should be defined to handle them) and, most importantly, many are supposed to return values. What arguments are passed, if any, and what should be return, vary from callback to callback.

= Registering a callback =

To install a function in a callback one must use the <tt>callback.register()</tt> function, whose syntax is:

<pre>
callback.register(callback_name, function)
</pre>

where <tt>callback_name</tt> is a string representing a predefined callback name (see below), and <tt>function</tt> is either a function variable or an anonymous function. The former is just the name of a function previously defined, for instance:

<pre>
local my_function = function (an_arg)
do_this_or_that(an_arg)
end
callback.register("some_callback", my_function)
</pre>

An anonymous function is a function definition that is not assigned to a variable, e.g.:

<pre>
callback.register("some_callback", function (an_arg) do_this_or_that(an_arg) end)
</pre>

Even in the case of a function variable, what LuaTeX registers is the function itself, and any further assignment to the variable has no effect on the callback.

If registering is successful, <tt>callback.register()</tt> returns a number, which is the internal representation of the callback; if registering fails (for instance because a callback doesn't exist, or what is registered isn't a function), the function returns <tt>nil</tt> and a string representing an error message.

Besides functions, <tt>callback.register()</tt> can take <tt>nil</tt> or <tt>false</tt>. The first value clears the callback, and returns to the default behavior (i.e. either nothing or TeX's default processing); the second value prevents anything from happening at this point, even TeX's default operations, and should be used with great care (the benefits are minor speed gain).

== Several functions in a callback ==

Each call to <tt>callback.register()</tt> erases any function previously registered. I.e. after

<pre>
callback.register("some_callback", function_one)
callback.register("some_callback", function_two)
</pre>

only <tt>function_two</tt> is registered in <tt>some_callback</tt>. Hence, if one wants several successive functions to occur in a callback, <tt>callback.register()</tt> is clearly insufficient. Instead, one must register a metafunction which calls the functions one after the other. This is taken care of in ConTeXt, and there exists <tt>luatexbase-mcb</tt> for plain TeX and LaTeX, but here's how to do it oneself. We'll take the <tt>process_input_buffer</tt> callback has an example; it is called whenever TeX reads an input line, it receives a string (the line) and should return one. A rough approach would be:

<pre>
local function_table = { }

local function metafunction (str)
for _, fct in ipairs(function_table) do
str = fct(str)
end
return str
end

function add_function (fct)
table.insert(function_table, fct)
end

callback.register("process_input_buffer", metafunction)
</pre>

Then one can call <tt>add_function(my_function)</tt> to add <tt>my_function</tt> to the list of functions registered in <tt>process_input_buffer</tt>, and <tt>metafunction</tt> will pass the callback's argument (an input line) to the first function, set that argument to what the function returns, then repeat the process for all functions and finally return the resulting string.

If one also wants to be able to remove functions (and not the entire metafunction), a subtler approach is required, where names are associated with functions and can be used as handles.

= Information about callbacks =

LuaTeX has two functions to get some information about callbacks. The first is <tt>callback.list()</tt>, called without arguments and returning a table with the callback names as keys and booleans as values: if the value is <tt>true</tt>, it means that something is registered in the callback, or that it is set to <tt>false</tt> (see above). The other function is <tt>callback.find()</tt>, called with a callback name. If a function is registered, it is returned; if the callback is set to <tt>false</tt>, a boolean with that value is returned; finally, if nothing is registered, <tt>nil</tt> is returned (note that to distinguish between a boolean set to <tt>false</tt> and <tt>nil</tt>, you must use <tt>type()</tt>).

= List of callbacks =

For suggestions on how to document callbacks, see [[Talk:Callbacks|the discussion page]].

== File discovery callbacks ==

These are called whenever an input file is needed. Each receives the <tt>asked_name</tt>, a string representing what file name was called by the user (for instance, <tt>my_file.tex</tt> in <tt>\input my_file.tex</tt>) and should return the name of a file to be read (or written to). The <tt>id_number</tt> in the first two callbacks is the stream's number plus one, because 0 denotes <tt>log</tt> and <tt>\input</tt> files.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[find_read_file]] ||id_number, asked_name ||actual_name ||File called with <tt>\read</tt> or <tt>\input</tt>
|-
|[[find_write_file]] ||id_number, asked_name ||actual_name ||<tt>log</tt> file or file called with <tt>\write</tt>
|-
|[[find_font_file]] ||asked_name ||actual_name
|-
|[[find_output_file]] ||asked_name ||actual_name
|-
|[[find_format_file]] ||asked_name ||actual_name
|-
|[[find_vf_file]] ||asked_name ||actual_name
|-
|[[find_map_file]] ||asked_name ||actual_name
|-
|[[find_enc_file]] ||asked_name ||actual_name
|-
|[[find_sfd_file]] ||asked_name ||actual_name
|-
|[[find_pk_file]] ||asked_name ||actual_name
|-
|[[find_data_file]] ||asked_name ||actual_name
|-
|[[find_opentype_file]] ||asked_name ||actual_name
|-
|[[find_truetype_file]] ||asked_name ||actual_name
|-
|[[find_type_file]] ||asked_name ||actual_name
|-
|[[find_image_file]] ||asked_name ||actual_name
|}

== File reading callbacks ==

The previous callbacks were called when a file was asked for; the following are triggered when the file is read. In all, <tt>filename</tt> is the string returned by the file discovery callbacks. The first callback reads <tt>\input</tt> or <tt>\read</tt> files; it must return a table, whose first cell is a function that will be called whenever a line from the input file is readed; the second cell is an optional function executed when the input file terminates.

The other callbacks are used for binary files: <tt>success</tt> is a boolean, <tt>data</tt> is a string and <tt>data_size</tt> is a number (the length of <tt>data</tt> in bytes).

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[open_read_file]] ||filename ||{reader, close}
|-
|[[read_font_file]] ||filename ||success, data, data_size
|-
|[[read_vf_file]] ||filename ||success, data, data_size
|-
|[[read_map_file]] ||filename ||success, data, data_size
|-
|[[read_enc_file]] ||filename ||success, data, data_size
|-
|[[read_sfd_file]] ||filename ||success, data, data_size
|-
|[[read_pk_file]] ||filename ||success, data, data_size
|-
|[[read_data_file]] ||filename ||success, data, data_size
|-
|[[read_truetype_file]] ||filename ||success, data, data_size
|-
|[[read_type1_file]] ||filename ||success, data, data_size
|-
|[[read_opentype_file]] ||filename ||success, data, data_size
|}

== Data processing callbacks ==

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[process_input_buffer]] ||line ||modified_line ||Called for each input line (after <tt>reader</tt> in <tt>open_read_file</tt> if TeX is not reading the main file).
|-
|[[process_output_buffer]] ||line ||modified_line ||Called for each line TeX writes to an external file; this excludes the <tt>log</tt> file, the terminal and <tt>\write18</tt> calls.
|-
|[[token_filter]] || ||token ||Called when TeX needs token; no argument is passed to the callback, the next token must be fetched with <tt>token.get_next()</tt>; what is returned is immediately processed by TeX.
|}

== Node list processing callbacks ==

These callbacks deal with node manipulations and occur at various stages of TeX's typesetting processes.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[buildpage_filter]] ||information || ||Called when TeX moves material to the main vertical list. (This callback is an endangered species.)
|-
|[[hyphenate]] ||head, tail || || Inserts discretionary nodes in the node list; called in restricted and unrestricted horizontal mode (i.e. <tt>\hbox</tt>'es and paragraph building). A function doing what TeX does here by default is <tt>lang.hyphenate()</tt>.
|-
|[[ligaturing]] ||head, tail || || Same as <tt>hyphenate</tt>, but applying ligatures (and adding some discretionaries too). An associated function is <tt>node.ligaturing()</tt>.
|-
|[[kerning]] ||head, tail || || Same as above, with font kerns. The associated function is <tt>node.kerning()</tt>.
|-
|[[pre_linebreak_filter]] ||head, context ||boolean or newhead ||Called in unrestricted horizontal mode, after <tt>kerning</tt>, and before calling the paragraph builder; it receives the list of nodes that is the contents of the paragraph (plus the <tt>\parfillskip</tt> glue) and must return something similar.
|-
|[[linebreak_filter]] ||head, is_display ||newhead ||The paragraph builder. It receives what the previous callback has returned and is in charge of making a paragraph out of it; it must return what it has built. An associated function is <tt>tex.linebreak()</tt>.
|-
|[[Show the hyphenation points|post_linebreak_filter]] ||head, groupcode, ||boolean or newhead ||Receives what has been returned by <tt>linebreak_filter</tt> and must return something similar.
|-
|[[hpack_filter]] ||head, context, size, packtype[, dir] ||boolean or newhead ||Called when TeX is going to build an <tt>\hbox</tt>.
|-
|[[vpack_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Called when TeX is going to build a <tt>\vbox</tt>.
|-
|[[pre_output_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Same as <tt>vpack_filter</tt>, but when TeX packs box 255 (or any other value of <tt>\outputbox</tt>) and goes to the output routine.
|-
|[[mlist_to_hlist]] ||head, display_type, need_penalties ||newhead ||Called when TeX turns a math list into a horizontal list (as described in Appendix G of the TeXbook).
|}

== Information reporting callbacks ==

These callbacks are called in various places when TeX has something to say. None of them takes an argument nor returns anything.

{| cellpadding="5"
!Name !! Short description
|-
|[[pre_dump]] ||Called before dumping a format.
|-
|[[start_run]] ||LuaTeX's banner (`This is LuaTeX...'); must be changed in the initialization script, otherwise the banner has already been printed.
|-
|[[stop_run]] ||TeX's final words (statistics and `Output written to...').
|-
|[[start_page_number]] ||The opening bracket and the page number printed when a page is shipped out.
|-
|[[stop_page_number]] ||The closing bracket printed after the previous callback.
|-
|[[show_error_hook]] || Called at the beginning of each error message.
|}

== Miscellaneous ==
{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
||[[finish_pdffile]] || || ||Called when all pages have been written to the PDF file.
|-
|[[define_font]] ||name, size, id ||font_table ||Called when the <tt>\font</tt> command is executed; <tt>name</tt> is the filename asked for, <tt>size</tt> is the <tt>at</tt> size, <tt>id</tt> is the number that will be assigned as an internal representation to the font thus loaded. The callback should return a table representing a font.
|}

Writing Lua in TeX

2011-05-06T10:37:12Z

Paul: Added another version of \luacode with catcode-12 backslashes.

= Embedding Lua code in a TeX document =

Although it is simpler to put Lua code in Lua files, from time to time one may want or need to go Lua in the middle of a document. To this end, LuaTeX has two commands: <tt>\directlua</tt> and <tt>\latelua</tt>. They work the same, except <tt>\latelua</tt> is processed when the page where it appears is shipped out, whereas <tt>\directlua</tt> is processed at once; the distinction is immaterial here, and what is said of <tt>\directlua</tt> also applies to <tt>\latelua</tt>.

<tt>\directlua</tt> can be called in three ways:

<pre>
\directlua {<lua code>}
\directlua name {<name>} {<lua code>}
\directlua <number> {<lua code>}
</pre>

Those three ways are equivalent when it comes to process <tt><lua code></tt>, but in the second case processing will occur in a chunk named <tt><name></tt>, and in the third it will occur in a chunk whose name is the entry <tt><number></tt> in the table <tt>lua.name</tt>. The difference manifests itself only when errors occur, in which case the name of the chunk, if any, is reported.

Each call to <tt>\directlua</tt>, named or not, is processed in a separate chunk. That means that any <tt>local</tt> variable is defined for this call only and is lost afterward. Hence:

<pre>
\directlua{
one = 1
local two = 2
}
\directlua{
texio.write_nl(type(one))
texio.write_nl(type(two))
}
</pre>

will report <tt>number</tt> and <tt>nil</tt> (<tt>texio.write_nl</tt> writes to the log file). On the other hand, Lua code is completely insensitive to TeX's grouping mechanism. In other words, calling <tt>\directlua</tt> between <tt>\bgroup</tt> and <tt>\egroup</tt> doesn't affect the code to be processed.

= TeX catcodes in Lua =

By default, the code passed to <tt>\directlua</tt> is treated as normal TeX input and only then sent to the Lua interpreter. This may lead to unwanted results and must be acknowledged.

== Expansion ==

As with any other special, the code in <tt>\directlua</tt> (and the <tt><name></tt>, if specifed) is fully expanded. This means that macros can be safely passed to <tt>\directlua</tt> if one wants their values, but that they should also be properly escaped when needed. For instance:

<pre>
\def\macro{1}
\directlua{
myvar = \macro
}
</pre>

defines <tt>myvar</tt> as the number <tt>1</tt>. To store the control sequence <tt>\macro</tt> instead, another course of action is needed: see the [[#Backslash|section on backslash]] below.

It should be noted that <tt>\par</tt> tokens are removed from <tt>\directlua</tt> if they are unexpandable (i.e., most likely, if they have their original meaning). Hence, empty lines can be used in Lua code.

== Line ends ==

When TeX reads a file, it normally turns line ends into spaces. That means that what looks like several lines is actually fed to the Lua interpreter as one big line. For instance:

<pre>
\directlua{
myvar = 1
anothervar = 2
onelastvar = 3
}
</pre>

amounts to the following, if it were written in a separate Lua file:

<pre>
myvar = 1 anothervar = 2 onelastvar = 3
</pre>

That is perfectly legitimate, but strange things might happen. First, TeX macros gobble spaces as usual. Hence:

<pre>
\def\macro{1}
\directlua{
myvar = \macro
anothervar = 2
}
</pre>

will be fed to the interpreter as

<pre>
myvar = 1anothervar = 2
</pre>

which is not legitimate at all. Second, the Lua comment <tt>--</tt> will affect everything to the end of the <tt>\directlua</tt> call. That is:

<pre>
\directlua{
myvar = 1
-- anothervar = 2
onelastvar = 3
}
</pre>

will be processed as

<pre>
myvar = 1 -- anothervar = 2 onelastvar = 3
</pre>

which works but only defines <tt>myvar</tt>. Third, when reporting error, the Lua interpreter will always mention the line number as 1, since it processes one big line only; that isn't extremely useful when the code is large.

The solution is to set <tt>\endlinechar=10</tt> or <tt>\catcode`\^^M=12</tt>. In both cases, line ends will be preserved and the code will be processed as it is input.

== Special characters ==

In TeX, some characters have a special behavior. That must be taken into account when writing Lua code: one must change their catcodes beforehand if one wants to handle them as Lua would, as has just been done for line ends. That means that <tt>\directlua</tt>, as such, is clearly insufficient to write any extended chunk of code. It is thus better to devise a special macro that sets the catcodes to the appropriate values, reads the Lua code, feeds it to <tt>\directlua</tt>, and restores the catcodes. The following code does the job:

<pre>
\def\luacode{%
\bgroup
\catcode`\{=12
\catcode`\}=12
\catcode`\^^M=12
\catcode`\#=12
\catcode`\~=12
\catcode`\%=12
\doluacode
}

\bgroup
\catcode`\^^M=12 %
\long\gdef\doluacode#1^^M#2\endluacode{\directlua{#2}\egroup}%
\egroup
</pre>

Note that not all special characters are set to normal (catcode 12) characters; that is explained for each below. Note also that <tt>\doluacode</tt>, internally called by <tt>\luacode</tt>, is defined to get rid of anything up to the line end, and then pass anything up to <tt>\endluacode</tt> to <tt>\directlua</tt>. Discarding what follows <tt>\luacode</tt> is important, otherwise a simple code as

<pre>
\luacode
myvar = 1
\endluacode
</pre>

would actually create two lines, the first being empty; it is annoying because errors are then reported with the wrong line number (i.e. any error in this one-line code would be reported to happen on line 2).

However, the rest of the line after <tt>\luacode</tt> could also be processed, instead of discarded, to manage special effects (e.g. specifying a chunk's name, storing the code in a control sequence, or even setting which catcodes should be changed or not).

=== Backslash ===

The backslash in TeX is used to form control sequences. In the definition of <tt>\luacode</tt> above, it isn't changed and thus behaves as usual. It allows commands to be passed and expanded to the Lua code. Anyway a backslash in Lua is also an escape character in strings. Hence, if one wants to store the name of a macro in Lua code, the following won't work:

<pre>
\luacode
myvar = "\noexpand\macro"
\endluacode
</pre>

because to the Lua interpreter the string is made of <tt>\m</tt> followed by <tt>acro</tt>; since <tt>\m</tt> is not defined in Lua, the string is read as <tt>macro</tt>, but in other circumstances strange things might happen: for instance, <tt>\n</tt> is a newline. The proper way to pass a macro verbatim is:

<pre>
\luacode
myvar = "\noexpand\\macro"
\endluacode
</pre>

which Lua will correctly read as

<pre>
myvar = "\\macro"
</pre>

with the backslash escaped to represent itself. Another solution is:

<pre>
myvar = [[\noexpand\macro]]
</pre>

because the double brackets signals a string in Lua where no escape sequence occurs (and the string can also run on several lines). Note however that in the second case <tt>myvar</tt> will be defined with a trailing space, i.e. as <tt>"\macro "</tt>, because of TeX's habit to append a trailing space to unexpanded (or unexpandable) control sequences.

If one wants the backslash to be treated as any other character (not creating control sequences), then one has to rewrite <tt>\luacode</tt> as follows:

<pre>
\def\luacode{%
\bgroup
\catcode`\\=12
\catcode`\{=12
\catcode`\}=12
\catcode`\^^M=12
\catcode`\#=12
\catcode`\~=12
\catcode`\%=12
\doluacode
}

\bgroup
\catcode`\|=0
\catcode`\^^M=12 %
\catcode`\\=12 %
|long|gdef|doluacode#1^^M#2\endluacode{|directlua{#2}|egroup}%
|egroup
</pre>

Backslashes remain escape characters in Lua, though.

=== Braces ===

One may want to define a string in Lua which contains unbalanced braces, i.e.:

<pre>
\luacode
myvar = "{"
\endluacode
</pre>

If the braces' catcodes hadn't been changed beforehand, that would be impossible. Note, however, that this means that one can't feed arguments to commands in the usual way. I.e. the following will produce nothing good:

<pre>
\luacode
myvar = "\dosomething{\macro}"
\endluacode
</pre>

<tt>\dosomething</tt> will be expanded with the left brace (devoid of its usual delimiter-ness) as its argument, and the rest of the line might produce chaos. Thus, one may also choose not to change the catcodes of braces, depending on how <tt>\luacode</tt> is most likely to be used. Note that strings with unbalanced braces can still be defined, even if braces have their usual catcodes, thanks to the following trick:

<pre>
\luacode
myvar = "{" -- }
\endluacode
</pre>

When the code is passed to <tt>\directlua</tt>, braces are balanced because the Lua comment means nothing to TeX; when passed to the Lua interpreter, on the other hand, the right brace is ignored.

=== Hash and comment ===

The hash sign <tt>#</tt> in Lua is the length operator: prefixed to a string or table variable, it returns its length. If its catcode weren't taken care of, LuaTeX would pass to <tt>\directlua</tt> a double hash for each hash, i.e. each <tt>#</tt> would be turned into <tt>##</tt>. That is normal TeX behavior, but unwanted here.

As for the commen sign <tt>%</tt>, it is useful in Lua when manipulating strings. If it weren't escaped it would discard parts of the code when TeX reads it, and a mutilated version of the input would be passed to the Lua interpreter. In turn, discarding a line by commenting it in <tt>\luacode</tt> should be done with the Lua comment <tt>--</tt>.

=== Active characters ===

The <tt>~</tt> character is generally active and used as a no-break space in TeX. It it were passed as is to <tt>\directlua</tt>, it would expand to uninterpretable control sequences, whereas in Lua it is used to form the unequal operator <tt>~=</tt>.

Other possible active characters should be taken care of, but which characters are active is unpredictable; punctuation marks might be so to accommodate special spacing, as with LaTeX's ''babel'' package, but such tricks are unlikely to survive in LuaTeX (cleaner methods exist that add a space before punctuation marks when necessary).

=== Other characters ===

When processing verbatim text in TeX, one generally also changes the catcodes of <tt>$</tt>, <tt>&</tt>, <tt>^</tt>, <tt>_</tt> and the space character, because they too are special. When passed to the Lua interpreter, though, their usual catcodes won't do any harm, that is why they are left unmodified here.

== \luaescapestring ==

Although it can't do all of what's been explained, the <tt>\luaescapestring</tt> command might be useful in some cases: it expands its argument (which must be enclosed in ''real'' braces) fully, then modify it so that dangerous characters are escaped: backslashes, hashes, quotes and line ends. For instance:

<pre>
\def\macro{"\noexpand\foo"}
\luacode
myvar = "\luaescapestring{\macro}"
\endluacode
</pre>

will be passed to Lua as

<pre>
myvar = "\"\\foo \""
</pre>

so that <tt>myvar</tt> is defined as <tt>"\foo "</tt>, with the quotes as parts of it. Note that the trailing space after <tt>\foo</tt> still happens.

= From Lua to TeX =

Inside Lua code, one can pass strings to be processed by TeX with the functions <tt>tex.print()</tt>, <tt>tex.sprint()</tt> and <tt>tex.tprint()</tt>. All such calls are processed at the end of a <tt>\directlua</tt> call, even though they might happen in the middle of the code. This behavior is worth noting because it might be surprising in some cases, although it is generally harmless.

== tex.print() ==

This function receives as its argument(s) either one or more strings or an array of strings. Each string is processed as an input line: an end-of-line character is appended (except to the last string), and TeX is in state <tt>newline</tt> when processing it (i.e. leading spaces are skipped). Hence the two equivalent calls:

<pre>
tex.print("a", "b")
tex.print({"a", "b"})
</pre>

are both interpreted by TeX as would the following two lines:

<pre>
a
b
</pre>

Thus `a b' is produced, since line ends normally produce a space.

The function can also take an optional number as its first argument; it is interpreted as referring to a catcode table (as defined by <tt>\initcatcodetable</tt> and <tt>\savecatcodetable</tt>), and each line is processed by TeX with that catcode regime. For instance (note that with such a minimal catcode table, braces don't even have their usual values):

<pre>
\bgroup
\initcatcodetable1
\catcode`\_=0
\savecatcodetable1
\egroup

\directlua{tex.print(1, "_TeX")}
</pre>

The string will be read with <tt>_</tt> as an escape character, and thus interpreted as the command commonly known as <tt>\TeX</tt>. The catcode regime holds only for the strings passed to <tt>tex.print()</tt> and the rest of the document isn't affected.

If the optional number is <tt>-1</tt>, or points to an invalid (i.e. undefined) catcode table, then the strings are processed with the current catcodes, as if there was no optional argument. If it is <tt>-2</tt>, then the strings are read as if the result of <tt>\detokenize</tt>: all characters have catcode 12 (i.e. `other', characters that have no function beside representing themselves), except space, which has catcode 10 (as usual).

== tex.sprint() ==

Like <tt>tex.print()</tt>, this function can receive either one or more strings or an array of strings, with an optional number as its first argument pointing to a catcode table. Unlike <tt>tex.print()</tt>, however, each string is processed as if TeX were in the middle of a line and not at the beginning of a new one: spaces aren't skipped, no end-of-line character is added and trailing spaces aren't ignored. Thus:

<pre>
tex.sprint("a", "b")
</pre>

is interpreted by TeX as

<pre>
ab
</pre>

without any space inbetween.

== tex.tprint() ==

This function takes an unlimited number of tables as its arguments; each table must be an array of strings, with the first entry optionally being a number pointing to a catcode table. Then each table is processed as if passed to <tt>tex.sprint()</tt>. Thus:

<pre>
tex.tprint({1, "a", "b"}, {"c", "d"})
</pre>

is equivalent to

<pre>
tex.sprint(1, "a", "b")
tex.sprint("c", "d")
</pre>

= The expansion of \directlua =

A call to <tt>\directlua</tt> is fully expandable; i.e. it can be used in contexts where full expansion is required, as in:

<pre>
\csname\directlua{tex.print("TeX")}\endcsname
</pre>

which is a somewhat convoluted way of saying <tt>\TeX</tt>. Besides, since Lua code is processed at once, things that were previously unthinkable can now be done easily. For instance, it is impossible to perform an assignment in an <tt>\edef</tt> by TeX's traditional means. I.e. the following:

<pre>
\edef\macro{\count1=5}
</pre>

defines <tt>\macro</tt> as <tt>\count1=5</tt> but doesn't perform the assignment (the <tt>\edef</tt> does nothing more than a simple <tt>\def</tt>). After the definition, the value of <tt>\count1</tt> hasn't changed. The same is not true, though, if such an assigment is made with Lua code. The following:

<pre>
\edef\macro{\directlua{tex.count[1] = 5}}
</pre>

defines <tt>\macro</tt> emptily (since nothing remains after <tt>\directlua</tt> has been processed) ''and'' sets count 1 to 5. Since such a behavior is totally unexpected in normal TeX, one should be wary when using <tt>\directlua</tt> in such contexts.

Callbacks

2011-01-17T17:16:28Z

Paul: Added reference to discussion page.

Callbacks allow the user to interact with TeX's internal operations; those can be enhanced by additional ones, but also replaced altogether. All callbacks are written in Lua.

Since functions registered in callbacks occur in a sequence of actions that makes up TeX's processing, many of them will receive arguments (and thus should be defined to handle them) and, most importantly, many are supposed to return values. What arguments are passed, if any, and what should be return, vary from callback to callback.

= Registering a callback =

To install a function in a callback one must use the <tt>callback.register()</tt> function, whose syntax is:

<pre>
callback.register(callback_name, function)
</pre>

where <tt>callback_name</tt> is a string representing a predefined callback name (see below), and <tt>function</tt> is either a function variable or an anonymous function. The former is just the name of a function previously defined, for instance:

<pre>
local my_function = function (an_arg)
do_this_or_that(an_arg)
end
callback.register("some_callback", my_function)
</pre>

An anonymous function is a function definition that is not assigned to a variable, e.g.:

<pre>
callback.register("some_callback", function (an_arg) do_this_or_that(an_arg) end)
</pre>

Even in the case of a function variable, what LuaTeX registers is the function itself, and any further assignment to the variable has no effect on the callback.

If registering is successful, <tt>callback.register()</tt> returns a number, which is the internal representation of the callback; if registering fails (for instance because a callback doesn't exist, or what is registered isn't a function), the function returns <tt>nil</tt> and a string representing an error message.

Besides functions, <tt>callback.register()</tt> can take <tt>nil</tt> or <tt>false</tt>. The first value clears the callback, and returns to the default behavior (i.e. either nothing or TeX's default processing); the second value prevents anything from happening at this point, even TeX's default operations, and should be used with great care (the benefits are minor speed gain).

== Several functions in a callback ==

Each call to <tt>callback.register()</tt> erases any function previously registered. I.e. after

<pre>
callback.register("some_callback", function_one)
callback.register("some_callback", function_two)
</pre>

only <tt>function_two</tt> is registered in <tt>some_callback</tt>. Hence, if one wants several successive functions to occur in a callback, <tt>callback.register()</tt> is clearly insufficient. Instead, one must register a metafunction which calls the functions one after the other. This is taken care of in ConTeXt, and there exists <tt>luatexbase-mcb</tt> for plain TeX and LaTeX, but here's how to do it oneself. We'll take the <tt>process_input_buffer</tt> callback has an example; it is called whenever TeX reads an input line, it receives a string (the line) and should return one. A rough approach would be:

<pre>
local function_table = { }

local function metafunction (str)
for _, fct in ipairs(function_table) do
str = fct(str)
end
return str
end

function add_function (fct)
table.insert(function_table, fct)
end

callback.register("process_input_buffer", metafunction)
</pre>

Then one can call <tt>add_function(my_function)</tt> to add <tt>my_function</tt> to the list of functions registered in <tt>process_input_buffer</tt>, and <tt>metafunction</tt> will pass the callback's argument (an input line) to the first function, set that argument to what the function returns, then repeat the process for all functions and finally return the resulting string.

If one also wants to be able to remove functions (and not the entire metafunction), a subtler approach is required, where names are associated with functions and can be used as handles.

= Information about callbacks =

LuaTeX has two functions to get some information about callbacks. The first is <tt>callback.list()</tt>, called without arguments and returning a table with the callback names as keys and booleans as values: if the value is <tt>true</tt>, it means that something is registered in the callback, or that it is set to <tt>false</tt> (see above). The other function is <tt>callback.find()</tt>, called with a callback name. If a function is registered, it is returned; if the callback is set to <tt>false</tt>, a boolean with that value is returned; finally, if nothing is registered, <tt>nil</tt> is returned (note that to distinguish between a boolean set to <tt>false</tt> and <tt>nil</tt>, you must use <tt>type()</tt>).

= List of callbacks =

For suggestions on how to document callbacks, see [[Talk:Callbacks|the discussion page]].

== File discovery callbacks ==

These are called whenever an input file is needed. Each receives the <tt>asked_name</tt>, a string representing what file name was called by the user (for instance, <tt>my_file.tex</tt> in <tt>\input my_file.tex</tt>) and should return the name of a file to be read (or written to). The <tt>id_number</tt> in the first two callbacks is the stream's number plus one, because 0 denotes <tt>log</tt> and <tt>\input</tt> files.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[find_read_file]] ||id_number, asked_name ||actual_name ||File called with <tt>\read</tt> or <tt>\input</tt>
|-
|[[find_write_file]] ||id_number, asked_name ||actual_name ||<tt>log</tt> file or file called with <tt>\write</tt>
|-
|[[find_font_file]] ||asked_name ||actual_name
|-
|[[find_output_file]] ||asked_name ||actual_name
|-
|[[find_format_file]] ||asked_name ||actual_name
|-
|[[find_vf_file]] ||asked_name ||actual_name
|-
|[[find_map_file]] ||asked_name ||actual_name
|-
|[[find_enc_file]] ||asked_name ||actual_name
|-
|[[find_sfd_file]] ||asked_name ||actual_name
|-
|[[find_pk_file]] ||asked_name ||actual_name
|-
|[[find_data_file]] ||asked_name ||actual_name
|-
|[[find_opentype_file]] ||asked_name ||actual_name
|-
|[[find_truetype_file]] ||asked_name ||actual_name
|-
|[[find_type_file]] ||asked_name ||actual_name
|-
|[[find_image_file]] ||asked_name ||actual_name
|}

== File reading callbacks ==

The previous callbacks were called when a file was asked for; the following are triggered when the file is read. In all, <tt>filename</tt> is the string returned by the file discovery callbacks. The first callback reads <tt>\input</tt> or <tt>\read</tt> files; it must return a table, whose first cell is a function that will be called whenever a line from the input file is readed; the second cell is an optional function executed when the input file terminates.

The other callbacks are used for binary files: <tt>success</tt> is a boolean, <tt>data</tt> is a string and <tt>data_size</tt> is a number (the length of <tt>data</tt> in bytes).

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[open_read_file]] ||filename ||{reader, close}
|-
|[[read_font_file]] ||filename ||success, data, data_size
|-
|[[read_vf_file]] ||filename ||success, data, data_size
|-
|[[read_map_file]] ||filename ||success, data, data_size
|-
|[[read_enc_file]] ||filename ||success, data, data_size
|-
|[[read_sfd_file]] ||filename ||success, data, data_size
|-
|[[read_pk_file]] ||filename ||success, data, data_size
|-
|[[read_data_file]] ||filename ||success, data, data_size
|-
|[[read_truetype_file]] ||filename ||success, data, data_size
|-
|[[read_type1_file]] ||filename ||success, data, data_size
|-
|[[read_opentype_file]] ||filename ||success, data, data_size
|}

== Data processing callbacks ==

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[process_input_buffer]] ||line ||modified_line ||Called for each input line (after <tt>reader</tt> in <tt>open_read_file</tt> if TeX is not reading the main file).
|-
|[[process_output_buffer]] ||line ||modified_line ||Called for each line TeX writes to an external file; this excludes the <tt>log</tt> file, the terminal and <tt>\write18</tt> calls.
|-
|[[token_filter]] || ||token ||Called when TeX needs token; no argument is passed to the callback, the next token must be fetched with <tt>token.get_next()</tt>; what is returned is immediately processed by TeX.
|}

== Node list processing callbacks ==

These callbacks deal with node manipulations and occur at various stages of TeX's typesetting processes.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[buildpage_filter]] ||information || ||Called when TeX moves material to the main vertical list. (This callback is an endangered species.)
|-
|[[hyphenate]] ||head, tail || || Inserts discretionary nodes in the node list; called in restricted and unrestricted horizontal mode (i.e. <tt>\hbox</tt>'es and paragraph building). A function doing what TeX does here by default is <tt>lang.hyphenate()</tt>.
|-
|[[ligaturing]] ||head, tail || || Same as <tt>hyphenate</tt>, but applying ligatures (and adding some discretionaries too). An associated function is <tt>node.ligaturing()</tt>.
|-
|[[kerning]] ||head, tail || || Same as above, with font kerns. The associated function is <tt>node.kerning()</tt>.
|-
|[[pre_linebreak_filter]] ||head, context ||boolean or newhead ||Called in unrestricted horizontal mode, after <tt>kerning</tt>, and before calling the paragraph builder; it receives the list of nodes that is the contents of the paragraph (plus the <tt>\parfillskip</tt> glue) and must return something similar.
|-
|[[linebreak_filter]] ||head, is_display ||newhead ||The paragraph builder. It receives what the previous callback has returned and is in charge of making a paragraph out of it; it must return what it has built. An associated function is <tt>tex.linebreak()</tt>.
|-
|[[Show the hyphenation points|post_linebreak_filter]] ||head, groupcode, ||boolean or newhead ||Receives what has been returned by <tt>linebreak_filter</tt> and must return something similar.
|-
|[[hpack_filter]] ||head, context, size, packtype[, dir] ||boolean or newhead ||Called when TeX is going to build an <tt>\hbox</tt>.
|-
|[[vpack_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Called when TeX is going to build a <tt>\vbox</tt>.
|-
|[[pre_output_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Same as <tt>vpack_filter</tt>, but when TeX packs box 255 (or any other value of <tt>\outputbox</tt>) and goes to the output routine.
|-
|[[mlist_to_hlist]] ||head, display_type, need_penalties ||newhead ||Called when TeX turns a math list into a node list.
|}

== Information reporting callbacks ==

These callbacks are called in various places when TeX has something to say. None of them takes an argument nor returns anything.

{| cellpadding="5"
!Name !! Short description
|-
|[[pre_dump]] ||Called before dumping a format.
|-
|[[start_run]] ||LuaTeX's banner (`This is LuaTeX...'); must be changed in the initialization script, otherwise the banner has already been printed.
|-
|[[stop_run]] ||TeX's final words (statistics and `Output written to...').
|-
|[[start_page_number]] ||The opening bracket and the page number printed when a page is shipped out.
|-
|[[stop_page_number]] ||The closing bracket printed after the previous callback.
|-
|[[show_error_hook]] || Called at the beginning of each error message.
|}

== Miscellaneous ==
{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
||[[finish_pdffile]] || || ||Called when all pages have been written to the PDF file.
|-
|[[define_font]] ||name, size, id ||font_table ||Called when the <tt>\font</tt> command is executed; <tt>name</tt> is the filename asked for, <tt>size</tt> is the <tt>at</tt> size, <tt>id</tt> is the number that will be assigned as an internal representation to the font thus loaded. The callback should return a table representing a font.
|}

Talk:Callbacks

2011-01-17T17:09:56Z

Paul: Proposed guidelines for articles on callbacks.

Suggested guidelines to document callbacks:

* Have an article for each callback, with a minimal example.

* Let a page's title be the name of the callback it describes, so it is automatically linked to this page, meant as a general introduction. If the page title differs, change the link here from <pre>[[callback_name]]</pre> to <pre>[[page_title|callback_name]]</pre>

* Link all pages on callbacks to this general introduction, which should gather general information about callbacks (e.g. no need to reexplain <tt>callback.register()</tt> on each page).

* Make code low-level, which means: no format-specific tricks, and nothing but <tt>callback.register()</tt> (we can redirect to the general introduction for discussion about the limitations of <tt>callback.register()</tt>). Basically, this means avoiding things like <tt>\documentclass{article}</tt> or even plain TeX's <tt>\bye</tt>. The code should be pure Lua and pure TeX (probably we can allow allocation macros like \newif and \newbox, though, since they are common to all formats), so the reader can simply paste it and process it as s/he prefers.

* Commented code is better than pure code, although pure code is better than no code.

Callbacks

2011-01-17T12:13:37Z

Paul: Introduction to callbacks and list of callbacks.

Callbacks allow the user to interact with TeX's internal operations; those can be enhanced by additional ones, but also replaced altogether. All callbacks are written in Lua.

Since functions registered in callbacks occur in a sequence of actions that makes up TeX's processing, many of them will receive arguments (and thus should be defined to handle them) and, most importantly, many are supposed to return values. What arguments are passed, if any, and what should be return, vary from callback to callback.

= Registering a callback =

To install a function in a callback one must use the <tt>callback.register()</tt> function, whose syntax is:

<pre>
callback.register(callback_name, function)
</pre>

where <tt>callback_name</tt> is a string representing a predefined callback name (see below), and <tt>function</tt> is either a function variable or an anonymous function. The former is just the name of a function previously defined, for instance:

<pre>
local my_function = function (an_arg)
do_this_or_that(an_arg)
end
callback.register("some_callback", my_function)
</pre>

An anonymous function is a function definition that is not assigned to a variable, e.g.:

<pre>
callback.register("some_callback", function (an_arg) do_this_or_that(an_arg) end)
</pre>

Even in the case of a function variable, what LuaTeX registers is the function itself, and any further assignment to the variable has no effect on the callback.

If registering is successful, <tt>callback.register()</tt> returns a number, which is the internal representation of the callback; if registering fails (for instance because a callback doesn't exist, or what is registered isn't a function), the function returns <tt>nil</tt> and a string representing an error message.

Besides functions, <tt>callback.register()</tt> can take <tt>nil</tt> or <tt>false</tt>. The first value clears the callback, and returns to the default behavior (i.e. either nothing or TeX's default processing); the second value prevents anything from happening at this point, even TeX's default operations, and should be used with great care (the benefits are minor speed gain).

== Several functions in a callback ==

Each call to <tt>callback.register()</tt> erases any function previously registered. I.e. after

<pre>
callback.register("some_callback", function_one)
callback.register("some_callback", function_two)
</pre>

only <tt>function_two</tt> is registered in <tt>some_callback</tt>. Hence, if one wants several successive functions to occur in a callback, <tt>callback.register()</tt> is clearly insufficient. Instead, one must register a metafunction which calls the functions one after the other. This is taken care of in ConTeXt, and there exists <tt>luatexbase-mcb</tt> for plain TeX and LaTeX, but here's how to do it oneself. We'll take the <tt>process_input_buffer</tt> callback has an example; it is called whenever TeX reads an input line, it receives a string (the line) and should return one. A rough approach would be:

<pre>
local function_table = { }

local function metafunction (str)
for _, fct in ipairs(function_table) do
str = fct(str)
end
return str
end

function add_function (fct)
table.insert(function_table, fct)
end

callback.register("process_input_buffer", metafunction)
</pre>

Then one can call <tt>add_function(my_function)</tt> to add <tt>my_function</tt> to the list of functions registered in <tt>process_input_buffer</tt>, and <tt>metafunction</tt> will pass the callback's argument (an input line) to the first function, set that argument to what the function returns, then repeat the process for all functions and finally return the resulting string.

If one also wants to be able to remove functions (and not the entire metafunction), a subtler approach is required, where names are associated with functions and can be used as handles.

= Information about callbacks =

LuaTeX has two functions to get some information about callbacks. The first is <tt>callback.list()</tt>, called without arguments and returning a table with the callback names as keys and booleans as values: if the value is <tt>true</tt>, it means that something is registered in the callback, or that it is set to <tt>false</tt> (see above). The other function is <tt>callback.find()</tt>, called with a callback name. If a function is registered, it is returned; if the callback is set to <tt>false</tt>, a boolean with that value is returned; finally, if nothing is registered, <tt>nil</tt> is returned (note that to distinguish between a boolean set to <tt>false</tt> and <tt>nil</tt>, you must use <tt>type()</tt>).

= List of callbacks =

== File discovery callbacks ==

These are called whenever an input file is needed. Each receives the <tt>asked_name</tt>, a string representing what file name was called by the user (for instance, <tt>my_file.tex</tt> in <tt>\input my_file.tex</tt>) and should return the name of a file to be read (or written to). The <tt>id_number</tt> in the first two callbacks is the stream's number plus one, because 0 denotes <tt>log</tt> and <tt>\input</tt> files.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[find_read_file]] ||id_number, asked_name ||actual_name ||File called with <tt>\read</tt> or <tt>\input</tt>
|-
|[[find_write_file]] ||id_number, asked_name ||actual_name ||<tt>log</tt> file or file called with <tt>\write</tt>
|-
|[[find_font_file]] ||asked_name ||actual_name
|-
|[[find_output_file]] ||asked_name ||actual_name
|-
|[[find_format_file]] ||asked_name ||actual_name
|-
|[[find_vf_file]] ||asked_name ||actual_name
|-
|[[find_map_file]] ||asked_name ||actual_name
|-
|[[find_enc_file]] ||asked_name ||actual_name
|-
|[[find_sfd_file]] ||asked_name ||actual_name
|-
|[[find_pk_file]] ||asked_name ||actual_name
|-
|[[find_data_file]] ||asked_name ||actual_name
|-
|[[find_opentype_file]] ||asked_name ||actual_name
|-
|[[find_truetype_file]] ||asked_name ||actual_name
|-
|[[find_type_file]] ||asked_name ||actual_name
|-
|[[find_image_file]] ||asked_name ||actual_name
|}

== File reading callbacks ==

The previous callbacks were called when a file was asked for; the following are triggered when the file is read. In all, <tt>filename</tt> is the string returned by the file discovery callbacks. The first callback reads <tt>\input</tt> or <tt>\read</tt> files; it must return a table, whose first cell is a function that will be called whenever a line from the input file is readed; the second cell is an optional function executed when the input file terminates.

The other callbacks are used for binary files: <tt>success</tt> is a boolean, <tt>data</tt> is a string and <tt>data_size</tt> is a number (the length of <tt>data</tt> in bytes).

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[open_read_file]] ||filename ||{reader, close}
|-
|[[read_font_file]] ||filename ||success, data, data_size
|-
|[[read_vf_file]] ||filename ||success, data, data_size
|-
|[[read_map_file]] ||filename ||success, data, data_size
|-
|[[read_enc_file]] ||filename ||success, data, data_size
|-
|[[read_sfd_file]] ||filename ||success, data, data_size
|-
|[[read_pk_file]] ||filename ||success, data, data_size
|-
|[[read_data_file]] ||filename ||success, data, data_size
|-
|[[read_truetype_file]] ||filename ||success, data, data_size
|-
|[[read_type1_file]] ||filename ||success, data, data_size
|-
|[[read_opentype_file]] ||filename ||success, data, data_size
|}

== Data processing callbacks ==

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[process_input_buffer]] ||line ||modified_line ||Called for each input line (after <tt>reader</tt> in <tt>open_read_file</tt> if TeX is not reading the main file).
|-
|[[process_output_buffer]] ||line ||modified_line ||Called for each line TeX writes to an external file; this excludes the <tt>log</tt> file, the terminal and <tt>\write18</tt> calls.
|-
|[[token_filter]] || ||token ||Called when TeX needs token; no argument is passed to the callback, the next token must be fetched with <tt>token.get_next()</tt>; what is returned is immediately processed by TeX.
|}

== Node list processing callbacks ==

These callbacks deal with node manipulations and occur at various stages of TeX's typesetting processes.

{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
|[[buildpage_filter]] ||information || ||Called when TeX moves material to the main vertical list. (This callback is an endangered species.)
|-
|[[hyphenate]] ||head, tail || || Inserts discretionary nodes in the node list; called in restricted and unrestricted horizontal mode (i.e. <tt>\hbox</tt>'es and paragraph building). A function doing what TeX does here by default is <tt>lang.hyphenate()</tt>.
|-
|[[ligaturing]] ||head, tail || || Same as <tt>hyphenate</tt>, but applying ligatures (and adding some discretionaries too). An associated function is <tt>node.ligaturing()</tt>.
|-
|[[kerning]] ||head, tail || || Same as above, with font kerns. The associated function is <tt>node.kerning()</tt>.
|-
|[[pre_linebreak_filter]] ||head, context ||boolean or newhead ||Called in unrestricted horizontal mode, after <tt>kerning</tt>, and before calling the paragraph builder; it receives the list of nodes that is the contents of the paragraph (plus the <tt>\parfillskip</tt> glue) and must return something similar.
|-
|[[linebreak_filter]] ||head, is_display ||newhead ||The paragraph builder. It receives what the previous callback has returned and is in charge of making a paragraph out of it; it must return what it has built. An associated function is <tt>tex.linebreak()</tt>.
|-
|[[Show the hyphenation points|post_linebreak_filter]] ||head, groupcode, ||boolean or newhead ||Receives what has been returned by <tt>linebreak_filter</tt> and must return something similar.
|-
|[[hpack_filter]] ||head, context, size, packtype[, dir] ||boolean or newhead ||Called when TeX is going to build an <tt>\hbox</tt>.
|-
|[[vpack_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Called when TeX is going to build a <tt>\vbox</tt>.
|-
|[[pre_output_filter]] ||head, context, size, packtype, maxdepth[, dir] ||boolean or newhead ||Same as <tt>vpack_filter</tt>, but when TeX packs box 255 (or any other value of <tt>\outputbox</tt>) and goes to the output routine.
|-
|[[mlist_to_hlist]] ||head, display_type, need_penalties ||newhead ||Called when TeX turns a math list into a node list.
|}

== Information reporting callbacks ==

These callbacks are called in various places when TeX has something to say. None of them takes an argument nor returns anything.

{| cellpadding="5"
!Name !! Short description
|-
|[[pre_dump]] ||Called before dumping a format.
|-
|[[start_run]] ||LuaTeX's banner (`This is LuaTeX...'); must be changed in the initialization script, otherwise the banner has already been printed.
|-
|[[stop_run]] ||TeX's final words (statistics and `Output written to...').
|-
|[[start_page_number]] ||The opening bracket and the page number printed when a page is shipped out.
|-
|[[stop_page_number]] ||The closing bracket printed after the previous callback.
|-
|[[show_error_hook]] || Called at the beginning of each error message.
|}

== Miscellaneous ==
{| cellpadding="5"
!Name !! Argument(s) !! Return value(s) !! Short description
|-
||[[finish_pdffile]] || || ||Called when all pages have been written to the PDF file.
|-
|[[define_font]] ||name, size, id ||font_table ||Called when the <tt>\font</tt> command is executed; <tt>name</tt> is the filename asked for, <tt>size</tt> is the <tt>at</tt> size, <tt>id</tt> is the number that will be assigned as an internal representation to the font thus loaded. The callback should return a table representing a font.
|}

Writing Lua in TeX

2010-12-18T21:59:50Z

Paul: Added description of tex.print() and friends, and a note on the expansion of \directlua.

= Embedding Lua code in a TeX document =

Although it is simpler to put Lua code in Lua files, from time to time one may want or need to go Lua in the middle of a document. To this end, LuaTeX has two commands: <tt>\directlua</tt> and <tt>\latelua</tt>. They work the same, except <tt>\latelua</tt> is processed when the page where it appears is shipped out, whereas <tt>\directlua</tt> is processed at once; the distinction is immaterial here, and what is said of <tt>\directlua</tt> also applies to <tt>\latelua</tt>.

<tt>\directlua</tt> can be called in three ways:

<pre>
\directlua {<lua code>}
\directlua name {<name>} {<lua code>}
\directlua <number> {<lua code>}
</pre>

Those three ways are equivalent when it comes to process <tt><lua code></tt>, but in the second case processing will occur in a chunk named <tt><name></tt>, and in the third it will occur in a chunk whose name is the entry <tt><number></tt> in the table <tt>lua.name</tt>. The difference manifests itself only when errors occur, in which case the name of the chunk, if any, is reported.

Each call to <tt>\directlua</tt>, named or not, is processed in a separate chunk. That means that any <tt>local</tt> variable is defined for this call only and is lost afterward. Hence:

<pre>
\directlua{
one = 1
local two = 2
}
\directlua{
texio.write_nl(type(one))
texio.write_nl(type(two))
}
</pre>

will report <tt>number</tt> and <tt>nil</tt> (<tt>texio.write_nl</tt> writes to the log file). On the other hand, Lua code is completely insensitive to TeX's grouping mechanism. In other words, calling <tt>\directlua</tt> between <tt>\bgroup</tt> and <tt>\egroup</tt> doesn't affect the code to be processed.

= TeX catcodes in Lua =

By default, the code passed to <tt>\directlua</tt> is treated as normal TeX input and only then sent to the Lua interpreter. This may lead to unwanted results and must be acknowledged.

== Expansion ==

As with any other special, the code in <tt>\directlua</tt> (and the <tt><name></tt>, if specifed) is fully expanded. This means that macros can be safely passed to <tt>\directlua</tt> if one wants their values, but that they should also be properly escaped when needed. For instance:

<pre>
\def\macro{1}
\directlua{
myvar = \macro
}
</pre>

defines <tt>myvar</tt> as the number <tt>1</tt>. To store the control sequence <tt>\macro</tt> instead, another course of action is needed: see the [[#Backslash|section on backslash]] below.

== Line ends ==

When TeX reads a file, it normally turns line ends into spaces. That means that what looks like several lines is actually fed to the Lua interpreter as one big line. For instance:

<pre>
\directlua{
myvar = 1
anothervar = 2
onelastvar = 3
}
</pre>

amounts to the following, if it were written in a separate Lua file:

<pre>
myvar = 1 anothervar = 2 onelastvar = 3
</pre>

That is perfectly legitimate, but strange things might happen. First, TeX macros gobble spaces as usual. Hence:

<pre>
\def\macro{1}
\directlua{
myvar = \macro
anothervar = 2
}
</pre>

will be fed to the interpreter as

<pre>
myvar = 1anothervar = 2
</pre>

which is not legitimate at all. Second, the Lua comment <tt>--</tt> will affect everything to the end of the <tt>\directlua</tt> call. That is:

<pre>
\directlua{
myvar = 1
-- anothervar = 2
onelastvar = 3
}
</pre>

will be processed as

<pre>
myvar = 1 -- anothervar = 2 onelastvar = 3
</pre>

which works but only defines <tt>myvar</tt>. Third, when reporting error, the Lua interpreter will always mention the line number as 1, since it processes one big line only; that isn't extremely useful when the code is large.

The solution is to set <tt>\endlinechar=10</tt> or <tt>\catcode`\^^M=12</tt>. In both cases, line ends will be preserved and the code will be processed as it is input.

== Special characters ==

In TeX, some characters have a special behavior. That must be taken into account when writing Lua code: one must change their catcodes beforehand if one wants to handle them as Lua would, as has just been done for line ends. That means that <tt>\directlua</tt>, as such, is clearly insufficient to write any extended chunk of code. It is thus better to devise a special macro that sets the catcodes to the appropriate values, reads the Lua code, feeds it to <tt>\directlua</tt>, and restores the catcodes. The following code does the job:

<pre>
\def\luacode{%
\bgroup
\catcode`\{=12
\catcode`\}=12
\catcode`\^^M=12
\catcode`\#=12
\catcode`\~=12
\catcode`\%=12
\doluacode
}

\bgroup
\catcode`\^^M=12 %
\long\gdef\doluacode#1^^M#2\endluacode{\directlua{#2}\egroup}%
\egroup
</pre>

Note that not all special characters are set to normal (catcode 12) characters; that is explained for each below. Note also that <tt>\doluacode</tt>, internally called by <tt>\luacode</tt>, is defined to get rid of anything up to the line end, and then pass anything up to <tt>\endluacode</tt> to <tt>\directlua</tt>. Discarding what follows <tt>\luacode</tt> is important, otherwise a simple code as

<pre>
\luacode
myvar = 1
\endluacode
</pre>

would actually create two lines, the first being empty; it is annoying because errors are then reported with the wrong line number (i.e. any error in this one-line code would be reported to happen on line 2).

However, the rest of the line after <tt>\luacode</tt> could also be processed, instead of discarded, to manage special effects (e.g. specifying a chunk's name, storing the code in a control sequence, or even setting which catcodes should be changed or not).

=== Backslash ===

The backslash in TeX is used to form control sequences. In the definition of <tt>\luacode</tt> above, it isn't changed and thus behaves as usual. It allows commands to be passed and expanded to the Lua code. Anyway a backslash in Lua is also an escape character in strings. Hence, if one wants to store the name of a macro in Lua code, the following won't work:

<pre>
\luacode
myvar = "\noexpand\macro"
\endluacode
</pre>

because to the Lua interpreter the string is made of <tt>\m</tt> followed by <tt>acro</tt>; since <tt>\m</tt> is not defined in Lua, the string is read as <tt>macro</tt>, but in other circumstances strange things might happen: for instance, <tt>\n</tt> is a newline. The proper way to pass a macro verbatim is:

<pre>
\luacode
myvar = "\noexpand\\macro"
\endluacode
</pre>

which Lua will correctly read as

<pre>
myvar = "\\macro"
</pre>

with the backslash escaped to represent itself. Another solution is:

<pre>
myvar = [[\noexpand\macro]]
</pre>

because the double brackets signals a string in Lua where no escape sequence occurs (and the string can also run on several lines). Note however that in the second case <tt>myvar</tt> will be defined with a trailing space, i.e. as <tt>"\macro "</tt>, because of TeX's habit to append a trailing space to unexpanded (or unexpandable) control sequences.

=== Braces ===

One may want to define a string in Lua which contains unbalanced braces, i.e.:

<pre>
\luacode
myvar = "{"
\endluacode
</pre>

If the braces' catcodes hadn't been changed beforehand, that would be impossible. Note, however, that this means that one can't feed arguments to commands in the usual way. I.e. the following will produce nothing good:

<pre>
\luacode
myvar = "\dosomething{\macro}"
\endluacode
</pre>

<tt>\dosomething</tt> will be expanded with the left brace (devoid of its usual delimiter-ness) as its argument, and the rest of the line might produce chaos. Thus, one may also choose not to change the catcodes of braces, depending on how <tt>\luacode</tt> is most likely to be used. Note that strings with unbalanced braces can still be defined, even if braces have their usual catcodes, thanks to the following trick:

<pre>
\luacode
myvar = "{" -- }
\endluacode
</pre>

When the code is passed to <tt>\directlua</tt>, braces are balanced because the Lua comment means nothing to TeX; when passed to the Lua interpreter, on the other hand, the right brace is ignored.

=== Hash and comment ===

The hash sign <tt>#</tt> in Lua is the length operator: prefixed to a string or table variable, it returns its length. If its catcode weren't taken care of, LuaTeX would pass to <tt>\directlua</tt> a double hash for each hash, i.e. each <tt>#</tt> would be turned into <tt>##</tt>. That is normal TeX behavior, but unwanted here.

As for the commen sign <tt>%</tt>, it is useful in Lua when manipulating strings. If it weren't escaped it would discard parts of the code when TeX reads it, and a mutilated version of the input would be passed to the Lua interpreter. In turn, discarding a line by commenting it in <tt>\luacode</tt> should be done with the Lua comment <tt>--</tt>.

=== Active characters ===

The <tt>~</tt> character is generally active and used as a no-break space in TeX. It it were passed as is to <tt>\directlua</tt>, it would expand to uninterpretable control sequences, whereas in Lua it is used to form the unequal operator <tt>~=</tt>.

Other possible active characters should be taken care of, but which characters are active is unpredictable; punctuation marks might be so to accommodate special spacing, as with LaTeX's ''babel'' package, but such tricks are unlikely to survive in LuaTeX (cleaner methods exist that add a space before punctuation marks when necessary).

=== Other characters ===

When processing verbatim text in TeX, one generally also changes the catcodes of <tt>$</tt>, <tt>&</tt>, <tt>^</tt>, <tt>_</tt> and the space character, because they too are special. When passed to the Lua interpreter, though, their usual catcodes won't do any harm, that is why they are left unmodified here.

== \luaescapestring ==

Although it can't do all of what's been explained, the <tt>\luaescapestring</tt> command might be useful in some cases: it expands its argument (which must be enclosed in ''real'' braces) fully, then modify it so that dangerous characters are escaped: backslashes, hashes, quotes and line ends. For instance:

<pre>
\def\macro{"\noexpand\foo"}
\luacode
myvar = "\luaescapestring{\macro}"
\endluacode
</pre>

will be passed to Lua as

<pre>
myvar = "\"\\foo \""
</pre>

so that <tt>myvar</tt> is defined as <tt>"\foo "</tt>, with the quotes as parts of it. Note that the trailing space after <tt>\foo</tt> still happens.

= From Lua to TeX =

Inside Lua code, one can pass strings to be processed by TeX with the functions <tt>tex.print()</tt>, <tt>tex.sprint()</tt> and <tt>tex.tprint()</tt>. All such calls are processed at the end of a <tt>\directlua</tt> call, even though they might happen in the middle of the code. This behavior is worth noting because it might be surprising in some cases, although it is generally harmless.

== tex.print() ==

This function receives as its argument(s) either one or more strings or an array of strings. Each string is processed as an input line: an end-of-line character is appended (except to the last string), and TeX is in state <tt>newline</tt> when processing it (i.e. leading spaces are skipped). Hence the two equivalent calls:

<pre>
tex.print("a", "b")
tex.print({"a", "b"})
</pre>

are both interpreted by TeX as would the following two lines:

<pre>
a
b
</pre>

Thus `a b' is produced, since line ends normally produce a space.

The function can also take an optional number as its first argument; it is interpreted as referring to a catcode table (as defined by <tt>\initcatcodetable</tt> and <tt>\savecatcodetable</tt>), and each line is processed by TeX with that catcode regime. For instance (note that with such a minimal catcode table, braces don't even have their usual values):

<pre>
\bgroup
\initcatcodetable1
\catcode`\_=0
\savecatcodetable1
\egroup

\directlua{tex.print(1, "_TeX")}
</pre>

The string will be read with <tt>_</tt> as an escape character, and thus interpreted as the command commonly known as <tt>\TeX</tt>. The catcode regime holds only for the strings passed to <tt>tex.print()</tt> and the rest of the document isn't affected.

If the optional number is <tt>-1</tt>, or points to an invalid (i.e. undefined) catcode table, then the strings are processed with the current catcodes, as if there was no optional argument. If it is <tt>-2</tt>, then the strings are read as if the result of <tt>\detokenize</tt>: all characters have catcode 12 (i.e. `other', characters that have no function beside representing themselves), except space, which has catcode 10 (as usual).

== tex.sprint() ==

Like <tt>tex.print()</tt>, this function can receive either one or more strings or an array of strings, with an optional number as its first argument pointing to a catcode table. Unlike <tt>tex.print()</tt>, however, each string is processed as if TeX were in the middle of a line and not at the beginning of a new one: spaces aren't skipped, no end-of-line character is added and trailing spaces aren't ignored. Thus:

<pre>
tex.sprint("a", "b")
</pre>

is interpreted by TeX as

<pre>
ab
</pre>

without any space inbetween.

== tex.tprint() ==

This function takes an unlimited number of tables as its arguments; each table must be an array of strings, with the first entry optionally being a number pointing to a catcode table. Then each table is processed as if passed to <tt>tex.sprint()</tt>. Thus:

<pre>
tex.tprint({1, "a", "b"}, {"c", "d"})
</pre>

is equivalent to

<pre>
tex.sprint(1, "a", "b")
tex.sprint("c", "d")
</pre>

= The expansion of \directlua =

A call to <tt>\directlua</tt> is fully expandable; i.e. it can be used in contexts where full expansion is required, as in:

<pre>
\csname\directlua{tex.print("TeX")}\endcsname
</pre>

which is a somewhat convoluted way of saying <tt>\TeX</tt>. Besides, since Lua code is processed at once, things that were previously unthinkable can now be done easily. For instance, it is impossible to perform an assignment in an <tt>\edef</tt> by TeX's traditional means. I.e. the following:

<pre>
\edef\macro{\count1=5}
</pre>

defines <tt>\macro</tt> as <tt>\count1=5</tt> but doesn't perform the assignment (the <tt>\edef</tt> does nothing more than a simple <tt>\def</tt>). After the definition, the value of <tt>\count1</tt> hasn't changed. The same is not true, though, if such an assigment is made with Lua code. The following:

<pre>
\edef\macro{\directlua{tex.count[1] = 5}}
</pre>

defines <tt>\macro</tt> emptily (since nothing remains after <tt>\directlua</tt> has been processed) ''and'' sets count 1 to 5. Since such a behavior is totally unexpected in normal TeX, one should be wary when using <tt>\directlua</tt> in such contexts.

Writing Lua in TeX

2010-12-18T15:37:57Z

Paul:

Writing Lua in TeX

2010-12-18T14:41:03Z

Paul:

Writing Lua in TeX

2010-12-18T14:39:00Z

Paul: Explain how to use \directlua.

Talk:Use a TrueType font

2010-12-08T09:14:45Z

Paul:

Old location. http://luatex.bluwiki.com/go/Use_a_TrueType_font

I've cleaned up the code, but more is to be done. Some things are useless, and it would be good to have a really minimal code. Other stuffs are missing (e.g. most kerns are in tables not stored with characters).

Use a TrueType font

2010-12-08T09:11:12Z

Paul: Cleaned up the code.

A TrueType (or Openype) font can be used without going through a TFM or an OFM file, like in XeTeX.

The best way to do so is to use the package luaotfload, on the CTAN, and developped here: http://github.com/mpg/luaotfload/

The code shows how we can use the table of a TrueType font to produce an internal TeX font table:

<pre>
\pdfoutput1
\directlua{
callback.register("define_font",
function(name, size)
local fonttype, f

filename = kpse.find_file(name, "opentype fonts")
if filename then
fonttype = "opentype"
else
filename = kpse.find_file(name, "truetype fonts")
end
if filename and not fonttype then
fonttype = "truetype"
end

if fonttype then
if size < 0 then
size = (- 655.36) * size
end
ttffont = fontloader.to_table(fontloader.open(filename))
if ttffont then
f = { }
f.name = ttffont.fontname
f.fullname = ttffont.names[1].names.fullname
f.parameters = { }
f.designsize = size
f.size = size
direction = 0
f.parameters.slant = 0
f.parameters.space = size * 0.25
f.parameters.space_stretch = 0.3 * size
f.parameters.space_shrink = 0.1 * size
f.parameters.x_height = 0.4 * size
f.parameters.quad = 1.0 * size
f.parameters.extra_space = 0
f.characters = { }
local mag = size / ttffont.units_per_em

local names_of_char = { }
for char, glyph in pairs(ttffont.map.map) do
names_of_char[ttffont.glyphs[glyph].name] = ttffont.map.backmap[glyph]
end

for char, glyph in pairs(ttffont.map.map) do
local glyph_table = ttffont.glyphs[glyph]

f.characters[char] = {
index = glyph,
width = glyph_table.width * mag,
name = glyph_table.name }
if glyph_table.boundingbox[4] then
f.characters[char].height = glyph_table.boundingbox[4] * mag
end
if glyph_table.boundingbox[2] then
f.characters[char].depth = -glyph_table.boundingbox[2] * mag
end

if glyph_table.kerns then
local kerns = { }
for _, kern in pairs(glyph_table.kerns) do
kerns[names_of_char[kern.char]] = kern.off * mag
end
f.characters[char].kerns = kerns
end
end

f.filename = filename
f.type = "real"
f.format = fonttype
f.embedding = "subset"
f.cidinfo = {
registry = "Adobe",
ordering = "Identity",
supplement = 0,
version = 1 }
end
else
f = font.read_tfm(name, size)
end
return f
end)
}
</pre>

The index of the <tt>f.characters</tt> table is the Unicode value, among the properties of each entry of that table we have <tt>index</tt> which is the glyph index. Using this property we get the mapping between Unicode characters and glyphs. Kerning is handled via TeX font LKP.

What is missing in the code above is better calculation of standard TeX font parameters and OpenType features.

Use a TrueType font

2010-12-08T09:00:26Z

Paul: Removed the useless definition of names_of_glyph (not used in the code).

A TrueType (or Openype) font can be used without going through a TFM or an OFM file, like in XeTeX.

The best way to do so is to use the package luaotfload, on the CTAN, and developped here: http://github.com/mpg/luaotfload/

The code shows how we can use the table of a TrueType font to produce an internal TeX font table:

<pre>
\pdfoutput1
\directlua{
callback.register('define_font',
function(name, size)
fonttype = nil
filename = kpse.find_file(name,"opentype fonts")
if (filename)
then fonttype = 'opentype'
else filename = kpse.find_file(name, "truetype fonts")
end
if filename and not fonttype then fonttype = 'truetype' end
if fonttype then
if (size < 0) then size = (- 655.36) * size end
ttffont = fontforge.to_table(fontforge.open(filename))
if ttffont then
f = { }
f.name = ttffont.fontname
f.fullname = ttffont.names[1].names.fullname
f.parameters = { }
f.designsize = size
f.size = size
direction = 0
f.parameters.slant = 0
f.parameters.space = size * 0.25
f.parameters.space_stretch = 0.3 * size
f.parameters.space_shrink = 0.1 * size
f.parameters.x_height = 0.4 * size
f.parameters.quad = 1.0 * size
f.parameters.extra_space = 0
f.characters = { }
local mag = size / ttffont.units_per_em

names_of_char = { }
for char, glyph in pairs(ttffont.map.map)
do
names_of_char[ttffont.glyphs[glyph].name]
= ttffont.map.backmap[glyph]
end

for char, glyph in pairs(ttffont.map.map)
do
glyph_table = ttffont.glyphs[glyph]

f.characters[char] = {
index = glyph,
width = glyph_table.width * mag,
name = glyph_table.name,
}
if glyph_table.boundingbox[4] then
f.characters[char].height = glyph_table.boundingbox[4] * mag
end
if glyph_table.boundingbox[2] then
f.characters[char].depth = -glyph_table.boundingbox[2] * mag
end

if glyph_table.kerns then
local kerns = { }
for _, kern in pairs(glyph_table.kerns)
do
kerns[names_of_char[kern.char]] = kern.off * mag
end
f.characters[char].kerns = kerns
end
end
f.filename = filename
f.type = 'real'
f.format = fonttype
f.embedding = "subset"
f.cidinfo = {
registry = "Adobe",
ordering = "Identity",
supplement = 0,
version = 1
}
end
else
f = font.read_tfm(name, size)
end
return f
end
)
}
</pre>

The index of the <tt>f.characters</tt> table is the Unicode value, among the properties of each entry of that table we have <tt>index</tt> which is the glyph index. Using this property we get the mapping between Unicode characters and glyphs. Kerning is handled via TeX font LKP.

What is missing in the code above is better calculation of standard TeX font parameters and OpenType features. � suivre...