Token filter - Revision history

Javier Bezos: /* Syntax */

2016-06-16T06:11:42Z

‎Syntax

Javier Bezos: /* Faking \XeTeXinterchartoks */

2016-06-15T11:07:18Z

‎Faking \XeTeXinterchartoks

Javier Bezos: /* Faking \XeTeXinterchartoks */

2016-06-15T08:03:51Z

‎Faking \XeTeXinterchartoks

80.26.88.176: /* Faking \XeTeXinterchartoks */

2016-06-15T07:57:10Z

‎Faking \XeTeXinterchartoks

Paul: Corrections.

2011-07-01T04:59:10Z

Corrections.

Taco: with \XeTeXinterchartoks replacement example

2011-06-30T11:03:13Z

with \XeTeXinterchartoks replacement example

New page

= Syntax =

This [[Callbacks|callback]] is called whenever TeX needs a new token. No arguments are passed to the callback, and the return value should be either a Lua table representing a to|-|be|-|processed token, a table consisting of a list of such tokens, or
something else like \type{nil} or an empty table. Your lua function should therefore look like this:

<pre>
function()
return <table> token
end
</pre>

If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by luatex immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level.

The following examples aren't mutually exclusive; however, if one wanted to register them all in the same callback, special care would be needed, as explained [[Callbacks#Several_functions_in_a_callback|in the main page on callbacks]]. Here each call to <tt>callback.register()</tt> removes whatever there might have been in the callback beforehand.

= Examples =

== Faking \XeTeXinterchartoks ==

Lua code:

<pre>
% luatexinterchartoks.lua

charclasses = charclasses or {}

function setcharclass (a,b)
charclasses[a] = b
end

local i = 0
while i < 65536 do
charclasses[i] = 0
i = i + 1
end

interchartoks = interchartoks or {}

function setinterchartoks (a,b,c)
interchartoks[a] = interchartoks[a] or {}
interchartoks[a][b] = c
end

local nc, oc
oc = 255

function do_intertoks ()
local tok = token.get_next()
if tex.count['XeTeXinterchartokenstate'] == 1 then
if tok[1] == 11 or tok[1] == 12 then
nc = charclasses[tok[2]]
newchar = tok[2]
else
nc = 255
newchar = ''
end
local insert = ''
if interchartoks[oc] and interchartoks[oc][nc] then
insert = interchartoks[oc][nc]
local newtok = tok
if insert<100 then
local dec = math.floor(insert / 10) + 48;
local unit = math.floor(insert % 10) + 48;
newtok = {
-- \XeTeXinterchartokenstate=0 \the\toks<n> \XeTeXinterchartokenstate=1
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('0'),12),
token.create(string.byte(' '),10),
token.create('the'),
token.create('toks'),
token.create(dec,12),
token.create(unit,12),
token.create(string.byte(' '),10),
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('1'),12),
token.create(string.byte(' '),10),
{tok[1], tok[2], tok[3]}}
end
tok = newtok
end
oc = nc
end
return tok
end

callback.register ('token_filter', do_intertoks)
</pre>

LaTeX package file:

<pre>
% luatexinterchartoks.sty
\newcount\XeTeXinterchartokenstate

\newcount\charclasses
\def\newXeTeXintercharclass#1%
{\global\advance\charclasses1\relax
\newcount#1
\global#1=\the\charclasses
}

\newcount\cchone
\newcount\cchtwo

\def\dodoXeTeXcharclass
{\directlua{setcharclass(\the\cchone,\the\cchtwo)}}

\def\doXeTeXcharclass%
{\afterassignment\dodoXeTeXcharclass\cchtwo }

\def\XeTeXcharclass%
{\afterassignment\doXeTeXcharclass\cchone }

\protected\def\XeTeXdointerchartoks%
{\directlua{setinterchartoks(\the\cchone,\the\cchtwo,\the\allocationnumber)}}

\protected\def\dodoXeTeXinterchartoks%
{\newtoks\mytoks\afterassignment\XeTeXdointerchartoks\global\mytoks }

\protected\def\doXeTeXinterchartoks%
{\afterassignment\dodoXeTeXinterchartoks\cchtwo }

\def\XeTeXinterchartoks%
{\afterassignment\doXeTeXinterchartoks\cchone }

\luatexdirectlua{require('luatexinterchartoks.lua')}

\endinput
</pre>

Test document:

<pre>
\documentclass{article}

\usepackage{luatexinterchartoks}
\usepackage{color}

\begin{document}

\newXeTeXintercharclass \mycharclassa
\newXeTeXintercharclass \mycharclassA
\newXeTeXintercharclass \mycharclassB
\XeTeXcharclass `\a \mycharclassa
\XeTeXcharclass `\A \mycharclassA
\XeTeXcharclass `\B \mycharclassB
% between "a" and "A":
\XeTeXinterchartoks \mycharclassa \mycharclassA = {[\itshape}
\XeTeXinterchartoks \mycharclassA \mycharclassa = {\upshape]}
% between " " and "B":
\XeTeXinterchartoks 255 \mycharclassB = {\bgroup\color{blue}}
\XeTeXinterchartoks \mycharclassB 255 = {\egroup}
% between "B" and "B":
\XeTeXinterchartoks \mycharclassB \mycharclassB = {.}

\begingroup
\XeTeXinterchartokenstate = 1
aAa A a B aBa BB
\endgroup

\end{document}

</pre>

← Older revision		Revision as of 06:11, 16 June 2016
Line 10:		Line 10:

	If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by LuaTeX immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level. If what is passed to TeX is expandable, then the result of expansion is inserted into input and <tt>token.get_next()</tt> will grab it.		If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by LuaTeX immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level. If what is passed to TeX is expandable, then the result of expansion is inserted into input and <tt>token.get_next()</tt> will grab it.
		+
		+	'''Note.''' This callback will be eventually removed and replaced by a new token library. See [http://tug.org/pipermail/luatex/2016-February/005651.html this message] on the luatex list.

	= Examples =		= Examples =

@@ Line 15: / Line 15: @@
 == Faking \XeTeXinterchartoks ==
-Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character level. So, <code>a\AA a</code>, with <code>\def\AA{A}</code>, doesn't work as expected.
+Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character lever and as implemented a macro breaks the char “chain”. So, <code>a\AA a</code>, with <code>\def\AA{A}</code>, doesn't work as expected.
 Lua code:

@@ Line 15: / Line 15: @@
 == Faking \XeTeXinterchartoks ==
-Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character level. So, <code>a\AA a</code>, with <code>\def\AA{A}</code> doesn't work as expected.
+Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character level. So, <code>a\AA a</code>, with <code>\def\AA{A}</code>, doesn't work as expected.
 Lua code:

← Older revision		Revision as of 07:57, 15 June 2016
Line 14:		Line 14:

	== Faking \XeTeXinterchartoks ==		== Faking \XeTeXinterchartoks ==
		+
		+	Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character level. So, <code>a\AA a</code>, with <code>\def\AA{A}</code> doesn't work as expected.

	Lua code:		Lua code:

@@ Line 1: / Line 1: @@
 = Syntax =
-This [[Callbacks|callback]] is called whenever TeX needs a new token. No arguments are passed to the callback, and the return value should be either a Lua table representing a to|-|be|-|processed token, a table consisting of a list of such tokens, or
+This [[Callbacks|callback]] is called whenever TeX needs a new token. No arguments are passed to the callback, and the return value should be either a Lua table representing a to-be-processed token, a table consisting of a list of such tokens, or something else like <tt>nil</tt> or an empty table. Since no argument is passed, if you want to get the next token in the document, you should use the <tt>token.get_next()</tt> function. Your lua function should therefore look like this:
 <pre>
@@ Line 10: / Line 9: @@
 </pre>
-If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by luatex immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level.
+If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by LuaTeX immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level. If what is passed to TeX is expandable, then the result of expansion is inserted into input and <tt>token.get_next()</tt> will grab it.
 = Examples =