LuaTeXWiki - User contributions [en]

Changing the font for a script

2016-06-30T14:44:41Z

Javier Bezos:

Typically, fonts contain only a selection of scripts, and changing them automatically (as well as the writing direction) to make sure the document is properly typeset would be a nice feature. Here is a mock up.

It is based on the <code>pre_linebreak_filter</code> callback, which contains the horizontal list of glyphs, kerns, whatsits, etc., before building the paragraph. This callback is used by the font loader, too. It works with LaTeX 2016 and LuaTeX 0.95, but it should work with previous versions with minimal changes.

It is intended just for small chunks of text (a proper noun, a few words, etc.). For proper bidi writing, see typo-dha.lua, typo-dua.lua, typo-dub.lua and typo-duc.lua in context.

You may also want to readjust the vertical positioning (the baselines don't always match). This part resorts to a pdf literal and it's commented out (because it's a hack - the glyphs are moved, but the corresponding boxes are not).

As an alternative approach, you may use combo fonts (provided by luaotfload 2.7).

<pre>
\documentclass{article}

\directlua{
local DIR = node.id("dir")
function arabic_text (head)
local prevglyf = nil
local isArab = false
for item in node.traverse_id(node.id"glyph", head) do
if item.char > 1536 and item.char < 1792 then
item.font = arabicfont % Defined below
if isArab == false then
isArab = true
local d = node.new(DIR)
d.dir = '+TRT'
node.insert_before(head, item, d)
% local r = node.new(node.id'whatsit', node.subtype'pdf_literal')
% r.data = '2 Ts'
% node.insert_before(head, item, r)
end
else
if isArab == true then
isArab = false
local d = node.new(DIR)
d.dir = '-TRT'
node.insert_before(head, prevglyf, d)
% local r = node.new(node.id'whatsit', node.subtype'pdf_literal')
% r.data = '0 Ts'
% node.insert_before(head, item, r)
end
end
prevglyf = item % A bit dangerous, but usually OK.
end
return head
end
luatexbase.add_to_callback("pre_linebreak_filter", arabic_text, "Arabic text")
luatexbase.add_to_callback("hpack_filter", arabic_text, "Arabic text")
}

% The contextual analysis for arabic must be after the font
% is changed. If fontspec is loaded before, then the callbacks must be
% reorderer, which is feasible.

\usepackage{fontspec}
\setmainfont{Iwona}
\newfontfamily\amiri[Script=Arabic]{Amiri}

\begin{document}

{\amiri (نص قصير)\directlua{ arabicfont = font.current() }} Some text

Some text نص قصير Some text

\begin{tabular}{rr}
One & Two \\
Three & أربعة % :-/ Extra space
\end{tabular}

Some text.

\end{document}
</pre>

Changing the font for a script

2016-06-21T06:06:59Z

Javier Bezos:

Typically, fonts contain only a selection of scripts, and changing them automatically (as well as the writing direction) to make sure the document is properly typeset would be a nice feature. Here is a mock up.

It is based on the <code>pre_linebreak_filter</code> callback, which contains the horizontal list of glyphs, kerns, whatsits, etc., before building the paragraph. This callback is used by the font loader, too. It works with LaTeX 2016 and LuaTeX 0.95, but it should work with previous versions with minimal changes.

It is intended just for small chunks of text (a proper noun, a few words, etc.). For proper bidi writing, see typo-dha.lua, typo-dua.lua, typo-dub.lua and typo-duc.lua in context.

You may also want to readjust the vertical positioning (the baselines don't always match). This part resorts to a pdf literal and it's commented out (because it's a hack - the glyphs are moved, but the corresponding boxes are not).

<pre>
\documentclass{article}

\directlua{
local DIR = node.id("dir")
function arabic_text (head)
local prevglyf = nil
local isArab = false
for item in node.traverse_id(node.id"glyph", head) do
if item.char > 1536 and item.char < 1792 then
item.font = arabicfont % Defined below
if isArab == false then
isArab = true
local d = node.new(DIR)
d.dir = '+TRT'
node.insert_before(head, item, d)
% local r = node.new(node.id'whatsit', node.subtype'pdf_literal')
% r.data = '2 Ts'
% node.insert_before(head, item, r)
end
else
if isArab == true then
isArab = false
local d = node.new(DIR)
d.dir = '-TRT'
node.insert_before(head, prevglyf, d)
% local r = node.new(node.id'whatsit', node.subtype'pdf_literal')
% r.data = '0 Ts'
% node.insert_before(head, item, r)
end
end
prevglyf = item % A bit dangerous, but usually OK.
end
return head
end
luatexbase.add_to_callback("pre_linebreak_filter", arabic_text, "Arabic text")
luatexbase.add_to_callback("hpack_filter", arabic_text, "Arabic text")
}

% The contextual analysis for arabic must be after the font
% is changed. If fontspec is loaded before, then the callbacks must be
% reorderer, which is feasible.

\usepackage{fontspec}
\setmainfont{Iwona}
\newfontfamily\amiri[Script=Arabic]{Amiri}

\begin{document}

{\amiri (نص قصير)\directlua{ arabicfont = font.current() }} Some text

Some text نص قصير Some text

\begin{tabular}{rr}
One & Two \\
Three & أربعة % :-/ Extra space
\end{tabular}

Some text.

\end{document}
</pre>

Changing the font for a script

2016-06-20T14:04:55Z

Javier Bezos:

Typically, fonts contain only a selection of scripts, and changing them automatically (as well as the writing direction) to make sure the document is properly typeset would be a nice feature. Here is a mock up.

It is based on the <code>pre_linebreak_filter</code> callback, which contains the horizontal list of glyphs, kerns, whatsits, etc., before building the paragraph. This callback is used by the font loader, too. It works with LaTeX 2016 and LuaTeX 0.95, but it should work with previous versions with minimal changes.

It is intended just for small chunks of text (a proper noun, a few words, etc.). For proper bidi writing, see typo-dha.lua, typo-dua.lua, typo-dub.lua and typo-duc.lua in context.

You may also want to readjust the vertical positioning (the baselines don't always match). This part resorts to a pdf literal and it's commented out.

<pre>
\documentclass{article}

\directlua{
local DIR = node.id("dir")
function arabic_text (head)
local prevglyf = nil
local isArab = false
for item in node.traverse_id(node.id"glyph", head) do
if item.char > 1536 and item.char < 1792 then
item.font = arabicfont % Defined below
if isArab == false then
isArab = true
local d = node.new(DIR)
d.dir = '+TRT'
node.insert_before(head, item, d)
% local r = node.new(node.id'whatsit', node.subtype'pdf_literal')
% r.data = '2 Ts'
% node.insert_before(head, item, r)
end
else
if isArab == true then
isArab = false
local d = node.new(DIR)
d.dir = '-TRT'
node.insert_before(head, prevglyf, d)
% local r = node.new(node.id'whatsit', node.subtype'pdf_literal')
% r.data = '0 Ts'
% node.insert_before(head, item, r)
end
end
prevglyf = item % A bit dangerous, but usually OK.
end
return head
end
luatexbase.add_to_callback("pre_linebreak_filter", arabic_text, "Arabic text")
luatexbase.add_to_callback("hpack_filter", arabic_text, "Arabic text")
}

% The contextual analysis for arabic must be after the font
% is changed. If fontspec is loaded before, then the callbacks must be
% reorderer, which is feasible.

\usepackage{fontspec}
\setmainfont{Iwona}
\newfontfamily\amiri[Script=Arabic]{Amiri}

\begin{document}

{\amiri (نص قصير)\directlua{ arabicfont = font.current() }} Some text

Some text نص قصير Some text

\begin{tabular}{rr}
One & Two \\
Three & أربعة % :-/ Extra space
\end{tabular}

Some text.

\end{document}
</pre>

Changing the font for a script

2016-06-20T13:51:37Z

Javier Bezos: Added vertical adjusting

Typically, fonts contain only a selection of scripts, and changing them automatically (as well as the writing direction) to make sure the document is properly typeset would be a nice feature. Here is a mock up.

It is based on the <code>pre_linebreak_filter</code> callback, which contains the horizontal list of glyphs, kerns, whatsits, etc., before building the paragraph. This callback is used by the font loader, too. It works with LaTeX 2016 and LuaTeX 0.95, but it should work with previous versions with minimal changes.

It is intended just for small chunks of text (a proper noun, a few words, etc.). For proper bidi writing, see typo-dha.lua, typo-dua.lua, typo-dub.lua and typo-duc.lua in context.

You may also want to readjust the vertical positioning (the baselines don't always match). This part resorts to a pdf literal and it's commented out.

<pre>
\documentclass{article}

\directlua{
local DIR = node.id("dir")
local prevglyf = nil
function arabic_text (head)
local isArab = false
for item in node.traverse_id(node.id"glyph", head) do
if item.char > 1536 and item.char < 1792 then
item.font = arabicfont % Defined below
if isArab == false then
isArab = true
local d = node.new(DIR)
d.dir = '+TRT'
node.insert_before(head, item, d)
% local r = node.new(node.id'whatsit', node.subtype'pdf_literal')
% r.data = '2 Ts'
% node.insert_before(head, item, r)
end
else
if isArab == true then
isArab = false
local d = node.new(DIR)
d.dir = '-TRT'
node.insert_before(head, prevglyf, d)
% local r = node.new(node.id'whatsit', node.subtype'pdf_literal')
% r.data = '0 Ts'
% node.insert_before(head, item, r)
end
end
prevglyf = item % A bit dangerous, but usually OK.
end
return head
end
luatexbase.add_to_callback("pre_linebreak_filter", arabic_text, "Arabic text")
luatexbase.add_to_callback("hpack_filter", arabic_text, "Arabic text")
}

% The contextual analysis for arabic must be after the font
% is changed. If fontspec is loaded before, then the callbacks must be
% reorderer, which is feasible.

\usepackage{fontspec}
\setmainfont{Iwona}
\newfontfamily\amiri[Script=Arabic]{Amiri}

\begin{document}

{\amiri (نص قصير)\directlua{ arabicfont = font.current() }} Some text

Some text نص قصير Some text

\begin{tabular}{rr}
One & Two \\
Three & أربعة % :-/ Extra space
\end{tabular}

Some text.

\end{document}
</pre>

Changing the font for a script

2016-06-19T08:18:53Z

Javier Bezos:

Typically, fonts contain only a selection of scripts, and changing them automatically (as well as the writing direction) to make sure the document is properly typeset would be a nice feature. Here is a mock up.

It is based on the <code>pre_linebreak_filter</code> callback, which contains the horizontal list of glyphs, kerns, whatsits, etc., before building the paragraph. This callback is used by the font loader, too. It works with LaTeX 2016 and LuaTeX 0.95, but it should work with previous versions with minimal changes.

It is intended just for small chunks of text (a proper noun, a few words, etc.). For proper bidi writing, see typo-dha.lua, typo-dua.lua, typo-dub.lua and typo-duc.lua in context.

<pre>
\documentclass{article}

\directlua{
local DIR = node.id("dir")
function arabic_text (head)
local isArab = false
for item in node.traverse_id(node.id"glyph", head) do
if item.char > 1536 and item.char < 1792 then
item.font = arabicfont % Defined below
if isArab == false then
isArab = true
d = node.new(DIR)
d.dir = '+TRT'
node.insert_before(head, item, d)
end
else
if isArab == true then
isArab = false
d = node.new(DIR)
d.dir = '-TRT'
node.insert_after(head, prevglyf, d)
end
end
prevglyf = item % A bit dangerous, but usually OK.
end
return head
end
luatexbase.add_to_callback("pre_linebreak_filter", arabic_text, "Arabic text")
luatexbase.add_to_callback("hpack_filter", arabic_text, "Arabic text")
}

% The contextual analysis for arabic must be after the font
% changes. It fontspec is loaded before, then the callbacks must be
% reorderer, which is feasible.

\usepackage{fontspec}
\setmainfont{Iwona}
\newfontfamily\amiri[Script=Arabic]{Amiri}

\begin{document}

{\amiri (نص قصير)\directlua{ arabicfont = font.current() }} Some text

Some text نص قصير Some text

\begin{tabular}{rr}
One & Two \\
Three & أربعة % :-/ Estra space
\end{tabular}

Some text.

\end{document}
</pre>

Changing the font for a script

2016-06-19T08:16:13Z

Javier Bezos:

Typically, fonts contain only a selection of scripts, and changing them automatically (as well as the writing direction) to make sure the document is properly typeset would be a nice feature. Here is a mock up.

It is based on the <code>pre_linebreak_filter</code> callback, which contains the horizontal list of glyphs, kerns, whatsits, etc., before building the paragraph. This callback is used by the font loader, too. It works with LaTeX 2016 and LuaTeX 0.95, but it should work with previous versions with minimal changes.

It is intended just for small chunks of text (a proper noun, a few words, etc.). For proper bidi writing, see typo-dha.lua, typo-dua.lua, typo-dub.lua and typo-duc.lua in context.

<pre>
\documentclass{article}

\directlua{
local DIR = node.id("dir")
function show_nodes (head)
local isArab = false
for item in node.traverse_id(node.id"glyph", head) do
if item.char > 1536 and item.char < 1792 then
item.font = arabicfont % Defined below
if isArab == false then
isArab = true
d = node.new(DIR)
d.dir = '+TRT'
node.insert_before(head, item, d)
end
else
if isArab == true then
isArab = false
d = node.new(DIR)
d.dir = '-TRT'
node.insert_after(head, prevglyf, d)
end
end
prevglyf = item % A bit dangerous, but usually OK.
end
return head
end
luatexbase.add_to_callback("pre_linebreak_filter", show_nodes, 'desc')
luatexbase.add_to_callback("hpack_filter", show_nodes, 'desc')
}

% The contextual analysis for arabic must be after the font
% changes. It fontspec is loaded before, then the callbacks must be
% reorderer, which is feasible.

\usepackage{fontspec}
\setmainfont{Iwona}
\newfontfamily\amiri[Script=Arabic]{Amiri}

\begin{document}

{\amiri (نص قصير)\directlua{ arabicfont = font.current() }} Some text

Some text نص قصير Some text

\begin{tabular}{rr}
One & Two \\
Three & أربعة % :-/ Estra space
\end{tabular}

Some text.

\end{document}
</pre>

Changing the font for a script

2016-06-19T08:09:12Z

Javier Bezos: Created page with "Typically, fonts contain only a selection of scripts, and changing them automatically (as well as the writing direction) to make sure the document is properly typeset would be a ..."

Typically, fonts contain only a selection of scripts, and changing them automatically (as well as the writing direction) to make sure the document is properly typeset would be a nice feature. Here is a mock up.

It is based on the <code>pre_linebreak_filter</code> callback, which contains the horizontal list of glyphs, kerns, whatsits, etc., before building the paragraph. This callback is used by the font loader, too. It works with LaTeX 2016 and LuaTeX 0.95, but it should work with previous versions with minimal changes.

It is intended just for small chunks of text (a proper noun, a few words, etc.). For proper bidi writing, see typo-dha.lua, typo-dua.lua, typo-dub.lua and typo-duc.lua in context.

<pre>
\documentclass{article}

\directlua{
local DIR = node.id("dir")
function show_nodes (head)
isArab = false
for item in node.traverse_id(node.id"glyph", head) do
if item.char > 1536 and item.char < 1792 then
item.font = arabicfont % Defined below
if isArab == false then
isArab = true
d = node.new(DIR)
d.dir = '+TRT'
node.insert_before(head, item, d)
end
else
if isArab == true then
isArab = false
d = node.new(DIR)
d.dir = '-TRT'
node.insert_after(head, prevglyf, d)
end
end
prevglyf = item % A bit dangerous, but usually OK.
end
return head
end
luatexbase.add_to_callback("pre_linebreak_filter", show_nodes, 'desc')
luatexbase.add_to_callback("hpack_filter", show_nodes, 'desc')
}

% The contextual analysis for arabic must be after the font
% changes. It fontspec is loaded before, then the callbacks must be
% reorderer, which is feasible.

\usepackage{fontspec}
\setmainfont{Iwona}
\newfontfamily\amiri[Script=Arabic]{Amiri}

\begin{document}

{\amiri (نص قصير)\directlua{ arabicfont = font.current() }} Some text

Some text نص قصير Some text

\begin{tabular}{rr}
One & Two \\
Three & أربعة % :-/ Estra space
\end{tabular}

Some text.

\end{document}
</pre>

Token filter

2016-06-16T06:11:42Z

Javier Bezos: /* Syntax */

= Syntax =

This [[Callbacks|callback]] is called whenever TeX needs a new token. No arguments are passed to the callback, and the return value should be either a Lua table representing a to-be-processed token, a table consisting of a list of such tokens, or something else like <tt>nil</tt> or an empty table. Since no argument is passed, if you want to get the next token in the document, you should use the <tt>token.get_next()</tt> function. Your lua function should therefore look like this:

<pre>
function()
return <table> token
end
</pre>

If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by LuaTeX immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level. If what is passed to TeX is expandable, then the result of expansion is inserted into input and <tt>token.get_next()</tt> will grab it.

'''Note.''' This callback will be eventually removed and replaced by a new token library. See [http://tug.org/pipermail/luatex/2016-February/005651.html this message] on the luatex list.

= Examples =

== Faking \XeTeXinterchartoks ==

Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character lever and as implemented a macro breaks the char “chain”. So, <code>a\AA a</code>, with <code>\def\AA{A}</code>, doesn't work as expected.

Lua code:

<pre>
% luatexinterchartoks.lua

charclasses = charclasses or {}

function setcharclass (a,b)
charclasses[a] = b
end

local i = 0
while i < 65536 do
charclasses[i] = 0
i = i + 1
end

interchartoks = interchartoks or {}

function setinterchartoks (a,b,c)
interchartoks[a] = interchartoks[a] or {}
interchartoks[a][b] = c
end

local nc, oc
oc = 255

function do_intertoks ()
local tok = token.get_next()
if tex.count['XeTeXinterchartokenstate'] == 1 then
if tok[1] == 11 or tok[1] == 12 then
nc = charclasses[tok[2]]
newchar = tok[2]
else
nc = 255
newchar = ''
end
local insert = ''
if interchartoks[oc] and interchartoks[oc][nc] then
insert = interchartoks[oc][nc]
local newtok = tok
if insert<100 then
local dec = math.floor(insert / 10) + 48;
local unit = math.floor(insert % 10) + 48;
newtok = {
-- \XeTeXinterchartokenstate=0 \the\toks<n> \XeTeXinterchartokenstate=1
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('0'),12),
token.create(string.byte(' '),10),
token.create('the'),
token.create('toks'),
token.create(dec,12),
token.create(unit,12),
token.create(string.byte(' '),10),
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('1'),12),
token.create(string.byte(' '),10),
{tok[1], tok[2], tok[3]}}
end
tok = newtok
end
oc = nc
end
return tok
end

callback.register ('token_filter', do_intertoks)
</pre>

LaTeX package file:

<pre>
% luatexinterchartoks.sty
\newcount\XeTeXinterchartokenstate

\newcount\charclasses
\def\newXeTeXintercharclass#1%
{\global\advance\charclasses1\relax
\newcount#1
\global#1=\the\charclasses
}

\newcount\cchone
\newcount\cchtwo

\def\dodoXeTeXcharclass
{\directlua{setcharclass(\the\cchone,\the\cchtwo)}}

\def\doXeTeXcharclass%
{\afterassignment\dodoXeTeXcharclass\cchtwo }

\def\XeTeXcharclass%
{\afterassignment\doXeTeXcharclass\cchone }

\protected\def\XeTeXdointerchartoks%
{\directlua{setinterchartoks(\the\cchone,\the\cchtwo,\the\allocationnumber)}}

\protected\def\dodoXeTeXinterchartoks%
{\newtoks\mytoks\afterassignment\XeTeXdointerchartoks\global\mytoks }

\protected\def\doXeTeXinterchartoks%
{\afterassignment\dodoXeTeXinterchartoks\cchtwo }

\def\XeTeXinterchartoks%
{\afterassignment\doXeTeXinterchartoks\cchone }

\luatexdirectlua{require('luatexinterchartoks.lua')}

\endinput
</pre>

Test document:

<pre>
\documentclass{article}

\usepackage{luatexinterchartoks}
\usepackage{color}

\begin{document}

\newXeTeXintercharclass \mycharclassa
\newXeTeXintercharclass \mycharclassA
\newXeTeXintercharclass \mycharclassB
\XeTeXcharclass `\a \mycharclassa
\XeTeXcharclass `\A \mycharclassA
\XeTeXcharclass `\B \mycharclassB
% between "a" and "A":
\XeTeXinterchartoks \mycharclassa \mycharclassA = {[\itshape}
\XeTeXinterchartoks \mycharclassA \mycharclassa = {\upshape]}
% between " " and "B":
\XeTeXinterchartoks 255 \mycharclassB = {\bgroup\color{blue}}
\XeTeXinterchartoks \mycharclassB 255 = {\egroup}
% between "B" and "B":
\XeTeXinterchartoks \mycharclassB \mycharclassB = {.}

\begingroup
\XeTeXinterchartokenstate = 1
aAa A a B aBa BB
\endgroup

\end{document}

</pre>

Token filter

2016-06-15T11:07:18Z

Javier Bezos: /* Faking \XeTeXinterchartoks */

= Syntax =

This [[Callbacks|callback]] is called whenever TeX needs a new token. No arguments are passed to the callback, and the return value should be either a Lua table representing a to-be-processed token, a table consisting of a list of such tokens, or something else like <tt>nil</tt> or an empty table. Since no argument is passed, if you want to get the next token in the document, you should use the <tt>token.get_next()</tt> function. Your lua function should therefore look like this:

<pre>
function()
return <table> token
end
</pre>

If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by LuaTeX immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level. If what is passed to TeX is expandable, then the result of expansion is inserted into input and <tt>token.get_next()</tt> will grab it.

= Examples =

== Faking \XeTeXinterchartoks ==

Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character lever and as implemented a macro breaks the char “chain”. So, <code>a\AA a</code>, with <code>\def\AA{A}</code>, doesn't work as expected.

Lua code:

<pre>
% luatexinterchartoks.lua

charclasses = charclasses or {}

function setcharclass (a,b)
charclasses[a] = b
end

local i = 0
while i < 65536 do
charclasses[i] = 0
i = i + 1
end

interchartoks = interchartoks or {}

function setinterchartoks (a,b,c)
interchartoks[a] = interchartoks[a] or {}
interchartoks[a][b] = c
end

local nc, oc
oc = 255

function do_intertoks ()
local tok = token.get_next()
if tex.count['XeTeXinterchartokenstate'] == 1 then
if tok[1] == 11 or tok[1] == 12 then
nc = charclasses[tok[2]]
newchar = tok[2]
else
nc = 255
newchar = ''
end
local insert = ''
if interchartoks[oc] and interchartoks[oc][nc] then
insert = interchartoks[oc][nc]
local newtok = tok
if insert<100 then
local dec = math.floor(insert / 10) + 48;
local unit = math.floor(insert % 10) + 48;
newtok = {
-- \XeTeXinterchartokenstate=0 \the\toks<n> \XeTeXinterchartokenstate=1
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('0'),12),
token.create(string.byte(' '),10),
token.create('the'),
token.create('toks'),
token.create(dec,12),
token.create(unit,12),
token.create(string.byte(' '),10),
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('1'),12),
token.create(string.byte(' '),10),
{tok[1], tok[2], tok[3]}}
end
tok = newtok
end
oc = nc
end
return tok
end

callback.register ('token_filter', do_intertoks)
</pre>

LaTeX package file:

<pre>
% luatexinterchartoks.sty
\newcount\XeTeXinterchartokenstate

\newcount\charclasses
\def\newXeTeXintercharclass#1%
{\global\advance\charclasses1\relax
\newcount#1
\global#1=\the\charclasses
}

\newcount\cchone
\newcount\cchtwo

\def\dodoXeTeXcharclass
{\directlua{setcharclass(\the\cchone,\the\cchtwo)}}

\def\doXeTeXcharclass%
{\afterassignment\dodoXeTeXcharclass\cchtwo }

\def\XeTeXcharclass%
{\afterassignment\doXeTeXcharclass\cchone }

\protected\def\XeTeXdointerchartoks%
{\directlua{setinterchartoks(\the\cchone,\the\cchtwo,\the\allocationnumber)}}

\protected\def\dodoXeTeXinterchartoks%
{\newtoks\mytoks\afterassignment\XeTeXdointerchartoks\global\mytoks }

\protected\def\doXeTeXinterchartoks%
{\afterassignment\dodoXeTeXinterchartoks\cchtwo }

\def\XeTeXinterchartoks%
{\afterassignment\doXeTeXinterchartoks\cchone }

\luatexdirectlua{require('luatexinterchartoks.lua')}

\endinput
</pre>

Test document:

<pre>
\documentclass{article}

\usepackage{luatexinterchartoks}
\usepackage{color}

\begin{document}

\newXeTeXintercharclass \mycharclassa
\newXeTeXintercharclass \mycharclassA
\newXeTeXintercharclass \mycharclassB
\XeTeXcharclass `\a \mycharclassa
\XeTeXcharclass `\A \mycharclassA
\XeTeXcharclass `\B \mycharclassB
% between "a" and "A":
\XeTeXinterchartoks \mycharclassa \mycharclassA = {[\itshape}
\XeTeXinterchartoks \mycharclassA \mycharclassa = {\upshape]}
% between " " and "B":
\XeTeXinterchartoks 255 \mycharclassB = {\bgroup\color{blue}}
\XeTeXinterchartoks \mycharclassB 255 = {\egroup}
% between "B" and "B":
\XeTeXinterchartoks \mycharclassB \mycharclassB = {.}

\begingroup
\XeTeXinterchartokenstate = 1
aAa A a B aBa BB
\endgroup

\end{document}

</pre>

Token filter

2016-06-15T08:03:51Z

Javier Bezos: /* Faking \XeTeXinterchartoks */

= Syntax =

This [[Callbacks|callback]] is called whenever TeX needs a new token. No arguments are passed to the callback, and the return value should be either a Lua table representing a to-be-processed token, a table consisting of a list of such tokens, or something else like <tt>nil</tt> or an empty table. Since no argument is passed, if you want to get the next token in the document, you should use the <tt>token.get_next()</tt> function. Your lua function should therefore look like this:

<pre>
function()
return <table> token
end
</pre>

If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by LuaTeX immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level. If what is passed to TeX is expandable, then the result of expansion is inserted into input and <tt>token.get_next()</tt> will grab it.

= Examples =

== Faking \XeTeXinterchartoks ==

Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character level. So, <code>a\AA a</code>, with <code>\def\AA{A}</code>, doesn't work as expected.

Lua code:

<pre>
% luatexinterchartoks.lua

charclasses = charclasses or {}

function setcharclass (a,b)
charclasses[a] = b
end

local i = 0
while i < 65536 do
charclasses[i] = 0
i = i + 1
end

interchartoks = interchartoks or {}

function setinterchartoks (a,b,c)
interchartoks[a] = interchartoks[a] or {}
interchartoks[a][b] = c
end

local nc, oc
oc = 255

function do_intertoks ()
local tok = token.get_next()
if tex.count['XeTeXinterchartokenstate'] == 1 then
if tok[1] == 11 or tok[1] == 12 then
nc = charclasses[tok[2]]
newchar = tok[2]
else
nc = 255
newchar = ''
end
local insert = ''
if interchartoks[oc] and interchartoks[oc][nc] then
insert = interchartoks[oc][nc]
local newtok = tok
if insert<100 then
local dec = math.floor(insert / 10) + 48;
local unit = math.floor(insert % 10) + 48;
newtok = {
-- \XeTeXinterchartokenstate=0 \the\toks<n> \XeTeXinterchartokenstate=1
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('0'),12),
token.create(string.byte(' '),10),
token.create('the'),
token.create('toks'),
token.create(dec,12),
token.create(unit,12),
token.create(string.byte(' '),10),
token.create('XeTeXinterchartokenstate'),
token.create(string.byte('='),12),
token.create(string.byte('1'),12),
token.create(string.byte(' '),10),
{tok[1], tok[2], tok[3]}}
end
tok = newtok
end
oc = nc
end
return tok
end

callback.register ('token_filter', do_intertoks)
</pre>

LaTeX package file:

<pre>
% luatexinterchartoks.sty
\newcount\XeTeXinterchartokenstate

\newcount\charclasses
\def\newXeTeXintercharclass#1%
{\global\advance\charclasses1\relax
\newcount#1
\global#1=\the\charclasses
}

\newcount\cchone
\newcount\cchtwo

\def\dodoXeTeXcharclass
{\directlua{setcharclass(\the\cchone,\the\cchtwo)}}

\def\doXeTeXcharclass%
{\afterassignment\dodoXeTeXcharclass\cchtwo }

\def\XeTeXcharclass%
{\afterassignment\doXeTeXcharclass\cchone }

\protected\def\XeTeXdointerchartoks%
{\directlua{setinterchartoks(\the\cchone,\the\cchtwo,\the\allocationnumber)}}

\protected\def\dodoXeTeXinterchartoks%
{\newtoks\mytoks\afterassignment\XeTeXdointerchartoks\global\mytoks }

\protected\def\doXeTeXinterchartoks%
{\afterassignment\dodoXeTeXinterchartoks\cchtwo }

\def\XeTeXinterchartoks%
{\afterassignment\doXeTeXinterchartoks\cchone }

\luatexdirectlua{require('luatexinterchartoks.lua')}

\endinput
</pre>

Test document:

<pre>
\documentclass{article}

\usepackage{luatexinterchartoks}
\usepackage{color}

\begin{document}

\newXeTeXintercharclass \mycharclassa
\newXeTeXintercharclass \mycharclassA
\newXeTeXintercharclass \mycharclassB
\XeTeXcharclass `\a \mycharclassa
\XeTeXcharclass `\A \mycharclassA
\XeTeXcharclass `\B \mycharclassB
% between "a" and "A":
\XeTeXinterchartoks \mycharclassa \mycharclassA = {[\itshape}
\XeTeXinterchartoks \mycharclassA \mycharclassa = {\upshape]}
% between " " and "B":
\XeTeXinterchartoks 255 \mycharclassB = {\bgroup\color{blue}}
\XeTeXinterchartoks \mycharclassB 255 = {\egroup}
% between "B" and "B":
\XeTeXinterchartoks \mycharclassB \mycharclassB = {.}

\begingroup
\XeTeXinterchartokenstate = 1
aAa A a B aBa BB
\endgroup

\end{document}

</pre>