https://wiki.luatex.org/index.php?title=Token_filter&feed=atom&action=history
Token filter - Revision history
2024-03-28T21:37:51Z
Revision history for this page on the wiki
MediaWiki 1.31.1
https://wiki.luatex.org/index.php?title=Token_filter&diff=184&oldid=prev
Javier Bezos: /* Syntax */
2016-06-16T06:11:42Z
<p><span dir="auto"><span class="autocomment">Syntax</span></span></p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 06:11, 16 June 2016</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l10" >Line 10:</td>
<td colspan="2" class="diff-lineno">Line 10:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by LuaTeX immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level. If what is passed to TeX is expandable, then the result of expansion is inserted into input and <tt>token.get_next()</tt> will grab it.</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by LuaTeX immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level. If what is passed to TeX is expandable, then the result of expansion is inserted into input and <tt>token.get_next()</tt> will grab it.</div></td></tr>
<tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr>
<tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">'''Note.''' This callback will be eventually removed and replaced by a new token library. See [http://tug.org/pipermail/luatex/2016-February/005651.html this message] on the luatex list.</ins></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= Examples =</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= Examples =</div></td></tr>
</table>
Javier Bezos
https://wiki.luatex.org/index.php?title=Token_filter&diff=183&oldid=prev
Javier Bezos: /* Faking \XeTeXinterchartoks */
2016-06-15T11:07:18Z
<p><span dir="auto"><span class="autocomment">Faking \XeTeXinterchartoks</span></span></p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 11:07, 15 June 2016</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l15" >Line 15:</td>
<td colspan="2" class="diff-lineno">Line 15:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Faking \XeTeXinterchartoks ==</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Faking \XeTeXinterchartoks ==</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character <del class="diffchange diffchange-inline">level</del>. So, <code>a\AA a</code>, with <code>\def\AA{A}</code>, doesn't work as expected.</div></td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character <ins class="diffchange diffchange-inline">lever and as implemented a macro breaks the char “chain”</ins>. So, <code>a\AA a</code>, with <code>\def\AA{A}</code>, doesn't work as expected.</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Lua code:</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Lua code:</div></td></tr>
</table>
Javier Bezos
https://wiki.luatex.org/index.php?title=Token_filter&diff=182&oldid=prev
Javier Bezos: /* Faking \XeTeXinterchartoks */
2016-06-15T08:03:51Z
<p><span dir="auto"><span class="autocomment">Faking \XeTeXinterchartoks</span></span></p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 08:03, 15 June 2016</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l15" >Line 15:</td>
<td colspan="2" class="diff-lineno">Line 15:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Faking \XeTeXinterchartoks ==</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Faking \XeTeXinterchartoks ==</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character level. So, <code>a\AA a</code>, with <code>\def\AA{A}</code> doesn't work as expected.</div></td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character level. So, <code>a\AA a</code>, with <code>\def\AA{A}</code><ins class="diffchange diffchange-inline">, </ins>doesn't work as expected.</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Lua code:</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Lua code:</div></td></tr>
</table>
Javier Bezos
https://wiki.luatex.org/index.php?title=Token_filter&diff=181&oldid=prev
80.26.88.176: /* Faking \XeTeXinterchartoks */
2016-06-15T07:57:10Z
<p><span dir="auto"><span class="autocomment">Faking \XeTeXinterchartoks</span></span></p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 07:57, 15 June 2016</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l14" >Line 14:</td>
<td colspan="2" class="diff-lineno">Line 14:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Faking \XeTeXinterchartoks ==</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Faking \XeTeXinterchartoks ==</div></td></tr>
<tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr>
<tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">Note this is not strictly equivalent to the real XeTeX thing, because the latter operates on the character level. So, <code>a\AA a</code>, with <code>\def\AA{A}</code> doesn't work as expected.</ins></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Lua code:</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Lua code:</div></td></tr>
</table>
80.26.88.176
https://wiki.luatex.org/index.php?title=Token_filter&diff=111&oldid=prev
Paul: Corrections.
2011-07-01T04:59:10Z
<p>Corrections.</p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 04:59, 1 July 2011</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td>
<td colspan="2" class="diff-lineno">Line 1:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= Syntax =</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= Syntax =</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>This [[Callbacks|callback]] is called whenever TeX needs a new token. No arguments are passed to the callback, and the return value should be either a Lua table representing a to<del class="diffchange diffchange-inline">|</del>-<del class="diffchange diffchange-inline">|</del>be<del class="diffchange diffchange-inline">|</del>-<del class="diffchange diffchange-inline">|</del>processed token, a table consisting of a list of such tokens, or</div></td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>This [[Callbacks|callback]] is called whenever TeX needs a new token. No arguments are passed to the callback, and the return value should be either a Lua table representing a to-be-processed token, a table consisting of a list of such tokens, or something else like <ins class="diffchange diffchange-inline"><tt></ins>nil<ins class="diffchange diffchange-inline"></tt> </ins>or an empty table<ins class="diffchange diffchange-inline">. Since no argument is passed, if you want to get the next token in the document, you should use the <tt>token.get_next()</tt> function</ins>. Your lua function should therefore look like this:</div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>something else like <del class="diffchange diffchange-inline">\type{</del>nil<del class="diffchange diffchange-inline">} </del>or an empty table. Your lua function should therefore look like this:</div></td><td colspan="2"> </td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><pre></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><pre></div></td></tr>
<tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l10" >Line 10:</td>
<td colspan="2" class="diff-lineno">Line 9:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div></pre></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div></pre></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by <del class="diffchange diffchange-inline">luatex </del>immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level.</div></td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by <ins class="diffchange diffchange-inline">LuaTeX </ins>immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level. <ins class="diffchange diffchange-inline">If what is passed </ins>to <ins class="diffchange diffchange-inline">TeX is expandable</ins>, <ins class="diffchange diffchange-inline">then </ins>the <ins class="diffchange diffchange-inline">result of expansion is inserted into input and </ins><tt><ins class="diffchange diffchange-inline">token</ins>.<ins class="diffchange diffchange-inline">get_next</ins>()</tt> <ins class="diffchange diffchange-inline">will grab it</ins>.</div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div> </div></td><td colspan="2"> </td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div> </div></td><td colspan="2"> </td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del class="diffchange diffchange-inline">The following examples aren't mutually exclusive; however, if one wanted </del>to <del class="diffchange diffchange-inline">register them all in the same callback</del>, <del class="diffchange diffchange-inline">special care would be needed, as explained [[Callbacks#Several_functions_in_a_callback|in </del>the <del class="diffchange diffchange-inline">main page on callbacks]]. Here each call to </del><tt><del class="diffchange diffchange-inline">callback</del>.<del class="diffchange diffchange-inline">register</del>()</tt> <del class="diffchange diffchange-inline">removes whatever there might have been in the callback beforehand</del>.</div></td><td colspan="2"> </td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= Examples =</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= Examples =</div></td></tr>
</table>
Paul
https://wiki.luatex.org/index.php?title=Token_filter&diff=110&oldid=prev
Taco: with \XeTeXinterchartoks replacement example
2011-06-30T11:03:13Z
<p>with \XeTeXinterchartoks replacement example</p>
<p><b>New page</b></p><div>= Syntax =<br />
<br />
This [[Callbacks|callback]] is called whenever TeX needs a new token. No arguments are passed to the callback, and the return value should be either a Lua table representing a to|-|be|-|processed token, a table consisting of a list of such tokens, or<br />
something else like \type{nil} or an empty table. Your lua function should therefore look like this:<br />
<br />
<pre><br />
function()<br />
return <table> token<br />
end<br />
</pre><br />
<br />
If your Lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). If your function returns a single usable token, then that token will be processed by luatex immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack at a completely new token list level.<br />
<br />
<br />
The following examples aren't mutually exclusive; however, if one wanted to register them all in the same callback, special care would be needed, as explained [[Callbacks#Several_functions_in_a_callback|in the main page on callbacks]]. Here each call to <tt>callback.register()</tt> removes whatever there might have been in the callback beforehand.<br />
<br />
= Examples =<br />
<br />
== Faking \XeTeXinterchartoks ==<br />
<br />
Lua code:<br />
<br />
<pre><br />
% luatexinterchartoks.lua<br />
<br />
charclasses = charclasses or {}<br />
<br />
function setcharclass (a,b)<br />
charclasses[a] = b<br />
end<br />
<br />
local i = 0<br />
while i < 65536 do<br />
charclasses[i] = 0<br />
i = i + 1<br />
end<br />
<br />
interchartoks = interchartoks or {}<br />
<br />
function setinterchartoks (a,b,c)<br />
interchartoks[a] = interchartoks[a] or {}<br />
interchartoks[a][b] = c<br />
end<br />
<br />
local nc, oc<br />
oc = 255<br />
<br />
function do_intertoks () <br />
local tok = token.get_next() <br />
if tex.count['XeTeXinterchartokenstate'] == 1 then<br />
if tok[1] == 11 or tok[1] == 12 then<br />
nc = charclasses[tok[2]] <br />
newchar = tok[2]<br />
else <br />
nc = 255<br />
newchar = ''<br />
end<br />
local insert = ''<br />
if interchartoks[oc] and interchartoks[oc][nc] then<br />
insert = interchartoks[oc][nc] <br />
local newtok = tok<br />
if insert<100 then<br />
local dec = math.floor(insert / 10) + 48;<br />
local unit = math.floor(insert % 10) + 48;<br />
newtok = {<br />
-- \XeTeXinterchartokenstate=0 \the\toks<n> \XeTeXinterchartokenstate=1<br />
token.create('XeTeXinterchartokenstate'),<br />
token.create(string.byte('='),12),<br />
token.create(string.byte('0'),12),<br />
token.create(string.byte(' '),10),<br />
token.create('the'),<br />
token.create('toks'),<br />
token.create(dec,12),<br />
token.create(unit,12),<br />
token.create(string.byte(' '),10),<br />
token.create('XeTeXinterchartokenstate'),<br />
token.create(string.byte('='),12),<br />
token.create(string.byte('1'),12),<br />
token.create(string.byte(' '),10),<br />
{tok[1], tok[2], tok[3]}} <br />
end<br />
tok = newtok<br />
end<br />
oc = nc<br />
end<br />
return tok<br />
end<br />
<br />
callback.register ('token_filter', do_intertoks)<br />
</pre><br />
<br />
LaTeX package file:<br />
<br />
<pre><br />
% luatexinterchartoks.sty<br />
\newcount\XeTeXinterchartokenstate<br />
<br />
\newcount\charclasses<br />
\def\newXeTeXintercharclass#1%<br />
{\global\advance\charclasses1\relax<br />
\newcount#1<br />
\global#1=\the\charclasses<br />
}<br />
<br />
\newcount\cchone<br />
\newcount\cchtwo<br />
<br />
\def\dodoXeTeXcharclass<br />
{\directlua{setcharclass(\the\cchone,\the\cchtwo)}}<br />
<br />
\def\doXeTeXcharclass%<br />
{\afterassignment\dodoXeTeXcharclass\cchtwo }<br />
<br />
\def\XeTeXcharclass%<br />
{\afterassignment\doXeTeXcharclass\cchone }<br />
<br />
\protected\def\XeTeXdointerchartoks%<br />
{\directlua{setinterchartoks(\the\cchone,\the\cchtwo,\the\allocationnumber)}}<br />
<br />
\protected\def\dodoXeTeXinterchartoks%<br />
{\newtoks\mytoks\afterassignment\XeTeXdointerchartoks\global\mytoks }<br />
<br />
\protected\def\doXeTeXinterchartoks%<br />
{\afterassignment\dodoXeTeXinterchartoks\cchtwo }<br />
<br />
\def\XeTeXinterchartoks%<br />
{\afterassignment\doXeTeXinterchartoks\cchone }<br />
<br />
\luatexdirectlua{require('luatexinterchartoks.lua')}<br />
<br />
\endinput<br />
</pre><br />
<br />
Test document:<br />
<br />
<pre><br />
\documentclass{article}<br />
<br />
\usepackage{luatexinterchartoks}<br />
\usepackage{color}<br />
<br />
\begin{document}<br />
<br />
\newXeTeXintercharclass \mycharclassa<br />
\newXeTeXintercharclass \mycharclassA<br />
\newXeTeXintercharclass \mycharclassB<br />
\XeTeXcharclass `\a \mycharclassa<br />
\XeTeXcharclass `\A \mycharclassA<br />
\XeTeXcharclass `\B \mycharclassB<br />
% between "a" and "A":<br />
\XeTeXinterchartoks \mycharclassa \mycharclassA = {[\itshape}<br />
\XeTeXinterchartoks \mycharclassA \mycharclassa = {\upshape]}<br />
% between " " and "B":<br />
\XeTeXinterchartoks 255 \mycharclassB = {\bgroup\color{blue}}<br />
\XeTeXinterchartoks \mycharclassB 255 = {\egroup}<br />
% between "B" and "B":<br />
\XeTeXinterchartoks \mycharclassB \mycharclassB = {.}<br />
<br />
\begingroup<br />
\XeTeXinterchartokenstate = 1<br />
aAa A a B aBa BB<br />
\endgroup<br />
<br />
\end{document}<br />
<br />
</pre></div>
Taco