https://wiki.luatex.org/index.php?title=Traversing_tokens&feed=atom&action=history
Traversing tokens - Revision history
2024-03-28T11:04:43Z
Revision history for this page on the wiki
MediaWiki 1.31.1
https://wiki.luatex.org/index.php?title=Traversing_tokens&diff=41&oldid=prev
Patrick: minor correction
2010-12-08T08:40:14Z
<p>minor correction</p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 08:40, 8 December 2010</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l4" >Line 4:</td>
<td colspan="2" class="diff-lineno">Line 4:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><pre></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><pre></div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>\<del class="diffchange diffchange-inline">directlua0</del>{</div></td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>\<ins class="diffchange diffchange-inline">directlua</ins>{</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>callback.register('token_filter',</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>callback.register('token_filter',</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>function()</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>function()</div></td></tr>
</table>
Patrick
https://wiki.luatex.org/index.php?title=Traversing_tokens&diff=13&oldid=prev
Patrick: Copied from the old bluwiki.com luatex wiki
2010-12-07T21:06:31Z
<p>Copied from the old bluwiki.com luatex wiki</p>
<p><b>New page</b></p><div>= Traversing (or tracing) tokens =<br />
<br />
Add the following where you want to start tracing tokens:<br />
<br />
<pre><br />
\directlua0{<br />
callback.register('token_filter',<br />
function()<br />
t = token.get_next()<br />
if (t[3]==0) then<br />
texio.write_nl('term and log', 'CHAR TOKEN char=' .. unicode.utf8.char(t[2]) .. ' catcode=' .. t[1])<br />
else<br />
texio.write_nl('term and log', 'CSEQ TOKEN name=' .. token.command_name(t) .. ' code=' .. t[1] .. ' id=' .. t[3])<br />
end<br />
return t<br />
end )<br />
}<br />
</pre><br />
<br />
The anonymous function reads tokens using <tt>token.get_next()</tt>, processes them, and then supplies them (or different ones) to TeX by its <tt>return</tt> value. The function unicode.utf8.char reads a numeric value and produces a UTF-8 character. For all functions in the <b>string</b> library of lua you have the equivalents functions in the <b>unicode.utf8</b> library which is loaded by luatex.<br />
<br />
When I apply this to file<br />
<br />
<pre><br />
\font\x=omlgc<br />
\x VAT bl? bla bla bla <br />
</pre><br />
<br />
I get the following:<br />
<br />
<pre><br />
This is luaTeX, Version 3.141592-snapshot-2007062922 (Web2C 7.5.6) (format= 2007.6.25) 4 JUL 2007 00:14<br />
**&plain test<br />
(<br />
CHAR TOKEN char= catcode=10<br />
CSEQ TOKEN name=register code=93 id=1115710<br />
CSEQ TOKEN name=expand_after code=118 id=1114234<br />
CHAR TOKEN char== catcode=12<br />
CHAR TOKEN char=o catcode=11<br />
CHAR TOKEN char=o catcode=11<br />
CHAR TOKEN char=m catcode=11<br />
CHAR TOKEN char=l catcode=11<br />
CHAR TOKEN char=g catcode=11<br />
CHAR TOKEN char=c catcode=11<br />
CHAR TOKEN char= catcode=10<br />
CSEQ TOKEN name=def_font code=92 id=1114234<br />
CSEQ TOKEN name=def_font code=92 id=1114234<br />
CSEQ TOKEN name=def_font code=92 id=1114234<br />
CSEQ TOKEN name=def_font code=92 id=1114234<br />
CSEQ TOKEN name=def_font code=92 id=1114234<br />
CHAR TOKEN char=V catcode=11<br />
CHAR TOKEN char=V catcode=11<br />
CHAR TOKEN char=A catcode=11<br />
CHAR TOKEN char=T catcode=11<br />
CHAR TOKEN char= catcode=10<br />
CHAR TOKEN char=b catcode=11<br />
CHAR TOKEN char=l catcode=11<br />
CHAR TOKEN char=? catcode=12<br />
CHAR TOKEN char= catcode=10<br />
CHAR TOKEN char=b catcode=11<br />
CHAR TOKEN char=l catcode=11<br />
CHAR TOKEN char=a catcode=11<br />
CHAR TOKEN char= catcode=10<br />
CHAR TOKEN char=b catcode=11<br />
CHAR TOKEN char=l catcode=11<br />
CHAR TOKEN char=a catcode=11<br />
CHAR TOKEN char= catcode=10<br />
CHAR TOKEN char=b catcode=11<br />
CHAR TOKEN char=l catcode=11<br />
CHAR TOKEN char=a catcode=11<br />
CHAR TOKEN char= catcode=10<br />
CSEQ TOKEN name=par_end code=13 id=1114870<br />
...<br />
</pre><br />
<br />
If you wondering why the letters o and V appear twice, here are the answers I received from Taco:<br />
<br />
=== why do I get the o character token twice?===<br />
<br />
This is because of how the filename scanning works: it has<br />
to pre-scan the first character to decide between the two<br />
allowed syntaxes<br />
<br />
<pre><br />
\font\f=omlgc<br />
</pre><br />
and<br />
<pre><br />
\font\f={omlgc}<br />
</pre><br />
<br />
=== why do I get the V character token twice? ===<br />
<br />
Because it is a character encoutered in vmode. TeX switches<br />
to hmode, then re-reads the character.<br />
<br />
[[User:Yannis.Haralambous|Yannis]] 04:40, 4 July 2007 (EDT)</div>
Patrick