LuaTeXWiki - User contributions [en]

TeX without TeX

2015-04-02T11:09:00Z

Patrick: /* A first example */ fixed link

== Use TeX's power without TeX macros ==

With the Lua-part in LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

[[Writing_Lua_in_TeX|Remember]] that you can jump into Lua mode from TeX with <tt>\directlua{...Lua code...}</tt>. For example this plain TeX document prints the first digits of the value of pi: <tt>$\pi = \directlua{ tex.print(math.pi)}$ \bye</tt>. You can get the result PDF by running <tt>luatex testfile.tex</tt>.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://gist.github.com/pgundlach/1062041 here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (86 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other. If they were not connected, only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

With this approach you are able to put some glyphs into the PDF but you will run into problems when you want to create a whole paragraph. It is much easier to ask TeX to do the line breaking. The steps for paragraph creation are

# put glyph nodes and glue nodes in a list (= chain them together with the prev/next pointers)
# add trailing infinite penalty and parfillskip
# call lang.hyphenate(), node.kerning() and node.ligaturing() to prepare the list for the linebreaking job
# call tex.linebreak()

For the next task we want to create the following paragraph:

[[File:Sample_paragraph.png|600px]]

We start with the small TeX stub as above:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

and myprogram.lua ([https://gist.github.com/1063228 download]) defines a function <tt>mknodes()</tt> that creates the list of nodes to be passed into the typesetter.

<pre>
function mknodes( text )
local current_font = font.current()
local font_parameters = font.getfont(current_font).parameters
local n, head, last
-- we should insert the paragraph indentation at the beginning
head = node.new("glue")
head.spec = node.new("glue_spec")
head.spec.width = 20 * 2^16
last = head

for s in string.utfvalues( text ) do
local char = unicode.utf8.char(s)
if unicode.utf8.match(char,"^%s$") then
-- its a space
n = node.new("glue")
n.spec = node.new("glue_spec")
n.spec.width = font_parameters.space
n.spec.shrink = font_parameters.space_shrink
n.spec.stretch = font_parameters.space_stretch
else -- a glyph
n = node.new("glyph")
n.font = current_font
n.subtype = 1
n.char = s
n.lang = tex.language
n.uchyph = 1
n.left = tex.lefthyphenmin
n.right = tex.righthyphenmin
end

last.next = n
last = n
end

-- now add the final parts: a penalty and the parfillskip glue
local penalty = node.new("penalty")
penalty.penalty = 10000

local parfillskip = node.new("glue")
parfillskip.spec = node.new("glue_spec")
parfillskip.spec.stretch = 2^16
parfillskip.spec.stretch_order = 2

last.next = penalty
penalty.next = parfillskip

-- just to create the prev pointers for tex.linebreak
node.slide(head)
return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, ... like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox = tex.linebreak(head,{ hsize = tex.sp("3in")})
node.write(vbox)
</pre>

(You can safely skip this section if you are not interested in the details.)

What has happened in detail? We have created a list of glyph and glue nodes and fed those into the line breaking part of TeX which gave us the vbox back. This vbox contains horizontal lists glued together:

[[File:Paragraph_with_boxes.png|600px]]

The dashed rectangle (vbox) should fit tight to the inner hboxes. It has a small offset just for demonstration purpose.

To show you some of the nodes that TeX inserted for you, have a look at the following image:

[[File:Nodes_in_sample_paragraph.png]]

Here is an explanation of the nodes:

# This is the dashed line in the image above. A paragraph is always vertical
# The first line. It has a width of three inches, a height and depth of together 8.888 pt
# The 20pt paragraph indent. This is the first item in the first line (note that the pointer comes from the head item of the node #2
# The first letter in the first line
# The interword glue.
# The second letter in the first line
# The clubpenalty
# The baselineskip minus the height of the hbox "above" (8.888pt + 5.111pt is approx. 14pt. The rounding error is due to the display of only few significant digits.) Note that TeX is in vertical mode, as it is the third item in the vbox (#1).
# The hbox of the second line
# The first letter of the second line
# A kern between the m and the y of the first word (second line)
# The last letter shown here

There are many nodes created in the paragraph which are not shown here. A full view can be found in [[Media:Nodes_in_sample_paragraph_full.pdf|a separate PDF file]].

== Inserting images is easy ==

Now that you have survived the hard part of this tutorial, we get to something easier. Image inclusion. There are several ways to do this, and in my opinion this is the easiest. We again start with our plain (lua)TeX stub:

<pre>
\directlua{dofile("includeimage.lua")}
\end
</pre>

and includeimage.lua is this small lua chunk:

<pre>
local image_fixed = img.scan({filename = "oilpainting.jpg"})

local image = img.copy(image_fixed)
local halfimage = img.copy(image_fixed)

halfimage.height = halfimage.height / 2
halfimage.width = halfimage.width / 2

node.write(img.node(image))
node.write(img.node(halfimage))
</pre>

You might be able to guess what happens here. We load the image [[Media:Oilpainting.jpg|oilpainting.jpg]] with <tt>img.scan</tt> but do not write it out to the PDF file yet. We create a copy of the image data so we can manipulate the size for example (another option would be to obtain a specific page of a PDF document or do some rotation). After that we write a reference to that image to the PDF file. In the case above, two references to the image (XObject) get written but the image is only written once. That is easy, right?

You can get an impression how the resulting document looks like:

[[File:Oilpaintingdocument.jpg]]

== Hyphenation ==

Every now and then you have to hyphenate your words in other languages. When you create the paragraph to be hyphenated (with <tt>lang.hyphenate()</tt>), you assemble it by creating glyph nodes which have a field lang (see above). This field accepts a number, the language id. So: how do we "create" a language? The first steps are simple, we start without TeX stub again:

<pre>
\directlua{dofile("hyphenation.lua")}
\end
</pre>

and the file hyphenation.lua is:
<pre>
local path = kpse.find_file("hyph-de-1996.pat.txt")
local l = lang.new()

local hyph_file = io.open(path)
lang.patterns(l,hyph_file:read("*all"))
hyph_file:close()
</pre>

Now the function <tt>lang.id(l)</tt> returns the language id of the language <tt>l</tt>, which we can use in the glyph nodes. A peculiarity of TeX prevents that we can start with that right away. All characters that are in the paragraph get lowercased before hyphenation rules apply. So how will a word like "Œuvre" be converted to lowercase? TeX has an internal table called ''lccode'' where it looks up a character and the output is its lowercase variant. So the lccode of Œ would be œ, but as TeX stores numbers, the lccode of 338 is 339. But only lccodes of the letters A-Z and a-z are set, so we need to set the codes for all other characters that appear in a text. In TeX you would write:

<pre>
\lccode`\Œ=`\œ
\lccode`\œ=`\œ
</pre>
or

<pre>
\lccode338=339
\lccode339=339
</pre>

You see that you need to set the lccode of the lowercase characters as well. In Lua you do something similar:

<pre>
tex.lccode[unicode.utf8.byte("Œ")] = [unicode.utf8.byte("œ")]
tex.lccode[unicode.utf8.byte("œ")] = [unicode.utf8.byte("œ")]
</pre>

or shorter:

<pre>
tex.lccode[338] = [339]
tex.lccode[339] = [339]
</pre>

Once you have set all the necessary lccodes (remember you don't need to do this with a to z and A to Z), you can expect the hyphenation to work.

TeX without TeX

2014-03-29T18:21:49Z

Patrick: Anchor %s regexp - important

== Use TeX's power without TeX macros ==

With the Lua-part in LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

[[Writing_Lua_in_TeX|Remember]] that you can jump into Lua mode from TeX with <tt>\directlua{...Lua code...}</tt>. For example this plain TeX document prints the first digits of the value of pi: <tt>$\pi = \directlua{ tex.print(math.pi)}$ \bye</tt>. You can get the result PDF by running <tt>luatex testfile.tex</tt>.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (86 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other. If they were not connected, only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

With this approach you are able to put some glyphs into the PDF but you will run into problems when you want to create a whole paragraph. It is much easier to ask TeX to do the line breaking. The steps for paragraph creation are

# put glyph nodes and glue nodes in a list (= chain them together with the prev/next pointers)
# add trailing infinite penalty and parfillskip
# call lang.hyphenate(), node.kerning() and node.ligaturing() to prepare the list for the linebreaking job
# call tex.linebreak()

For the next task we want to create the following paragraph:

[[File:Sample_paragraph.png|600px]]

We start with the small TeX stub as above:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

and myprogram.lua ([https://gist.github.com/1063228 download]) defines a function <tt>mknodes()</tt> that creates the list of nodes to be passed into the typesetter.

<pre>
function mknodes( text )
local current_font = font.current()
local font_parameters = font.getfont(current_font).parameters
local n, head, last
-- we should insert the paragraph indentation at the beginning
head = node.new("glue")
head.spec = node.new("glue_spec")
head.spec.width = 20 * 2^16
last = head

for s in string.utfvalues( text ) do
local char = unicode.utf8.char(s)
if unicode.utf8.match(char,"^%s$") then
-- its a space
n = node.new("glue")
n.spec = node.new("glue_spec")
n.spec.width = font_parameters.space
n.spec.shrink = font_parameters.space_shrink
n.spec.stretch = font_parameters.space_stretch
else -- a glyph
n = node.new("glyph")
n.font = current_font
n.subtype = 1
n.char = s
n.lang = tex.language
n.uchyph = 1
n.left = tex.lefthyphenmin
n.right = tex.righthyphenmin
end

last.next = n
last = n
end

-- now add the final parts: a penalty and the parfillskip glue
local penalty = node.new("penalty")
penalty.penalty = 10000

local parfillskip = node.new("glue")
parfillskip.spec = node.new("glue_spec")
parfillskip.spec.stretch = 2^16
parfillskip.spec.stretch_order = 2

last.next = penalty
penalty.next = parfillskip

-- just to create the prev pointers for tex.linebreak
node.slide(head)
return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, ... like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox = tex.linebreak(head,{ hsize = tex.sp("3in")})
node.write(vbox)
</pre>

(You can safely skip this section if you are not interested in the details.)

What has happened in detail? We have created a list of glyph and glue nodes and fed those into the line breaking part of TeX which gave us the vbox back. This vbox contains horizontal lists glued together:

[[File:Paragraph_with_boxes.png|600px]]

The dashed rectangle (vbox) should fit tight to the inner hboxes. It has a small offset just for demonstration purpose.

To show you some of the nodes that TeX inserted for you, have a look at the following image:

[[File:Nodes_in_sample_paragraph.png]]

Here is an explanation of the nodes:

# This is the dashed line in the image above. A paragraph is always vertical
# The first line. It has a width of three inches, a height and depth of together 8.888 pt
# The 20pt paragraph indent. This is the first item in the first line (note that the pointer comes from the head item of the node #2
# The first letter in the first line
# The interword glue.
# The second letter in the first line
# The clubpenalty
# The baselineskip minus the height of the hbox "above" (8.888pt + 5.111pt is approx. 14pt. The rounding error is due to the display of only few significant digits.) Note that TeX is in vertical mode, as it is the third item in the vbox (#1).
# The hbox of the second line
# The first letter of the second line
# A kern between the m and the y of the first word (second line)
# The last letter shown here

There are many nodes created in the paragraph which are not shown here. A full view can be found in [[Media:Nodes_in_sample_paragraph_full.pdf|a separate PDF file]].

== Inserting images is easy ==

Now that you have survived the hard part of this tutorial, we get to something easier. Image inclusion. There are several ways to do this, and in my opinion this is the easiest. We again start with our plain (lua)TeX stub:

<pre>
\directlua{dofile("includeimage.lua")}
\end
</pre>

and includeimage.lua is this small lua chunk:

<pre>
local image_fixed = img.scan({filename = "oilpainting.jpg"})

local image = img.copy(image_fixed)
local halfimage = img.copy(image_fixed)

halfimage.height = halfimage.height / 2
halfimage.width = halfimage.width / 2

node.write(img.node(image))
node.write(img.node(halfimage))
</pre>

You might be able to guess what happens here. We load the image [[Media:Oilpainting.jpg|oilpainting.jpg]] with <tt>img.scan</tt> but do not write it out to the PDF file yet. We create a copy of the image data so we can manipulate the size for example (another option would be to obtain a specific page of a PDF document or do some rotation). After that we write a reference to that image to the PDF file. In the case above, two references to the image (XObject) get written but the image is only written once. That is easy, right?

You can get an impression how the resulting document looks like:

[[File:Oilpaintingdocument.jpg]]

== Hyphenation ==

Every now and then you have to hyphenate your words in other languages. When you create the paragraph to be hyphenated (with <tt>lang.hyphenate()</tt>), you assemble it by creating glyph nodes which have a field lang (see above). This field accepts a number, the language id. So: how do we "create" a language? The first steps are simple, we start without TeX stub again:

<pre>
\directlua{dofile("hyphenation.lua")}
\end
</pre>

and the file hyphenation.lua is:
<pre>
local path = kpse.find_file("hyph-de-1996.pat.txt")
local l = lang.new()

local hyph_file = io.open(path)
lang.patterns(l,hyph_file:read("*all"))
hyph_file:close()
</pre>

Now the function <tt>lang.id(l)</tt> returns the language id of the language <tt>l</tt>, which we can use in the glyph nodes. A peculiarity of TeX prevents that we can start with that right away. All characters that are in the paragraph get lowercased before hyphenation rules apply. So how will a word like "Œuvre" be converted to lowercase? TeX has an internal table called ''lccode'' where it looks up a character and the output is its lowercase variant. So the lccode of Œ would be œ, but as TeX stores numbers, the lccode of 338 is 339. But only lccodes of the letters A-Z and a-z are set, so we need to set the codes for all other characters that appear in a text. In TeX you would write:

<pre>
\lccode`\Œ=`\œ
\lccode`\œ=`\œ
</pre>
or

<pre>
\lccode338=339
\lccode339=339
</pre>

You see that you need to set the lccode of the lowercase characters as well. In Lua you do something similar:

<pre>
tex.lccode[unicode.utf8.byte("Œ")] = [unicode.utf8.byte("œ")]
tex.lccode[unicode.utf8.byte("œ")] = [unicode.utf8.byte("œ")]
</pre>

or shorter:

<pre>
tex.lccode[338] = [339]
tex.lccode[339] = [339]
</pre>

Once you have set all the necessary lccodes (remember you don't need to do this with a to z and A to Z), you can expect the hyphenation to work.

TeX without TeX

2011-08-11T12:15:37Z

Patrick: /* Inserting images is easy */ Show example resulting document

== Use TeX's power without TeX macros ==

With the Lua-part in LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

[[Writing_Lua_in_TeX|Remember]] that you can jump into Lua mode from TeX with <tt>\directlua{...Lua code...}</tt>. For example this plain TeX document prints the first digits of the value of pi: <tt>$\pi = \directlua{ tex.print(math.pi)}$ \bye</tt>. You can get the result PDF by running <tt>luatex testfile.tex</tt>.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (86 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other. If they were not connected, only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

With this approach you are able to put some glyphs into the PDF but you will run into problems when you want to create a whole paragraph. It is much easier to ask TeX to do the line breaking. The steps for paragraph creation are

# put glyph nodes and glue nodes in a list (= chain them together with the prev/next pointers)
# add trailing infinite penalty and parfillskip
# call lang.hyphenate(), node.kerning() and node.ligaturing() to prepare the list for the linebreaking job
# call tex.linebreak()

For the next task we want to create the following paragraph:

[[File:Sample_paragraph.png|600px]]

We start with the small TeX stub as above:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

and myprogram.lua ([https://gist.github.com/1063228 download]) defines a function <tt>mknodes()</tt> that creates the list of nodes to be passed into the typesetter.

<pre>
function mknodes( text )
local current_font = font.current()
local font_parameters = font.getfont(current_font).parameters
local n, head, last
-- we should insert the paragraph indentation at the beginning
head = node.new("glue")
head.spec = node.new("glue_spec")
head.spec.width = 20 * 2^16
last = head

for s in string.utfvalues( text ) do
local char = unicode.utf8.char(s)
if unicode.utf8.match(char,"%s") then
-- its a space
n = node.new("glue")
n.spec = node.new("glue_spec")
n.spec.width = font_parameters.space
n.spec.shrink = font_parameters.space_shrink
n.spec.stretch = font_parameters.space_stretch
else -- a glyph
n = node.new("glyph")
n.font = current_font
n.subtype = 1
n.char = s
n.lang = tex.language
n.uchyph = 1
n.left = tex.lefthyphenmin
n.right = tex.righthyphenmin
end

last.next = n
last = n
end

-- now add the final parts: a penalty and the parfillskip glue
local penalty = node.new("penalty")
penalty.penalty = 10000

local parfillskip = node.new("glue")
parfillskip.spec = node.new("glue_spec")
parfillskip.spec.stretch = 2^16
parfillskip.spec.stretch_order = 2

last.next = penalty
penalty.next = parfillskip

-- just to create the prev pointers for tex.linebreak
node.slide(head)
return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, ... like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox = tex.linebreak(head,{ hsize = tex.sp("3in")})
node.write(vbox)
</pre>

(You can safely skip this section if you are not interested in the details.)

What has happened in detail? We have created a list of glyph and glue nodes and fed those into the line breaking part of TeX which gave us the vbox back. This vbox contains horizontal lists glued together:

[[File:Paragraph_with_boxes.png|600px]]

The dashed rectangle (vbox) should fit tight to the inner hboxes. It has a small offset just for demonstration purpose.

To show you some of the nodes that TeX inserted for you, have a look at the following image:

[[File:Nodes_in_sample_paragraph.png]]

Here is an explanation of the nodes:

# This is the dashed line in the image above. A paragraph is always vertical
# The first line. It has a width of three inches, a height and depth of together 8.888 pt
# The 20pt paragraph indent. This is the first item in the first line (note that the pointer comes from the head item of the node #2
# The first letter in the first line
# The interword glue.
# The second letter in the first line
# The clubpenalty
# The baselineskip minus the height of the hbox "above" (8.888pt + 5.111pt is approx. 14pt. The rounding error is due to the display of only few significant digits.) Note that TeX is in vertical mode, as it is the third item in the vbox (#1).
# The hbox of the second line
# The first letter of the second line
# A kern between the m and the y of the first word (second line)
# The last letter shown here

There are many nodes created in the paragraph which are not shown here. A full view can be found in [[Media:Nodes_in_sample_paragraph_full.pdf|a separate PDF file]].

== Inserting images is easy ==

Now that you have survived the hard part of this tutorial, we get to something easier. Image inclusion. There are several ways to do this, and in my opinion this is the easiest. We again start with our plain (lua)TeX stub:

<pre>
\directlua{dofile("includeimage.lua")}
\end
</pre>

and includeimage.lua is this small lua chunk:

<pre>
local image_fixed = img.scan({filename = "oilpainting.jpg"})

local image = img.copy(image_fixed)
local halfimage = img.copy(image_fixed)

halfimage.height = halfimage.height / 2
halfimage.width = halfimage.width / 2

node.write(img.node(image))
node.write(img.node(halfimage))
</pre>

You might be able to guess what happens here. We load the image [[Media:Oilpainting.jpg|oilpainting.jpg]] with <tt>img.scan</tt> but do not write it out to the PDF file yet. We create a copy of the image data so we can manipulate the size for example (another option would be to obtain a specific page of a PDF document or do some rotation). After that we write a reference to that image to the PDF file. In the case above, two references to the image (XObject) get written but the image is only written once. That is easy, right?

You can get an impression how the resulting document looks like:

[[File:Oilpaintingdocument.jpg]]

== Hyphenation ==

Every now and then you have to hyphenate your words in other languages. When you create the paragraph to be hyphenated (with <tt>lang.hyphenate()</tt>), you assemble it by creating glyph nodes which have a field lang (see above). This field accepts a number, the language id. So: how do we "create" a language? The first steps are simple, we start without TeX stub again:

<pre>
\directlua{dofile("hyphenation.lua")}
\end
</pre>

and the file hyphenation.lua is:
<pre>
local path = kpse.find_file("hyph-de-1996.pat.txt")
local l = lang.new()

local hyph_file = io.open(path)
lang.patterns(l,hyph_file:read("*all"))
hyph_file:close()
</pre>

Now the function <tt>lang.id(l)</tt> returns the language id of the language <tt>l</tt>, which we can use in the glyph nodes. A peculiarity of TeX prevents that we can start with that right away. All characters that are in the paragraph get lowercased before hyphenation rules apply. So how will a word like "Œuvre" be converted to lowercase? TeX has an internal table called ''lccode'' where it looks up a character and the output is its lowercase variant. So the lccode of Œ would be œ, but as TeX stores numbers, the lccode of 338 is 339. But only lccodes of the letters A-Z and a-z are set, so we need to set the codes for all other characters that appear in a text. In TeX you would write:

<pre>
\lccode`\Œ=`\œ
\lccode`\œ=`\œ
</pre>
or

<pre>
\lccode338=339
\lccode339=339
</pre>

You see that you need to set the lccode of the lowercase characters as well. In Lua you do something similar:

<pre>
tex.lccode[unicode.utf8.byte("Œ")] = [unicode.utf8.byte("œ")]
tex.lccode[unicode.utf8.byte("œ")] = [unicode.utf8.byte("œ")]
</pre>

or shorter:

<pre>
tex.lccode[338] = [339]
tex.lccode[339] = [339]
</pre>

Once you have set all the necessary lccodes (remember you don't need to do this with a to z and A to Z), you can expect the hyphenation to work.

File:Oilpaintingdocument.jpg

2011-08-11T12:14:02Z

Patrick: Sample document of two oilpaintings

Sample document of two oilpaintings

File:Fontsampleroutput.png

2011-08-09T10:02:35Z

Patrick: example output of the fontsampler code

example output of the fontsampler code

TeX without TeX

2011-07-07T11:14:59Z

Patrick: Language and Hyphenation

== Use TeX's power without TeX macros ==

With the Lua-part in LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

[[Writing_Lua_in_TeX|Remember]] that you can jump into Lua mode from TeX with <tt>\directlua{...Lua code...}</tt>. For example this plain TeX document prints the first digits of the value of pi: <tt>$\pi = \directlua{ tex.print(math.pi)}$ \bye</tt>. You can get the result PDF by running <tt>luatex testfile.tex</tt>.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (86 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other. If they were not connected, only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

With this approach you are able to put some glyphs into the PDF but you will run into problems when you want to create a whole paragraph. It is much easier to ask TeX to do the line breaking. The steps for paragraph creation are

# put glyph nodes and glue nodes in a list (= chain them together with the prev/next pointers)
# add trailing infinite penalty and parfillskip
# call lang.hyphenate(), node.kerning() and node.ligaturing() to prepare the list for the linebreaking job
# call tex.linebreak()

For the next task we want to create the following paragraph:

[[File:Sample_paragraph.png|600px]]

We start with the small TeX stub as above:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

and myprogram.lua ([https://gist.github.com/1063228 download]) defines a function <tt>mknodes()</tt> that creates the list of nodes to be passed into the typesetter.

<pre>
function mknodes( text )
local current_font = font.current()
local font_parameters = font.getfont(current_font).parameters
local n, head, last
-- we should insert the paragraph indentation at the beginning
head = node.new("glue")
head.spec = node.new("glue_spec")
head.spec.width = 20 * 2^16
last = head

for s in string.utfvalues( text ) do
local char = unicode.utf8.char(s)
if unicode.utf8.match(char,"%s") then
-- its a space
n = node.new("glue")
n.spec = node.new("glue_spec")
n.spec.width = font_parameters.space
n.spec.shrink = font_parameters.space_shrink
n.spec.stretch = font_parameters.space_stretch
else -- a glyph
n = node.new("glyph")
n.font = current_font
n.subtype = 1
n.char = s
n.lang = tex.language
n.uchyph = 1
n.left = tex.lefthyphenmin
n.right = tex.righthyphenmin
end

last.next = n
last = n
end

-- now add the final parts: a penalty and the parfillskip glue
local penalty = node.new("penalty")
penalty.penalty = 10000

local parfillskip = node.new("glue")
parfillskip.spec = node.new("glue_spec")
parfillskip.spec.stretch = 2^16
parfillskip.spec.stretch_order = 2

last.next = penalty
penalty.next = parfillskip

-- just to create the prev pointers for tex.linebreak
node.slide(head)
return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, ... like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox = tex.linebreak(head,{ hsize = tex.sp("3in")})
node.write(vbox)
</pre>

(You can safely skip this section if you are not interested in the details.)

What has happened in detail? We have created a list of glyph and glue nodes and fed those into the line breaking part of TeX which gave us the vbox back. This vbox contains horizontal lists glued together:

[[File:Paragraph_with_boxes.png|600px]]

The dashed rectangle (vbox) should fit tight to the inner hboxes. It has a small offset just for demonstration purpose.

To show you some of the nodes that TeX inserted for you, have a look at the following image:

[[File:Nodes_in_sample_paragraph.png]]

Here is an explanation of the nodes:

# This is the dashed line in the image above. A paragraph is always vertical
# The first line. It has a width of three inches, a height and depth of together 8.888 pt
# The 20pt paragraph indent. This is the first item in the first line (note that the pointer comes from the head item of the node #2
# The first letter in the first line
# The interword glue.
# The second letter in the first line
# The clubpenalty
# The baselineskip minus the height of the hbox "above" (8.888pt + 5.111pt is approx. 14pt. The rounding error is due to the display of only few significant digits.) Note that TeX is in vertical mode, as it is the third item in the vbox (#1).
# The hbox of the second line
# The first letter of the second line
# A kern between the m and the y of the first word (second line)
# The last letter shown here

There are many nodes created in the paragraph which are not shown here. A full view can be found in [[Media:Nodes_in_sample_paragraph_full.pdf|a separate PDF file]].

== Inserting images is easy ==

Now that you have survived the hard part of this tutorial, we get to something easier. Image inclusion. There are several ways to do this, and in my opinion this is the easiest. We again start with our plain (lua)TeX stub:

<pre>
\directlua{dofile("includeimage.lua")}
\end
</pre>

and includeimage.lua is this small lua chunk:

<pre>
local image_fixed = img.scan({filename = "oilpainting.jpg"})

local image = img.copy(image_fixed)
local halfimage = img.copy(image_fixed)

halfimage.height = halfimage.height / 2
halfimage.width = halfimage.width / 2

node.write(img.node(image))
node.write(img.node(halfimage))
</pre>

You might be able to guess what happens here. We load the image [[Media:Oilpainting.jpg|oilpainting.jpg]] with <tt>img.scan</tt> but do not write it out to the PDF file yet. We create a copy of the image data so we can manipulate the size for example (another option would be to obtain a specific page of a PDF document or do some rotation). After that we write a reference to that image to the PDF file. In the case above, two references to the image (XObject) get written but the image is only written once. That is easy, right?

== Hyphenation ==

Every now and then you have to hyphenate your words in other languages. When you create the paragraph to be hyphenated (with <tt>lang.hyphenate()</tt>), you assemble it by creating glyph nodes which have a field lang (see above). This field accepts a number, the language id. So: how do we "create" a language? The first steps are simple, we start without TeX stub again:

<pre>
\directlua{dofile("hyphenation.lua")}
\end
</pre>

and the file hyphenation.lua is:
<pre>
local path = kpse.find_file("hyph-de-1996.pat.txt")
local l = lang.new()

local hyph_file = io.open(path)
lang.patterns(l,hyph_file:read("*all"))
hyph_file:close()
</pre>

Now the function <tt>lang.id(l)</tt> returns the language id of the language <tt>l</tt>, which we can use in the glyph nodes. A peculiarity of TeX prevents that we can start with that right away. All characters that are in the paragraph get lowercased before hyphenation rules apply. So how will a word like "Œuvre" be converted to lowercase? TeX has an internal table called ''lccode'' where it looks up a character and the output is its lowercase variant. So the lccode of Œ would be œ, but as TeX stores numbers, the lccode of 338 is 339. But only lccodes of the letters A-Z and a-z are set, so we need to set the codes for all other characters that appear in a text. In TeX you would write:

<pre>
\lccode`\Œ=`\œ
\lccode`\œ=`\œ
</pre>
or

<pre>
\lccode338=339
\lccode339=339
</pre>

You see that you need to set the lccode of the lowercase characters as well. In Lua you do something similar:

<pre>
tex.lccode[unicode.utf8.byte("Œ")] = [unicode.utf8.byte("œ")]
tex.lccode[unicode.utf8.byte("œ")] = [unicode.utf8.byte("œ")]
</pre>

or shorter:

<pre>
tex.lccode[338] = [339]
tex.lccode[339] = [339]
</pre>

Once you have set all the necessary lccodes (remember you don't need to do this with a to z and A to Z), you can expect the hyphenation to work.

TeX without TeX

2011-07-06T06:53:42Z

Patrick: Introduction of \directlua

== Use TeX's power without TeX macros ==

With the Lua-part in LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

[[Writing_Lua_in_TeX|Remember]] that you can jump into Lua mode from TeX with <tt>\directlua{...Lua code...}</tt>. For example this plain TeX document prints the first digits of the value of pi: <tt>$\pi = \directlua{ tex.print(math.pi)}$ \bye</tt>. You can get the result PDF by running <tt>luatex testfile.tex</tt>.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (86 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other. If they were not connected, only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

With this approach you are able to put some glyphs into the PDF but you will run into problems when you want to create a whole paragraph. It is much easier to ask TeX to do the line breaking. The steps for paragraph creation are

# put glyph nodes and glue nodes in a list (= chain them together with the prev/next pointers)
# add trailing infinite penalty and parfillskip
# call lang.hyphenate(), node.kerning() and node.ligaturing() to prepare the list for the linebreaking job
# call tex.linebreak()

For the next task we want to create the following paragraph:

[[File:Sample_paragraph.png|600px]]

We start with the small TeX stub as above:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

and myprogram.lua ([https://gist.github.com/1063228 download]) defines a function <tt>mknodes()</tt> that creates the list of nodes to be passed into the typesetter.

<pre>
function mknodes( text )
local current_font = font.current()
local font_parameters = font.getfont(current_font).parameters
local n, head, last
-- we should insert the paragraph indentation at the beginning
head = node.new("glue")
head.spec = node.new("glue_spec")
head.spec.width = 20 * 2^16
last = head

for s in string.utfvalues( text ) do
local char = unicode.utf8.char(s)
if unicode.utf8.match(char,"%s") then
-- its a space
n = node.new("glue")
n.spec = node.new("glue_spec")
n.spec.width = font_parameters.space
n.spec.shrink = font_parameters.space_shrink
n.spec.stretch = font_parameters.space_stretch
else -- a glyph
n = node.new("glyph")
n.font = current_font
n.subtype = 1
n.char = s
n.lang = tex.language
n.uchyph = 1
n.left = tex.lefthyphenmin
n.right = tex.righthyphenmin
end

last.next = n
last = n
end

-- now add the final parts: a penalty and the parfillskip glue
local penalty = node.new("penalty")
penalty.penalty = 10000

local parfillskip = node.new("glue")
parfillskip.spec = node.new("glue_spec")
parfillskip.spec.stretch = 2^16
parfillskip.spec.stretch_order = 2

last.next = penalty
penalty.next = parfillskip

-- just to create the prev pointers for tex.linebreak
node.slide(head)
return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, ... like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox = tex.linebreak(head,{ hsize = tex.sp("3in")})
node.write(vbox)
</pre>

(You can safely skip this section if you are not interested in the details.)

What has happened in detail? We have created a list of glyph and glue nodes and fed those into the line breaking part of TeX which gave us the vbox back. This vbox contains horizontal lists glued together:

[[File:Paragraph_with_boxes.png|600px]]

The dashed rectangle (vbox) should fit tight to the inner hboxes. It has a small offset just for demonstration purpose.

To show you some of the nodes that TeX inserted for you, have a look at the following image:

[[File:Nodes_in_sample_paragraph.png]]

Here is an explanation of the nodes:

# This is the dashed line in the image above. A paragraph is always vertical
# The first line. It has a width of three inches, a height and depth of together 8.888 pt
# The 20pt paragraph indent. This is the first item in the first line (note that the pointer comes from the head item of the node #2
# The first letter in the first line
# The interword glue.
# The second letter in the first line
# The clubpenalty
# The baselineskip minus the height of the hbox "above" (8.888pt + 5.111pt is approx. 14pt. The rounding error is due to the display of only few significant digits.) Note that TeX is in vertical mode, as it is the third item in the vbox (#1).
# The hbox of the second line
# The first letter of the second line
# A kern between the m and the y of the first word (second line)
# The last letter shown here

There are many nodes created in the paragraph which are not shown here. A full view can be found in [[Media:Nodes_in_sample_paragraph_full.pdf|a separate PDF file]].

== Inserting images is easy ==

Now that you have survived the hard part of this tutorial, we get to something easier. Image inclusion. There are several ways to do this, and in my opinion this is the easiest. We again start with our plain (lua)TeX stub:

<pre>
\directlua{dofile("includeimage.lua")}
\end
</pre>

and includeimage.lua is this small lua chunk:

<pre>
local image_fixed = img.scan({filename = "oilpainting.jpg"})

local image = img.copy(image_fixed)
local halfimage = img.copy(image_fixed)

halfimage.height = halfimage.height / 2
halfimage.width = halfimage.width / 2

node.write(img.node(image))
node.write(img.node(halfimage))
</pre>

You might be able to guess what happens here. We load the image [[Media:Oilpainting.jpg|oilpainting.jpg]] with <tt>img.scan</tt> but do not write it out to the PDF file yet. We create a copy of the image data so we can manipulate the size for example (another option would be to obtain a specific page of a PDF document or do some rotation). After that we write a reference to that image to the PDF file. In the case above, two references to the image (XObject) get written but the image is only written once. That is easy, right?

TeX without TeX

2011-07-04T16:38:50Z

Patrick: Image inclusion

== Use TeX's power without TeX macros ==

With the Lua-part in LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (86 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other. If they were not connected, only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

With this approach you are able to put some glyphs into the PDF but you will run into problems when you want to create a whole paragraph. It is much easier to ask TeX to do the line breaking. The steps for paragraph creation are

# put glyph nodes and glue nodes in a list (= chain them together with the prev/next pointers)
# add trailing infinite penalty and parfillskip
# call lang.hyphenate(), node.kerning() and node.ligaturing() to prepare the list for the linebreaking job
# call tex.linebreak()

For the next task we want to create the following paragraph:

[[File:Sample_paragraph.png|600px]]

We start with the small TeX stub as above:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

and myprogram.lua ([https://gist.github.com/1063228 download]) defines a function <tt>mknodes()</tt> that creates the list of nodes to be passed into the typesetter.

<pre>
function mknodes( text )
local current_font = font.current()
local font_parameters = font.getfont(current_font).parameters
local n, head, last
-- we should insert the paragraph indentation at the beginning
head = node.new("glue")
head.spec = node.new("glue_spec")
head.spec.width = 20 * 2^16
last = head

for s in string.utfvalues( text ) do
local char = unicode.utf8.char(s)
if unicode.utf8.match(char,"%s") then
-- its a space
n = node.new("glue")
n.spec = node.new("glue_spec")
n.spec.width = font_parameters.space
n.spec.shrink = font_parameters.space_shrink
n.spec.stretch = font_parameters.space_stretch
else -- a glyph
n = node.new("glyph")
n.font = current_font
n.subtype = 1
n.char = s
n.lang = tex.language
n.uchyph = 1
n.left = tex.lefthyphenmin
n.right = tex.righthyphenmin
end

last.next = n
last = n
end

-- now add the final parts: a penalty and the parfillskip glue
local penalty = node.new("penalty")
penalty.penalty = 10000

local parfillskip = node.new("glue")
parfillskip.spec = node.new("glue_spec")
parfillskip.spec.stretch = 2^16
parfillskip.spec.stretch_order = 2

last.next = penalty
penalty.next = parfillskip

-- just to create the prev pointers for tex.linebreak
node.slide(head)
return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, ... like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox = tex.linebreak(head,{ hsize = tex.sp("3in")})
node.write(vbox)
</pre>

(You can safely skip this section if you are not interested in the details.)

What has happened in detail? We have created a list of glyph and glue nodes and fed those into the line breaking part of TeX which gave us the vbox back. This vbox contains horizontal lists glued together:

[[File:Paragraph_with_boxes.png|600px]]

The dashed rectangle (vbox) should fit tight to the inner hboxes. It has a small offset just for demonstration purpose.

To show you some of the nodes that TeX inserted for you, have a look at the following image:

[[File:Nodes_in_sample_paragraph.png]]

Here is an explanation of the nodes:

# This is the dashed line in the image above. A paragraph is always vertical
# The first line. It has a width of three inches, a height and depth of together 8.888 pt
# The 20pt paragraph indent. This is the first item in the first line (note that the pointer comes from the head item of the node #2
# The first letter in the first line
# The interword glue.
# The second letter in the first line
# The clubpenalty
# The baselineskip minus the height of the hbox "above" (8.888pt + 5.111pt is approx. 14pt. The rounding error is due to the display of only few significant digits.) Note that TeX is in vertical mode, as it is the third item in the vbox (#1).
# The hbox of the second line
# The first letter of the second line
# A kern between the m and the y of the first word (second line)
# The last letter shown here

There are many nodes created in the paragraph which are not shown here. A full view can be found in [[Media:Nodes_in_sample_paragraph_full.pdf|a separate PDF file]].

== Inserting images is easy ==

Now that you have survived the hard part of this tutorial, we get to something easier. Image inclusion. There are several ways to do this, and in my opinion this is the easiest. We again start with our plain (lua)TeX stub:

<pre>
\directlua{dofile("includeimage.lua")}
\end
</pre>

and includeimage.lua is this small lua chunk:

<pre>
local image_fixed = img.scan({filename = "oilpainting.jpg"})

local image = img.copy(image_fixed)
local halfimage = img.copy(image_fixed)

halfimage.height = halfimage.height / 2
halfimage.width = halfimage.width / 2

node.write(img.node(image))
node.write(img.node(halfimage))
</pre>

You might be able to guess what happens here. We load the image [[Media:Oilpainting.jpg|oilpainting.jpg]] with <tt>img.scan</tt> but do not write it out to the PDF file yet. We create a copy of the image data so we can manipulate the size for example (another option would be to obtain a specific page of a PDF document or do some rotation). After that we write a reference to that image to the PDF file. In the case above, two references to the image (XObject) get written but the image is only written once. That is easy, right?

File:Oilpainting.jpg

2011-07-04T16:30:38Z

Patrick: some sample jpeg image

some sample jpeg image

TeX without TeX

2011-07-04T15:50:29Z

Patrick: forgot parindent

== Use TeX's power without TeX macros ==

With the Lua-part in LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (86 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other. If they were not connected, only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

With this approach you are able to put some glyphs into the PDF but you will run into problems when you want to create a whole paragraph. It is much easier to ask TeX to do the line breaking. The steps for paragraph creation are

# put glyph nodes and glue nodes in a list (= chain them together with the prev/next pointers)
# add trailing infinite penalty and parfillskip
# call lang.hyphenate(), node.kerning() and node.ligaturing() to prepare the list for the linebreaking job
# call tex.linebreak()

For the next task we want to create the following paragraph:

[[File:Sample_paragraph.png|600px]]

We start with the small TeX stub as above:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

and myprogram.lua ([https://gist.github.com/1063228 download]) defines a function <tt>mknodes()</tt> that creates the list of nodes to be passed into the typesetter.

<pre>
function mknodes( text )
local current_font = font.current()
local font_parameters = font.getfont(current_font).parameters
local n, head, last
-- we should insert the paragraph indentation at the beginning
head = node.new("glue")
head.spec = node.new("glue_spec")
head.spec.width = 20 * 2^16
last = head

for s in string.utfvalues( text ) do
local char = unicode.utf8.char(s)
if unicode.utf8.match(char,"%s") then
-- its a space
n = node.new("glue")
n.spec = node.new("glue_spec")
n.spec.width = font_parameters.space
n.spec.shrink = font_parameters.space_shrink
n.spec.stretch = font_parameters.space_stretch
else -- a glyph
n = node.new("glyph")
n.font = current_font
n.subtype = 1
n.char = s
n.lang = tex.language
n.uchyph = 1
n.left = tex.lefthyphenmin
n.right = tex.righthyphenmin
end

last.next = n
last = n
end

-- now add the final parts: a penalty and the parfillskip glue
local penalty = node.new("penalty")
penalty.penalty = 10000

local parfillskip = node.new("glue")
parfillskip.spec = node.new("glue_spec")
parfillskip.spec.stretch = 2^16
parfillskip.spec.stretch_order = 2

last.next = penalty
penalty.next = parfillskip

-- just to create the prev pointers for tex.linebreak
node.slide(head)
return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, ... like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox = tex.linebreak(head,{ hsize = tex.sp("3in")})
node.write(vbox)
</pre>

(You can safely skip this section if you are not interested in the details.)

What has happened in detail? We have created a list of glyph and glue nodes and fed those into the line breaking part of TeX which gave us the vbox back. This vbox contains horizontal lists glued together:

[[File:Paragraph_with_boxes.png|600px]]

The dashed rectangle (vbox) should fit tight to the inner hboxes. It has a small offset just for demonstration purpose.

To show you some of the nodes that TeX inserted for you, have a look at the following image:

[[File:Nodes_in_sample_paragraph.png]]

Here is an explanation of the nodes:

# This is the dashed line in the image above. A paragraph is always vertical
# The first line. It has a width of three inches, a height and depth of together 8.888 pt
# The 20pt paragraph indent. This is the first item in the first line (note that the pointer comes from the head item of the node #2
# The first letter in the first line
# The interword glue.
# The second letter in the first line
# The clubpenalty
# The baselineskip minus the height of the hbox "above" (8.888pt + 5.111pt is approx. 14pt. The rounding error is due to the display of only few significant digits.) Note that TeX is in vertical mode, as it is the third item in the vbox (#1).
# The hbox of the second line
# The first letter of the second line
# A kern between the m and the y of the first word (second line)
# The last letter shown here

There are many nodes created in the paragraph which are not shown here. A full view can be found in [[Media:Nodes_in_sample_paragraph_full.pdf|a separate PDF file]].

TeX without TeX

2011-07-04T13:03:12Z

Patrick: New image size

== Use TeX's power without TeX macros ==

With the Lua-part in LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (86 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other. If they were not connected, only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

With this approach you are able to put some glyphs into the PDF but you will run into problems when you want to create a whole paragraph. It is much easier to ask TeX to do the line breaking. The steps for paragraph creation are

# put glyph nodes and glue nodes in a list (= chain them together with the prev/next pointers)
# add trailing infinite penalty and parfillskip
# call lang.hyphenate(), node.kerning() and node.ligaturing() to prepare the list for the linebreaking job
# call tex.linebreak()

For the next task we want to create the following paragraph:

[[File:Sample_paragraph.png|600px]]

We start with the small TeX stub as above:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

and myprogram.lua ([https://gist.github.com/1063228 download]) defines a function <tt>mknodes()</tt> that creates the list of nodes to be passed into the typesetter.

<pre>
function mknodes( text )
local current_font = font.current()
local font_parameters = font.getfont(current_font).parameters
local n, head, last
for s in string.utfvalues( text ) do
local char = unicode.utf8.char(s)
if unicode.utf8.match(char,"%s") then
-- its a space
n = node.new("glue")
n.spec = node.new("glue_spec")
n.spec.width = font_parameters.space
n.spec.shrink = font_parameters.space_shrink
n.spec.stretch = font_parameters.space_stretch
else -- a glyph
n = node.new("glyph")
n.font = current_font
n.subtype = 1
n.char = s
n.lang = tex.language
n.uchyph = 1
n.left = tex.lefthyphenmin
n.right = tex.righthyphenmin
end
if head then
last.next = n
else
head = n
end
last = n
end

-- now add the final parts: a penalty and the parfillskip glue
local penalty = node.new("penalty")
penalty.penalty = 10000

local parfillskip = node.new("glue")
parfillskip.spec = node.new("glue_spec")
parfillskip.spec.stretch = 2^16
parfillskip.spec.stretch_order = 2

last.next = penalty
penalty.next = parfillskip

-- just to create the prev pointers for tex.linebreak
node.slide(head)
return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, ... like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox = tex.linebreak(head,{ hsize = tex.sp("3in")})
node.write(vbox)
</pre>

(You can safely skip this section if you are not interested in the details.)

What has happened in detail? We have created a list of glyph and glue nodes and fed those into the line breaking part of TeX which gave us the vbox back. This vbox contains horizontal lists glued together:

[[File:Paragraph_with_boxes.png|600px]]

The dashed rectangle (vbox) should fit tight to the inner hboxes. It has a small offset just for demonstration purpose.

To show you some of the nodes that TeX inserted for you, have a look at the following image:

[[File:Nodes_in_sample_paragraph.png]]

Here is an explanation of the nodes:

# This is the dashed line in the image above. A paragraph is always vertical
# The first line. It has a width of three inches, a height and depth of together 8.888 pt
# The 20pt paragraph indent. This is the first item in the first line (note that the pointer comes from the head item of the node #2
# The first letter in the first line
# The interword glue.
# The second letter in the first line
# The clubpenalty
# The baselineskip minus the height of the hbox "above" (8.888pt + 5.111pt is approx. 14pt. The rounding error is due to the display of only few significant digits.) Note that TeX is in vertical mode, as it is the third item in the vbox (#1).
# The hbox of the second line
# The first letter of the second line
# A kern between the m and the y of the first word (second line)
# The last letter shown here

There are many nodes created in the paragraph which are not shown here. A full view can be found in [[Media:Nodes_in_sample_paragraph_full.pdf|a separate PDF file]].

File:Paragraph with boxes.png

2011-07-04T13:01:14Z

Patrick: uploaded a new version of "File:Paragraph with boxes.png": Numbering as in explanation

A sample paragraph with hboxes and bounding box shown

TeX without TeX

2011-07-04T12:49:39Z

Patrick: More explanation on the sample paragraph

== Use TeX's power without TeX macros ==

With LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (86 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other. If they were not connected, only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

With this approach you are able to put some glyphs into the PDF but you will run into problems when you want to create a whole paragraph. It is much easier to ask TeX to do the line breaking. The steps for paragraph creation are

# put glyph nodes and glue nodes in a list (= chain them together with the prev/next pointers)
# add trailing infinite penalty and parfillskip
# call lang.hyphenate(), node.kerning() and node.ligaturing() to prepare the list for the linebreaking job
# call tex.linebreak()

For the next task we want to create the following paragraph:

[[File:Sample_paragraph.png|600px]]

We start with the small TeX stub as above:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

and myprogram.lua ([https://gist.github.com/1063228 download]) defines a function <tt>mknodes()</tt> that creates the list of nodes to be passed into the typesetter.

<pre>
function mknodes( text )
local current_font = font.current()
local font_parameters = font.getfont(current_font).parameters
local n, head, last
for s in string.utfvalues( text ) do
local char = unicode.utf8.char(s)
if unicode.utf8.match(char,"%s") then
-- its a space
n = node.new("glue")
n.spec = node.new("glue_spec")
n.spec.width = font_parameters.space
n.spec.shrink = font_parameters.space_shrink
n.spec.stretch = font_parameters.space_stretch
else -- a glyph
n = node.new("glyph")
n.font = current_font
n.subtype = 1
n.char = s
n.lang = tex.language
n.uchyph = 1
n.left = tex.lefthyphenmin
n.right = tex.righthyphenmin
end
if head then
last.next = n
else
head = n
end
last = n
end

-- now add the final parts: a penalty and the parfillskip glue
local penalty = node.new("penalty")
penalty.penalty = 10000

local parfillskip = node.new("glue")
parfillskip.spec = node.new("glue_spec")
parfillskip.spec.stretch = 2^16
parfillskip.spec.stretch_order = 2

last.next = penalty
penalty.next = parfillskip

-- just to create the prev pointers for tex.linebreak
node.slide(head)
return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, ... like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox = tex.linebreak(head,{ hsize = tex.sp("3in")})
node.write(vbox)
</pre>

(You can safely skip this section if you are not interested in the details.)

What has happened in detail? We have created a list of glyph and glue nodes and fed those into the line breaking part of TeX which gave us the vbox back. This vbox contains horizontal lists glued together:

[[File:Paragraph_with_boxes.png]]

The dashed rectangle (vbox) should fit tight to the inner hboxes. It has a small offset just for demonstration purpose.

To show you some of the nodes that TeX inserted for you, have a look at the following image:

[[File:Nodes_in_sample_paragraph.png]]

Here is an explanation of the nodes:

# This is the dashed line in the image above. A paragraph is always vertical
# The first line. It has a width of three inches, a height and depth of together 8.888 pt
# The 20pt paragraph indent. This is the first item in the first line (note that the pointer comes from the head item of the node #2
# The first letter in the first line
# The interword glue.
# The second letter in the first line
# The clubpenalty
# The baselineskip minus the height of the hbox "above" (8.888pt + 5.111pt is approx. 14pt. The rounding error is due to the display of only few significant digits.) Note that TeX is in vertical mode, as it is the third item in the vbox (#1).
# The hbox of the second line
# The first letter of the second line
# A kern between the m and the y of the first word (second line)
# The last letter shown here

There are many nodes created in the paragraph which are not shown here. A full view can be found in [[Media:Nodes_in_sample_paragraph_full.pdf|a separate PDF file]].

File:Nodes in sample paragraph full.pdf

2011-07-04T12:44:17Z

Patrick: lots of nodes

lots of nodes

File:Nodes in sample paragraph.png

2011-07-04T12:30:26Z

Patrick: The nodes in the sample paragraph

The nodes in the sample paragraph

File:Paragraph with boxes.png

2011-07-04T11:34:29Z

Patrick: A sample paragraph with hboxes and bounding box shown

A sample paragraph with hboxes and bounding box shown

TeX without TeX

2011-07-04T11:31:27Z

Patrick: paragraph with tex.shipout()

File:Sample paragraph.png

2011-07-04T10:36:21Z

Patrick: a simple paragraph

a simple paragraph

TeX without TeX

2011-07-03T20:04:56Z

Patrick: typo

== Use TeX's power without TeX macros ==

With LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (86 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other, otherwise only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

(todo: glyph subtype, ligaturing)

TeX without TeX

2011-07-03T09:50:24Z

Patrick: typo (</pre>)

== Use TeX's power without TeX macros ==

With LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (65 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other, otherwise only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

(todo: glyph subtype, ligaturing)

TeX without TeX

2011-07-03T09:44:35Z

Patrick: how to create a glyph node

== Use TeX's power without TeX macros ==

With LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
-- this will hold the items that go onto the page
local pagelist

-- Call tex.shipout() with the contents of the pagelist
function shipout()

local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
tex.box[666] = vbox
tex.shipout(666)

pagelist = nil

-- Not strictly necessary. TeX holds the current page number in counter 0.
-- TeX displays the contents of this counter when it puts a page into
-- the pdf (tex.shipout()). If we don't change the counter, TeX will
-- display [1] [1], instead of [1] [2] for our two page document.
tex.count[0] = tex.count[0] + 1
end

function add_to_page( list )
-- We attach the nodelist 'list' to the end of the pagelist
-- if pagelist doesn't exist, 'list' is our new pagelist
-- if it exists, we go to the end with node.tail() and adjust
-- the prev and next pointers, so list becomes part
-- of pagelist.
if not pagelist then pagelist = list
else
local tail = node.tail(pagelist)
tail.next = list
list.prev = tail
end
end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
local r = node.new("rule")
r.width = size
r.height = size / 2
r.depth = size / 2
return r
end

do
local destcounter = 0
-- Create a pdf anchor (dest object). It returns a whatsit node and the
-- number of the anchor, so it can be used in a pdf link or an outline.
function mkdest()
destcounter = destcounter + 1
local d = node.new("whatsit","pdf_dest")
d.named_id = 0
d.dest_id = destcounter
d.dest_type = 3

return d, destcounter
end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
local start, tmp, cur
start = select(1,...)
tmp = start
for i=2,select("#",...) do
cur = select(i,...)
tmp.next = cur
cur.prev = tmp
tmp = cur
end
local h,b = node.hpack(start) -- ignore badness
return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link = node.new("whatsit","pdf_end_link")

start_link.width = tenpt
start_link.height = tenpt / 2
start_link.depth = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node.

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*216, a stretch of 216 and a stretch_order of 3.

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
<pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (65 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other, otherwise only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

(todo: glyph subtype, ligaturing)

File:Singleglyphnode.png

2011-07-03T09:43:36Z

Patrick: structure for a glyph node

structure for a glyph node

File:Simplegluenode.png

2011-07-03T09:42:13Z

Patrick: structure for glue /glue_spec nodes

structure for glue /glue_spec nodes

TeX without TeX

2011-07-02T20:18:45Z

Patrick: remove unnecessary code

TeX without TeX

2011-07-02T19:09:35Z

Patrick:

TeX without TeX

2011-07-02T19:07:34Z

Patrick: Continue page on 'notex'.

Help:Editing

2011-07-02T18:46:07Z

Patrick: Link to the mediwiki.org page

See [http://www.mediawiki.org/wiki/Help:Formatting the formatting page on the mediwiki.org site]

TeX without TeX

2011-07-02T12:05:02Z

Patrick: formatting the link to the reference manual

TeX without TeX

2011-07-02T11:39:10Z

Patrick: ^ -> sup

== Use TeX's power without TeX macros ==

With LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [[http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual]] is about this topic.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=216) scaled points are 1 point, so 20pt becomes 20 * 216. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

TeX without TeX

2011-07-02T11:11:38Z

Patrick: Started page on TeX without TeX

== Use TeX's power without TeX macros ==

With LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [[http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual]] is about this topic.

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures.

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width = 20 * 65536
r.height = 10 * 65536
r.depth = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=2^16) scaled points are 1 point, so 20pt becomes 20 * 2^16. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
\hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:

[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width = 20 * 65536
r1.height = 10 * 65536
r1.depth = 10 * 65536

r2 = node.new("rule")
r2.width = 20 * 65536
r2.height = 10 * 65536
r2.depth = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".

File:Rulenode2.png

2011-07-02T11:07:26Z

Patrick: vlist/rule/rule node visualized

vlist/rule/rule node visualized

File:Rulenode.png

2011-07-02T10:54:48Z

Patrick: uploaded a new version of "File:Rulenode.png": Smaller size

simple rule node

File:Rulenode.png

2011-07-02T09:45:01Z

Patrick: simple rule node

simple rule node

Main Page

2011-07-02T09:27:49Z

Patrick: /* Some articles */ Link to TeX without TeX

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

See [[Special:Allpages|all the articles]] or the [[Special:Recentchanges|recent changes]].

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Some articles ==

* [[Documentation and help]] points to other online resources (manuals, mailing list, etc.) related to LuaTeX.
* [[Writing Lua in TeX]] explains how to write Lua code in a TeX document (and back).
* [[Attributes]] introduces LuaTeX's thrilling new concept.
* Pages on callbacks:
** [[Callbacks]] introduces callbacks and how to use them.
** There is a page on the [[Post linebreak filter|<tt>post_linebreak_filter</tt>]] callback, explaining and illustrating it with a couple of examples. The [[Show the hyphenation points]] article is another example of use.
** The page on [[process input buffer|the <tt>process_input_buffer</tt> callback]] illustrates how to read documents with non-UTF-8 encoding, and how to write TeX with lightweight markup instead of the usual commands.
** Another callback is [[show error hook|<tt>show_error_hook</tt>]], which lets you enliven your error messages!
** You can fake XeTeX's interchar tokens with the [[token filter|<tt>token_filter</tt>]].
* [[TeX without TeX]] is about using TeX's functionality (typesetting, pdf writing) only using Lua code (no <tt>\TeX</tt> macros).

== From the old bluwiki.com ==

(Mostly by and thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

User:Patrick

2011-01-11T16:09:18Z

Patrick: Link to luatex.de

LuaTeX addict. Contact me at patrick <at> gundla <dot> ch.

If you can read german, see more LuaTeX stuff at http://www.luatex.de.

Show the hyphenation points

2011-01-11T15:35:07Z

Patrick: show the disc nodes

This LaTeX example adds a function to the <tt>post_linebreak_filter</tt> callback. The function iterates through all the nodes in the paragraph (and descends into the vlist and hlist nodes) and inserts a small rule where the discretionary nodes are.

<pre>
\documentclass[12pt,a4paper]{scrartcl}
\usepackage[english]{babel}
\usepackage{blindtext}
\usepackage{fontspec}
\directlua{
show_hyph = function(head)
while head do
if head.id == 0 or head.id == 1 then % hlist, vlist
show_hyph(head.list) % should be head.head in a newer luatex than 0.64
elseif head.id == 7 then % disc
local n = node.new("whatsit","pdf_literal")
n.mode = 0
n.data = "q 0.3 w 0 2 m 0 7 l S Q"
n.next = head.next
n.prev = head
head.next = n
head = n
end
head = head.next
end
return true
end
luatexbase.add_to_callback("post_linebreak_filter",show_hyph,"show_hyph")

}

\begin{document}
\begin{minipage}{5cm}
\blindtext
\end{minipage}
\end{document}
</pre>

Writing Lua in TeX

2010-12-27T19:04:29Z

Patrick: Undo revision 49 by 89.162.236.186 (talk)

= Embedding Lua code in a TeX document =

Although it is simpler to put Lua code in Lua files, from time to time one may want or need to go Lua in the middle of a document. To this end, LuaTeX has two commands: <tt>\directlua</tt> and <tt>\latelua</tt>. They work the same, except <tt>\latelua</tt> is processed when the page where it appears is shipped out, whereas <tt>\directlua</tt> is processed at once; the distinction is immaterial here, and what is said of <tt>\directlua</tt> also applies to <tt>\latelua</tt>.

<tt>\directlua</tt> can be called in three ways:

<pre>
\directlua {<lua code>}
\directlua name {<name>} {<lua code>}
\directlua <number> {<lua code>}
</pre>

Those three ways are equivalent when it comes to process <tt><lua code></tt>, but in the second case processing will occur in a chunk named <tt><name></tt>, and in the third it will occur in a chunk whose name is the entry <tt><number></tt> in the table <tt>lua.name</tt>. The difference manifests itself only when errors occur, in which case the name of the chunk, if any, is reported.

Each call to <tt>\directlua</tt>, named or not, is processed in a separate chunk. That means that any <tt>local</tt> variable is defined for this call only and is lost afterward. Hence:

<pre>
\directlua{
one = 1
local two = 2
}
\directlua{
texio.write_nl(type(one))
texio.write_nl(type(two))
}
</pre>

will report <tt>number</tt> and <tt>nil</tt> (<tt>texio.write_nl</tt> writes to the log file). On the other hand, Lua code is completely insensitive to TeX's grouping mechanism. In other words, calling <tt>\directlua</tt> between <tt>\bgroup</tt> and <tt>\egroup</tt> doesn't affect the code to be processed.

= TeX catcodes in Lua =

By default, the code passed to <tt>\directlua</tt> is treated as normal TeX input and only then sent to the Lua interpreter. This may lead to unwanted results and must be acknowledged.

== Expansion ==

As with any other special, the code in <tt>\directlua</tt> (and the <tt><name></tt>, if specifed) is fully expanded. This means that macros can be safely passed to <tt>\directlua</tt> if one wants their values, but that they should also be properly escaped when needed. For instance:

<pre>
\def\macro{1}
\directlua{
myvar = \macro
}
</pre>

defines <tt>myvar</tt> as the number <tt>1</tt>. To store the control sequence <tt>\macro</tt> instead, another course of action is needed: see the [[#Backslash|section on backslash]] below.

== Line ends ==

When TeX reads a file, it normally turns line ends into spaces. That means that what looks like several lines is actually fed to the Lua interpreter as one big line. For instance:

<pre>
\directlua{
myvar = 1
anothervar = 2
onelastvar = 3
}
</pre>

amounts to the following, if it were written in a separate Lua file:

<pre>
myvar = 1 anothervar = 2 onelastvar = 3
</pre>

That is perfectly legitimate, but strange things might happen. First, TeX macros gobble spaces as usual. Hence:

<pre>
\def\macro{1}
\directlua{
myvar = \macro
anothervar = 2
}
</pre>

will be fed to the interpreter as

<pre>
myvar = 1anothervar = 2
</pre>

which is not legitimate at all. Second, the Lua comment <tt>--</tt> will affect everything to the end of the <tt>\directlua</tt> call. That is:

<pre>
\directlua{
myvar = 1
-- anothervar = 2
onelastvar = 3
}
</pre>

will be processed as

<pre>
myvar = 1 -- anothervar = 2 onelastvar = 3
</pre>

which works but only defines <tt>myvar</tt>. Third, when reporting error, the Lua interpreter will always mention the line number as 1, since it processes one big line only; that isn't extremely useful when the code is large.

The solution is to set <tt>\endlinechar=10</tt> or <tt>\catcode`\^^M=12</tt>. In both cases, line ends will be preserved and the code will be processed as it is input.

== Special characters ==

In TeX, some characters have a special behavior. That must be taken into account when writing Lua code: one must change their catcodes beforehand if one wants to handle them as Lua would, as has just been done for line ends. That means that <tt>\directlua</tt>, as such, is clearly insufficient to write any extended chunk of code. It is thus better to devise a special macro that sets the catcodes to the appropriate values, reads the Lua code, feeds it to <tt>\directlua</tt>, and restores the catcodes. The following code does the job:

<pre>
\def\luacode{%
\bgroup
\catcode`\{=12
\catcode`\}=12
\catcode`\^^M=12
\catcode`\#=12
\catcode`\~=12
\catcode`\%=12
\doluacode
}

\bgroup
\catcode`\^^M=12 %
\long\gdef\doluacode#1^^M#2\endluacode{\directlua{#2}\egroup}%
\egroup
</pre>

Note that not all special characters are set to normal (catcode 12) characters; that is explained for each below. Note also that <tt>\doluacode</tt>, internally called by <tt>\luacode</tt>, is defined to get rid of anything up to the line end, and then pass anything up to <tt>\endluacode</tt> to <tt>\directlua</tt>. Discarding what follows <tt>\luacode</tt> is important, otherwise a simple code as

<pre>
\luacode
myvar = 1
\endluacode
</pre>

would actually create two lines, the first being empty; it is annoying because errors are then reported with the wrong line number (i.e. any error in this one-line code would be reported to happen on line 2).

However, the rest of the line after <tt>\luacode</tt> could also be processed, instead of discarded, to manage special effects (e.g. specifying a chunk's name, storing the code in a control sequence, or even setting which catcodes should be changed or not).

=== Backslash ===

The backslash in TeX is used to form control sequences. In the definition of <tt>\luacode</tt> above, it isn't changed and thus behaves as usual. It allows commands to be passed and expanded to the Lua code. Anyway a backslash in Lua is also an escape character in strings. Hence, if one wants to store the name of a macro in Lua code, the following won't work:

<pre>
\luacode
myvar = "\noexpand\macro"
\endluacode
</pre>

because to the Lua interpreter the string is made of <tt>\m</tt> followed by <tt>acro</tt>; since <tt>\m</tt> is not defined in Lua, the string is read as <tt>macro</tt>, but in other circumstances strange things might happen: for instance, <tt>\n</tt> is a newline. The proper way to pass a macro verbatim is:

<pre>
\luacode
myvar = "\noexpand\\macro"
\endluacode
</pre>

which Lua will correctly read as

<pre>
myvar = "\\macro"
</pre>

with the backslash escaped to represent itself. Another solution is:

<pre>
myvar = [[\noexpand\macro]]
</pre>

because the double brackets signals a string in Lua where no escape sequence occurs (and the string can also run on several lines). Note however that in the second case <tt>myvar</tt> will be defined with a trailing space, i.e. as <tt>"\macro "</tt>, because of TeX's habit to append a trailing space to unexpanded (or unexpandable) control sequences.

=== Braces ===

One may want to define a string in Lua which contains unbalanced braces, i.e.:

<pre>
\luacode
myvar = "{"
\endluacode
</pre>

If the braces' catcodes hadn't been changed beforehand, that would be impossible. Note, however, that this means that one can't feed arguments to commands in the usual way. I.e. the following will produce nothing good:

<pre>
\luacode
myvar = "\dosomething{\macro}"
\endluacode
</pre>

<tt>\dosomething</tt> will be expanded with the left brace (devoid of its usual delimiter-ness) as its argument, and the rest of the line might produce chaos. Thus, one may also choose not to change the catcodes of braces, depending on how <tt>\luacode</tt> is most likely to be used. Note that strings with unbalanced braces can still be defined, even if braces have their usual catcodes, thanks to the following trick:

<pre>
\luacode
myvar = "{" -- }
\endluacode
</pre>

When the code is passed to <tt>\directlua</tt>, braces are balanced because the Lua comment means nothing to TeX; when passed to the Lua interpreter, on the other hand, the right brace is ignored.

=== Hash and comment ===

The hash sign <tt>#</tt> in Lua is the length operator: prefixed to a string or table variable, it returns its length. If its catcode weren't taken care of, LuaTeX would pass to <tt>\directlua</tt> a double hash for each hash, i.e. each <tt>#</tt> would be turned into <tt>##</tt>. That is normal TeX behavior, but unwanted here.

As for the commen sign <tt>%</tt>, it is useful in Lua when manipulating strings. If it weren't escaped it would discard parts of the code when TeX reads it, and a mutilated version of the input would be passed to the Lua interpreter. In turn, discarding a line by commenting it in <tt>\luacode</tt> should be done with the Lua comment <tt>--</tt>.

=== Active characters ===

The <tt>~</tt> character is generally active and used as a no-break space in TeX. It it were passed as is to <tt>\directlua</tt>, it would expand to uninterpretable control sequences, whereas in Lua it is used to form the unequal operator <tt>~=</tt>.

Other possible active characters should be taken care of, but which characters are active is unpredictable; punctuation marks might be so to accommodate special spacing, as with LaTeX's ''babel'' package, but such tricks are unlikely to survive in LuaTeX (cleaner methods exist that add a space before punctuation marks when necessary).

=== Other characters ===

When processing verbatim text in TeX, one generally also changes the catcodes of <tt>$</tt>, <tt>&</tt>, <tt>^</tt>, <tt>_</tt> and the space character, because they too are special. When passed to the Lua interpreter, though, their usual catcodes won't do any harm, that is why they are left unmodified here.

== \luaescapestring ==

Although it can't do all of what's been explained, the <tt>\luaescapestring</tt> command might be useful in some cases: it expands its argument (which must be enclosed in ''real'' braces) fully, then modify it so that dangerous characters are escaped: backslashes, hashes, quotes and line ends. For instance:

<pre>
\def\macro{"\noexpand\foo"}
\luacode
myvar = "\luaescapestring{\macro}"
\endluacode
</pre>

will be passed to Lua as

<pre>
myvar = "\"\\foo \""
</pre>

so that <tt>myvar</tt> is defined as <tt>"\foo "</tt>, with the quotes as parts of it. Note that the trailing space after <tt>\foo</tt> still happens.

= From Lua to TeX =

Inside Lua code, one can pass strings to be processed by TeX with the functions <tt>tex.print()</tt>, <tt>tex.sprint()</tt> and <tt>tex.tprint()</tt>. All such calls are processed at the end of a <tt>\directlua</tt> call, even though they might happen in the middle of the code. This behavior is worth noting because it might be surprising in some cases, although it is generally harmless.

== tex.print() ==

This function receives as its argument(s) either one or more strings or an array of strings. Each string is processed as an input line: an end-of-line character is appended (except to the last string), and TeX is in state <tt>newline</tt> when processing it (i.e. leading spaces are skipped). Hence the two equivalent calls:

<pre>
tex.print("a", "b")
tex.print({"a", "b"})
</pre>

are both interpreted by TeX as would the following two lines:

<pre>
a
b
</pre>

Thus `a b' is produced, since line ends normally produce a space.

The function can also take an optional number as its first argument; it is interpreted as referring to a catcode table (as defined by <tt>\initcatcodetable</tt> and <tt>\savecatcodetable</tt>), and each line is processed by TeX with that catcode regime. For instance (note that with such a minimal catcode table, braces don't even have their usual values):

<pre>
\bgroup
\initcatcodetable1
\catcode`\_=0
\savecatcodetable1
\egroup

\directlua{tex.print(1, "_TeX")}
</pre>

The string will be read with <tt>_</tt> as an escape character, and thus interpreted as the command commonly known as <tt>\TeX</tt>. The catcode regime holds only for the strings passed to <tt>tex.print()</tt> and the rest of the document isn't affected.

If the optional number is <tt>-1</tt>, or points to an invalid (i.e. undefined) catcode table, then the strings are processed with the current catcodes, as if there was no optional argument. If it is <tt>-2</tt>, then the strings are read as if the result of <tt>\detokenize</tt>: all characters have catcode 12 (i.e. `other', characters that have no function beside representing themselves), except space, which has catcode 10 (as usual).

== tex.sprint() ==

Like <tt>tex.print()</tt>, this function can receive either one or more strings or an array of strings, with an optional number as its first argument pointing to a catcode table. Unlike <tt>tex.print()</tt>, however, each string is processed as if TeX were in the middle of a line and not at the beginning of a new one: spaces aren't skipped, no end-of-line character is added and trailing spaces aren't ignored. Thus:

<pre>
tex.sprint("a", "b")
</pre>

is interpreted by TeX as

<pre>
ab
</pre>

without any space inbetween.

== tex.tprint() ==

This function takes an unlimited number of tables as its arguments; each table must be an array of strings, with the first entry optionally being a number pointing to a catcode table. Then each table is processed as if passed to <tt>tex.sprint()</tt>. Thus:

<pre>
tex.tprint({1, "a", "b"}, {"c", "d"})
</pre>

is equivalent to

<pre>
tex.sprint(1, "a", "b")
tex.sprint("c", "d")
</pre>

= The expansion of \directlua =

A call to <tt>\directlua</tt> is fully expandable; i.e. it can be used in contexts where full expansion is required, as in:

<pre>
\csname\directlua{tex.print("TeX")}\endcsname
</pre>

which is a somewhat convoluted way of saying <tt>\TeX</tt>. Besides, since Lua code is processed at once, things that were previously unthinkable can now be done easily. For instance, it is impossible to perform an assignment in an <tt>\edef</tt> by TeX's traditional means. I.e. the following:

<pre>
\edef\macro{\count1=5}
</pre>

defines <tt>\macro</tt> as <tt>\count1=5</tt> but doesn't perform the assignment (the <tt>\edef</tt> does nothing more than a simple <tt>\def</tt>). After the definition, the value of <tt>\count1</tt> hasn't changed. The same is not true, though, if such an assigment is made with Lua code. The following:

<pre>
\edef\macro{\directlua{tex.count[1] = 5}}
</pre>

defines <tt>\macro</tt> emptily (since nothing remains after <tt>\directlua</tt> has been processed) ''and'' sets count 1 to 5. Since such a behavior is totally unexpected in normal TeX, one should be wary when using <tt>\directlua</tt> in such contexts.

Traversing tokens

2010-12-08T08:40:14Z

Patrick: minor correction

= Traversing (or tracing) tokens =

Add the following where you want to start tracing tokens:

<pre>
\directlua{
callback.register('token_filter',
function()
t = token.get_next()
if (t[3]==0) then
texio.write_nl('term and log', 'CHAR TOKEN char=' .. unicode.utf8.char(t[2]) .. ' catcode=' .. t[1])
else
texio.write_nl('term and log', 'CSEQ TOKEN name=' .. token.command_name(t) .. ' code=' .. t[1] .. ' id=' .. t[3])
end
return t
end )
}
</pre>

The anonymous function reads tokens using <tt>token.get_next()</tt>, processes them, and then supplies them (or different ones) to TeX by its <tt>return</tt> value. The function unicode.utf8.char reads a numeric value and produces a UTF-8 character. For all functions in the string library of lua you have the equivalents functions in the unicode.utf8 library which is loaded by luatex.

When I apply this to file

<pre>
\font\x=omlgc
\x VAT bl? bla bla bla
</pre>

I get the following:

<pre>
This is luaTeX, Version 3.141592-snapshot-2007062922 (Web2C 7.5.6) (format= 2007.6.25) 4 JUL 2007 00:14
**&plain test
(
CHAR TOKEN char= catcode=10
CSEQ TOKEN name=register code=93 id=1115710
CSEQ TOKEN name=expand_after code=118 id=1114234
CHAR TOKEN char== catcode=12
CHAR TOKEN char=o catcode=11
CHAR TOKEN char=o catcode=11
CHAR TOKEN char=m catcode=11
CHAR TOKEN char=l catcode=11
CHAR TOKEN char=g catcode=11
CHAR TOKEN char=c catcode=11
CHAR TOKEN char= catcode=10
CSEQ TOKEN name=def_font code=92 id=1114234
CSEQ TOKEN name=def_font code=92 id=1114234
CSEQ TOKEN name=def_font code=92 id=1114234
CSEQ TOKEN name=def_font code=92 id=1114234
CSEQ TOKEN name=def_font code=92 id=1114234
CHAR TOKEN char=V catcode=11
CHAR TOKEN char=V catcode=11
CHAR TOKEN char=A catcode=11
CHAR TOKEN char=T catcode=11
CHAR TOKEN char= catcode=10
CHAR TOKEN char=b catcode=11
CHAR TOKEN char=l catcode=11
CHAR TOKEN char=? catcode=12
CHAR TOKEN char= catcode=10
CHAR TOKEN char=b catcode=11
CHAR TOKEN char=l catcode=11
CHAR TOKEN char=a catcode=11
CHAR TOKEN char= catcode=10
CHAR TOKEN char=b catcode=11
CHAR TOKEN char=l catcode=11
CHAR TOKEN char=a catcode=11
CHAR TOKEN char= catcode=10
CHAR TOKEN char=b catcode=11
CHAR TOKEN char=l catcode=11
CHAR TOKEN char=a catcode=11
CHAR TOKEN char= catcode=10
CSEQ TOKEN name=par_end code=13 id=1114870
...
</pre>

If you wondering why the letters o and V appear twice, here are the answers I received from Taco:

=== why do I get the o character token twice?===

This is because of how the filename scanning works: it has
to pre-scan the first character to decide between the two
allowed syntaxes

<pre>
\font\f=omlgc
</pre>
and
<pre>
\font\f={omlgc}
</pre>

=== why do I get the V character token twice? ===

Because it is a character encoutered in vmode. TeX switches
to hmode, then re-reads the character.

[[User:Yannis.Haralambous|Yannis]] 04:40, 4 July 2007 (EDT)

User:Patrick

2010-12-08T08:23:47Z

Patrick: User page

LuaTeX addict. Contact me at patrick <at> gundla <dot> ch.

Explore the table obtained from a TrueType font

2010-12-08T08:22:10Z

Patrick: forgot macro

Define the following macro in your TeX file:

<pre>
\def\studyttffont#1{\directlua{
function printtable (ind,tableau,prof)
local k,l
texio.write_nl(string.rep("...",prof) .. ind .. " -> {")
prof=prof+1
for k,l in pairs(tableau) do
if (type(l)=="table") then
printtable(k,l,prof)
else
texio.write_nl(string.rep("...",prof) .. k .. " -> " .. tostring(l))
end
end
texio.write_nl(string.rep("...",prof-1) .. "}")
end

filename = kpse.find_file("#1","truetype fonts")
ttffont = fontloader.to_table(fontloader.open(filename))
printtable("#1",ttffont,0)
}}
</pre>

and call it giving the name of the font you want to explore (the font must be in some directory known by kpse). Here is what you get for the "Gentium" font (strongly truncated, otherwise it would be too long for wiki):

<pre>
(
gentium -> {
...mark_class_cnt -> 0
...glyphs -> {
......1 -> {
.........boundingbox -> {
............1 -> 0
............2 -> 0
............3 -> 0
............4 -> 0
.........}
.........name -> .null
.........unicodeenc -> 0
.........width -> 0
......}
......2 -> {
.........boundingbox -> {
............1 -> 0
............2 -> 0
............3 -> 0
............4 -> 0
.........}
.........name -> nonmarkingreturn
.........unicodeenc -> 12
.........width -> 451
......}
......3 -> {
.........boundingbox -> {
............1 -> 0
............2 -> 0
............3 -> 0
............4 -> 0
.........}
.........name -> space
.........unicodeenc -> 32
.........width -> 451
......}
......4 -> {
.........boundingbox -> {
............1 -> 154
............2 -> -40
............3 -> 391
............4 -> 1480
.........}
.........name -> exclam
.........unicodeenc -> 33
.........width -> 557
......}
......5 -> {
.........boundingbox -> {
............1 -> 160
............2 -> 813
............3 -> 711
............4 -> 1480
.........}
.........name -> quotedbl
.........unicodeenc -> 34
.........width -> 870
......}
</pre>

Explanation: the <tt>glyphs</tt> table is indexed by glyph indexes. Each glyph index maps to a bounding box, a potential (PostScript) name, a potential Unicode character, and to its width. Bounding box and width are given in font units.

<pre>
......6 -> {
.........boundingbox -> {
............1 -> 87
............2 -> 127
............3 -> 982
............4 -> 1292
.........}
.........name -> numbersign
.........unicodeenc -> 35
.........width -> 961
......}
......7 -> {
.........boundingbox -> {
............1 -> 88
............2 -> -174
............3 -> 862
............4 -> 1384
.........}
.........name -> dollar
.........unicodeenc -> 36
.........width -> 961
......}
......8 -> {
.........boundingbox -> {
............1 -> 76
............2 -> -34
............3 -> 1337
............4 -> 1237
.........}
.........name -> percent
.........unicodeenc -> 37
.........width -> 1413
......}
......9 -> {
.........boundingbox -> {
............1 -> 80
............2 -> -30
............3 -> 1346
............4 -> 1450
.........}
.........name -> ampersand
.........unicodeenc -> 38
.........width -> 1366
......}
......10 -> {
.........boundingbox -> {
............1 -> 160
............2 -> 813
............3 -> 352
............4 -> 1480
.........}
.........name -> quotesingle
.........unicodeenc -> 39
.........width -> 512
......}
......11 -> {
.........boundingbox -> {
............1 -> 121
............2 -> -315
............3 -> 610
............4 -> 1600
.........}
.........name -> parenleft
.........unicodeenc -> 40
.........possub -> {
............1 -> {
...............flags -> 0
...............type -> substitution
...............subs -> {
..................variant -> parenright
...............}
...............script_lang_index -> 1
...............tag -> rtla
............}
.........}
.........width -> 649
......}
</pre>

Here we have the special case of a glyph which has an attached OpenType transformation. In this case it is a (simple) substitution from the GSUB table. This substitution should happen when we are in the script+language combination indexed by number 1, and the OpenType which calls it is <tt>rtla</tt>, also known as �Right-to-left alternates�. The glyph �parenleft� is substituted by the glyph �parenright�.

<pre>
......12 -> {
.........boundingbox -> {
............1 -> 41
............2 -> -315
............3 -> 530
............4 -> 1600
.........}
.........name -> parenright
.........unicodeenc -> 41
.........possub -> {
............1 -> {
...............flags -> 0
...............type -> substitution
...............subs -> {
..................variant -> parenleft
...............}
...............script_lang_index -> 1
...............tag -> rtla
............}
.........}
.........width -> 649
......}
......13 -> {
.........boundingbox -> {
............1 -> 70
............2 -> 709
............3 -> 846
............4 -> 1561
.........}
.........name -> asterisk
.........unicodeenc -> 42
.........width -> 915
......}
......14 -> {
.........boundingbox -> {
............1 -> 61
............2 -> 205
............3 -> 778
............4 -> 920
.........}
.........name -> plus
.........unicodeenc -> 43
.........width -> 840
......}
......15 -> {
.........boundingbox -> {
............1 -> 88
............2 -> -305
............3 -> 391
............4 -> 230
.........}
.........kerns -> {
............1 -> {
...............char -> quotedblright
...............off -> -220
...............flags -> 0
...............sli -> 0
............}
............2 -> {
...............char -> quoteright
...............off -> -220
...............flags -> 0
...............sli -> 0
............}
.........}
.........name -> comma
.........unicodeenc -> 44
.........width -> 469
......}
</pre>

Here we have some kernings: the �comma� glyph gets kerned by -220 units when in front of �quoteright� or �quotedblright�.

<pre>
......16 -> {
.........boundingbox -> {
............1 -> 61
............2 -> 455
............3 -> 629
............4 -> 569
.........}
.........name -> hyphen
.........unicodeenc -> 45
.........width -> 690
......}
......17 -> {
.........boundingbox -> {
............1 -> 131
............2 -> -40
............3 -> 369
............4 -> 236
.........}
.........kerns -> {
............1 -> {
...............char -> quotedblright
...............off -> -220
...............flags -> 0
...............sli -> 0
............}
............2 -> {
...............char -> quoteright
...............off -> -220
...............flags -> 0
...............sli -> 0
............}
.........}
.........name -> period
.........unicodeenc -> 46
.........width -> 469
......}
......18 -> {
.........boundingbox -> {
............1 -> 49
............2 -> -335
............3 -> 909
............4 -> 1600
.........}
.........name -> slash
.........unicodeenc -> 47
.........width -> 961
......}
......19 -> {
.........boundingbox -> {
............1 -> 76
............2 -> -30
............3 -> 885
............4 -> 1235
.........}
.........name -> zero
.........unicodeenc -> 48
.........width -> 961
......}
......20 -> {
.........boundingbox -> {
............1 -> 139
............2 -> 0
............3 -> 840
............4 -> 1245
.........}
.........name -> one
.........unicodeenc -> 49
.........width -> 961
......}
......21 -> {
.........boundingbox -> {
............1 -> 98
............2 -> 0
............3 -> 836
............4 -> 1235
.........}
.........name -> two
.........unicodeenc -> 50
.........width -> 961
......}
......22 -> {
.........boundingbox -> {
............1 -> 68
............2 -> -30
............3 -> 825
............4 -> 1235
.........}
.........name -> three
.........unicodeenc -> 51
.........width -> 961
......}
......23 -> {
.........boundingbox -> {
............1 -> 55
............2 -> 0
............3 -> 874
............4 -> 1245
.........}
.........name -> four
.........unicodeenc -> 52
.........width -> 961
......}
......24 -> {
.........boundingbox -> {
............1 -> 78
............2 -> -30
............3 -> 836
............4 -> 1223
.........}
.........name -> five
.........unicodeenc -> 53
.........width -> 961
......}
......25 -> {
.........boundingbox -> {
............1 -> 111
............2 -> -30
............3 -> 881
............4 -> 1249
.........}
.........name -> six
.........unicodeenc -> 54
.........width -> 961
......}
......26 -> {
.........boundingbox -> {
............1 -> 104
............2 -> -30
............3 -> 885
............4 -> 1204
.........}
.........name -> seven
.........unicodeenc -> 55
.........width -> 961
......}
......27 -> {
.........boundingbox -> {
............1 -> 94
............2 -> -30
............3 -> 864
............4 -> 1235
.........}
.........name -> eight
.........unicodeenc -> 56
.........width -> 961
......}
......28 -> {
.........boundingbox -> {
............1 -> 100
............2 -> -43
............3 -> 870
............4 -> 1235
.........}
.........name -> nine
.........unicodeenc -> 57
.........width -> 961
......}
......29 -> {
.........boundingbox -> {
............1 -> 131
............2 -> -40
............3 -> 369
............4 -> 961
.........}
.........name -> colon
.........unicodeenc -> 58
.........width -> 469
......}
......30 -> {
.........boundingbox -> {
............1 -> 88
............2 -> -305
............3 -> 391
............4 -> 961
.........}
.........name -> semicolon
.........unicodeenc -> 59
.........width -> 469
......}
......31 -> {
.........boundingbox -> {
............1 -> 61
............2 -> 256
............3 -> 838
............4 -> 899
.........}
.........name -> less
.........unicodeenc -> 60
.........possub -> {
............1 -> {
...............flags -> 0
...............type -> substitution
...............subs -> {
..................variant -> greater
...............}
...............script_lang_index -> 1
...............tag -> rtla
............}
.........}
.........width -> 899
......}
......32 -> {
.........boundingbox -> {
............1 -> 61
............2 -> 354
............3 -> 838
............4 -> 768
.........}
.........name -> equal
.........unicodeenc -> 61
.........width -> 899
......}
......33 -> {
.........boundingbox -> {
............1 -> 61
............2 -> 256
............3 -> 838
............4 -> 899
.........}
.........name -> greater
.........unicodeenc -> 62
.........possub -> {
............1 -> {
...............flags -> 0
...............type -> substitution
...............subs -> {
..................variant -> less
...............}
...............script_lang_index -> 1
...............tag -> rtla
............}
.........}
.........width -> 899
......}
......34 -> {
.........boundingbox -> {
............1 -> 80
............2 -> -40
............3 -> 807
............4 -> 1480
.........}
.........name -> question
.........unicodeenc -> 63
.........width -> 887
......}
......35 -> {
.........boundingbox -> {
............1 -> 70
............2 -> -375
............3 -> 1583
............4 -> 1333
.........}
.........name -> at
.........unicodeenc -> 64
.........width -> 1653
......}
......36 -> {
.........boundingbox -> {
............1 -> 0
............2 -> 0
............3 -> 1198
............4 -> 1317
.........}
.........kerns -> {
............1 -> {
...............char -> quotedblright
...............off -> -200
...............flags -> 0
...............sli -> 0
............}
............2 -> {
...............char -> quoteright
...............off -> -200
...............flags -> 0
...............sli -> 0
............}
............3 -> {
...............char -> y
...............off -> -100
...............flags -> 0
...............sli -> 0
............}
............4 -> {
...............char -> w
...............off -> -80
...............flags -> 0
...............sli -> 0
............}
............5 -> {
...............char -> v
...............off -> -100
...............flags -> 0
...............sli -> 0
............}
............6 -> {
...............char -> Y
...............off -> -180
...............flags -> 0
...............sli -> 0
............}
............7 -> {
...............char -> W
...............off -> -120
...............flags -> 0
...............sli -> 0
............}
............8 -> {
...............char -> V
...............off -> -160
...............flags -> 0
...............sli -> 0
............}
............9 -> {
...............char -> U
...............off -> -80
...............flags -> 0
...............sli -> 0
............}
............10 -> {
...............char -> T
...............off -> -100
...............flags -> 0
...............sli -> 0
............}
............11 -> {
...............char -> Q
...............off -> -30
...............flags -> 0
...............sli -> 0
............}
............12 -> {
...............char -> O
...............off -> -30
...............flags -> 0
...............sli -> 0
............}
............13 -> {
...............char -> G
...............off -> -30
...............flags -> 0
...............sli -> 0
............}
............14 -> {
...............char -> C
...............off -> -30
...............flags -> 0
...............sli -> 0
............}
.........}
.........name -> A
.........unicodeenc -> 65
.........width -> 1219
......}
......37 -> {
.........boundingbox -> {
............1 -> 41
............2 -> -14
............3 -> 1016
............4 -> 1290
.........}
.........name -> B
.........unicodeenc -> 66
.........width -> 1126
......}
......38 -> {
.........boundingbox -> {
............1 -> 70
............2 -> -30
............3 -> 1018
............4 -> 1290
.........}
.........kerns -> {
............1 -> {
...............char -> A
...............off -> -60
...............flags -> 0
...............sli -> 0
............}
.........}
.........name -> C
.........unicodeenc -> 67
.........width -> 1098
......}
......39 -> {
.........boundingbox -> {
............1 -> 41
............2 -> 0
............3 -> 1124
............4 -> 1290
.........}
.........kerns -> {
............1 -> {
...............char -> Y
...............off -> -60
...............flags -> 0
...............sli -> 0
............}
............2 -> {
...............char -> W
...............off -> -20
...............flags -> 0
...............sli -> 0
............}
............3 -> {
...............char -> V
...............off -> 40
...............flags -> 0
...............sli -> 0
............}
............4 -> {
...............char -> A
...............off -> -60
...............flags -> 0
...............sli -> 0
............}
.........}
.........name -> D
.........unicodeenc -> 68
.........width -> 1214
......}

another 35,850 lines of code

...}
...fontstyle_id -> 0
...units_per_em -> 2048
</pre>

2,048 units per em means that to obtain TeX quads we need to divide by that number.

<pre>
...glyphcnt -> 1699
...gasp_cnt -> 3
...weight -> Book
...pfminfo -> {
......vlinegap -> 0
......os2_subyoff -> 1200
......hheadascent_add -> 1
......os2_strikeypos -> 512
......hhead_descent -> -565
......os2_typolinegap -> 90
......typodescent_add -> 1
......weight -> 400
......hhead_ascent -> 1759
......os2_strikeysize -> 102
......panose_set -> 1
......os2_breakchar -> 0
......pfmset -> 1
......os2_typoascent -> -88
......winascent_add -> 1
......firstchar -> 32
......os2_capheight -> 0
......windescent_add -> 1
......vheadset -> 0
......os2_subxsize -> 856
......os2_xheight -> 0
......os2_supxoff -> 0
......hheaddescent_add -> 1
......os2_defaultchar -> 0
......linegap -> 45
......os2_family_class -> 0
......os2_vendor -> SIL
......os2_supyoff -> 1200
......os2_winascent -> 1759
......os2_subysize -> 856
......avgwidth -> 845
......os2_subxoff -> 0
......os2_supysize -> 856
......os2_windescent -> 565
......os2_typodescent -> -69
......hheadset -> 1
......fstype -> 0
......os2_supxsize -> 856
......lastchar -> 64260
......panose -> {
.........serifstyle -> Any
.........armstyle -> Any
.........contrast -> Medium
.........familytype -> Text and Display
.........proportion -> Modern
.........strokevariation -> Any
.........weight -> Book
.........xheight -> Constant/Large
.........midline -> Any
.........letterform -> Normal/Contact
......}
......width -> 5
......pfmfamily -> 17
......subsuper_set -> 1
......typoascent_add -> 1
...}
...fullname -> GentiumAlt
...map -> {
......map -> {
.........32 -> 3
.........33 -> 4
.........34 -> 5
.........35 -> 6
.........36 -> 7
.........37 -> 8
.........38 -> 9
.........39 -> 10
.........40 -> 11

another 1,679 lines

.........65764 -> 1688
.........65765 -> 1689
.........65766 -> 1690
.........65767 -> 1691
.........65768 -> 1692
.........65769 -> 1693
.........65770 -> 1694
.........65771 -> 1695
.........65772 -> 1696
.........65773 -> 1697
.........65774 -> 1698
......}
......enc -> {
.........1 -> {
............has_2byte -> 1
............char_max -> 0
............only_1byte -> 0
............is_unicodebmp -> 1
............char_cnt -> 65536
............high_page -> 0
............has_1byte -> 0
............low_page -> 0
............builtin -> 1
............enc_name -> UnicodeBmp
............hidden -> 1
.........}
.........2 -> {
............has_2byte -> 0
............char_max -> 0
............only_1byte -> 0
............is_original -> 1
............char_cnt -> 0
............high_page -> 0
............has_1byte -> 0
............low_page -> 0
............builtin -> 1
............enc_name -> Original
............hidden -> 1
.........}
.........3 -> {
............has_2byte -> 0
............char_max -> 0
............only_1byte -> 0
............char_cnt -> 0
............is_custom -> 1
............has_1byte -> 0
............low_page -> 0
............builtin -> 1
............enc_name -> Custom
............high_page -> 0
............hidden -> 1
.........}
.........4 -> {
............has_2byte -> 0
............char_max -> 0
............only_1byte -> 1
............char_cnt -> 256
............high_page -> 0
............has_1byte -> 1
............low_page -> 0
............builtin -> 1
............enc_name -> TeX-Base-Encoding
............unicode -> {
...............1 -> 729
...............2 -> 64257
...............3 -> 64258
...............4 -> 8260
...............5 -> 733
...............6 -> 321
...............7 -> 322
...............8 -> 731

another 238 lines

...............247 -> 247
...............248 -> 248
...............249 -> 249
...............250 -> 250
...............251 -> 251
...............252 -> 252
...............253 -> 253
...............254 -> 254
...............255 -> 255
...............0 -> 0
............}
............hidden -> 1
.........}
......}
......backmax -> 1699
......enccount -> 65775
......encmax -> 65775
......backmap -> {
.........1 -> 65537
.........2 -> 65538
.........3 -> 32
.........4 -> 33
.........5 -> 34
.........6 -> 35
.........7 -> 36
.........8 -> 37
.........9 -> 38
.........10 -> 39
.........11 -> 40
.........12 -> 41
.........13 -> 42

another 1,679 lines

.........1693 -> 65769
.........1694 -> 65770
.........1695 -> 65771
.........1696 -> 65772
.........1697 -> 65773
.........1698 -> 65774
.........0 -> 65536
......}
...}
...italicangle -> 0
...strokedfont -> 0
...cidinfo -> {
......supplement -> 0
......version -> 0
...}
...familyname -> GentiumAlt
...script_lang -> {
......1 -> {
.........script -> latn
.........langs -> {
............1 -> dflt
.........}
......}
...}
</pre>

The table <tt>script_lang</tt> contains scripts and languages. Here we have script <tt>latn</tt> (Latin) with default language.

<pre>
...upos -> -48
...version -> 1.01; 2003; initial SIL release
...glyphmax -> 1699
...names -> {
......1 -> {
.........lang -> English (US)
.........names -> {
............designerurl -> http://www.sil.org/~gaultney
............venderurl -> http://www.sil.org/
............compatfull -> GentiumAlt
............trademark -> Gentium is a trademark of SIL International.
............license -> This font is the property of SIL International. It is distributed as copyrighted freeware. You may use this software without any charge and may distribute it, as is, to others. Commercial distribution of this software is restricted
without prior written permission. If you wish to distribute this software commercially, contact SIL for details on obtaining a license. You may not rent or lease the software, nor may you modify, adapt, translate, reverse engineer, decompile, or disassemb
le the software. You may not make derivative fonts from this software. THE SOFTWARE AND RELATED FILES ARE PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND.
............fullname -> GentiumAlt
............preffamilyname -> GentiumAlt
............designer -> J. Victor Gaultney
............copyright -> Copyright (c) SIL International 2003. All rights reserved.
............version -> Version 1.01; 2003; initial SIL release
............prefmodifiers -> Regular
............family -> GentiumAlt
............postscriptname -> GentiumAlt
............subfamily -> Regular
............uniqueid -> SIL: GentiumAlt: 2003
.........}
......}
...}
...gentags -> {
......tt_cur -> 0
...}
...modificationtime -> 1063146346
...fontname -> GentiumAlt
...changed -> 0
...vertical_origin -> 0
...table_version -> 0.1
...hasvmetrics -> 0
...creationtime -> 1032220800
...order2 -> 1
...head_optimized_for_cleartype -> 0
...strokewidth -> 0
...design_range_bottom -> 0
...design_size -> 0
...design_range_top -> 0
...sli_cnt -> 1
...macstyle -> -1
...ttf_tables -> {
......1 -> {
.........len -> 32
.........data ->
.........maxlen -> 0
.........tag -> maxp
......}
...}
...copyright -> Copyright (c) SIL International 2003. All rights reserved.
...os2_version -> 1
...gasp_version -> 0
...uni_interp -> none
...uwidth -> 14
...weight_width_slope_only -> 0
...descent -> 410
...ascent -> 1638
...uniqueid -> 0
} )
</pre>

Explore the table obtained from a TrueType font

2010-12-08T08:21:18Z

Patrick: fontforge -> fontloader

Define the following macro in your TeX file:

<pre>
function printtable (ind,tableau,prof)
local k,l
texio.write_nl(string.rep("...",prof) .. ind .. " -> {")
prof=prof+1
for k,l in pairs(tableau) do
if (type(l)=="table") then
printtable(k,l,prof)
else
texio.write_nl(string.rep("...",prof) .. k .. " -> " .. tostring(l))
end
end
texio.write_nl(string.rep("...",prof-1) .. "}")
end

filename = kpse.find_file("#1","truetype fonts")
ttffont = fontloader.to_table(fontloader.open(filename))
printtable("#1",ttffont,0)
</pre>

and call it giving the name of the font you want to explore (the font must be in some directory known by kpse). Here is what you get for the "Gentium" font (strongly truncated, otherwise it would be too long for wiki):

<pre>
(
gentium -> {
...mark_class_cnt -> 0
...glyphs -> {
......1 -> {
.........boundingbox -> {
............1 -> 0
............2 -> 0
............3 -> 0
............4 -> 0
.........}
.........name -> .null
.........unicodeenc -> 0
.........width -> 0
......}
......2 -> {
.........boundingbox -> {
............1 -> 0
............2 -> 0
............3 -> 0
............4 -> 0
.........}
.........name -> nonmarkingreturn
.........unicodeenc -> 12
.........width -> 451
......}
......3 -> {
.........boundingbox -> {
............1 -> 0
............2 -> 0
............3 -> 0
............4 -> 0
.........}
.........name -> space
.........unicodeenc -> 32
.........width -> 451
......}
......4 -> {
.........boundingbox -> {
............1 -> 154
............2 -> -40
............3 -> 391
............4 -> 1480
.........}
.........name -> exclam
.........unicodeenc -> 33
.........width -> 557
......}
......5 -> {
.........boundingbox -> {
............1 -> 160
............2 -> 813
............3 -> 711
............4 -> 1480
.........}
.........name -> quotedbl
.........unicodeenc -> 34
.........width -> 870
......}
</pre>

Explanation: the <tt>glyphs</tt> table is indexed by glyph indexes. Each glyph index maps to a bounding box, a potential (PostScript) name, a potential Unicode character, and to its width. Bounding box and width are given in font units.

<pre>
......6 -> {
.........boundingbox -> {
............1 -> 87
............2 -> 127
............3 -> 982
............4 -> 1292
.........}
.........name -> numbersign
.........unicodeenc -> 35
.........width -> 961
......}
......7 -> {
.........boundingbox -> {
............1 -> 88
............2 -> -174
............3 -> 862
............4 -> 1384
.........}
.........name -> dollar
.........unicodeenc -> 36
.........width -> 961
......}
......8 -> {
.........boundingbox -> {
............1 -> 76
............2 -> -34
............3 -> 1337
............4 -> 1237
.........}
.........name -> percent
.........unicodeenc -> 37
.........width -> 1413
......}
......9 -> {
.........boundingbox -> {
............1 -> 80
............2 -> -30
............3 -> 1346
............4 -> 1450
.........}
.........name -> ampersand
.........unicodeenc -> 38
.........width -> 1366
......}
......10 -> {
.........boundingbox -> {
............1 -> 160
............2 -> 813
............3 -> 352
............4 -> 1480
.........}
.........name -> quotesingle
.........unicodeenc -> 39
.........width -> 512
......}
......11 -> {
.........boundingbox -> {
............1 -> 121
............2 -> -315
............3 -> 610
............4 -> 1600
.........}
.........name -> parenleft
.........unicodeenc -> 40
.........possub -> {
............1 -> {
...............flags -> 0
...............type -> substitution
...............subs -> {
..................variant -> parenright
...............}
...............script_lang_index -> 1
...............tag -> rtla
............}
.........}
.........width -> 649
......}
</pre>

Here we have the special case of a glyph which has an attached OpenType transformation. In this case it is a (simple) substitution from the GSUB table. This substitution should happen when we are in the script+language combination indexed by number 1, and the OpenType which calls it is <tt>rtla</tt>, also known as �Right-to-left alternates�. The glyph �parenleft� is substituted by the glyph �parenright�.

<pre>
......12 -> {
.........boundingbox -> {
............1 -> 41
............2 -> -315
............3 -> 530
............4 -> 1600
.........}
.........name -> parenright
.........unicodeenc -> 41
.........possub -> {
............1 -> {
...............flags -> 0
...............type -> substitution
...............subs -> {
..................variant -> parenleft
...............}
...............script_lang_index -> 1
...............tag -> rtla
............}
.........}
.........width -> 649
......}
......13 -> {
.........boundingbox -> {
............1 -> 70
............2 -> 709
............3 -> 846
............4 -> 1561
.........}
.........name -> asterisk
.........unicodeenc -> 42
.........width -> 915
......}
......14 -> {
.........boundingbox -> {
............1 -> 61
............2 -> 205
............3 -> 778
............4 -> 920
.........}
.........name -> plus
.........unicodeenc -> 43
.........width -> 840
......}
......15 -> {
.........boundingbox -> {
............1 -> 88
............2 -> -305
............3 -> 391
............4 -> 230
.........}
.........kerns -> {
............1 -> {
...............char -> quotedblright
...............off -> -220
...............flags -> 0
...............sli -> 0
............}
............2 -> {
...............char -> quoteright
...............off -> -220
...............flags -> 0
...............sli -> 0
............}
.........}
.........name -> comma
.........unicodeenc -> 44
.........width -> 469
......}
</pre>

Here we have some kernings: the �comma� glyph gets kerned by -220 units when in front of �quoteright� or �quotedblright�.

<pre>
......16 -> {
.........boundingbox -> {
............1 -> 61
............2 -> 455
............3 -> 629
............4 -> 569
.........}
.........name -> hyphen
.........unicodeenc -> 45
.........width -> 690
......}
......17 -> {
.........boundingbox -> {
............1 -> 131
............2 -> -40
............3 -> 369
............4 -> 236
.........}
.........kerns -> {
............1 -> {
...............char -> quotedblright
...............off -> -220
...............flags -> 0
...............sli -> 0
............}
............2 -> {
...............char -> quoteright
...............off -> -220
...............flags -> 0
...............sli -> 0
............}
.........}
.........name -> period
.........unicodeenc -> 46
.........width -> 469
......}
......18 -> {
.........boundingbox -> {
............1 -> 49
............2 -> -335
............3 -> 909
............4 -> 1600
.........}
.........name -> slash
.........unicodeenc -> 47
.........width -> 961
......}
......19 -> {
.........boundingbox -> {
............1 -> 76
............2 -> -30
............3 -> 885
............4 -> 1235
.........}
.........name -> zero
.........unicodeenc -> 48
.........width -> 961
......}
......20 -> {
.........boundingbox -> {
............1 -> 139
............2 -> 0
............3 -> 840
............4 -> 1245
.........}
.........name -> one
.........unicodeenc -> 49
.........width -> 961
......}
......21 -> {
.........boundingbox -> {
............1 -> 98
............2 -> 0
............3 -> 836
............4 -> 1235
.........}
.........name -> two
.........unicodeenc -> 50
.........width -> 961
......}
......22 -> {
.........boundingbox -> {
............1 -> 68
............2 -> -30
............3 -> 825
............4 -> 1235
.........}
.........name -> three
.........unicodeenc -> 51
.........width -> 961
......}
......23 -> {
.........boundingbox -> {
............1 -> 55
............2 -> 0
............3 -> 874
............4 -> 1245
.........}
.........name -> four
.........unicodeenc -> 52
.........width -> 961
......}
......24 -> {
.........boundingbox -> {
............1 -> 78
............2 -> -30
............3 -> 836
............4 -> 1223
.........}
.........name -> five
.........unicodeenc -> 53
.........width -> 961
......}
......25 -> {
.........boundingbox -> {
............1 -> 111
............2 -> -30
............3 -> 881
............4 -> 1249
.........}
.........name -> six
.........unicodeenc -> 54
.........width -> 961
......}
......26 -> {
.........boundingbox -> {
............1 -> 104
............2 -> -30
............3 -> 885
............4 -> 1204
.........}
.........name -> seven
.........unicodeenc -> 55
.........width -> 961
......}
......27 -> {
.........boundingbox -> {
............1 -> 94
............2 -> -30
............3 -> 864
............4 -> 1235
.........}
.........name -> eight
.........unicodeenc -> 56
.........width -> 961
......}
......28 -> {
.........boundingbox -> {
............1 -> 100
............2 -> -43
............3 -> 870
............4 -> 1235
.........}
.........name -> nine
.........unicodeenc -> 57
.........width -> 961
......}
......29 -> {
.........boundingbox -> {
............1 -> 131
............2 -> -40
............3 -> 369
............4 -> 961
.........}
.........name -> colon
.........unicodeenc -> 58
.........width -> 469
......}
......30 -> {
.........boundingbox -> {
............1 -> 88
............2 -> -305
............3 -> 391
............4 -> 961
.........}
.........name -> semicolon
.........unicodeenc -> 59
.........width -> 469
......}
......31 -> {
.........boundingbox -> {
............1 -> 61
............2 -> 256
............3 -> 838
............4 -> 899
.........}
.........name -> less
.........unicodeenc -> 60
.........possub -> {
............1 -> {
...............flags -> 0
...............type -> substitution
...............subs -> {
..................variant -> greater
...............}
...............script_lang_index -> 1
...............tag -> rtla
............}
.........}
.........width -> 899
......}
......32 -> {
.........boundingbox -> {
............1 -> 61
............2 -> 354
............3 -> 838
............4 -> 768
.........}
.........name -> equal
.........unicodeenc -> 61
.........width -> 899
......}
......33 -> {
.........boundingbox -> {
............1 -> 61
............2 -> 256
............3 -> 838
............4 -> 899
.........}
.........name -> greater
.........unicodeenc -> 62
.........possub -> {
............1 -> {
...............flags -> 0
...............type -> substitution
...............subs -> {
..................variant -> less
...............}
...............script_lang_index -> 1
...............tag -> rtla
............}
.........}
.........width -> 899
......}
......34 -> {
.........boundingbox -> {
............1 -> 80
............2 -> -40
............3 -> 807
............4 -> 1480
.........}
.........name -> question
.........unicodeenc -> 63
.........width -> 887
......}
......35 -> {
.........boundingbox -> {
............1 -> 70
............2 -> -375
............3 -> 1583
............4 -> 1333
.........}
.........name -> at
.........unicodeenc -> 64
.........width -> 1653
......}
......36 -> {
.........boundingbox -> {
............1 -> 0
............2 -> 0
............3 -> 1198
............4 -> 1317
.........}
.........kerns -> {
............1 -> {
...............char -> quotedblright
...............off -> -200
...............flags -> 0
...............sli -> 0
............}
............2 -> {
...............char -> quoteright
...............off -> -200
...............flags -> 0
...............sli -> 0
............}
............3 -> {
...............char -> y
...............off -> -100
...............flags -> 0
...............sli -> 0
............}
............4 -> {
...............char -> w
...............off -> -80
...............flags -> 0
...............sli -> 0
............}
............5 -> {
...............char -> v
...............off -> -100
...............flags -> 0
...............sli -> 0
............}
............6 -> {
...............char -> Y
...............off -> -180
...............flags -> 0
...............sli -> 0
............}
............7 -> {
...............char -> W
...............off -> -120
...............flags -> 0
...............sli -> 0
............}
............8 -> {
...............char -> V
...............off -> -160
...............flags -> 0
...............sli -> 0
............}
............9 -> {
...............char -> U
...............off -> -80
...............flags -> 0
...............sli -> 0
............}
............10 -> {
...............char -> T
...............off -> -100
...............flags -> 0
...............sli -> 0
............}
............11 -> {
...............char -> Q
...............off -> -30
...............flags -> 0
...............sli -> 0
............}
............12 -> {
...............char -> O
...............off -> -30
...............flags -> 0
...............sli -> 0
............}
............13 -> {
...............char -> G
...............off -> -30
...............flags -> 0
...............sli -> 0
............}
............14 -> {
...............char -> C
...............off -> -30
...............flags -> 0
...............sli -> 0
............}
.........}
.........name -> A
.........unicodeenc -> 65
.........width -> 1219
......}
......37 -> {
.........boundingbox -> {
............1 -> 41
............2 -> -14
............3 -> 1016
............4 -> 1290
.........}
.........name -> B
.........unicodeenc -> 66
.........width -> 1126
......}
......38 -> {
.........boundingbox -> {
............1 -> 70
............2 -> -30
............3 -> 1018
............4 -> 1290
.........}
.........kerns -> {
............1 -> {
...............char -> A
...............off -> -60
...............flags -> 0
...............sli -> 0
............}
.........}
.........name -> C
.........unicodeenc -> 67
.........width -> 1098
......}
......39 -> {
.........boundingbox -> {
............1 -> 41
............2 -> 0
............3 -> 1124
............4 -> 1290
.........}
.........kerns -> {
............1 -> {
...............char -> Y
...............off -> -60
...............flags -> 0
...............sli -> 0
............}
............2 -> {
...............char -> W
...............off -> -20
...............flags -> 0
...............sli -> 0
............}
............3 -> {
...............char -> V
...............off -> 40
...............flags -> 0
...............sli -> 0
............}
............4 -> {
...............char -> A
...............off -> -60
...............flags -> 0
...............sli -> 0
............}
.........}
.........name -> D
.........unicodeenc -> 68
.........width -> 1214
......}

another 35,850 lines of code

...}
...fontstyle_id -> 0
...units_per_em -> 2048
</pre>

2,048 units per em means that to obtain TeX quads we need to divide by that number.

<pre>
...glyphcnt -> 1699
...gasp_cnt -> 3
...weight -> Book
...pfminfo -> {
......vlinegap -> 0
......os2_subyoff -> 1200
......hheadascent_add -> 1
......os2_strikeypos -> 512
......hhead_descent -> -565
......os2_typolinegap -> 90
......typodescent_add -> 1
......weight -> 400
......hhead_ascent -> 1759
......os2_strikeysize -> 102
......panose_set -> 1
......os2_breakchar -> 0
......pfmset -> 1
......os2_typoascent -> -88
......winascent_add -> 1
......firstchar -> 32
......os2_capheight -> 0
......windescent_add -> 1
......vheadset -> 0
......os2_subxsize -> 856
......os2_xheight -> 0
......os2_supxoff -> 0
......hheaddescent_add -> 1
......os2_defaultchar -> 0
......linegap -> 45
......os2_family_class -> 0
......os2_vendor -> SIL
......os2_supyoff -> 1200
......os2_winascent -> 1759
......os2_subysize -> 856
......avgwidth -> 845
......os2_subxoff -> 0
......os2_supysize -> 856
......os2_windescent -> 565
......os2_typodescent -> -69
......hheadset -> 1
......fstype -> 0
......os2_supxsize -> 856
......lastchar -> 64260
......panose -> {
.........serifstyle -> Any
.........armstyle -> Any
.........contrast -> Medium
.........familytype -> Text and Display
.........proportion -> Modern
.........strokevariation -> Any
.........weight -> Book
.........xheight -> Constant/Large
.........midline -> Any
.........letterform -> Normal/Contact
......}
......width -> 5
......pfmfamily -> 17
......subsuper_set -> 1
......typoascent_add -> 1
...}
...fullname -> GentiumAlt
...map -> {
......map -> {
.........32 -> 3
.........33 -> 4
.........34 -> 5
.........35 -> 6
.........36 -> 7
.........37 -> 8
.........38 -> 9
.........39 -> 10
.........40 -> 11

another 1,679 lines

.........65764 -> 1688
.........65765 -> 1689
.........65766 -> 1690
.........65767 -> 1691
.........65768 -> 1692
.........65769 -> 1693
.........65770 -> 1694
.........65771 -> 1695
.........65772 -> 1696
.........65773 -> 1697
.........65774 -> 1698
......}
......enc -> {
.........1 -> {
............has_2byte -> 1
............char_max -> 0
............only_1byte -> 0
............is_unicodebmp -> 1
............char_cnt -> 65536
............high_page -> 0
............has_1byte -> 0
............low_page -> 0
............builtin -> 1
............enc_name -> UnicodeBmp
............hidden -> 1
.........}
.........2 -> {
............has_2byte -> 0
............char_max -> 0
............only_1byte -> 0
............is_original -> 1
............char_cnt -> 0
............high_page -> 0
............has_1byte -> 0
............low_page -> 0
............builtin -> 1
............enc_name -> Original
............hidden -> 1
.........}
.........3 -> {
............has_2byte -> 0
............char_max -> 0
............only_1byte -> 0
............char_cnt -> 0
............is_custom -> 1
............has_1byte -> 0
............low_page -> 0
............builtin -> 1
............enc_name -> Custom
............high_page -> 0
............hidden -> 1
.........}
.........4 -> {
............has_2byte -> 0
............char_max -> 0
............only_1byte -> 1
............char_cnt -> 256
............high_page -> 0
............has_1byte -> 1
............low_page -> 0
............builtin -> 1
............enc_name -> TeX-Base-Encoding
............unicode -> {
...............1 -> 729
...............2 -> 64257
...............3 -> 64258
...............4 -> 8260
...............5 -> 733
...............6 -> 321
...............7 -> 322
...............8 -> 731

another 238 lines

...............247 -> 247
...............248 -> 248
...............249 -> 249
...............250 -> 250
...............251 -> 251
...............252 -> 252
...............253 -> 253
...............254 -> 254
...............255 -> 255
...............0 -> 0
............}
............hidden -> 1
.........}
......}
......backmax -> 1699
......enccount -> 65775
......encmax -> 65775
......backmap -> {
.........1 -> 65537
.........2 -> 65538
.........3 -> 32
.........4 -> 33
.........5 -> 34
.........6 -> 35
.........7 -> 36
.........8 -> 37
.........9 -> 38
.........10 -> 39
.........11 -> 40
.........12 -> 41
.........13 -> 42

another 1,679 lines

.........1693 -> 65769
.........1694 -> 65770
.........1695 -> 65771
.........1696 -> 65772
.........1697 -> 65773
.........1698 -> 65774
.........0 -> 65536
......}
...}
...italicangle -> 0
...strokedfont -> 0
...cidinfo -> {
......supplement -> 0
......version -> 0
...}
...familyname -> GentiumAlt
...script_lang -> {
......1 -> {
.........script -> latn
.........langs -> {
............1 -> dflt
.........}
......}
...}
</pre>

The table <tt>script_lang</tt> contains scripts and languages. Here we have script <tt>latn</tt> (Latin) with default language.

<pre>
...upos -> -48
...version -> 1.01; 2003; initial SIL release
...glyphmax -> 1699
...names -> {
......1 -> {
.........lang -> English (US)
.........names -> {
............designerurl -> http://www.sil.org/~gaultney
............venderurl -> http://www.sil.org/
............compatfull -> GentiumAlt
............trademark -> Gentium is a trademark of SIL International.
............license -> This font is the property of SIL International. It is distributed as copyrighted freeware. You may use this software without any charge and may distribute it, as is, to others. Commercial distribution of this software is restricted
without prior written permission. If you wish to distribute this software commercially, contact SIL for details on obtaining a license. You may not rent or lease the software, nor may you modify, adapt, translate, reverse engineer, decompile, or disassemb
le the software. You may not make derivative fonts from this software. THE SOFTWARE AND RELATED FILES ARE PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND.
............fullname -> GentiumAlt
............preffamilyname -> GentiumAlt
............designer -> J. Victor Gaultney
............copyright -> Copyright (c) SIL International 2003. All rights reserved.
............version -> Version 1.01; 2003; initial SIL release
............prefmodifiers -> Regular
............family -> GentiumAlt
............postscriptname -> GentiumAlt
............subfamily -> Regular
............uniqueid -> SIL: GentiumAlt: 2003
.........}
......}
...}
...gentags -> {
......tt_cur -> 0
...}
...modificationtime -> 1063146346
...fontname -> GentiumAlt
...changed -> 0
...vertical_origin -> 0
...table_version -> 0.1
...hasvmetrics -> 0
...creationtime -> 1032220800
...order2 -> 1
...head_optimized_for_cleartype -> 0
...strokewidth -> 0
...design_range_bottom -> 0
...design_size -> 0
...design_range_top -> 0
...sli_cnt -> 1
...macstyle -> -1
...ttf_tables -> {
......1 -> {
.........len -> 32
.........data ->
.........maxlen -> 0
.........tag -> maxp
......}
...}
...copyright -> Copyright (c) SIL International 2003. All rights reserved.
...os2_version -> 1
...gasp_version -> 0
...uni_interp -> none
...uwidth -> 14
...weight_width_slope_only -> 0
...descent -> 410
...ascent -> 1638
...uniqueid -> 0
} )
</pre>

Main Page

2010-12-07T21:28:43Z

Patrick: Link corrected

== Welcome to the LuaTeX wiki ==

This is a wiki for [http://www.luatex.org LuaTeX], a typesetting engine derived from
[http://en.wikipedia.org/wiki/TeX TeX] that includes [http://www.lua.org Lua] as an embedded scripting language.

We are [[Special:Recentchanges|currently editing]] [[Special:Allpages|{{NUMBEROFARTICLES}}]] articles.

* [http://www.mediawiki.org/wiki/Help:Formatting How to edit wiki pages]
* [[Bug tracking|Bug tracking]]: Mantis [http://tracker.luatex.org bug tracker]

== Getting help ==

* There are two mailing lists related to LuaTeX, [http://tug.org/mailman/listinfo/luatex one for general discussions and questions about LuaTeX ("luatex users")] and [http://www.ntg.nl/mailman/listinfo/dev-luatex one more related to the development of LuaTeX ("dev-luatex")]
* There is also [http://www.tug.org/mailman/listinfo/lualatex-dev a LuaLaTeX (LaTeX with LuaTeX) mailing list].
* [http://tex.stackexchange.com/ tex.stackexchange.com] is a q&a site for questions related to TeX, and you can ask (and answer) LuaTeX related questions there.
* The [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf online reference manual (pdf)] describes every aspect of LuaTeX, but is rather technical and concise.

== Some topics from the old wiki at bluwiki.com ==

(Thanks to [http://omega.enstb.org/yannis/ Yannis Haralambous])

* An example of code [[traversing TeX nodes]] before an horizontal list goes through the line breaking engine;
* An example of code [[traversing tokens]] just before execution or expansion;
* you want to [[explore the table obtained from a TrueType font]], loaded by <tt>font.read_ttf</tt>;
* you want to [[explore the internal font table]] of a pre-loaded font or of a font you have loaded by <tt>\font</tt> and then used for at least one glyph;
* how to [[use a TrueType font]] without going through a TFM or a OFM file;
* how to do kinsoku ([[Japanese and more generally CJK typesetting]]);
* you want a newline in your log file or on the terminal? add <tt>\string\n</tt> to your string;
* you want to [[sort a token list]];
* you want to [[split a comma-separated list]];
* you want to [[encrypt your document using ROT13]];
* you want to [[typeset non-TeX files by converting them using Lua code]];
* example code to [[mirror characters with Bidi_Mirrored property]];
* using mplib to write [[metapost with LuaTeX]]

Talk:Metapost with LuaTeX

2010-12-07T21:27:34Z

Patrick: Link to old location

Old location: http://luatex.bluwiki.com/go/Metapost_with_LuaTeX

Metapost with LuaTeX

2010-12-07T21:27:15Z

Patrick: Copied from the old bluwiki.com luatex wiki

Writing directly some metapost code is now possible in LuaTeX, through MPLib. It only works in pdf mode.

To get it working, you will need three files: [http://source.contextgarden.net/tex/context/base/supp-mpl.lua supp-mpl.lua] and [http://source.contextgarden.net/tex/context/base/supp-mpl.tex supp-mpl.tex] (you need to copy/paste them) from the ConTeXt distribution, and a stupid .sty file:

<pre>
\NeedsTeXFormat{LaTeX2e}
\ProvidesPackage{mplib}

\input supp-mpl.tex

\directlua0{dofile(kpse.find_file("formathack.lua"))}
</pre>

And create a file formathack.lua containing:

<pre>
local preamble = [[
input %s ; dump ;
]]
metapost.make = function (name, mem_name)
local mpx = mplib.new {
ini_version = true,
find_file = metapost.finder,
job_name = name}
if mpx then
local result = mpx:execute(string.format(preamble,name))
mpx:finish()
end
end
function replacesuffix(filename, suffix)
return (string.gsub(filename,"%.[%a%d]+$","")) .. "." .. suffix
end
metapost.load = function (name)
local mpx = mplib.new {
ini_version = false,
mem_name = replacesuffix(name,"mem"),
find_file = finder
}
local result
if not mpx then
metapost.make(name)
metapost.report("creating %s.mem", name)
mpx = mplib.new {
ini_version = false,
mem_name = replacesuffix(name,"mem"),
find_file = finder
}
end
if not mpx then
result = { status = 99, error = "out of memory"}
end
return mpx, result
end
</pre>

Once you have all four files, the following example should work:

<pre>
\mplibcode
beginfig(1);
pickup pencircle xscaled 5 rotated 30;
draw unitsquare scaled 20 withcmykcolor (0.3,0.4,0.6,0);
endfig;
\endmplibcode
</pre>

as well as any metapost code.

Talk:Mirror characters with Bidi Mirrored property

2010-12-07T21:23:52Z

Patrick: Link to old location

Old location: http://luatex.bluwiki.com/go/Mirror_characters_with_Bidi_Mirrored_property

Mirror characters with Bidi Mirrored property

2010-12-07T21:23:34Z

Patrick: Copied from the old bluwiki.com luatex wiki

This example code for mirroring characters that has Bidi_Mirrored property (as defined by Unicode) in RTL contexts. It uses characters.data table defined in [http://source.contextgarden.net/tex/context/base/char-def.lua char-def.lua] file from ConTeXt distribution to get character properties.

<pre>
dofile("char-def.lua") -- character definition table

local function mirror(hlist)
local rtl = false
if tex.textdir == "TRT" then
rtl = true
elseif tex.textdir == "TLT" then
rtl = false
end
for n in node.traverse(hlist) do
if n.id == node.id('hlist') then
mirror(n.list)
end
if n.id == node.id("whatsit") and n.subtype == 7 then
if n.dir == "+TRT" or n.dir == "-TLT" then
rtl = true
elseif n.dir == "-TRT" or n.dir == "+TLT" then
rtl = false
end
end
if n.id == node.id('glyph') then
if rtl then
local char = n.char
local mirror = characters.data[char].mirror
if mirror then
n.char = mirror
end
end
end
end
end

callback.register('pre_linebreak_filter',
function(h)
mirror(h)
return true
end
)
</pre>

Example usage:
<pre>
\directlua{dofile("mirror.lua")}

(Left to right) {\textdir TRT (RIGHT TO LEFT)}

\textdir TRT

RIGHT [TO] LEFT (TEXT)

\textdir TLT

left [to] right (text)
\bye
</pre>

--khaledhosny

Talk:Typeset non-TeX files by converting them using Lua code

2010-12-07T21:21:37Z

Patrick: Link to old location

Old location: http://luatex.bluwiki.com/go/Typeset_non-TeX_files_by_converting_them_using_Lua_code

Typeset non-TeX files by converting them using Lua code

2010-12-07T21:21:16Z

Patrick: Copied from the old bluwiki.com luatex wiki

The following code example demonstrates how to use the file reading callback and coroutines to convert a non-TeX file into TeX code which is then typeset.

First the Lua code ("Preprocessor.lua"):

local function readFile(fn)
local file = assert(io.open(fn, "r"))
local contents = file:read("*a")
file:close()
return contents
end

local function processInputFile(contents)
-- Preamble
coroutine.yield("\\pdfoutput=1 ")

-- Process the file: Return each line
for line in contents:gmatch("(.-)[\n\r]") do
coroutine.yield(line)
end

-- Postamble
coroutine.yield("\\bye")

-- Done
coroutine.yield(nil)
end

callback.register("open_read_file", function(fileName)
local contents = readFile(fileName)
return {
reader = coroutine.wrap(function()
processInputFile(contents .. "\n")
end)
}
end)

Then a small test file ("Preprocessor.txt"):

Hello World1

Hello World2

Paragraph 3,
continuing right now
and going on over
four lines

And finally the bash shell script to tie everything together ("Preprocessor.sh"):

luatex -fmt=luatex --jobname=Preprocessor '\directlua0{dofile("Preprocessor.lua")}\input Preprocessor.txt'

As you can see, the file to be converted is simply read using "\input" and then converted
in the Lua function "processInputFile". By using coroutines, we can pretend to convert the file
in one go, when really we are supplying it line by line to LuaTeX.

It would of course be possible to check the file type/extension inside the "open_read_file" callback and perform different conversions (or none at all if a TeX file) depending on the
file type, i.e. to typeset source code.

Talk:Encrypt your document using ROT13

2010-12-07T21:20:12Z

Patrick: Link to old location

Old location: http://luatex.bluwiki.com/go/Encrypt_your_document_using_ROT13

Encrypt your document using ROT13

2010-12-07T21:19:47Z

Patrick: Copied from the old bluwiki.com luatex wiki

The following PlainTeX code example uses the ligaturing callback to "encrypt" the document using ROT13-"encryption".

%&luatex

% Notes:
%
% - Hyphenation is applied before rotating.
% - Ligaturing is performed after rotating.

\pdfoutput=1
\begingroup
\catcode`\%=12
\directlua0{
callback.register('ligaturing', function(head)
for v in node.traverse_id(33, head) do
if v.subtype == 1 then
local c = v.char
if c >= 65 and c <= 90 then
v.char = (c - 65 + 13) % 26 + 65
elseif c >= 97 and c <= 122 then
v.char = (c - 97 + 13) % 26 + 97
end
end
end
node.ligaturing(head)
end)
}
\endgroup

% Hello World; fi -- fl -- ffl -- ffi -- enough with the ligatures!
% Better do some kerning instead: AV

Uryyb Jbeyq; sv -- sy -- ssy -- ssv -- rabhtu jvgu gur yvtngherf!
Orggre qb fbzr xreavat vafgrnq: NI

\bye