Editing TeX without TeX

== Use TeX's power without TeX macros ==

With LuaTeX you have access to most of TeX's capabilities for creating PDF documents. Most of the [http://www.luatex.org/svn/trunk/manual/luatexref-t.pdf LuaTeX reference manual] is about this topic. 

The underlying idea is to create TeX's internal data structure as TeX would do by transforming macros and primitives into something called nodes. These nodes are then transformed into instructions for the PDF file (PDF objects). So the following pages deal with node creation and the structures. 

Let's start with a rule node. When you write <tt>\hrule width 20pt height 10pt depth 10pt </tt> in TeX, a node of type rule will be created, which has three fields: height, depth and width. We can visualize this node by a simple box like this:

[[File:rulenode.png]]

The same can be done directly in Lua:
<pre>
r = node.new("rule")
r.width  = 20 * 65536
r.height = 10 * 65536
r.depth  = 10 * 65536
</pre>

Some remarks: the size is in scaled points, and 65536 (=2<sup>16</sup>) scaled points are 1 point, so 20pt becomes 20 * 2<sup>16</sup>. Not shown in the example are (among others) the next and prev fields. These fields hold a pointer (the node address) to the following and the previous nodes. A node that exist by itself has its prev and next fields set to nil.

A slightly more complicated example is when you create two rules and pack then into a vertical box. This would look like this in plain TeX:

<pre>
\vbox {\hrule width 20pt height 10pt depth 10pt
           \hrule width 20pt height 10pt depth 10pt }
</pre>

This would generate a node list like:


[[File:Rulenode2.png]]

The same can be done directly in Lua:
<pre>
r1 = node.new("rule")
r1.width  = 20 * 65536
r1.height = 10 * 65536
r1.depth  = 10 * 65536

r2 = node.new("rule")
r2.width  = 20 * 65536
r2.height = 10 * 65536
r2.depth  = 10 * 65536

r1.next = r2
r2.prev = r1

vbox = node.vpack(r1)
</pre>

The result is a vbox containing the two rule nodes. The nodes are connected by their next and prev pointers, so the vbox contains them both. The size (width, height and depth) is adjusted automatically.

This also applies to all other things you can put on a TeX page: glyphs, horizontal boxes, math, images, hyperlinks, glue .... For a complete list of the nodes, see the reference manual Chapter 8 "Nodes".


== Beyond reports and articles ==

Perhaps the question that comes into your mind is: why should I bother writing Lua code instead of TeX code? For many documents, including articles, reports and books, the classic way to create a PDF with TeX is to use LaTeX or ConTeXt. But there are applications that use TeX as an 100% automatic PDF generation software. If you want to create data sheets or product listings, your normal input is not a classic document, but most likely an Excel spreadsheet or an XML file, perhaps extracted from a database. You won't have any TeX markup within these documents. So: if you don't have TeX markup in these documents, you don't need the TeX interpreter to read these files. More likely is an XML parser or a spreadsheet 2 ... converter.

When you use regular LaTeX or ConTeXt code to put the contents of the database in your PDF (database publishing), you run into several problems:

* Exact positioning of items on the page can be troublesome
* catcodes / command escapes are to be considered and surely get into your way
* TeX is no fun to program in (well it is, but only for some people) - calculations, tests, exception handling, control flow
* The packages that exist may reach their limit soon. You need a super flexible image-between-columns-with-parshape package? Write it yourself.

With Lua you have a very decent programming language that ''is'' fun to program in. When you use the Lua interface to TeX, you have full control of what is happening, which is usually the level of control you need for high quality output.

One approach is to create a stub TeX file and let Lua do the rest:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

The file <tt>myprogram.lua</tt> then contains all the code to a) read the source file(s), b) extract the information and create the appropriate nodes and c) instruct TeX to create PDF pages from these nodes.

In a future version of LuaTeX you will be able to run the Lua interpreter without the stub TeX file above.

== A first example ==

The example document given below creates two pages by using Lua code alone. You will learn how to access TeX's boxes and counters from the Lua side, shipout a page into the PDF file, create horizontal and vertical boxes (hbox and vbox), create new nodes and manipulate the nodes links structure. The example covers the following node types: rule, whatsit, vlist, hlist and action.

In the example code we use black squares as the contents of the pages and not normal text, because character handling takes more code and will be covered later on.

Save the following file into <tt>myprogram.lua</tt> and use the TeX stub from above (or get the source [https://raw.github.com/gist/1062041/d3de082e955f6535f3820742cef76fe9e50d5a6f/myprogram.lua here]).
<pre>
do
  -- this will hold the items that go onto the page
  local pagelist
 
  -- Call tex.shipout() with the contents of the pagelist
  function shipout()

    local vbox,b = node.vpack(pagelist) -- we ignore the badness 'b'
    tex.box[666] = vbox
    tex.shipout(666)

    pagelist = nil
    
    -- Not strictly necessary. TeX holds the current page number in counter 0.
    -- TeX displays the contents of this counter when it puts a page into
    -- the pdf (tex.shipout()). If we don't change the counter, TeX will
    -- display [1] [1], instead of [1] [2] for our two page document.
    tex.count[0] = tex.count[0] + 1
 end

  function add_to_page( list )
    -- We attach the nodelist 'list' to the end of the pagelist
    -- if pagelist doesn't exist, 'list' is our new pagelist
    -- if it exists, we go to the end with node.tail() and adjust
    -- the prev and next pointers, so list becomes part
    -- of pagelist.
    if not pagelist then pagelist = list
    else
      local tail = node.tail(pagelist)
      tail.next = list
      list.prev  = tail
    end
  end
end

-- This creates a new square rule and returns the pointer to it.
function mkrule( size )
  local r = node.new("rule")
  r.width  = size 
  r.height = size / 2
  r.depth  = size / 2
  return r
end

do
  local destcounter = 0
  -- Create a pdf anchor (dest object). It returns a whatsit node and the 
  -- number of the anchor, so it can be used in a pdf link or an outline.
  function mkdest()
    destcounter = destcounter + 1
    local d = node.new("whatsit","pdf_dest")
    d.named_id = 0
    d.dest_id = destcounter
    d.dest_type = 3

    return d, destcounter
  end
end

-- Take a list of nodes and put them into an hbox. The prev and next fields
-- of the nodes will be set automatically. Return a pointer to the hbox.
function hpack( ... )
  local start, tmp, cur
  start = select(1,...)
  tmp = start
  for i=2,select("#",...) do
    cur = select(i,...)
    tmp.next = cur
    cur.prev = tmp
    tmp = cur
  end
  local h,b = node.hpack(start) -- ignore badness
  return h
end

local tenpt = 10 * 2^16
---------------------------
-- page 1
---------------------------
local n,dest = mkdest() -- dest is needed for the link to this anchor

add_to_page(n)
add_to_page(mkrule(2 * tenpt))

-- The pagelist contains a pdf dest node (a link destination) and a rule of size 20pt x 20pt.

shipout()
---------------------------
-- page 2
---------------------------
-- This is the page with the link to the anchor (dest) on page one. A
-- link consists of three nodes: a pdf_start_link, a pdf_end_link and and 
-- action node that specifies the action to perform when the user clicks on
-- the link.
-- The pdf link must be inside a horizontal box, that's why we hpack() it.
-- The link_attr (link attributes) is optional, here it draws a yellowish border
-- around the link.
local start_link = node.new("whatsit","pdf_start_link")
local end_link   = node.new("whatsit","pdf_end_link")

start_link.width     = tenpt
start_link.height    = tenpt / 2
start_link.depth     = tenpt / 2
start_link.link_attr = "/C [0.9 1 0] /Border [0 0 2]"

start_link.action = node.new("action")
start_link.action.action_type = 1
start_link.action.action_id   = dest

local rule = mkrule(tenpt)
local hbox = hpack(start_link, rule, end_link)
add_to_page(hbox)

-- This pagelist consists of an hbox whose contents is "start_link",
-- the 10pt x 10pt rule and the "end_link" node. 

shipout()
---------------------------

-- Just to show you that you can get some memory usage statistics:
print(string.format("\nnode_mem_usage=%s",status.node_mem_usage))
</pre>

The idea is to create nodes, collect them in a list (here: pagelist) and use <tt>tex.shipout()</tt> to put the page into the PDF.

== And what about ''real'' content? ==

Black squares won't make anyone happy. So we need a typeset paragraph. For that we start with a single glyph and a glue first. A glyph node has this structure:

[[File:Singleglyphnode.png]]

whereas a glue is more complicated:

[[File:Simplegluenode.png]]

The glue item comes in pairs: a glue node and a glue_spec node. The information about the shrink/stretch values go into the glue_spec node. For example, the TeX glue <tt>2pt plus 1fill</tt> has a width of 2*2<sup>16</sup>, a stretch of 2<sup>16</sup> and a stretch_order of 3. 

<pre>
local g = node.new("glyph")
g.font = font.current()
g.lang = tex.language
g.char = 86 -- V

local hbox = node.hpack(g)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

We create a glyph, pack it into an hbox (horizontal box), pack that hbox into a vbox (a vertical box) and write this box to the main vertical list. The char number (86 in the example above) is based on an internal encoding, so we have to keep an eye on what character is on what position. Now we do this for two glyphs:

<pre>
local g1 = node.new("glyph")
g1.font = font.current()
g1.lang = tex.language
g1.char = 86

local g2 = node.new("glyph")
g2.font = font.current()
g2.lang = tex.language
g2.char = 97

g1.next = g2
g2.prev = g1

local hbox = node.hpack(g1)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

Pretty much the same as above. The glyphs g1 and g2 are chained together by setting the next and prev pointer to each other. If they were not connected, only glyph g1 gets into the hbox.

If you take a close look at the PDF, you see that the two glyphs are too far away, a (negative) kern should be inserted. This can be done by inserting a kern-node manually or you can ask TeX to do that for you. The last lines of the example above should then read:

<pre>
local head,tail,success = node.kerning(g1)
local hbox = node.hpack(head)
local vbox = node.vpack(hbox)

node.write(vbox)
</pre>

With this approach you are able to put some glyphs into the PDF but you will run into problems when you want to create a whole paragraph. It is much easier to ask TeX to do the line breaking. The steps for paragraph creation are

# put glyph nodes and glue nodes in a list (= chain them together with the prev/next pointers)
# add trailing infinite penalty and parfillskip
# call lang.hyphenate(), node.kerning() and node.ligaturing() to prepare the list for the linebreaking job
# call tex.linebreak()

For the next task we want to create the following paragraph:

[[File:Sample_paragraph.png|600px]]


We start with the small TeX stub as above:

<pre>
\directlua{dofile("myprogram.lua")}
\end
</pre>

and myprogram.lua ([https://gist.github.com/1063228 download]) defines a function <tt>mknodes()</tt> that creates the list of nodes to be passed into the typesetter.

<pre>
function mknodes( text )
  local current_font = font.current()
  local font_parameters = font.getfont(current_font).parameters
  local n, head, last
  for s in string.utfvalues( text ) do
    local char = unicode.utf8.char(s)
    if unicode.utf8.match(char,"%s") then
      -- its a space
      n = node.new("glue")
      n.spec = node.new("glue_spec")
      n.spec.width   = font_parameters.space
      n.spec.shrink  = font_parameters.space_shrink
      n.spec.stretch = font_parameters.space_stretch
    else -- a glyph
      n = node.new("glyph")
      n.font = current_font
      n.subtype = 1
      n.char = s
      n.lang = tex.language
      n.uchyph = 1
      n.left = tex.lefthyphenmin
      n.right = tex.righthyphenmin
    end
    if head then
      last.next = n
    else
      head = n
    end
    last = n
  end
  
  -- now add the final parts: a penalty and the parfillskip glue
  local penalty = node.new("penalty")
  penalty.penalty = 10000

  local parfillskip = node.new("glue")
  parfillskip.spec = node.new("glue_spec")
  parfillskip.spec.stretch = 2^16
  parfillskip.spec.stretch_order = 2
  
  last.next = penalty
  penalty.next = parfillskip

  -- just to create the prev pointers for tex.linebreak
  node.slide(head)
  return head
end

local txt = "A wonderful serenity has taken possession of my entire soul, ... like mine."

tex.baselineskip = node.new("glue_spec")
tex.baselineskip.width = 14 * 2^16

local head = mknodes(txt)
lang.hyphenate(head)
head = node.kerning(head)
head = node.ligaturing(head)

local vbox = tex.linebreak(head,{ hsize = tex.sp("3in")})
node.write(vbox)
</pre>