Highlight Support for Asciidoctor

Asciidoctor extensions to support syntax highlighting via Highlight in the HTML backend.

highlight-treeprocessor.rb
highlight-treeprocessor_mod.rb v1.3.0 (2019-03-30)

Introduction

AsciiDoc is a very powerful documentation markup syntax for writing documents and books in plaintext sources files which can be version controlled and converted to HTML5, DocBook, PDF, and many other formats — think markdown with tons of extra features.

The most used AsciiDoc implementation today is Asciidoctor, the Ruby successors of the original Python AsciiDoc. Asciidoctor is a fast, open source AsciiDoc text processor and publishing toolchain, with a rich echosystem of additional tools, extensions and resources.

Unfortunately, Asciidoctor doesn’t natively support using Highlight for coloring code blocks in documentation, although other highlighters are natively supported.

However, the Asciidoctor team has published on the Asciidoctor Extensions Lab project an extension called HighlightTreeprocessor that adds some basic support for Highlight integration:

https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/highlight-treeprocessor.rb

In this folder I’ll be including and documenting both the original extension as well as a tweaked version which I use in many documentation projects.

Folder Contents

`highlight-treeprocessor.rb`	The original extension.
`highlight-treeprocessor_mod.rb`	The modded extension.
`example.asciidoc` `example.html`	Original extension usage example.
`example_mod.asciidoc` `example_mod.html` `example_mod.css`	Modded extension usage example.
`/haml/block_listing.html.haml` shared	Asciidoctor assets for the modded extension (only required during conversion).
`/sass/`	Sass sources and assets to customize the Highlight CSS themes for the modded extension examples.
`build.sh`	Converts the examples from AsciiDoc to HTML

Original vs Modded Extension

Here’s a comparison table of the differences between the original extension and my modded version:

feature	original	modded
linenums	✓	✓
subs	✗	✓
external CSS	✗	✓
multiple themes	✗	✓
callouts	✗	✗
code in tables	✗	✗

feature

original

modded

linenums

✓

subs

✗

✓

external CSS

✗

✓

multiple themes

✗

✓

callouts

✗

code in tables

✗

Custom CSS and Multiple Themes

The reason I’ve tweaked the original extension was to allow using custom CSS stylesheets to provide each language with its own theme, instead of having a single coloring theme for every code block in the document. This is the main difference between the original extension and my modded version, and it comes at the price of having to provide your own CSS to the document — while the original extension will automatically inject into the document the CSS of an Highlight theme.

So, if you’re happy highlighting all the code with one of Highlight native themes, then you should use the original HighlightTreeprocessor extension; but if you’d like more control over CSS styles, and to theme each language separately, then the modded extension is the way to go.

In the /sass/ folder you’ll find some useful Sass/SCSS templates and examples on how to easily create multiple themes targeting specific languages.

Substitutions

The other main difference is that the modded extension supports substitutions in the code block.

Example 1. Code with substitutions

Asciidoc source

[source,alan,subs=quotes]
------------------------
The 'dance hall' #IsA location#
  Description
    "The hall is #_bigger_ than you expected#, and
     far too crowded. You feel the urge to leave."
End the.
------------------------

HTML result

The 'dance hall' IsA location
  Description
    "The hall is bigger than you expected, and
     far too crowded. You feel the urge to leave."
End the.

`HIGHLIGHT_DATADIR` Enforcement

Another extra (optional) feature of the modded extension is that, if the HIGHLIGHT_DATADIR environment variable is set, it will enforce it by invoking Highlight with the --data-dir=<HIGHLIGHT_DATADIR> option so that it gets the highest override priority.

This ensures that custom langDefs inside HIGHLIGHT_DATADIR will always override the predefined ones, whereas ordinarily the path of the HIGHLIGHT_DATADIR env var would be searched after Highlight installation folder.

Since it’s not possible to control the --data-dir option from within AsciiDoc documents, this hardcoded workaround is a convenient solution for projects which need to override Highlight default configurations.

Missing Features

Neither the original nor my modded version of the extension support callouts or highlighting listing blocks inside tables.

Callouts

The extension doesn’t currently process callouts, and Asciidoctor will raise a warning if you use them (indeed, the build.sh script to convert this document does warn about missing callouts).

Example 2. Code with callouts

The 'dance hall' IsA location <1><2>
  Description <3>
    "The hall is bigger_ than you expected, and
     far too crowded. You feel the urge to leave."
End the.

1	`'dance hall'` is a quoted identifier, it will be show as the location name.
2	The dance hall is an instance of the `location` class.
3	The `Description` of a location will be shown every time the player enters the room or examines it via the `look` verb.

In the original extension support for callouts extraction and processing was never implemented. I’ve attempted to add callouts support myself, but without success.

Code Inside Tables

Listing blocks inside table cells are not processed by the treeprocessor extension. The listing block will just get some basic styling but the code won’t be processed by Highlight.

Example 3. Code listing inside a table cell

problem example

problem	example
Listing blocks inside tables won’t be highlighted.	This Lua block isn’t processed by Highlight: `if newState==HL_KEYWORD and kwgroup==5 then AddKeyword(token, 5) end`

Listing blocks inside tables won’t be highlighted.

This Lua block isn’t processed by Highlight:

if newState==HL_KEYWORD and kwgroup==5 then
  AddKeyword(token, 5)
end

In the above table the rightmost column is set to be styled as AsciiDoc (using the a specifier):

[cols="1*<d,1*<a",options="header,autowidth"]

But the treeprocessor extension doesn’t process listings inside table cells.

For more details regarding this limitation, see Dan Allen’s reply in Issue #2990.

Development Roadmap

Bringing full Highlight support to Asciidoctor has always ranked high in my priorities list since I work a lot with code documentation and I enjoy the power of AsciiDoc. Unfortunately, I don’t have any knowledge of Ruby and the Asciidoctor API, so all my tweaks to the HighlightTreeprocessor extension so far are nothing more than trial and error attempts to hack my way through it.

Lack of support for callouts is a serious limitation which hinders using Highlight in Asciidoctor projects — or the other way around, hinder using Asciidoctor in project which depend on Highlight. I’ve tried several times to add callouts support to the HighlightTreeprocessor extension, but to no avail.

Also, as Dan Allen pointed out to me in Issue #2990, the new Asciidoctor API makes the approach of the HighlightTreeprocessor extension somewhat obsolete:

Asciidoctor 2.0 introduces a new API for registering a syntax highlighter adapter. I recommend that you use this for your highlight extension. Here’s an example:

https://github.com/asciidoctor/asciidoctor/blob/master/lib/asciidoctor/syntax_highlighter/coderay.rb

Instead of having to find nodes in a tree processor extension, the adapter gets called by the processor to highlight the relevant nodes, contribution markup to the output, and write a custom stylesheet.

You can find the API docs here:

https://github.com/asciidoctor/asciidoctor/blob/master/lib/asciidoctor/syntax_highlighter.rb

— Dan Allen (@mojavelinux)
Issue #2990 at asciidoctor

So, ultimately, any further development in the direction of fully supporting Highlight in Asciidoctor (whether natively or by a third party extension) will need to shift to the new Asciidoctor 2.0 API.

Hopefully, someone with more Ruby competence than me will take on this task, and soon we might see Highlight natively supported by Asciidoctor. In the meantime, I haven’t given up on this goal, and I’ll be trying to learn a bit of Ruby and study the new Asciidoctor API, and see where I can get.

Credits

Due credits and licenses for third party assets used in this directory tree.

Asciidoctor Highlight Extensions

The MIT License

Copyright (C) 2014-2016 The Asciidoctor Project

The highlight-treeprocessor_mod.rb file was adapted by Tristano Ajmone from the original file highlight-treeprocessor.rb, also released under MIT License:

The MIT License

Copyright (C) 2018 Tristano Ajmone.
Copyright (C) 2014-2016 The Asciidoctor Project

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

Asciidoctor Haml Templates

/haml/

The files inside the /haml/ folder were adapted by Tristano Ajmone from the original Haml HTML5 templates taken from the Asciidoctor Backends project, Copyright © 2012-2016 Dan Allen and the Asciidoctor Project, released under MIT License:

The MIT License

Copyright (C) 2018 Tristano Ajmone.
Copyright (C) 2012-2016 Dan Allen and the Asciidoctor Project

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

Base16 Color Schemes

The Sass/CSS themes use the following Base16 color schemes:

Base16 Eighties, by Chris Kempson
Base16 Brewer, by Timothée Poisot

The Base16 project was created by Chris Kempson:

http://chriskempson.com/projects/base16/

Base16 Sass

The following Base16 SCSS files:

./sass/_base16-brewer.scss

were taken from the Base16 Sass project by Tristano Ajmone:

https://github.com/tajmone/Base16-Sass

also released under MIT License.

MIT License

Copyright (c) 2019 Tristano Ajmone <tajmone@gmail.com>
https://github.com/tajmone/Base16-Sass

Copyright (c) 2012 Chris Kempson (http://chriskempson.com)
https://github.com/chriskempson/base16-builder

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.