Adopting the Metanorma toolchain for publications

Summary

Tip

.Summary The easiest way to adopt Metanorma is via Simple Adoption, which allows you to supply stylesheets and HTML files for styling.

To customize behaviour further than Simple Adoption, you need to create a custom gem:

Clone the correct base gem: we recommend using the `metanorma-ribose` gem, which has the added advantage of allowing you to use the config file.
Either:

Directly use the configuration YAML file as documented for Simple Adoption; `metanorma-ribose` has already renamed `metanorma.yml.example` to `metanorma.yml`; or Change the root tag and namespace to a namespace specific to your organization's document standard (XMLROOTTAG and XML_NAMESPACE located in .../lib/metanorma/{flavour}/converter.rb)

Change any references to ribose or Ribose in the gem to your organization's document standard.
Change the styling of the document outputs (located in .../lib/isodoc/{flavour}/html).
customize any code you need to inherited from the gem's parents.

General

The toolchains currently available proceed in two steps:

map an input markup language (currently AsciiDoc only) into Metanorma Standoc Semantic XML; and
map Metanorma Standoc XML into various output formats (currently Word doc, HTML, PDF, all via Metanorma Standoc Presentation XML; flavor-specific XML output formats such as IETF RFC XML and ISO STS are prospectively to be generated through document models, see document-model-outputs).

Running the metanorma CLI tool involves a third step, of exposing the capabilities available in the first two in a consistent format.

These two steps are represented as three separate modules, which are included in the same gem; for the Generic gem, they are Isodoc::Generic, and Metanorma::Generic.

Your adaptation of the toolchain will need to instantiate these three modules. The connection between the two first steps is taken care of in the toolchain, and metanorma explicitly invokes the two steps, feeding the XML output of the first step as input into the second. The metanorma-sample gem outputs both Word and HTML; you can choose to output only Word, or only HTML, and you can choose to generate PDF as well.

The modules involve classes which rely on inheritance from other classes; the current gems use Metanorma::{Standoc, ISO, Generic}::Converter, Isodoc::{PresentationXML, Metadata, HtmlConvert, WordConvert}, and Metanorma::Processor as their base classes. This allows the standards-specific classes to be quite succinct, as most of their behaviour is inherited from other classes; but it also means that you need to be familiar with the underlying gems, in order to do most customization.

In the case of Metanorma::X classes, the changes you will need to make involve the intermediate XML representation of your document, which is built up through Nokogiri Builder; e.g. adding different enums, or adding new elements. The adaptations in Metanorma::Generic::Converter are limited (and are almost all to do with reading in properties from a config file), and most projects can take them across as is.

The customizations needed for Metanorma::Generic::Processor are minor, and involve invoking methods specific to the gem for document generation.

The customizations needed for Isodoc::Generic are more extensive. Five base classes are involved:

Isodoc::PresentationXMLConvert converts Metanorma Standoc Semantic XML to Metanorma Standoc Presentation XML.
Isodoc::Metadata processes the metadata about the document stored in //bibdata. This information typically ends up in the document title page, as opposed to the document body. For that reason, metadata is extracted into a Hash, which is passed to document output (title page, Word header) via the Liquid template language. See Metadata and Predefined text for more information. The Metadata class is only used for HTML and DOC; PDF parses the Presentation XML in one go.
Isodoc::HtmlConvert converts Metanorma Standoc Presentation XML to HTML.
Isodoc::PDFConvert converts Metanorma Standoc Presentation XML to PDF.
Isodoc::WordConvert converts Metanorma Standoc Presentation XML to Word HTML; the html2doc gem then converts this to a .doc document.

The Isodoc::HtmlConvert and Isodoc::WordConvert are expected to be near-identical, since any rendering differences between the two are addressed in the HTML CSS stylesheet. The Isodoc::HtmlConvert and Isodoc::WordConvert overlap substantially, as both use variants of HTML. However there is no reason not to make substantially different rendering choices in the HTML and Word branches of the code.

In addition to these, Relaton::Render is used to provide a rendering of the Relaton XML bibliographic references for the document in the Presentation XML, based on a stylesheet provided for the flavour. The Relaton::Render methods may themselves be customised for the current flavour. The configuration of Relaton::Render is described in relaton.org.

Document-model output formats (prospective)

Note

This mechanism is a prospective change: it has been merged into metanorma-core but is not yet part of a released Metanorma version, and no flavor output format is generated through it in production yet. This section will be updated once the mechanism and its adopters go live.

Besides the Presentation XML based rendering described above, a flavor will be able to register output formats that are rendered through a document model: the Metanorma XML is read into a metanorma-document model by a flavor-specific reader class, transformed into a model of the target format by an output transformer class, and the resulting model is serialized to the output file:

(Semantic | Presentation) XML → reader.from_xml → transformer.transform → target.to_xml

This is the intended mechanism for IETF RFC XML output and ISO STS output.

A flavor adds such a format by overriding Metanorma::Processor#document_transformers in its processor class, mapping the output format symbol to the two injected classes, plus serialization options:

def document_transformers
  {
    rfc: {
      # required: responds to .from_xml(String), returning a document model
      reader: Metanorma::IetfDocument::Root,
      # required: .new(model, options); the instance responds to #transform,
      # returning a target model that responds to #to_xml
      transformer: Metanorma::Ietf::Transformer::IetfToRfcV3,
      # optional: strip the default XML namespace before reading (default false)
      strip_default_namespace: true,
      # optional: keyword arguments splatted into target.to_xml (default {})
      to_xml_options: { pretty: true, declaration: true, encoding: "utf-8" },
      # optional: callable (xml, transformer, options) -> xml,
      # run after serialization (default: identity)
      post_process: ->(xml, _transformer, _options) { xml },
    },
  }
end

The base Metanorma::Processor#output consults this registry before falling back to writing out its input as is, and drives the registered chain through Metanorma::Processor#renderviadocument_model. Whether the format is fed Semantic XML or Presentation XML is decided, as for all output formats, by Metanorma::Processor#usepresentationxml.

The reader and transformer classes are duck-typed and need not live inside the Metanorma:: namespace, so they can be supplied by any gem. A flavor that renders only through document models needs no output override at all; a flavor with mixed output formats keeps its case dispatch in output and lets super route the document-model formats.

Metanorma::Standoc customization examples

In the following snippets, the parameter node represents the current node of the AsciiDoc document, and xml represents the Nokogiri Builder node of the XML output.

The predefined text representation of the document's author, publisher and copyright holder names Acme as the responsible organization.

def metadata_author(node, xml)
  xml.contributor do |c|
    c.role **{ type: "author" }
    c.organization do |a|
      a.name "Acme"
    end
  end
end

The editorial committees are represented as a single element. (node.attr() recovers Asciidoctor document attribute values.)

def metadata_committee(node, xml)
  xml.editorialgroup do |a|
    a.committee node.attr("committee"),
      **attr_code(type: node.attr("committee-type"))
  end
end

The document identifier concatenates the document number, the abbreviation of the document status (retrieved via IsoDoc::Generic::Metadata), and the document year.

def metadata_id(node, xml)
  docstatus = node.attr("status")
  dn = node.attr("docnumber")
  if docstatus
    abbr = IsoDoc::Generic::Metadata.new("en", "Latn", {}).
      status_abbr(docstatus)
    dn = "#{dn}(#{abbr})" unless abbr.empty?
  end
  node.attr("copyright-year") and dn += ":#{node.attr("copyright-year")}"
  xml.docidentifier dn, **{type: "acme"}
  xml.docnumber { |i| i << node.attr("docnumber") }
end

A security element is added to the document metadata, at the metadata extension point (where flavour-specific metadata is entered).

def metadata_security(node, xml)
  security = node.attr("security") || return
  xml.security security
end
def metadata_ext(node, xml)
  super
  metadata_security(node, xml)
end

Title validation and style validation is disabled.

def title_validate(root)
  nil
end

The document type attribute is restricted to a prescribed set of options.

def doctype(node)
  d = node.attr("doctype")
  unless %w{policy-and-procedures best-practices
    supporting-document report legal directives proposal
    standard}.include? d
    warn "#{d} is not a legal document type: reverting to 'standard'"
    d = "standard"
  end
  d
end

Inline headers are ignored.

def sections_cleanup(x)
  super
  x.xpath("//*[@inline-header]").each do |h|
    h.delete("inline-header")
  end
end

Metanorma::Processor customization examples

initialize names the token by which Asciidoctor registers the standard

def initialize
  @short = :sample
  @input_format = :asciidoc
  @asciidoctor_backend = :sample
end

output_formats names the available output formats (including XML, which is inherited from the parent class)

def output_formats
  super.merge(
    html: "html",
    doc: "doc",
    pdf: "pdf"
  )
end

version gives the current version string for the gem

def version
  "Metanorma::Generic #{Metanorma::Generic::VERSION}"
end

inputtoisodoc is the call which converts Metanorma AsciiDoc input into Metanorma XML

def input_to_isodoc(file, filename)
  Metanorma::Input::Asciidoc.new.process(file, filename, @asciidoctor_backend)
end

output is the call which converts Metanorma XML into various nominated output formats

def output(isodoc_node, outname, format, options={})
  case format
  when :html
    IsoDoc::Generic::HtmlConvert.new(options).convert(outname, isodoc_node)
  when :doc
    IsoDoc::Generic::WordConvert.new(options).convert(outname, isodoc_node)
  when :pdf
    IsoDoc::Generic::PdfConvert.new(options).convert(outname, isodoc_node)
  else
    super
  end
end

Isodoc::Standoc customization examples

In Metadata-processing code:

Restrict author processing to the editorial committee: do not process any other contributors, including persons as authors:

def author(isoxml, _out)
  tc = isoxml.at(ns("//bibdata/ext/editorialgroup/committee"))
  set(:tc, tc.text) if tc
end

Create abbreviations for the recognised statuses of documents:

def status_abbr(status)
  case status
  when "working-draft" then "wd"
  when "committee-draft" then "cd"
  when "draft-standard" then "d"
  else
    ""
  end
end

Add the month/year revision date to the metadata associated with the document version:

def version(isoxml, _out)
  super
  revdate = get[:revdate]
  set(:revdate_monthyear, monthyr(revdate))
end

Add a security element to metadata:

def security(isoxml, _out)
  security = isoxml.at(ns("//bibdata/ext/security")) || return
  set(:security, security.text)
end

In code common to all of HTML, PDF and Word (BaseConvert module):

Add the security element to the extraction of metadata:

def info(isoxml, out)
  @meta.security isoxml, out
  super
end

Add two line breaks between the annex label and the annex title:

def annex_name(annex, name, div)
  div.h1 **{ class: "Annex" } do |t|
    t << "#{get_anchors[annex['id']][:label]} "
    t.br
    t.b do |b|
      name&.children&.each { |c2| parse(c2, b) }
    end
  end
end

Change the default label for annexes from "Annex" to "Appendix".

def i18n_init(lang, script)
  super
  @annex_lbl = "Appendix"
end

Simplify the processing of predefined text for terms and definitions: do not add a trailing predefined text section. applicable whether or no the terms and definitions section is empty:

def term_defs_boilerplate(div, source, term, preface)
  if source.empty? && term.nil?
    div << @no_terms_boilerplate
  else
    div << term_defs_boilerplate_cont(source, term)
  end
end

Render term headings in the same paragraph as the term heading number

def term_cleanup(docxml)
  docxml.xpath("//p[@class = 'Terms']").each do |d|
    h2 = d.at("./preceding-sibling::*[@class = 'TermNum'][1]")
    h2.add_child("&nbsp;")
    h2.add_child(d.remove)
  end
  docxml
end

Initialise the HTML Converter:

Set the default fonts for the HTML rendering, which will be used to populate the HTML CSS stylesheet. Also add default font sizes isodoc ≥ v1.3.0.

def default_fonts(options)
  {
    bodyfont: (options[:script] == "Hans" ? '"SimSun",serif' : '"Overpass",sans-serif'),
    headerfont: (options[:script] == "Hans" ? '"SimHei",sans-serif' : '"Overpass",sans-serif'),
    monospacefont: '"Space Mono",monospace'
    monospacefont: '"Space Mono",monospace',
    normalfontsize: "1.0em",
    monospacefontsize: "0.8em",
    smallerfontsize: "0.9em",
    footnotefontsize: "0.8em"
  }
end

Set the default HTML assets for the HTML rendering.

def default_file_locations(_options)
  {
    htmlstylesheet: html_doc_path("htmlstyle.css"),
    htmlcoverpage: html_doc_path("html_sample_titlepage.html"),
    htmlintropage: html_doc_path("html_sample_intro.html"),
    scripts: html_doc_path("scripts.html"),
  }
end

Access Google Fonts for the HTML rendering.

def googlefonts
  <<~HEAD.freeze
<link href="https://fonts.googleapis.com/css?family=Open+Sans:300,300i,400,400i,600,600i|Space+Mono:400,700" rel="stylesheet">
<link href="https://fonts.googleapis.com/css?family=Overpass:300,300i,600,900" rel="stylesheet">
  HEAD
end

Set distinct default fonts and HTML assets for the Word rendering. Also add default font sizes isodoc ≥ v1.3.0.

class WordConvert < IsoDoc::WordConvert
  def default_fonts(options)
    {
      bodyfont: (options[:script] == "Hans" ? '"SimSun",serif' : '"Arial",sans-serif'),
      headerfont: (options[:script] == "Hans" ? '"SimHei",sans-serif' : '"Arial",sans-serif'),
      monospacefont: '"Courier New",monospace'
      normalfontsize: "12.0pt",
      monospacefontsize: "11.0pt",
      smallerfontsize: "10.0pt",
      footnotefontsize: "9.0pt"
    }
  end
  def default_file_locations(_options)
    {
      wordstylesheet: html_doc_path("wordstyle.css"),
      standardstylesheet: html_doc_path("sample.css"),
      header: html_doc_path("header.html"),
      wordcoverpage: html_doc_path("word_sample_titlepage.html"),
      wordintropage: html_doc_path("word_sample_intro.html"),
      ulstyle: "l3",
      olstyle: "l2",
    }
  end
end

Edit this page on GitHub →