Collections configuration

Collection manifest

The collection manifest contains the following keys. The format has been streamlined metanorma ≥ v2.0.0, but is backwards compatible with earlier instances of the manifest; the legacy equivalent keys are given in [brackets].

output-folder

The folder where output is generated to; same value as option --output-folder above. The output folder contains those preprocessed individual files, and files named collection in the target formats. If the output folder given is a relative path, the path is relative to the location of the YAML collection manifest metanorma ≥ v2.1.5.

directives

Directives on how the collection should be generated. Accepted values are listed below.

documents-inline

(default) Indicates that the files should be concatenated into a single XML file for processing. If neither documents-inline nor documents-external is supplied, documents-inline is presupposed.

Currently, if you are generating PDF or DOC output, you can only specify documents-inline.

documents-external

Indicates that the files are kept separate in processing, and the overall collection XML file does not contain the XML of the individual documents.

presentation-xml

Indicates that the XML supplied is in Metanorma Presentation XML format, and does not need to be converted to Presentation XML by Metanorma metanorma ≥ v1.3.5.

This is automatically included when sectionsplit is set in the Metanorma file, to break a single document up into multiple HTML files.

coverpage: {file-path}

Gives the location of the HTML coverpage, as an alternative to the -c argument of metanorma collection metanorma ≥ v1.5.7.

Any locally specified coverpage will be treated as a Liquid template, which can be populated with metadata extracted from the bibliographic metadata about the collection; refer to Metadata and boilerplate.

collection-word-coverpage: {file-path}

Gives the location of the Word DOC coverpage for the entire collection metanorma ≥ v1.7.5.

If left out, or given as null, no such coverpage is added.

collection-word-intropage: {file-path}

Gives the location of the Word DOC introductory section for the entire collection metanorma ≥ v1.7.5.

As with HTML and Word DOC output of standalone documents, this is intended for prefatory material appearing before user-specified prefatory content: typically boilerplate content, and a placeholder for the table of contents.

If left out, or given as null, no such introductory material is added.

document-word-coverpage: {file-path}

Gives the location of the Word DOC coverpage for each individual document metanorma ≥ v1.7.5.

If left out, or given as null, the default cover page that would appear in a standalone document is used. If given as ”“, no such coverpage is added.

document-word-intropage: {file-path}

Gives the location of the DOC introductory section for each individual document metanorma ≥ v1.7.5.

If left out, or given as null, the default introductory section that would appear in a standalone document is used. If given as ”“, no such introductory section is added.

coverpage-style

Gives the style of the PDF and HTML coverpage, if multiple styles are offered metanorma ≥ v1.5.7.

bare-after-first

Compiles the first HTML document in the collection complete (with coverpage and boilerplate), and all subsequent files with the bare option (i.e. without coverpage and boilerplate) metanorma ≥ v1.3.5.

This is automatically included when sectionsplit is set in the Metanorma file, to break a single document up into multiple HTML files.

recompile-xml

If a file is specified as being *.adoc, Metanorma always recompiles it into XML, whether or not a corresponding XML file is already there.

This document attribute is only used to refresh XML files: by default, the last XML generated from Metanorma AsciiDoc is cached for time savings metanorma ≥ v2.0.6.

The attribute is set by default to true, and needs to be explicitly set to false in order to prevent recompilation metanorma ≥ v2.1.10.

Warning

The recompilation of XML does not take into account into any change detection of source files, so if the source file has changed and an XML present, the XML will still not be recompiled.

coverpage-pdf-portflio

The path to the file name of the fallback cover page in PDF format, for the generated PDF portfolio output. This PDF file is included in the PDF Portfolio as the cover page rendered if the PDF reader does not not support reading PDF Portfolio files natively metanorma ≥ v2.3.1.

flavor

The flavor of Metanorma to use in processing the collection metadata and (by default) the documents it contains metanorma ≥ v2.0.6. The corresponding value of flavor is described in the Metanorma document attribute :mn-document-class:.

format

Specifies the possible output formats for the collection as a list, as an alternative to the -f argument of metanorma collection. Allowed values are:

xml

Metanorma Semantic XML

presentation

Metanorma Presentation XML. This is added automatically if any of the following formats is specified.

html

HTML.

doc

Word DOC.

pdf

PDF.

pdf-portfolio

PDF Portfolio. metanorma ≥ v2.3.1

bibdata

Metadata about the collection. Entered in the Relaton format.

docid

type

(mandatory) docid/type is used by Metanorma to determine the flavour of the collection. Currently a collection can only contain documents of one flavour.

entry [manifest]

A manifest listing the documents contained in the collection, in nested hierarchy.

entry can appear recursively in a entry. This allows users to specify hierarchic levels of documents in the collection. That hierarchy will be reflected in the index page navigation for the collection.

type [level]

Names the current hierarchical level of the manifest.

title

Gives the title of the current level of the manifest.

file [fileref]

The file path of a document in the collection relative to the manifest file. file and entry are mutually exclusive: file indicates the leaf nodes of the manifest entries.

format

By default, the format values specified at the collection level (or html, by default) are also used at any level of the manifest; so if the formats for the collection are specified as pdf, html, then all documents in the collection are also compiled as PDF and HTML. This can be overridden at any level of the manifest, using the same form of values as with the format attribute metanorma ≥ v2.1.6. Any values specified at a given level in the manifest are inherited by all entries within that level; so specifying pdf, html at a subcollection will apply to all documents in the subcollection, unless those documents too have a format attribute.

• The documents are expected to be Metanorma Semantic XML documents (++*++.xml), Metanorma Presentation XML documents (++*++.xml), attachments (see below), YAML files (++*++.yml, ++*++.yaml), or AsciiDoc source files (++*++.adoc).
• If the file is in AsciiDoc format:

The file is compiled to a Metanorma Semantic XML document in preprocessing [added in https://github.com/metanorma/metanorma/releases/tag/v2.0.0]. By default, for caching reasons, the document is not recompiled to a Metanorma Semantic XML document if a corresponding XML document has already been generated in the same directory metanorma ≥ v2.0.6.

* If the document attribute of recompile-xml is set, The document is always* recompiled to a Metanorma Semantic XML document, even if a corresponding XML document has already been generated.

• If a document is a YAML file, it is assumed to be a collection manifest itself, and its manifest is recursively read into the current manifest at that point of the entry metanorma ≥ v2.0.0.

** This allows manifests to include other manifests. If the YAML file is in a different directory, the file locations of any files it references are updated to be relative to the current manifest.

• A manifest can have both files and nested manifests as its children metanorma ≥ v1.7.7.

identifier

The document identifier, used to index the document in processing. It is also the identifier used to reference this document from other documents in the same collection, using bibliographic references (Cross-references). If the identifier is not supplied, and this is a Metanorma document, the identifier will be extracted from the document metanorma ≥ v2.0.0.

pdf-file

In PDF Portfolio output, specifies the file name to be used for the output PDF file. metanorma ≥ v2.3.1.

output-filename

The output name to be given to the document, without no file extension, e.g. document metanorma ≥ v2.3.1.

Site manifest

The collection manifest is expected to reference Metanorma Semantic XML documents.

The starting point for generating a collection is Metanorma AsciiDoc documents. In order to specify a collection and generate it as straightforwardly as possible, the collection manifest should be accompanied by a site manifest, named metanorma.yml, specifying both the component AsciiDoc files, and the collection manifest, as dependency files.

Site compilation will compile both the component files, and the collection depending on them. This is done by running metanorma site generate in the same directory as metanorma.yml.

Since Metanorma site compilation compiles documents to a _site/documents directory, the collection manifest needs to reference the Semantic XML documents in that same _site/documents directory.

---
metanorma:
  source:
    files:
      - document.1.adoc
      - document.2.adoc
      - collection.yml
  collection:
    organization: "British Standards Institute"
    name: "Retrofitting dwellings for improved energy efficiency -- Specification and guidance"

In the site manifest, the files to be compiled are listed under metanorma.source.files; any YAML file in the list is assumed to be a collection manifest.

The collection is specified in the site manifest with two attributes: a name for the collection document, and an organization treated as the corporate author of the collection. Both will feature in the index file of the documents generated in the site (_site/index.html), and correspond to bibdata.title.content and bibdata.copyright.owner.name in the collection manifest.

---
directives:
  - documents-inline
bibdata:
  type: collection
  docid:
    type: bsi
    id: bsidocs
format:
  - xml
  - html
  - presentation
  - pdf
entry:
  - file: _site/documents/document.1.xml
    identifier: bsidocs-1
  - file: _site/documents/document.2.xml
    identifier: bsidocs-2

Index page template

The HTML index page template is currently realised as a Liquid template, which forms a sidebar for the display of the HTML content of each file.

The following fields are defined:

doctitle, docnumber, etc.

Information derived from the Relaton YAML description in the manifest of the entire collection.

The field names are as defined for Liquid templates in Metanorma: see Metadata and Boilerplate.

navigation

A nested list giving hyperlinks to the constituent documents, following the specification in the manifest field of the collection manifest.

nav_object

The same nested list, presented as a recursive object, in order to allow users to select only a subset of the navigation list for presentation metanorma ≥ v1.6.4.

It contains the following fields:

title

The list title.

type

The list type (from entry.type in the manifest) metanorma ≥ v2.0.0.

docrefs

A hyperlinked list of the documents at that level of the manifest.

children

An array of child manifests. This list can be recursive.

prefatory-content

Prefatory content from the collection manifest metanorma ≥ v1.5.6.

final-content

Final content from the collection manifest metanorma ≥ v1.5.6.

bibdata

A hash representation of the bibdata element representing the bibliographic metadata of the manifest metanorma ≥ v2.0.8.

Why

A collection is normally compiled in a single process, with every component document resident in memory at once. For large collections this accumulates memory across documents until the build runs out of memory: the failure is cumulative, not per-document (each document's own peak may be modest), and no per-document optimization removes it. Collections in the hundreds of documents — for example large information-model libraries — hit this limit.

The incremental build stages the compilation so that peak memory is bounded by a single document rather than the whole collection. It is opt-in: the default collection build path is unchanged.

How it works

The incremental build has three stages.

1. Compile each document in isolation, preserving its outgoing cross-document references as stubs on the semantic XML — rather than stripping them, as an isolated document with absent siblings otherwise would — and writing the result to a durable, content-addressed on-disk store.
2. Run the isolated compiles one per process, sequentially. Process isolation is what bounds memory: each document is compiled in its own operating-system process, which exits and releases all its memory before the next document begins. Only one document is ever resident.
3. Reinflate: a final pass reads the stored stubs and resolves them into real cross-document links, producing the same output an all-present build would — across HTML, PDF and Word.

Because cross-document reference resolution lives in the collection layer, operating on the semantic XML upstream of where output branches into the three formats, one implementation serves all of them: HTML, PDF (via collection.presentation.xml) and Word.

Render options

The incremental build is driven by three collection render options:

preserve_unresolved

When compiling a document in isolation, keep its unresolved cross-document repo:() references as stubs (retaining the bibitemid and repository document identifier) instead of stripping them to plain text. This is what allows a document built without its siblings present to be relinked later.

artifactstoredir

The directory of the content-addressed on-disk store that each document's stub-bearing semantic XML and anchor catalogue are written to. It is optional: when preserve_unresolved is set and artifactstoredir is omitted, the store defaults to a directory named .metanorma-collection-cache in the working directory. That name begins with a dot, so it is a hidden directory — it does not clutter ordinary directory listings. Pass artifactstoredir only to place the store somewhere else.

reinflate

Run the final reinflation pass: resolve the stored stubs into real cross-document links over the stored semantic XML. Only the cross-document resolution is run — the intra-document rendering passes were already applied during the isolated compile, so nothing is re-inflated.

The content-addressed store

Each document's stored artefacts are named:

<document-id-slug>.<content-hash>.<stage>.<format>

For example:

ISO-99000-1-2024.fd4e958e5f82dc20.semantic.xml
ISO-99000-1-2024.fd4e958e5f82dc20.anchors.json

document-id-slug

The document identifier, made filesystem-safe (ISO 99000-1:2024 becomes ISO-99000-1-2024). Human-readable but lossy; the hash carries identity.

content-hash

A 16-hexadecimal-digit SHA-256 over the compiled input and the Metanorma version. This is the resumption key.

stage

semantic (the stub-bearing semantic XML), anchors (the document's anchor catalogue, contributed to the collection-wide index), or presentation (after reinflation).

format

xml for semantic and presentation, json for anchors.

The store defaults to a directory named .metanorma-collection-cache — a hidden directory, since the name begins with a dot — in the working directory, and persists between builds, which gives two properties:

Resumable

An interrupted build recompiles only what is missing: a document whose artefacts for the current content hash are already present is skipped.

Idempotent

Re-running produces the same output with no side effects. By default, when a document's content changes, only its latest hashed version is retained; previous-hash versions of the same document are pruned. Pass keep_cache to retain every version.

What resumability does and does not provide

The store provides memory-bounding and cross-document correctness — not make-style compile-time incrementality. Every incremental build still reinflates every document; unchanged documents are not skipped at compile time.

Worked example

Take a three-part collection whose collection.yml lists the members ISO 99000-1:2024, ISO 99000-2:2024 and ISO 99000-3:2024, cross-referencing one another.

# One member, in its own process (this is what the orchestration runs per document)
Metanorma::Collection.parse("iso-99000-1-only.yml").render(
  format: %i[xml],
  output_folder: "tmp",                      # discarded; the store is the output
  preserve_unresolved: true,
  # artifact_store_dir omitted, so the store defaults to the hidden
  # .metanorma-collection-cache directory; pass it only to place the store elsewhere.
)

.metanorma-collection-cache/
  ISO-99000-1-2024.fd4e958e5f82dc20.semantic.xml
  ISO-99000-1-2024.fd4e958e5f82dc20.anchors.json
  ISO-99000-2-2024.ca9a7421d97ecdda.semantic.xml
  ISO-99000-2-2024.ca9a7421d97ecdda.anchors.json
  ISO-99000-3-2024.0eeb3368552ba91f.semantic.xml
  ISO-99000-3-2024.0eeb3368552ba91f.anchors.json

# Reinflation reads the stored stubs and resolves cross-document links; it does
# not re-inflate the documents.
Metanorma::Collection.parse("reinflate.yml").render(
  format: %i[xml html],                      # add presentation/pdf/doc as needed
  output_folder: "_site",
  reinflate: true,
)