Streaming of Large Documents

Sometimes source documents are too large to hold in memory. Saxon-EE provides a range of facilities for processing such documents in streaming mode: that is, processing data as it is read by the XML parser, without building a complete tree representation of the document in memory.

Some of these facilities implement new features in the draft XSLT 3.0 recommendation. Some are specific to Saxon, and a few facilities are also available in XQuery.

Inevitably there are things that cannot be done in streaming mode - sorting is an obvious example. Sometimes, achieving a streaming transformation means rethinking the design of how it works - for example, splitting it into multiple phases. So streaming is rarely a case of simply taking your existing code and setting a simple switch to request streamed implementation.

There are basically two ways of doing streaming in Saxon:

Burst-mode streaming: with this approach, the transformation of a large file is broken up into a sequence of transformations of small pieces of the file. Each piece in turn is read from the input, turned into a small tree in memory, transformed, and written to the output file.

This approach works well for files that are fairly flat in structure, for example a log file holding millions of log records, where the processing of each log record is independent of the ones that went before.

A variant of this technique uses the new XSLT 3.0 xsl:iterate instruction to iterate over the records, in place of xsl:for-each. This allows working data to be maintained as the records are processed: this makes it possible, for example, to output totals or averages at the end of the run, or to make the processing of one record dependent on what came before it in the file. The xsl:iterate instruction also allows early exit from the loop, which makes it possible for a transformation to process data from the beginning of a large file without actually reading the whole file.

Burst-mode streaming is available in both XSLT and XQuery, but there is no equivalent in XQuery to the xsl:iterate construct.
Streaming templates: this approach follows the traditional XSLT processing pattern of performing a recursive descent of the input XML hierarchy by matching template rules to the nodes at each level, but does so one element at a time, without building the tree in memory.

Every template belongs to a mode (perhaps the default, unnamed mode), and streaming is a property of the mode that can be specified using the new xsl:mode declaration. If the mode is declared to be streamable, then every template rule within that mode must obey the rules for streamable processing.

The rules for what is allowed in streamed processing are quite complicated, but the essential principle is that the template rule for a given node can only read the descendants of that node once, in order. There are further rules imposed by limitations in the current Saxon implementation: for example, although grouping using <xsl:for-each-group group-adjacent="xxx"> is theoretically consistent with a streamed implementation, it is not currently implemented in Saxon.

The streamed template mechanism applies to XSLT only.

Both these facilities are available in Saxon-EE only. Streamed templates also require XSLT 3.0 to be enabled by setting the relevant configuration parameters or command line options.