Streaming XML->JS parser that extracts selected components from XML documents one-by-ony as they are found and pipes them downstream. This method ensures a low memory footprint as neither the XML file nor the corresponding JSON have to be fully loaded into memory at any point in time.
Usage:
// returns a stream
The selector can be any of the following:
String
=> a selected tag has to match the supplied selector exactlyRegExp
=> the fullpath (space delimited) has to pass the regexNumber
=> pipes down all element (with any child objects) that reside at a particular depth inside the document.Function
=> element selected when function returnstrue
. The function will be called with the following object:tag
: name of the tagpath
: an array of the tags of the full pathattr
: an object with the attributes of the elementdepth
: the depth of the object within the xml document
Array
=> Each element is a criteria - will be evaluated asor
Each selected element will include all children as a json object. Selecting depth 0 for example will just result in the whole document piped down as one json record. Each object (+ children) will contain a property
named __attr__
that contains any attributes defined in the XML document. By default the __attr__
object is non-enumerable, meaning that its not visible to screen or Object keys but can still be accessable.
The piped record will include the following properties: tag
, path
,depth
and value
with the element+children placed in the value
property.
The following options
can be defined
arrayNotation
: default false - enforces that all children are within an array, even when there is only one child nodecoerce
: default false - coerces numbers and booleans if turned on, both in text and attribute values. Ifcoerce
is an object then custom coercion methods can be defined for each element name / attribute nametextObject
: default false - Enforces that each text element is placed under.text
property. The default behavior is: if text object the only child of a node:|{text:'xxxx'} is simplified to `'xxxx'.valuesOnly
: pipes down just the matched record (i.e. the.value
of the default record)showAttr
: default false - show the__att__
object instead of hiding it behind the non-enumerable definitionmergeAttr
: default false - merges the attributes with the children objects (risking collisions when attribute name == children tag name)
Simple example
Given a sample file.xml
First 4.32 Second 5.73
Running the following:
fs ;
will console.log the following records one by one:
{
tag: 'item',
path: ['xml','item'],
depth: 1,
value: {
description: 'First',
price: 4.32
}
}
{
tag: 'item',
path: ['xml','item'],
depth: 1,
value: {
description: 'Second',
price: 4.73
}
}
Using .pipe(xmler('item'),{coerce:true})
will extract the same records. Using .pipe(xmler('price',{coerce:true}))
will return an array where the values will be 4.32 and 4.37 and the path
will be ['xml','item','price']
Real life example
This example streams the full Open Streetmap XML for North America into JSON records:
var request = ;var bz2 = ;var etl = ;var xmler = ;var fs = ; // Keep track of linecount and report every secondvar count = 0;; ;
Same example with the records piped to mongo (bulk = 100 records at a time and max concurrent connections = 5)
var mongo = ;var collection = mongo ; ;