tokenize

www.altova.com Print this Topic Previous Page Up One Level Next page

Home >  Functions > Function Library Reference > core | string functions >

tokenize

Result is the input string split into a sequence of chunks/sections defined by the delimiter parameter. The result can then be passed on for further processing.

fn-tokenize

E.g. Input string is A,B,C and delimiter is "," - then result is A B C.

 

Example

The tokenizeString1.mfd file available in the ...\MapForceExamples folder shows how the tokenize function is used.

tok-tok1

The XML source file is shown below. The Tool element has two attributes: Name and Code, with the Tool element data consisting of comma delimited text.

tok-tok01

 

What the mapping does:

 

The tokenize function receives data from the Tool element/item and uses the comma "," delimiter to split that data into separate chunks. I.e. the first chunk "XML editor".
As the result parameter is mapped to the Rows item in the target component, one row is generated for each chunk.
The result parameter is also mapped to the left-trim function which removes the leading white space of each chunk.
The result of the left-trim parameter (each chunk) is mapped to the Feature item of the target component.
The target component output file has been defined as a CSV file (AltovaToolFeatures.csv) with the field delimiter being a semicolon (double click component to see settings).

 

Result of the mapping:

 

For each Tool element of the source file
The (Tool) Name is mapped to the Tool item in the target component
Each chunk of the tokenized Tool content is appended to the (Tool Name) Feature item
E.g. The first tool, XMLSpy, gets the first Feature chunk "XML editor"
This is repeated for all chunks of the current Tool and then for all Tools.
Clicking the Output tab delivers the result shown below.

tok-tok2


© 2019 Altova GmbH