ProductsMapForceMapping DesignData Processing Functions

MapForce® 2012 delivers powerful, flexible support for defining custom data processing functions between source and target files.

MapForce provides an extensible library of data processing functions for filtering and manipulating data according to the needs of your data mapping project. To save time and leverage work you’ve already completed and tested, you can even import existing data conversion code or an XSLT 1.0 or 2.0 file for use as a function library. MapForce also includes a unique visual function builder for defining custom functions that combine multiple operations.

Custom Data Processing Functions

MapForce can handle the most advanced conversion scenarios thanks to a comprehensive data mapping function library, which allows you to define rules based on conditions, Boolean logic, string operations, mathematical computations, SQL and SQL/XML statements, or any user-defined function. You can even use an existing Web service to look up or process data in any mapping.

In the screenshot below, the box between the lines joining the data source with the target contains a data processing function, which performs custom data manipulations on data elements before completing the mapping.

In this simple XML-to-XML mapping, the source XML Schema (Exp Report) has a Person data element that has the separate child elements First (first name) and Last (last name). However, the target XML Schema (Marketing Expenses) only has a single data element, FullName, for both the first and last name. Defining a mapping that uses the concat (concatenation) data processing function takes the data contained in two separate elements and combines them into a single data element to fit in the target.

Parameterizing Transformations

MapForce also supports transformation input parameters, which allow the input of outside parameters to affect mapping transformations. This is extremely useful when you want to use MapForce-generated transformations in your own data integration applications, and your mapping project includes filters that extract information from the data source in response to those input parameters. The transformation input parameters can be passed to the main mapping function created by the MapForce code generator in Java, C#, or C++.

This parameterization of transformation code is a powerful new way to increase the flexibility of your mapping project and allows you to use MapForce for extremely complex and demanding applications.

Function Library

MapForce includes a comprehensive function library for building advanced data processing functions to perform any type of computational operation on data to make it adhere to the content model of the target. All functions are available in the library pane of the design window in a hierarchical tree structure that you can expand and collapse.

Many of the built-in functions, such as concat, add, multiply, etc., support an unlimited number of parameters, making it easy to perform mathematical manipulations and combine multiple parameters in one easy step. Aggregate functions allow you to perform computations on entire groups of data, and then return a single value to the mapping target. These include count, sum, min, average, join-string, and others.

Functions in the core library are generalized and not specific to any type of output. Using these core functions, you can create XSLT 1.0/2.0, XQuery, Java, C++, or C# data conversion code by simply selecting the language(s) you require. The figure below shows some of the functions available in the core library, including logical operators, mathematical functions, and common string operations.

Just a few examples of the powerful functions available in MapForce appear below:

Grouping

Grouping enables users to combine source data in groups for output and then apply processing instructions to those groups, essentially transforming flat data into a hierarchical structure. In the very simple example below, the group-by function has been used to convert a CSV flat file (shown here in Excel 2003) into XML.

A screenshot of this mapping in MapForce appears below:

Grouping allows an enormous amount of flexibility for working with and repurposing data in any number of different ways.

Grouping functions include:

• group-by - reorganizes data in the output document based on a specified common value, or grouping key
• group-adjacent - applies grouping to an alternating sequence of items, assigning items that have a common value to the same group
• group-starting-with - begins a new group based on a specified pattern and puts all subsequent items into the same group until another item matching that pattern is found
• group-ending-with - ends a new group based on a specified pattern and puts all subsequent items into the same group until another item matching that pattern is found

These are available as sequence functions in the function library.

Grouping is available for XSLT 2.0, Java, C#, and C++.

Formatting of numbers, dates, and times

MapForce offers a variety of different formatting functions that enable you to easily convert numbers, dates, and times to the equivalent string representation(s). For example, the simple format-number function below converts a numerical input (e.g., 12345) to a monetary one in with a USD prefix, comma separating the thousands digits, and two decimal places (e.g., $12,345.00).

Parsing of numbers, dates, and times

These are the inverse of the functions above and enable you to convert formatted numbers, dates, and times to numerical representations. For example, the parse-number function below parses the string input 31,234.50 to the simple number 31234.5.

Distinct-values function

The distinct-values function allows you to automatically remove duplicate values from the result-set of a mapping. This feature is available under sequence functions in the MapForce function library and can be easily inserted into any mapping by simply dragging it onto the design pane.

For example, in the screenshot below the output document will display each Department name only once.

Position function

The predicated position function in the MapForce function library gives you the ability to return output based on the context position of data in the input document (i.e. read/translate only the nth row/node).

In the example below, the output document will only display data about the first two people listed in the input.

This feature is available under node functions in the MapForce function library and can be easily inserted into any mapping by simply dragging it onto the design pane.

Tokenize Functions

MapForce supports the following functions, which are used for splitting data in different ways:

• tokenize - splits strings in mapped fields based on a common delimiter parameter (see example below)
• tokenize-regexp - splits strings in mapped fields based on a regular expression
• tokenize-by-length - splits data into chunks of equal size

These are available as string functions in the function library.

Node Name Function

The node-name function returns the QName of the selected node. This function is available under node functions in the function library.

Below the core library, MapForce displays only the libraries and functions that are available in the output language you have selected. If you switch languages and your mapping contains a function that is not available in that language, MapForce provides a clear warning.

The data processing functions available in MapForce are completely extensible, and you can easily write your own function libraries, import an XSLT file to use as a function library, or turn existing data conversion code into a function library. For example, you can import the XSLT Standard Library, which is available for download here. You can also connect to Web services to look up or manipulate data before returning it to the mapping target.

In addition, MapForce includes the powerful visual function builder for defining and saving complex functions.

To use a data processing function, simply drag and drop the function that you would like to use from the function library onto the MapForce design window. Connect the desired elements from the source data model to the inputs of the data processing function, and connect the output of the data processing function to the target data model.

MapForce supports advanced multi-stage data processing logic, meaning you can pass the output of one function into the input of another function, chaining them together as required, before completing the data transformation. Complex functions like this can be saved and reused via the visual function builder.

Filters and Conditions

Filters and conditions are available on the MapForce toolbar. Inserting these data processing functions into a mapping allows you to select data from the source based on Boolean conditions.

The if-else condition used in MapForce can accommodate for any number of variables. This feature, which is equivalent to a switch statement in many programming languages, enables you to easily control the flow of data in your mapping projects by matching a value to a selected criterion.

For example, the screenshot below shows the if-then component used in combination with other components to analyze temperature data and produce the following results:

• If temp is greater than 20, true is passed on to bool1 and "high" is passed on to value-true1.
• If temp is less than 5, true is passed on to bool2 and "low" is passed on to value-true2.
• Only one of the value-true parameters can be true for any given temperature, and that is the one that is passed on to the desc attribute in the target component.

When you are working with a database as your mapping source, MapForce provides advanced support for SQL SELECT and WHERE statements, allowing you to insert mapping components based on these operators.

Some data processing functions require you to supply a constant. For instance, you may need to insert a space when concatenating the data in two elements. You can add a constant to your mapping by clicking the “Insert constant” button on the toolbar.

MapForce also allows you to provide default values to use for a particular element if the output of a mapping is null or absent. This allows you to create stringent if/then data processing rules in your mapping project to ensure that output data conforms to the target model.

You may also need to transform the values of an input field into a different set of result data. For example, if months are represented by numbers in your mapping source, but you need them to be written as words in the target, you can configure the Value-Map function to map the numbers that correspond to the actual names of the months (1 to January, 2 to February, and so on). Double-clicking a value-map in a data mapping allows you to create a look up table representing the input and result values.

The Value-Map function can be used for many types of conditions; for example, if the source data is “Order Complete,” you can use the function to write “Send Invoice” to the target field, and so on. This function makes it easier to define precise mappings of XML, database, flat file, EDI, Excel 2007+, XBRL, and Web services data.

Intermediate Variables

Intermediate variables are a special type of component that store an intermediate mapping result for further processing and can be used to solve various advanced mapping problems. An intermediate variable is equivalent to a regular (non-inline) user-defined function, and is a structural component without an instance file.

Intermediate variables are identified in your mapping with a dashed border. The compute-when input item of the variable allows you to control its scope. In other words, compute-when specifies when and how often the variable value is computed as the mapping is executed. You will not need to connect this input in many cases, but it can be essential to override the default context, or to optimize mapping performance.

A new variable is inserted into a mapping via the Insert menu or the variable icon on the Insert Component toolbar. Either selection immediately opens the Create Variable dialog. A simple variable can represent any of dozens of common datatypes such as string, integer, boolean, decimal, dateTime, etc.

A complex variable can be based on an XML Schema, database, EDI, or FlexText structure.

MapForce users with a code-writing background frequently think in terms of storing and manipulating variables that contain partial results or even unknown content. Applying this strategy in a MapForce data mapping project can help you speed up and simplify mappings, or even facilitate re-purposing a partial mapping for new requirements.

Experience the ease with which you can create custom data processing functions for your data integration projects. Download a free 30-day trial of MapForce.