Please enable JavaScript to view this site.

Altova MapForce 2022 Professional Edition

Data streaming is a MapForce built-in mechanism that allows you to use large data sources as inputs or outputs to your mappings. Data streaming should not be confused with stream objects in MapForce generated code. Stream objects represent a possible way of handling data if you integrate MapForce generated code with a custom C# and Java application.

 

Data streaming applies to the following data sources:

 

XML files

CSV files

Fixed-length field files

Databases

 

When you use any of the above data sources as an input or output in your mappings, MapForce treats the data source as an open stream of data and processes its contents sequentially instead of loading all data into the memory.

 

Note:Data streaming is only possible if you have selected BUILT-IN as a transformation language.

 

Memory usage considerations

When you work with mapping inputs and outputs that are data streaming candidates, Out of memory errors can occur if your mapping requires random access to the input source. For example, your mapping contains a component that applies a group-by function to the source data. If you apply the group-by function to the entire tree structure of the input file, this will require the entire source file to be loaded into the memory. Consequently, file streaming will no longer be possible. The same is true for any operation which requires the whole contents of the mapping source to be loaded into the memory (e.g., sorting). In scenarios similar to this example, the transformation will complete successfully if there is enough virtual memory and disk space available in your system.

 

© 2015-2021 Altova GmbH