About Data Streaming

www.altova.com Print this Topic Previous Page Up One Level Next page

Home >  Common Tasks > Working with Mappings >

About Data Streaming

Data streaming is a MapForce built-in mechanism that allows you to use arbitrarily large data sources as input or output to your mappings. Data streaming should not be confused with stream objects in MapForce generated code. (The latter represent a possible way of handling data if you integrate MapForce generated code with a custom C# and Java application.)

 

Data streaming applies to the following data sources:

 

XML files
CSV files
Fixed-length field files
Databases

 

When you use any of the above data sources as input or output in your mappings, MapForce treats the data source as an open stream of data, and processes its contents sequentially, instead of loading all data into the memory.

 

Note:Data streaming is possible only if you have selected BUILT-IN as transformation language (see Selecting a transformation language ).

 

Memory usage considerations

When you work with mapping inputs and outputs that are data streaming candidates, “Out of memory” errors can occur if your mapping requires random access to the input source.

 

For example, let’s assume that your mapping contains a component that applies a group-by function on the source data. If you apply the group-by function on the entire tree structure of the input file, this would require the entire source file to be loaded into memory, and, consequently, file streaming would no longer be possible. The same is true for any operation which would require the whole contents of the mapping source to be loaded into memory, such as sorting.

 

When situations such as the one described above occur, the transformation will nevertheless complete successfully if there is enough virtual memory and disk space available on your system.


© 2019 Altova GmbH