Posts

XML Wildcard Data Mapping and Transformation in MapForce


MapForce 2014 now supports mapping of XML wildcards that create a designed-in extension mechanism for XML Schemas, implemented as <xs:any> or <xs:anyAttribute>.

Consider this XML fragment viewed in XMLSpy:

Fragment of an XML document following the GPX XML Schema

This small portion of a very long file shows two track points recorded during a road trip by a Garmin GPS device. The XML data follows the GPX XML Schema, with Garmin extensions to add speed and course heading values. XML wildcard mapping lets you map these extensions in MapForce just like any other elements explicitly defined in the XML Schema.

Let’s assume we want to read the .gpx file and extract the highest speed recorded for a trip:

Output of the completed mapping Read more…

Tags: , , ,

Stop by booth 535 next week at Oracle OpenWorld


clip_image004September here at Altova means ramping up for a busy Autumn, beautiful cool crisp days in New England and… flying out to San Francisco for Oracle OpenWorld 2013! If you are planning on attending come by booth #535 in Moscone South and see Altova from September 23 – 25th for a demo of Altova’s tools for Oracle Users and to see our new line of cross-platform server software products: RaptorXML Server, FlowForce Server, MapForce Server and StyleVision Server. See first-hand how thesclip_image002e new products offer high-speed automaton for projects designed using familiar Altova MissionKit developer tools.

We would love to hear from you about your latest projects and challenges, collaborate on best practices or let us show you some of the new exciting things Altova has to offer. While you are at our booth mention this blog post to receive a special giveaway. Hope to see you next week in San Francisco!

Tags: , , , , , ,

Use XPath Expressions to Refine Data Selection


In this era of big data we often need to simplify analysis and communication by creating customized views into sections of a larger file. XPath, short for XML Path Language, is designed to allow users to address parts of large XML documents. XMLSpy supports XPath with an XPath Analyzer window and in its interface for charting XML data, MapForce supports XPath functions for XML data mapping, and StyleVision supports XPath in conditional templates, extension templates, and template filtering.

faces-laptop

Read more…

Tags: , , , , ,

Download Example Files for FlowForce Server, MapForce, and StyleVision


In response to an interested reader’s suggestion, the Altova MapForce mapping files and StyleVision stylesheet we deployed to FlowForce Server for the job described in the blog post Taming Bad Input Data with FlowForce Server are now available for download on the Altova Web site at www.altova.com/documents/AltovaBlogExampleFiles.zip

Altova MapForce and StyleVision sample files

Simply unzip the archive into a new folder and you’ll have all the data mappings, the stylesheet, and other supporting data files all in one place. A ReadMe file explains the contents. You can download fully functional free trials of Altova MissionKit and FlowForce Server at https://www.altova.com/download.html and implement and test FlowForce Server yourself.

Or, execute the data mappings in MapForce and the stylesheet in StyleVision to see how easy it is to extract meaningful information from the GPS data recorded by your own digital camera or GPS device. The example files in the download were also used in Web Service as a Look-Up Table to Refine GPS Data, XPath Enhances XML Reports, and others in our series on working with XML and Global Positioning Systems.

If you’re already an Altova MissionKit user, you can download these files with examples of Web services and user functions for MapForce, and XPath calculations and chart features for StyleVision, and add them to the extensive libraries of MapForce and StyleVision samples installed with Altova MissionKit tools.

Tags: , , , ,

Taming Bad Input Data with FlowForce Server


Whenever you accept data from an outside source you risk encountering errors. We have blogged about this phenomenon in the past in Expect the Unexpected – Altova MissionKit Solves a Number Format Mystery and in the series of posts on Processing the Groupon API.

Bad data in an input file can cause the data transformation step of a FlowForce Server job to fail. When a FlowForce Server Job fails, further execution steps will not be performed. FlowForce Server is designed this way to prevent an error in one job step from cascading into a series of additional invalid results. Happily, FlowForce Server also includes features to help you recover from errors and keep production flowing.

In this post we will further extend the data mapping and report rendering job described in Customizing a FlowForce Server Job to gracefully handle bad data in an input file.

FlowForce Server New Job Steps
Read more…

Tags: , , , , , ,

Web Interface Simplifies Automation of Data Transformations


FlowForce Server is a new product designed to provide automation of data transformations performed by dedicated high-speed servers. FlowForce Server can start jobs based on a variety of triggers, runs multiple jobs simultaneously, and can even run multiple instances of the same job, depending on workflow.Monitoring all these complex activities is critical to success in a busy production environment. The FlowForce Server Web interface includes customizable views into operations, simplifying management from anywhere on the network.

FlowForce Server viewed in a Web browser window

The screen shot above shows the Home page of the FlowForce Server Web browser interface, displaying all currently running jobs and active triggers. Six instances of the gpxElevationUSGS job are running, each identified by a unique job ID. Four job triggers are also active, three watching hot folders, and one based on a timer. Each FlowForce Server job automates a MapForce data transformation.The orange arrows below each grid are clickable update buttons, and the job names link to the definition pages for each job.The blue headings at the top are also clickable buttons. The Log link displays the Log View, a detailed history of all system activity, shown in a truncated version below.

FlowForce Server Log View Shows History of Operations

Each message line describes one step in one instance of a FlowForce Server job, and may have links to more information, as we described in our earlier blog post titled Automate Data Transformation with FlowForce Server. If the status is anything other than 0, the more link opens a detailed error message.

Description of a single FlowForce Server job step

In a busy environment, all this can be just too much information. In that case, the Log View offers several alternatives to help you find critical information quickly. The options bar above the list lets you filter items displayed by a particular job or Message severity.

Filter FlowForce Server Log messages by date, by job, or by severity

If you only want to know if anything unexpected occurred, select Warning as shown above to hide all normal Info messages.You can also sort the list by any of the column headings Date, Severity, Module, User, or Instance ID in ascending or descending order. When the FlowForce Server is running multiple simultaneous jobs and instances, it’s very likely individual steps from different jobs will be shuffled. Sorting by Instance ID can let you more easily review the job history.Click here to learn more about FlowForce Server Beta 3 at the Altova Web site, or visit the FlowForce Server Beta 3 download page to get started automating data transformations in your data center!

Tags: , ,

Automate Data Transformation with FlowForce Server


Altova designed FlowForce Server to provide comprehensive automation, management, and control over data transformations performed by dedicated high-speed servers. FlowForce Server Beta 3 is currently available to users of MapForce Enterprise and Professional Editions at no charge during the beta test period.FlowForce Server can provide hot folder automation of data mappings and maintains a detailed activity log users can monitor remotely in a Web browser window. The screenshot below shows the log for FlowForce Server running the MapForce data mapping CameraLogToGPX we wrote about in the blog post titled Process Multiple Input Files in a Single Data Mapping. This mapping used wildcards to specify multiple input files for processing.
FlowForce Server job log

It only takes a few minutes to set up, run, and review the results of jobs like this on FlowForce Server. Wildcards or Hot Folders?Wildcards and hot folders increase the complexity of a data transformation workflow, and using them successfully requires careful planning. Let’s take a minute to look a little deeper at the scenario we want to implement.Assume we are the IT department in a company that publishes nature and hiking guides. We employ photographers who go out trekking and record their routes as they go, using the GPS tracking feature of their digital cameras. We want to convert the camera GPS log files to XML-based .gpx format for mapping and other processing.We will publish a folder on our network where photographers can drop off their GPS log files. This will be the hot folder FlowForce Server watches for new files to supply as input to the CameraLogToGPX mapping. We only need to process each input file once. So, after the data transformation is complete, we can remove the input file from the hot folder. We also want to place the output file in a separate folder. This suggests the following FlowForce Server job steps:

  • Look in the hot folder to see if new a input files has arrived
  • Perform the data mapping on the input file and place the output file in a separate folder
  • Move the input file to a permanent location

The diagram below shows a folder structure we can use for the workflow, with files ready to drop into the hot folder for processing:

Hot folder structure

The hot folder is C:CameraGPShotFolder and the generated .gpx files will be placed in C:CameraGPSoutputFiles. When the data mapping is done, input files will be moved to C:CameraGPScompletedInput. Deploy the Mapping on a ServerMapForce Beta is a component of FlowForce Server that works just like MapForce, and adds a feature to deploy mapping files to a FlowForce Server. Existing data mappings need little or no special preparation for deployment.The only thing we need to think about are the filenames of the input and output files. We will instruct FlowForce Server to provide the input filename as a job parameter as new files arrive in the hot folder for processing. The original mapping assigned a wildcard filename for the input component, which is no longer needed. We will also want FlowForce to specify the location of the output file.We can open the mapping in MapForce Beta, remove the filename from the input component, and add a remove-folder filepath function to the output file so FlowForce Server can set the destination. The screenshot below shows the new filename definitions in the mapping.

Mapping file names

In the MapForce Beta dialog to deploy the mapping, we can choose to immediately open the mapping in a FlowForce Server job definition window in a Web browser to finish defining the job operations.

Deploy mapping dialog

Defining the Job in FlowForce ServerThe screenshot below shows the complete job steps defined in a FlowForce Server job properties window:

FlowForce Server job definition

The job trigger is defined at the bottom of the window. Every 30 seconds, FlowForce Server will check the hot folder. If the contents have changed, FlowForce Server will execute the job steps. Each Execution step could be a deployed MapForce mapping, a system step, or even another FlowForce Server job.The name of each new file entering the hot folder becomes the parameter called {triggerfile} that we use in the mapping step as the input filename, and in the move step as the name of the file to be moved.The Working-directory parameter in the mapping step defines where the output files will be placed.FlowForce Server also includes features to set automatic run and stop times for jobs, user permissions and roles, and Queue settings to define minimum time between job runs and maximum parallel instances of a job.In our scenario, we are likely to receive multiple input files in groups as they are copied from photographers’ memory cards. Multiple parallel runs can greatly improve throughput. As a rule of thumb, you might want to match the number of cores or CPUs in the machine running FlowForce Server.

FlowForce Server job queue settings

Exploring the Job Log

Each time FlowForce Server runs our job, six lines are added to the Log View shown in the illustration at the top of this post. The first and last lines record the start and completion of the job, and each Execution step generated its own start and completion messages. The phrase “completed with status: 0” means the step was successful with no errors.

FlowForce job log

We can click the more links for a detailed report on each Execution step. The screenshot below shows the message for the MapForce mapping Execution step:

FlowForce job detail

Each file dropped into the hot folder generates an individual FlowForce Server job instance, even if multiple files are added as a group. This makes it easy to track any individual input file that generates an error.When we dropped four files into the hot folder, FlowForce Server ran four jobs, and the contents of the output folder looked like this:

Job results folder

Click here to read more about FlowForce Server Beta 3 at the Altova Web site, or visit the FlowForce Server Beta 3 download page to get started automating data transformations in your data center!

Tags: , ,