Ingenuity System Testing Services Case Study


Altova XMLSpy and MapForce help analyze results of 30,000 automated system tests.

Located approximately 100 miles west of London, Ingenuity System Testing Services Ltd. blends pure system testing and quality management techniques with business risk management initiatives to create and manage tests of complex software systems.

Ingenuity operates as a partner with developers and end-users during software development to enable clients to make qualified business decisions about the systems they depend on for success.

Ingenuity specializes in testing software systems that support the dynamic real-time market for electronic trading of insurance instruments. These insurance systems typically include a quotation engine that communicates the features of insurers’ individual products and an EDI (Electronic Data Interchange) business process component to complete the transactions.

While the EDI standard provides for fixed elements, such as question-sets and message formats, there are significant product-specific variations that must also be communicated, and it is critical that these systems represent each insurance product accurately.

Adding another level of complexity, the insurance system software in this case study was written by a third party contractor hired by the insurance companies, and the system had to prove capable of describing many product alternatives in an environment with frequent product changes.

Ingenuity has earned a reputation for timely delivery, integrity, and professionalism, with over ten years experience in the field and many clients among the top 10 insurers in various categories. Their actions have enabled them to maintain and enhance this reputation in the face of ever increasing technical requirements.

"It became clear that the Altova suite of XML tools had been devised to make things happen,"

Simon Mills, Managing Director of Ingenuity

The Challenge

The project required Ingenuity to collaborate with an independent software developer on a validation engine for a major upgrade of the developer’s existing insurance trading system. This utility would be used by Ingenuity to thoroughly test the developer’s system and would be provided to insurance companies to aid them in their own efforts to implement the upgrade and validate the updated system.

Successful completion of independent testing was a contractual obligation between the insurance company and the software developer. The deliverable required from Ingenuity was verification, through their test series, that the software validation engine created by the developer produced accurate results and that the insurance system software performed as the developer represented.

The validation engine was designed to send a set of parameters to the insurance system that simulate a product tender, a request from a user, or some other normal set of system inputs. The system processes the input and the validation engine records and delivers output to the tester that documents the results. The tester then compares the actual results to the expected results and reconciles any differences to determine whether the system is performing correctly or not.

Ingenuity did not simply run acceptance tests on the final release at the conclusion of the project. Ingenuity’s testers worked in tandem with the software developers, checking each development iteration as it was completed.

Based on previous efforts testing insurance trading systems, Ingenuity already had hundreds of test plans and countless thousands of unit tests that would be applicable to the new system. The test data of sample inputs and expected results was stored in specific formats used by the previous systems. The collection of test results contained complex variations of results that Ingenuity would extract, reformat into text using a rudimentary in-house conversion utility, and input into a Lotus-based comparator utility for analysis.

The new version of the system, however, was not compatible with the earlier data formats. The new system required XML file input and provided XML results output. The software development team used a rapid development process and required the testers to keep up. The developers also had a habit of altering the XML Schemas to suit demands of the moment and did not always adequately document their latest changes.

"We are testing experts, not full-time developers," said Simon Mills, Managing Director of Ingenuity. "But we are highly skilled at creating our own utilities. For our first round of tests against the new system we chose to use Lotus to map input and output data into the right format, then form it into XML."

"Our initial parsing tool for converting the previous format of our tests to XML and interpreting the result files, while manageable, was going to take five to six man-days to make robust," Mills continued. "Moreover, we had to create new parsing rules for each variation of the results output, and we could expect to repeat this effort with every software iteration as the developers changed their schemas."

The Solution

Since both the test input and results files were in XML, a native-XML approach to managing the test data seemed more logical and potentially more efficient. The Ingenuity team researched XML technologies by trying out all the XML manipulation utilities they could find on the Internet, according to Mills. "It became clear that the Altova suite of XML tools had been devised to make things happen," he said.

First, the Ingenuity team determined they did not have to rely on the software developer’s documentation of the XML Schemas for their input and results files. Since both the input and the results were XML files, Ingenuity used Altova XMLSpy to open sample input and results files provided by the developer and generate the schemas automatically using the application’s built-in ‘Generate DTD/Schema’ command.

Creating a schema in XMLSpy

The XMLSpy Generate DTD/Schema menu selection makes it easy to create a schema from a sample XML document.

Creating a schema in XMLSpy

The XMLSpy Generate DTD/Schema dialog lets you control enumeration scenarios and representation of complex elements during schema creation.

Next, Ingenuity utilized Altova MapForce to extract the relevant results, perform necessary calculations, and output an easily understandable table in the text-based format required by their homegrown analysis utility. To map the data into the necessary format, an XMLSpy generated schema from the previous step was first inserted into the application’s design pane and linked to a corresponding XML results document as the data source. Next, a text file was specified as the target of the data mapping with all appropriate settings defined. To perform calculations and manipulate the data into the required output structure, standard data processing rules were simply dragged from the MapForce function libraries window and dropped into the mapping design. Then, connecting lines were pulled from the XML Schema source data elements, through the data processing rules as needed, to the target data elements in the new text file. That’s all there was to it. The output of each data mapping was saved to a file and opened seamlessly in Ingenuity’s analysis utility. The total development time from creation of the schema through refinement of the mapping was less than four hours for a single Ingenuity test engineer.

MapForce text file dialog

Text files in CSV or tab-delimited format are often the "least common denominator" between incompatible systems. MapForce offers extensive control over text files.

MapForce function helper window

The MapForce Function Libraries helper window provides easy access to data processing functions that can be performed on source data as part of the integration.

Ingenuity MapForce mapping

Ingenuity testers created this MapForce mapping to extract critical information from the XML-based results file represented on the left and create text-based tables shown on the right.

Analyzing the results file of one benchmark test took 45 minutes with Ingenuity’s original parsing utility. In contrast, the MapForce method processed the same data in 11 seconds.

The approach Ingenuity took with MapForce allowed the results of 30,000 automated tests, creating 90,000 to 100,000 results, to be available for automated comparison in less than 6 minutes, from start to finish. Ingenuity estimates the difference this has made to the overall test effort is more than 100-fold, since the tests had to be repeated on every development iteration of the insurance trading system.

The Results

In the past, test plans were limited by deadline pressures.

"We have always had to concentrate on the best 100 to 200 tests using only two sample insurance products," explained Mills. "Where the full set of relevant tests is around 700 and the total number of insurance products handled by the system might be 36, the utter speed of our new process has allowed us to greatly expand the definition of ideal testing. This is not doing ‘more testing because we can,’ it is doing all the testing we would have liked to do, but that time limitations prevented in the past. We’re creating much greater certainty for our clients."

"Many of us had no more than a technical appreciation of XML before having to use it and live with it for this project," said Bill Gribble, Technical Director for Ingenuity. "The tutorials that come with Altova’s products proved perfectly sufficient to move the technically appreciative to technically adept in very short periods of time."

"We fully intend to move all our mapping exercises, of which we have hundreds, into the Altova MapForce environment," he concluded.

Find out how MapForce can help with your data integration challenge.