|
|
Rank: Newbie
Joined: 7/22/2011
Posts: 5
Location: Germany
|
To interface 2 line of business (LoB) applications (ie Finance and Order Processing) a common & simple design pattern is ETL:
1) extract all required data periodically from source system
2) transform source data by identifying added/deleted/changed data (delta data)
3) load only data identified as delta into target system
1) & 3) can be implemented using Altova tools (i.e. MapForce) but 2) is a gap because
DiffDog is currently no providing all required output functionality to fully support 2) even though the features to compare 'database content' as such exist!
Request features:
- output option to separate identified changes in files according to category of add/delete/change in order to feed another processing step
- the 3 output files (1 for each catagory) need to have the same & unchanged record format as the source data(optional headers are allowed (i.e. for CSV)
- command line option to specify behaviour and output file names
- supported database sources have to include flat file formats like CSV, XML & XLS
key points of feature request:
- evolve DiffDog from 'visualizing' differences into a tool to 'identify differences' in order to create input for other tools
Challenges:
- large data volumes (million of records) created by 1) require several days of continous processing even on modern hardware for 2) & 3)
- development skills available for each LOB application involved do not allow to cover 1), 2) & 3) in economical ways by one person requiring splitting of work into manageable pieces over a team
This would allow a tool chain like:
- Windows PowerShell export all relevant data from both LoB DBs to CSV
- DiffDog 2 CSV files to identify delta
- LoB provided import application to validate & load delta data into target LoB system
|
|
Rank: Advanced Member
Joined: 12/13/2005
Posts: 2,856
Location: Mauritius
|
1. Did you see export of changes functionality in DiffDog?
2. Why not use merge facility of DiffDog itself?
3. What is the ultimate goal of loading differences in XMLSpy?
4. Why don't you use MapForce for any of the steps, even though it is much better fit than XMLSpy at least for extracting and entering data back.
|
|
Rank: Newbie
Joined: 7/22/2011
Posts: 5
Location: Germany
|
Thanks vlad, I used your questions in order to revise my post with the intend to hopefully clarify what I'm really after.
re 1) http://manual.altova.com/DiffDog/diffdogenterprise/ doesn't offer anything to implement my required separation of changes into files according to change category
re 2-4) this is not really relevant for the feature request I try to make
|
|
Rank: Advanced Member
Joined: 12/13/2005
Posts: 2,856
Location: Mauritius
|
Ok... somehow I'm more confused than before - if I read your new post correctly your only problem is that DiffDog doesn't generate separate files per change category, but creates a single file which lists all changes, right?
If this is THE issue, you can use MapForce to generate multiple files out of a single difference file from DiffDog. Once mapped, this process can be automated via command line, for example.
|
|
|
guest |
