|
|
Rank: Member
Joined: 10/28/2009
Posts: 20
Location: Italy
|
Hi,
while i'm using more complex mapping between my databases, Mapforce is slowing down. The database are huge and contain a lot of objects, so i believe that is normal that Mapforce use a lot of RAM (i have only two gigabytes). There is something that i can do about this? It's possible to use a step-by-step mapping instead using only one (for example, a source table that contains header and summary of object, at the first time fill only destination header table and in the second time fill the summary destination table)? Or there is a configuration that is suitable for me?
Thanks in advance for your help.
|
|
Rank: Advanced Member
Joined: 6/16/2006
Posts: 502
Location: AT
|
To get an idea of what you mean with "huge": How many different tables are you using in your mapping, and how many records are you processing?
|
|
Rank: Member
Joined: 10/28/2009
Posts: 20
Location: Italy
|
The source db has 269 tables and more than 20000 records.
The destination db has 149 tables, has already more than 9000 record and at this stage of mapping i'm importing about 9500/10000 records.
|
|
Rank: Newbie
Joined: 11/19/2009
Posts: 8
Location: Australia
|
It is not huge, really. I am mapping XML to Access db with over 1 million record tables. The problem is not in your RAM size but Java VM memory allocation. I tried increasing Java VM memory according to the documentation, but it didn't help, though my RAM is 8GB. The only way is to split large XML files into small ones. MapForce can handle XMLs up to 70-80MB quite well. With my experience the troubles start after 100MB.
|
|
Rank: Advanced Member
Joined: 7/17/2008
Posts: 185
Location: Minutiae, Triviality
|
Quote:though my RAM is 8GB.
Just out of curiosity, did you only run the java under Windows? Windows 32bit limits an application to 2Gb of virtual memory, and there is an abstraction layer involved. So, on Windows 32bit, the amount of physical memory on your machine is irrelevant from the viewpoint of a single application -- that app has a 2Gb space to play in, no more, no less.
more memory = faster reactions, more applications open.
Using -Xmx to increase the JVMs max memory space on Windows 32bit is pretty much limited to 1500m as anything over that starts cannibalizing from the space where the JVMs overhead needs to fit.
So far as I know, linux/Mac OS X/Solaris, etc don't have this memory architecture limitation imposed by the Windows OS, and so it might actually work. I've run MapForce generated code on Mac OS X's java without problem, but don't have the scope of "millions of rows" so can't test this specific thing, sorry.
If you are mapping DB to DB, the combination of the amount of rows being worked with and the complexity of the input and output components' representations in memory may become too large. To work around this, rReduce the number of rows of data that you are accessing per run of the java app. The complexity of the abstract DOM is certainly not going to be so complex as to not fit in memory, its only when you start adding nodes that memory gets used up.
Sorry, dropped into lecture mode. I'll shut up now :)
|
|
Rank: Advanced Member
Joined: 6/16/2006
Posts: 502
Location: AT
|
MapForce should process databases record by record, so as long as there are only database components involved, memory consumption at runtime should be almost constant.
Or do the memory and performance problems already occur at design time?
|
|
|
guest |
