Three limitations to processing XML files are:

1. XML File Size as set by the OS.
2. RAM consumption.
3. CPU consumption.

Most XML Parsers can be used on big files (100GB) without exceeding these
limitations.  This is because XML Parsers are stream based - reading small
chunks at a time.  If you want to process the XML file, however, you will
need to use a streaming technology like SAX.  Other XML processing
technologies like many DOM implementations will cause you to exceed RAM.

Choosing a max like 1MB to 50MB will allow you to more freely use a wide
variety of XML and other technologies (like EMail attachments) making your
XML less constrained.  Again, it depends on your use cases for the XML.


Jim


-----Original Message-----
From: Koya, Suresh (Suresh Koya) [mailto:suresh.koya@h...] 
Sent: Thursday, September 03, 2009 10:15 AM
To: Mukul Gandhi; Simon St.Laurent
Cc: xml-dev@l...
Subject: RE: [xml-dev] XML not ideal for Big Data

I  too agree with you. I guess the idea which started the markup movement
was to make data human readable along with being machine readable. Someone
should probably think once before XMLizing a 50 MB size stream, if any
average person(I cannot) can read through a 50 MB document. The parsers are
meant to be human interpreters for machines and optimized(fast) xml parsers
are like speed readers. In this aspect the machines could be as
(in)efficient as us. That is why the parsers limitation on data size always
looked good to me. 
 
By the way, was there any information which tells what is the reasonable
file size which a person can read trough and understand? I found some
useless information from a google search or my google search criteria is
inefficient. Whatever.. 

/Suresh
This is my own thought and the company I work for has nothing to do with it.
 

-----Original Message-----
From: Mukul Gandhi [mailto:gandhi.mukul@g...] 
Sent: Thursday, September 03, 2009 9:38 AM
To: Simon St.Laurent
Cc: xml-dev@l...
Subject: Re: [xml-dev] XML not ideal for Big Data

I somewhat agree to this article, about not using XML for very large sizes.
I generally not like (or it doesn't work well with XML tools) working with
XML's larger than, say 50 MB! But different people, may have different
preferences.

But certainly XML is very valuable in smaller sizes (5 bytes to 50 MB I
guess :)). We can think of useful existing applications of XML, like small
sized messages that we send, configuration files for various purposes (EJB
descriptors, Eclipse meta data and so on). Almost all kinds of tools produce
XML in some form, for report generation, which can be transformed to various
targets (HTML, PDF etc).

If I am given a huge XML document for consuming by my application, I have no
choice to process and consume it :) But if I need to design a storage
format, and a data model for large quantities of data (say in giga bytes), I
would not like to store and model all the data as XML.
I would try to do it as a combination of XML and Relational format, perhaps.
I would try to use XML, for schema's which change frequently.

The beauty of XML today is, that every modern software system understands
XML, making XML as we all know, a great interoperability format.

On Thu, Sep 3, 2009 at 9:23 PM, Simon St.Laurent<simonstl@s...>
wrote:
> Perhaps there were better ways to have made XML work with his problems...
> but I think on the whole he's right.
>
> http://dataspora.com/blog/xml-and-big-data/
>
> --
> Simon St.Laurent
> http://simonstl.com/



--
Regards,
Mukul Gandhi

_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS to support
XML implementation and development. To minimize spam in the archives, you
must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@l...
subscribe: xml-dev-subscribe@l... List archive:
http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php


_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@l...
subscribe: xml-dev-subscribe@l...
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php




_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@l...
subscribe: xml-dev-subscribe@l...
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php