I just submitted a new idea:
The maximum size of XML file imports is limited to 30 MB. (This can be further limited using the MAX_IMPORT_SIZE system setting.)The maximum size should be increased, preferably to at least 100 MB. Failing that, an automated mechanism for splitting XML exports into multiple files should be provided.
If you like the idea, please vote for it.
As there will be always some limitations for RAM or file size, a splitting tool to create multiple XML files would be nice and doesn't need modifications on the current modules.
Of course this tool should create files with coherent object definitions in each file, don't split simply physically the number of bytes !
The size of the splitted files should also be a parameter, to enable quick load using multiple small files (multiple new application load i.e.) or a longer one for a few big files (mass update i.e.). It also can help to get file size usable by the AE Server regarding the availability of RAM (2 GB RAM is not 36 GB RAM when loading XML definitions !), workload on the server, etc ...
This is something already managed by the compressing tools (Zip, 7Zip, ...) so technically this should not be a real issue to create the tool. Just need to include the concept of object definition to ensure that the splitted files are usable independently for loading.