Extremely large texture packs push the Process to the very limits of memory allocation,
This can be worked around by enabling Large Address Aware, which allows a 32bit process to utilise memory addresses past 2GB on 64bit operating systems, and appropriately configured 32bit operating systems(*).
The default compilation setting for most emulators is to compile without support for Large Address Aware enabled.
To enable Large Address Aware on an already compiled version of the emulator, you can use either the Large Address Aware tool provided by FordGT90 or 4gb Patch.
Getting it working on a 32bit operating system is a lot trickier, in fact to the point of impossibility with larger packs requiring an address space past 3GB to even load the pack initially, in these cases its better to download a precompiled texture cache where possible.
*32bit Operating systems can have the user address space adjusted using either the /3Gb boot switch (XP only) or adjusting the UserVaLimit in the NT6 BC Editor.