The option is called "use file storage instead of memory cache." You need to get the latest WIP build if you don't see it now. If you enable file storage with the PNG pack then GlideN64 will read all of the PNG files and then create a new cache file that it can read from more quickly. Unlike Dolphin, it won't load new textures instantly because it's reading from that cache file instead of the PNGs.

I can't recommend any Mupen64 version because I've never gotten any of them to work. I think the only decent version with a GUI is also behind a paywall. It does seem like that 5GB memory figure is wrong, as I've since seen double that listed with even less of the pack loaded:

I think reloading the pack causes the figure to inaccurately double each time. Whatever is going on with the RAM use, a member of my Discord did successfully compile a 64-bit Project64 that might hopefully solve any memory issues for good (it's interpreter only right now).