Hmm, have you tried all sound plugins? With your specs, I would recommend using jabo's. Also make sure that when using jabos that you use the rsp from pj64.
I've lost sound recently. Since I had to reinstall XP my sound in 1964 & pj64 has disapeered, Mupen sound is only there for some plugins and it's very rough. I hadn't looked into it further as I've been playing N64 games on my xbox instead.