Alright, so I assumed. I assumed that the core was the cause of the problem and not the graphics plugin. I did, however, test it using various graphics plugins. For some reason, though, I must have made a mistake somewhere, because it turns out that the core does not fix the slowdown. I don't remember how I even reached my conclusion, anymore.
So, try this. Use 1964 with the Rice's Daedalus 5.2.0 plugin. It comes with 1964, so no need to download it. Just press Ctrl+P, and change your plugin to Rice's Daedalus 5.2.0.
I have noted, however, that some static appears in Project64 but not Nemu64 when both emulators use the same graphics plugin.