Well there is no such animal, the debugger would have to be integrated in with PJ64. Best thing I can think of is using GDB for your base set. Let me see.. that would allow you to inspect and step through each assembly level instruction in the N64's CPU. The graphics and sound coprocessors are horsies of a different color, though I suppose with the right added tools you can do the same for those as well. You can't debug it in Dynamic REC mode though.
If you plan on debugging the graphics plugin that's yet another matter, you need the GPU source code and to launch the main AP (PJ64) from within VC++ or BCB. It's possible to do this through a network (if you are using glide for example and thus have no window) but it's not exactly EASY to do. When you compile the DLL it generates debugging information in a seperate file, thus PJ64 must be run from the IDE application to debug the DLL (hmmm) I know it's a bit too exciting (hehe).
As for running your games at there best, lots of trial and error might be more practical.
Cyb