You might want to try changing the "deadzone" of your controller plugin to see if that helps, you might also try changing the "range" of your analog stick. Both can be done in the "Configure controller plugin" part of options.
I find a Deadzone of 5% and Range of 66% (Real N64 Range enabled) allows for you to get the aiming almost dead-on. You might also want to make sure you've set up your controller's calibration in Windows to eliminate any jittering that you might have going on there.