Sweet man. I don't think VB would work because it isn't fast enough because it is too high of a language. The N64 is quite a beast. Also, do you have any programming history? Try starting with an NES emu or simplier first, if you've never done anything before. NES isn't all cake you know, there a plently of mappers.
Anyway, get good at C and C++. Start with something simple like an NES (as I'm doing). Work your way up. If you just start with N64 you may become too frustrated to continue. Like Ice said though, check out some sources as well as documentation, espeacially on the op-codes as well as Zilmar spec and how the core interacts with the 3 other plugins, that is if you are writing for a Zilmar spec. emu.
If you have experience with C or other languages go for it! My advice is to get a real N64 , tools to open it, docs on N64, and open source emu's to see how it goes.
Best of Luck!