I don't remember where you can find Gameboy docs, it was a long time ago.. just use Google
Anyhow Gameboy isn't necessarily the "next" platform in complexity, there are a lot of 8-bit systems with similar complexity.. The SMS (Sega Master System) is said to be quite easy, and there's a lot of information about it out there too, and there are a lot of games out for it (probably not as many as gameboy, but still tons). Having lots of games to try in your emulator is important, since it will help you find bugs. I'd recommend that you should at least take a look at Master System

NES is however a lot more complex because of all the "mappers". There is still no 100% perfect NES emulator!