If you're just interested in 8-bit stuff, then I'd say go with the Dingoo. You can get them for fairly cheap (DealExtreme sells them for about $80), there's a respectable homebrew scene. While it runs software off SD card, you might have to do a bit of software hacking, as a lot of software requires Dingux (a version of Linux that runs on the Dingoo) to be installed. You might want to read some FAQs to see if this is going to be a problem for you or not. There are emulators installed right from the factory, so if they cover what you want, you might not have to worry at all. Rumor has it that a new model of Dingoo is on the way, though from what I could tell, the only differences were more memory and a different case style.
The PSP is alright, but you do have to hack it to install a custom firmware... certain models will be more difficult than others (avoid the 3000 and Go), and if you buy retail games, you might have to fight with official firmware update). I also have had poor luck with emulators randomly crashing, confusing directory structures required on the memory card, etc. I wouldn't buy a PSP specifically with emulation in mind. Of course, you have the advantage of playing commercial games, and the PSP is capable of doing all 16-bit systems full speed, and has the Sony-made PSX emulator.
You can also run homebrew on the DS, but this requires a flash adapter (the Acekard 2i is what you want, and only $15). The DS is less powerful than any of the other consoles listed, but there's a good chance you might already have one, so it's worth mentioning anyways. I haven't really done much emulation on DS (I already own a GP2X), but my understanding is that 8-bit systems should be full speed.
The GP2X is somewhat obsolete these days, but if you can find one for cheap, it's an excellent little system. All you have to do is copy whatever you want to an SD card and you're good to go. Very simple overall. The system is powerful enough to do Genesis and TG16 full speed, but SNES is a bit of a challenge for it (some games work great, others slow down a lot). There were a few different versions of the GP2X released, with some changes to the controls. If you decide to get one, I'd say go with the later F-200. None of the models have a "normal" d-pad, which is a little annoying. Keep in mind that you'll have to install the newest firmware update in order to use SDHC cards with it.
There is a newer system called the GP2X Wiz that came out a while back that is a bit more powerful (enough to do SNES emulation perfectly and PSX somewhat decently). Most GP2X software and emulators have been ported to the Wiz. Rather than getting a Wiz though, you might want to wait and see what happens with the Caanoo, which looks to be yet another replacement (same hardware as Wiz, except with more memory, a different case style, and a built-in accelerometer). Rumor has it they were going to announce it at E3, but someone leaked it early. One would hope that all Wiz software would work with it out of the box, but this company is known for making silly changes that break compatibility, so wait and see.
There's also the Pandora. This device is more powerful than any of the others listed (enough to do full-speed PSX, probably N64, and *maybe* even Dreamcast if you don't mind playing without sound and with frame skipping). The Pandora also has a keyboard, which comes in very handy if you want to do any computer-type stuff, or emulate classic computers like C64, Atari 8-bit, Amiga, etc. There have been ridiculous amounts of delays in this project, but they've finally started shipping. They're still working on getting the first 4000 preorders built (let alone shipped), and no one knows whey they're going to open up ordering for the next batch... if you're patient, this might be the better choice overall, but may be overkill if all you want is Atari 2600 and NES.
--Zero