If you are willing to put in some legwork, I would pick up an original xbox. Now now now. Hear me out.
There is a lot of literiture online about softmodding the xbox. Once you do that- you can emulate pretty much anything that came in a cartridge, and a ton of arcade games. Plus There are emulators for the psone, and sega cd. The xbox will actually run these games from the cd tray.
To answer your question though- i would go dreamcast. They are super super cheap these days, and there is a lot of sites out there giving you instructions to burn games that you might not be able to get ahold of like Ikaruga or rival schools project justice. No mod chip required. Plus- even though it is a small libary of games the quality is amazing. The games you want to play vs the games that no one plays is pretty much 1 good game for every 3 games. That's some good odds if you ever find one that you've never heard of before in a thrift store.
the reason i wouldn't go psone is that all of playstation's stuff has been backward compatible, and the playstation controller has changed very little. A psone is kinda a waste of money because you could always spend a little more and get a ps2.... or a lot more and get a 60gb ps3. Both have the same experience, plus more.