Here's a great article from Eurogamer that talks about installing Linux on the PS3 and what it can and can't do:
http://www.eurogamer.net/article.php?article_id=70611
According to that article, they were able to get MAME up and running. In that article they installed Fedora, but I'd go with Ubuntu myself. In fact, I'm going to make an effort to do that sometime this week. I just need to dig up a USB keyboard and mouse.
By typing "ubuntu on ps3" in Google you'll get a bunch of install guides to choose from. This seems like a pretty straight forward one:
http://news.softpedia.com/news/Ubunt...S3-52699.shtml