You're mixing up two different issues, though. You're completely ignoring that the genre itself has little to do with all the random PS2 and PSP RPGs being niche as hell. The fact that their entire presentation is what it is makes them niche as hell. That's a incredibly important difference you're not addressing. The genre really doesn't mean jack shit. You make an RPG with a huge budget, massive advertising, fantastic acting, and a dramatic story and you end up with the multimillion selling Mass Effect. You make a budget PSP RPG with anime cliches, wacky outfits, and rudimentary mechanics and, surprise, niche game is niche.
Turn Ar Tonelico into a platformer and it'll have the exact same appeal it already does. People aren't avoiding it because it's an RPG. They avoid it because it's...well...
THIS. You honestly think that people who pick up Mass Effect aren't picking up Ar Tonelico because they don't like RPGs?
That's like saying fighting games aren't mainstream because most people only play Street Fighter and Mortal Kombat while ignoring Guilty Gear, Arcana Heart, and Melty Blood.