If that's the case, then most genres are not mainstream for the simple fact that so few titles ever hit the big sales figures when compared to the larger quantity of titles in that genre. That takes us back to Madden. Regardless of Sega and Tecmo's previous success, it's pretty much entirely Madden at this point for various reasons. So football is niche now? Or at least is more niche than it was 15-20 years ago?
You keep talking about the "RPG genre" as if it's all or nothing. As if there has to be a majority of success on all fronts irrespective of the content of the games themselves. That's crazy. People don't buy games based on genre alone. They base their decisions on game content. People don't go "Oh, I like Mario...guess I like God of War, too, since it's also a platformer." They very well may like both games but one doesn't beget the other. Just like if people are playing Mass Effect but not Ar Tonelico, that doesn't mean the genre as a whole isn't mainstream. Correlation is not causation. There are too many variables among the titles in the genre, every one of them can alone determine why one game succeeds and another fails. You can't ignore all of that and write it off as the entire genre being niche. If anything it just means there's a niche within the larger genre.
Seriously, if it the vast majority of titles in a genre not being mainstream renders the genre itself not mainstream then we'd be left with...I dunno...one mainstream genre? If that?