Well, for one thing many of the games of today are designed to be "experiences" as opposed to just being "games". They are more about the story, the characters, the plot, etc. more so than actual game play.

Take a look at Alan Wake. Got great reviews, has a great story (I loved it-couldn't wait to get through the game to find out what happens) and the graphics and atmosphere are top notch. However the game play is practically old school Silent Hill combat with more linearity: Watch/listen some part that adds to the story, blast a bunch of evil dudes as you make your way to the next check point, rinse and repeat. If you got hurt, you could wait a bit or find a lamp post to heal under. Or the original BioShock, which does hearken back to the difficulty of the System Shock games but now you have Vitachambers which can bring you back to life if you buy the farm.

Accessibility, like what Robocop2 mentioned, is partly the reason. The other reason is flow. The game has to flow to keep the thrill of the story going. Getting offed every few minutes breaks that flow and could break the player's desire to continue the game. To have them equal the difficulty of games from yesteryear would make the game too unrelenting and would possibly cause today's gamer to just give up.

As for what games today equal the difficulty of the past, I'd say check out titles on XBLA, PSN, etc. Those with worthy challenge are, surprise, influenced by the classic design of yesteryear.

Now the big question is, what does everyone prefer-the gritty difficulty of the past or the heroic-feeling experience of today?