In the 8-bit era, Nintendo had about a 78% market share. For the 16-bit era, Nintendo's market share dropped to 51% but they still held a narrow lead despite the rise of a successful competitor, the Sega Genesis, and a more crowded console market. With the next generation, however, Nintendo's market share plummeted to 22% with the N64, worsening to 10% with the GameCube before rebounding to 37% with the Wii. See sales numbers below (source: Wikipedia)
I feel that Nintendo's fear of piracy is what cost them massive amounts of market share, leading to a sort of "lost decade" for the company from about 1998 to 2006. The fear manifested itself in the use of cartridges for the Nintendo 64 instead of the then industry standard CDs. I loved the feel of N64 carts in my hand as a kid; PS1 disks just seemed flimsy and insubstantial. But game development, and the market, is not based on a 6-year-old from Tennessee's whims; it's based on hard technical data. Those cartridges averaged 16 megabytes, which was a step up from the 16-bit generation's 2 megabytes (keep in mind these are mid-generation averages, cartridge sizes varied substantially) but tiny compared to a CD's 700 megabytes. A CD could fit about 44 average N64 games, or about 11 maximally-sized (64 megabyte) ones.
As if that's not bad enough, cartridges are more expensive to manufacture: N64 games typically went for $60-80 new, while PS1 games were typically $40-50. Need more space? A second disc for a PS1 game wouldn't add much to the price, while good luck trying to get people to pay well over $100 for a two-cart game (which was never tried). Third parties flocked to the PlayStation, and the only reason the N64 was able to even keep its 22% market share was because Nintendo put out some masterpieces on the console. Had the N64 gone with CDs, it probably would have sold twice as well. Nintendo went with the cartridges based on fear of piracy hurting game sales, but this fear ended up halving their sales on the system.
The GameCube? Same story. Nintendo went with one-gigabyte discs instead of the 5-gigabyte DVDs due to, you guessed it, fear of piracy.
Sales numbers:
8-bit gen: (NES 61.9 million, Master System 13 million*, Atari 7800 about 2 million, probably 2 million others such as Atari XEGS, Zemmix, etc)
16-bit gen: (SNES 49.1 million, Genesis 35.2 million, TG16 10 million, CD-i 1 million, Neo Geo 980,000, NG CD 570,000, Atari Jaguar 250,000)
32/64 bit gen: (PS1 102.5 million, N64 32.9 million, Sega Saturn 9.3 million, 3DO 2 million, others such as Playdia, Loopy, FM Towns Marty about 1 million)
PS2/N64/Xbox/DC gen: (PS2 155 million, Xbox 24 million, Gamecube 22 million, Dreamcast 9.13 million)
HD gen: (Wii 101.6 million, PS3 87.4 million, Xbox 360 84 million)
*not including recent Brazil sales