Nintendo and Sega both had strict restrictions on their 3rd parties during the 8-bit era of video gaming. Nintendo's are more well known, but basically boil down to: if you made NES games, you couldn't make games for other platforms, Nintendo had to manufacture all of the cartridges, you could only make 5 games a year. Sega didn't even allow third-parties until late summer 1988, two years after their console was released. By this time Nintendo had built up a large lineup of AAA games. By the 16-bit era, these restrictions were significantly relaxed, Nintendo's due to an FTC case. In real life, Nintendo got 80% of the market, Atari 12%, and Sega 8%.

But what if the third-party restrictions were less strict, say akin to what they were in the 16-bit era, from the start? There would still be the Nintendo Seal of Quality and Sega Seal of Quality, etc but things like cross-platform games would be allowed (as we saw in the 16-bit era)? There were a few cross-platform games in the 8-bit era but that was due to the efforts of companies like Tengen.

In my opinion, Nintendo would have still won, Sega would have benefited, and Atari wouldn't have come off much better and may have ended up off worse. The Master System was the more powerful console and games appearing on the NES and SMS would have helped the SMS library, getting it more sales, in addition to the games potentially looking/sounding slightly better. Ultimately, however, I think that due to Nintendo's first-party games, Nintendo still would have won the race. The 7800 would have looked even more outdated and behind with a successful NES and Master System. The games the 7800 offered were out of style, mostly gussied-up versions of what was popular in 1982. The biggest reasons people bought 7800s was that they were cheap and offered 2600 backward compatibility.

Something like the PS1/N64/Saturn race would have happened, only with fewer consoles sold overall due to the smaller size of the video game market in the late 1980s vs. mid-late 1990s. In that race, the PS1 shipped 41 million units in North America, the N64 21 million units, the Saturn 2 million units. Ironically, the PS1 got 64% of the market, the N64 33%, and the Saturn 3%.

In my scenario I think the NES would get something similar, within a few percent of 65% of the market, Sega maybe 25% or so, and Atari 10% (or less).