I think the main concern will always be with having non-genuine game and accessories ending up on the market that is not labelled as such. Yes it is your own property and if you have for example a damaged label, it's normal that you might want to take steps to fix and improve your property. But we all know there is a good chance this game might end up back on the market at some point, and I think this is what people feel uncomfortable with. What if when it gets sold the person forget to mention some parts are not genuine?
At this point you wonder where it will end. What will stop someone from making reproductions of Little Samson, printing his own professional looking label and sell the game as genuine? I think people fear that if we are not cautious with the subject of restoration we might reach the point where it will become more and more commonplace and in the end you will have to doubt everything you come across and start asking for picture opened cart with PCB showing.
TL;DR: Slippery slope.