If people don't understand your collection then they could mistake collecting for hording, which may or may not be the case here, but I think hoarders in the sense that show usually portrays have a deep psychological (rather than just a sentimental) attachment to the stuff cluttering up their lives.