If you look at the labeling on that box it won't work, that's a composite in to Coax out setup for back when TV's with composite jacks were in the minority. These days it's the other way around. Coax jacks are getting eliminated.
I believe that you can use the digital/analog converter boxes that started to pop up when UHF signals were phased out. Those have Coax in and both coax and composite (and it some cases S-Video) out.
They run about $30-$40 or less. Can anybody confirm that those work as a pass-through for analog signal from an RF system like a 2600? I would imagine that they do.