Revert "demux: improve stream selection state"

The stream selection state wasn't improved. I didn't realize this messed
with caches. All in all, just not a good idea. Back to drawing board I
guess.

This reverts commit f40bbfec4fcd2d9a787a4d98ec7698a646e5607e.