Jitter is the reason why the same digital signal (pattern of 0/1 bits) can sound so very different when converted to analog. And jitter is incredible difficult to get rid off. Let me try to explain why (I will simplify here and there to explain the core of the problem more clearly).
The digital signal has a very high frequency, so everything in the digital signal path will transmit electromagnetic waves (radiowaves). Even the power supply will start to transmit because it has to supply the aggressively switching digital loads (0V or 5V) with the same high frequencies.
These waves are picked up everywhere in the circuit including that same paths/parts that transmit them, causing many tiny bit timing errors: jitter. These timing errors are created in complex patterns related to the various digital signals (containing music and functional information), power supply regulation, etc. It’s these jitter patterns that distort the sound in an ugly way (random jitter is far less audible).
If you could see the electromagnetic waves/fields in digital equipment it would look like total chaos, even in very well designed equipment. Even with optimal optical isolation and/or reclocking the jitter patterns at the input will be partly copied to the output. Total isolation is simply not possible. If you want digital audio at the highest possible level, you will need to prevent jitter as much as possible.
In simple digital circuits it’s easier to limit jitter damage than in complex circuits. Note that adding a jitter reducing circuit will also introduce some new jitter pattern, it’s that complex.
Upsampling/oversampling adds a lot of complexity, so more usually more jitter patterns of the ugly sounding kind. Higher resolution formats also mean higher frequencies, increasing the problems. A perfect clock hardly improves anything because the jitter problems in the rest of the design are usually far worse. The actual point where digital is converted to analog is critical, but you can not expect to eliminate jitter from a dirty digital signal, no matter how perfect the clock is.
This is why some of the early, simple DAC designs can sound so great (when well implemented).
Unfortunately it’s extremely difficult to implement digital audio at a high level and very, very few designers understand the problems. No manufacturer has ever been able to produce an affordable high-quality CD player. Still it is possible. Now everything is shifting towards computer audio. Computers are a giant source of ugly jitter. Over the years we’ve tried many computer/streaming audio solutions, everything which is supposed to be “the best”. But we haven’t heard anything yet that could approach a good CD setup (starting at 5000 euro/dollar). Now, most CD equipment sucks (including very expensive ones), and computer audio is the future, so it’s not strange that many audiophiles are switching to computer audio. But if you’re aming for the highest quality then there’s still a gap. I hope it will be bridged, but it won’t be easy. Again the simplest solutions seem to be the best sounding ones (see: The search for the best USB DAC).