Played with this some on my machine and discovered something that may be obvious to you and others, but will share it just in case. Basically, and I'm paraphrasing some LLM's explanation here, if you encode on a GPU and decode on a CPU you can have issues due to rounding of floating points being done differently.
Is this common knowledge? It suggests for it to work with two parties, "...crucially the same compute backend (ideally deterministic CPU math)" should be used.