I have a Swedish "mm" that means "yes" or "go on", but when I instinctively use it speaking English, people tend to interpret it as a "hmm?", like "sorry, what was that, I was thinking of Ancient Rome", basically the opposite of what it's supposed to signal.

Over the years, I have tried to get rid of it or, failing that, modify it to sound more affirmative, but to this day, I still have it, and it's still causing confusion.

If it sounds more like a Japanese "un", native English speakers and Cantonese English speakers read it correctly. I can do it with effort, but unless I catch myself, that's not how it comes out.

Basically, the Swedish rising tone "mm" sounds like a distracted question mark to people, but a falling tone "un" sounds like an affirmative full stop.