Phonotactics.

I believe lots of people know, that different languages use different phonemes (sets of sounds). However, what often goes unnoticed is that different languages also use different combinations of phonemes. The rules dictating which combinations are allowed or not in a certain language are called phonotactics of that language.

So. What you’re noticing is a restriction in the English phonotactics: they don’t allow English syllables to start with /rj/ [see note], even if Japanese does. English is known for allowing some pretty weird stuff, but that sequence won’t pop up in English, so its speakers have a really hard time with it.

And note how all your examples have that sequence:

• ⟨遠慮⟩ enryo /eN.rjo/
• ⟨留学生⟩ ryūgakusei /rju:.ga.ku.se:/
• ⟨略奪⟩ ryakudatsu /rja.ku.da.tu/
• ⟨涼⟩ Ryō /rjo:/

There’s also the matter that Japanese /r/ is not the same as English /ɹ/. Japanese /r/ is typically pronounced as [ɾ]; I think a lot of English speakers are used to that sound only between vowels, because they associate it with /t/ or /d/. But even if the English speaker in question uses [ɹ] for that Japanese /r/, the sequence is still problematic.

For further info on English phonotactics, check this page. Wikipedia also has a page for the Japanese phonotactics, it’s useful to contrast both.

note: that “/j/” is the first sound in “yes” or in “やる”. It should not be confused with the sound in “jeans” /dʒ/.

The same reason that Japanese speakers often have trouble with L/R distinction, the English ‘th’ sound, and others; things that don’t exist in one language take time not only to learn to pronounce, but to learn to even hear. If you don’t grow up with those sounds, hearing them is harder.

The Japanese r-dan sounds also aren’t mapped directly to English R (nor L, and indeed I hear some speakers pronounce it much more l-like to my ear). It takes practice and exposure.

Edit: forgot to specifically mention things like りょ. If it’s written in romaji, there’s no clear way for an absolute beginner (or someone uninterested in the language) to know. This is why going to kana from the start is a better option.

To add on to what others have said, making sounds is heavily based on muscle memory for the position of your tongue. In English, R sounds typically have your tongue further back. In Japanese it’s kind of halfway between R and D and it flicks forward quickly so it sounds like a combination of R, D, and L to native English speakers. When they see that it’s an R sound based on the romanji they will default to the their native R tongue placement. It’s a much easier sound for people who can roll their R’s because the placement of the tongue is more similar.

It’s possible to learn how to make the sounds by practice and’s listening but often it will take years if you don’t get coaching on how to place your tongue correctly.

It’s not rocket science, they’re just sounds that don’t get used in English.

I’m not really sure either. I learned these sounds self-studying Japanese on my own in high school and I don’t consider them especially hard.

It’s probably just that those sounds aren’t used much in English so the first few attempts are going to sound bad.