Good question!

Hey, since here aren't a lot of people, I dare to write something for
discussion although I don't know much.

When designing rotor systems, you have to keep the tip speed subsonic
to prevent all sorts of problems and losses that would otherwise occur
at the sound barrier, right?

Well, the main reason is noise and then there are strength issues.

What about large turbofan engines such as on modern airliners? The fan
tips must be travelling at many orders of magnitude faster than the
speed of sound? Why are they not subject to the same limitation?

Why do you think, they must be travelling at Mach 1? AFAIK it's not
true at all. First there is no advancing blade, so the TAS does not add
to the fan blade tip speed because it is perpendicular. Second, even
Jets capable of supersonic speed have most parts of their engines
working with subsonic velocities. Especially the air intake is designed
to be at subsonic. Good to see at the Concorde.

It sounds unlogical, that the air intake can be subsonic when the whole
aircraft is supersonic, but I cannot recall the explanation. Supersonic
aerodynamics are rather simple, but the transition is extremely weird.

Having said that, the fans must be rotating at a reasonable speed
compared to a rotor. They may rotate a bit faster because their diameter
is smaller and they do not need a margin for an advancing blade.