I don't see why it's that annoying. I've done it a few times too, and none of the people I asked seemed bothered.
Admittedly, I tend to do it only when there's a genuine connection between the two Servants (like coming from the same myth-cycle or having similar motivations/backstory), and not nearly as often as Koga does.