That's now how it works. It's just that using a Servant makes them more Buddy, and upon reaching a certain number of being Buddy, they go up a Buddy Rank, which increases the effectiveness of power up and bond up. Basically, you're encouraged to use the same party all the time. Build your all Berserkers team.