Sure, if your servant is right next to you. But look at FSN when half the masters go to school without their servant.
For example in Fate/Zero in this scene Assassin could have taken out Kiritsugu effortlessly and Saber would have been left with no master and vanished.
Spoiler:
Assassin was actually there watching but just did nothing.