By the time of the Fourth War he's probably going to become bad news sooner or later even without Gil tipping him down. Amakusa could see this in the Apocrypha timeline, which is why he chose to spend as little time with him as possible, and that's despite Amakusa being a Servant.