Only works on the curse debuffs, not the clock debuffs. You need to bring some good debuff clear with you, because you have to hit him constantly, since each turn he recovers a curse stack and I believe he can use a skill to give himself some more curses, apart from recovering them when you break his bar.
He can also remove one curse stack himself, but he uses it to charge one NP tick and he doesn't do it that often.