The important part would be what Zouken was doing to the family and why he turned away from Zouken because of that.
Since again, we don't know what happened to Kariya and Byakuya's biological parents, but considering what happened to Shinji's mother we can speculate.