The solution to this bickering is very simple. Add a cast time/animation similar to the duration of the pet summoning. By the time you`ve done it, you`ll be dead if there`s enough people attacking you. Personally, theres far, far worse CSW issue`s than this. Mages TP`n parties into the back room from under the floor is just one example that springs to mind.
TL;DR The castle is 10 times easier to defend that it is to attack.