A little while before the fifth Patch of Baldur's Gate 3 came along, I wrote up this news post about a teaser image Larian Studios had shared on Twitter, featuring Withers in a silly little hat warning "fun is mandatory".
I joked about being scared at the time. Little did I know, I was very right to be. If you rain on Withers' carefully constructed parade, his cosmic wrath is mighty. This was actually brought to my attention by camilams on the Baldur's Gate 3 subreddit, but I figured I'd boot up my save to test it myself. (Sorry, Volo. You were the only person I could bring myself to harm.)
What pushes Withers over the edge is actually murdering one of the guests. I ran around, blew up some furniture, stabbed people, and even got into a scrap with my old party. Withers doesn't intervene until you actually cross the line and do a kill—even knocking someone out doesn't count.
I'm honestly not surprised he reacts like this, considering the effort he's put into the banquet. Obviously he has his ulterior motives—clearly wanting to make sure Faerun stays protected in the future. But he could've just grabbed you all when the moment arrived and not before. The banquet's him being nice. So to shank Volo just 'cause you were bored is bad manners, to put it lightly.
If you think plonking you in a cutscene and ripping you through a portal without so much as a saving throw is actually unfair, you probably aren't going to 1v1 Withers. As I mentioned during my psychic unravelling at a ship between Raphael and everyone's favourite mummy man, Withers is likely an Avatar of Jergal, the former god of Death.
When the adventurers Myrkul, Bhaal, and Bane came to his domain to tear his power away, he basically shrugged and said 'this'll be fun', handing the realms of Death, Murder and Strife over willingly. All this to say, he's used to dealing with upstart adventurers—and considering the savage dunking he dishes out to the three after the credits roll, Withers always gets his way. Just be thankful he was there to respec Shadowheart out of her baffling starting stats, and eat your soup.