The XP penalty inflicted by quitting Overwatch games before they end has been a source of confusion among players. Mostly the questions concern how to get rid of it. In response, Blizzard has detailed the punishment on Overwatch's official forums.
If your percentage of games left across the last 20 games rises above Blizzard's "pretty high" threshold, you'll be given a warning in the bottom-left corner of your screen. Crucially, this includes disconnects and inactivity kicks. The former seems rather cruel, but Overwatch can tell when the disconnect is server-side, and won't penalise in those instances.
Pass the threshold and you'll be hit with a 75% reduction in XP gain. It's not time-based—the penalty remains until your leave rate in the last 20 games falls back below the threshold.
Leaving the game before the 'Assemble' phase is over won't count against you. The same is true for leaving on the victory or defeat screen. What will add to your leave rate is being pulled from a game in progress by joining a group. On that count Blizzard has admitted "this process may not always feel great, and that's certainly feedback for us to consider."
Some players have reported getting the penalty for leaving games as spectators. This, we're told, is definitely not meant to happen, so file a bug report if you get hit.