They're hard on hits for QB's for the same reason that they're hard on hits for kickers. Both are often in a vulnerable position. Not like you're trying to tackle a 200 lb WR or something...