Well, when you make rules that everybody's supposed to play by and someone doesn't, it harms all the others who were supposed to be playing the game with the same information as everybody else but actually weren't. So everybody who could have benefited from knowing that Buffett was about to invest but didn't was harmed (most directly those on the other side of the trades, but also everybody else who with that information would have traded differently). That's a lot of people. But that's just the first degree; once you have enough of that stuff, it erodes confidence in the whole game and makes the whole system stop working, so that's a lot more damage, even if more indirect and diffuse.
I agree with Liberty.
If a precedence is set, then the system will stop working and give people incentive to rub elbows with politicians, not play by the rule and do some charity to cover their butt if caught.