Yes, the vibe is different. Both are wonderful.
Fairfax is smaller, but the if you attend the events NormR has posted you will likely find that most everyone you run in to will be extremely knowledgeable and serious, and the events are all first rate.
There are tons of smart capable investors and business people at Berkshire, but its just so large and there are so many people who are there for the party. At Fairfax, its smaller, but very high quality throughout. Yes you might bump it to someone who is just there at the Fairfax AGM because they heard the Keg was providing free food samples, but that is rare. Prem and Fairfax are famous to people "in the know", Buffett is just plain old famous.
FFH being held downtown in a major financial center your also more likely to find yourself at events that are higher cost or where you might feel more comfortable in business attire. Aside from that, there are many people who attend both, and much will feel familiar.
Also, depending on where you are coming from Toronto can be super easy to get to compared to Omaha. Toronto is served by two easily accessible international airports. Omaha is much more of a challenge for many people to get to.
I hope that's helpful.