My hypothesis on why it is hard to beat the index...

[jobyts]

May be I just reinvented the wheel, but here's my hypothesis on why it is hard to beat the index. Any feedback welcome...

 

* Technical analysis works very well as long as many people do it.

 

* Investing in index based on market cap/stock price is momentum investing, a type of technical analysis.

 

* Many people buy the index.

 

* Positive momentum helps the fundamentals too. Being part of a popular index thus creates a not so trivial moat.

 

* The momentum and the moat makes it hard to beat the index.

[frommi]

source:

https://can-turtles-fly.blogspot.de/2009/08/charlie-munger-stock-market-as-pari.html

 

The model I like to sort of simplify the notion of what goes o­n in a market for common stocks is the pari-mutuel system at the racetrack. If you stop to think about it, a pari-mutuel system is a market. Everybody goes there and bets and the odds change based o­n what's bet. That's what happens in the stock market.

 

But if you look at the odds, the bad horse pays 100 to 1, whereas the good horse pays 3 to 2. Then it's not clear which is statistically the best bet using the mathematics of Fermat and Pascal. The prices have changed in such a way that it's very hard to beat the system.

[Jurgis]

Right. And because of what frommi posted, markets may be somewhat efficient, i.e. it's hard to outperform by picking any horse - good or bad - without extra insight. Obviously there are exceptions as some people on this board and elsewhere demonstrate.

[Hielko]

My hypothesis: it isn't hard to beat the index.

[rb]

One of the main reasons why it's hard to beat an index is career risk. It's really hard to beat an index when you don't really try to beat an index. Being an IM/PM is actually a pretty sweet gig. You make at least 250K, the hours are good, the downside is that every now and then you need to go suck up to a big client.

 

In order to beat the index, your portfolio will look very different from the index. It will have low correlation to the index and thus big deviations both up and down from the index. When you have a positive deviation you'll get a pat on the back and an attaboy. Maybe you get to take a victory lap or 2. When you have a negative deviation you get fired.

 

So where's the incentive to beat the index?

[CorpRaider]

Costs. Matter.

[randomep]

My hypothesis: it isn't hard to beat the index.

 

yes and no.

 

Buffer and Munger have said it is hard and it isn't hard at the same time (although they used different words for hard).  It is not hard in the sense that it doesn't take high IQ, or book smarts, or even a computer to beat the index. So anyone in theory can do it.

 

It is hard in the sense that very few people have the wherewithal to apply a theory and wait decades.  The not-so-hard to beat the index theory says you just buy good stocks and wait and wait.  A reasonable duration to confidently test this hypothesis is say 30yrs.  Who can do an intellectual exercise for 30yrs.  To do this also requires that the investor does not change his small AUM.  You see investors who run small funds with amazing results and get killed as they grow to a big fund and have to invest large amounts of money (cough cough Pabrai)

 

There are people who do this but they are by definition small anonymous investors you don't hear about. In the last year or so I heard of one guy who was a janitor / gas station attendant and died with $8M to his name. He simply invested in good stocks for decades.  There are many people like that and if you analyze each of their trades I am sure in the aggregate they beat the market.

 

 

 

 

[Jurgis]

My hypothesis: it isn't hard to beat the index.

 

False.

 

Even highly survivor-biased polls on CoBF, which is stacked by the best-of-the-best, does not show that it isn't hard to beat the index.

[Gregmal]

One of the main reasons why it's hard to beat an index is career risk. It's really hard to beat an index when you don't really try to beat an index. Being an IM/PM is actually a pretty sweet gig. You make at least 250K, the hours are good, the downside is that every now and then you need to go suck up to a big client.

 

In order to beat the index, your portfolio will look very different from the index. It will have low correlation to the index and thus big deviations both up and down from the index. When you have a positive deviation you'll get a pat on the back and an attaboy. Maybe you get to take a victory lap or 2. When you have a negative deviation you get fired.

 

So where's the incentive to beat the index?

 

This is probably the most overlooked yet relevant thing.

 

Further if you look at guys whom supposedly beat the indexes regularly, ie a Klarman, they definitely have a "dgaf" approach as to what others think and their portfolios are rather unconventional. Second, they already have their financial future's taken care of.

[KJP]

May be I just reinvented the wheel, but here's my hypothesis on why it is hard to beat the index.

 

Your model doesn't take into account who it is that is trying to "beat the index" and the issues that they specifically face.

 

1) People who pay a fee to have their money managed by third parties:  They face the missive headwind of management fees; they also may be indirectly exposed to the "professional money manager issues"

 

2) Professional money managers: Career risk counsels against deviating too far from the index; size and liquidity limit investable universe; investor base places limits on the amount of volatility that can be experienced; varying styles across time may not be permitted by investors seeking specific factor exposure; also usually look at performance after fees or costs; random variance in the timing (e.g., a strategy/investor who would be successful over 20 years never gets the chance because s/he underperforms over the first three years, whereas someone who gets lucky in the first three can hang around a long time because time-weighted returns can still look, even if their dollar-weighted returns are below average).

 

3) People who manage their own money and whose portfolio size does not create significant limits:  Time and effort; sticking to a strategy, even one that has been shown to work over long periods, is psychologically difficult.

 

So, both the OP's hypothesis and Hielko's hypothesis may be correct.  The OP may be correct for most people in most situations.  Hielko may be correct for small subsets of the groups above.  For example, Hielko's hypothesis may be correct for the (likely very small subset) of (3) that is both willing to put in the time and effort and has the temperament to stick to a proven strategy.  Hielko's blog suggests he may be in that small subset.  Most people, however (me included), likely overestimate their chances of being in that subset.

 

Put another way, the hypothesis depends on the population you're looking at, just as the validity of the hypothesis "It's easy to not smoke" depends on whether the population you're looking at currently smokes. 

[KJP]

 

Further if you look at guys whom supposedly beat the indexes regularly, ie a Klarman, they definitely have a "dgaf" approach as to what others think and their portfolios are rather unconventional. Second, they already have their financial future's taken care of.

 

I would add that they have an investor base (or enough personal wealth) that allows them to act that way.  You can be a fantastic investor, but if you have an investor base that demands, for example, very low volatility, you likely will struggle.

[Paarslaars]

My hypothesis: it isn't hard to beat the index.

 

Hard is a relative term. What is easy for you can be hard for someone else.

I always thought math was easy but until college I was surrounded by people who disagreed.

So beating the index can be easy for you but hard for 90% of people, I assume the statement here refers to the average investor.

So far you're doing better :)

 

 

 

[Hielko]

My hypothesis: it isn't hard to beat the index.

 

False.

 

Even highly survivor-biased polls on CoBF, which is stacked by the best-of-the-best, does not show that it isn't hard to beat the index.

Well, the question is how do you exactly define what being hard is. Lets say you could explain a simple strategy to slightly outperform an index to a random person in one hour. Simply sticking to this simple strategy would be enough to outperform. That's not hard. On the other hand, having the mental capacity to just stick to that strategy and not do anything else might be super hard for a lot of people. Or maybe the hard part is not following one certain simple strategy, but figuring out that this strategy is better than the thousands of other alternative strategies that exist for investing. It can be easy on one level, but hard on another level.

[DTEJD1997]

Hey all:

 

Playing no-limit Texas Hold-Em poker is relatively simple.  I can explain it to anybody with an 85+ IQ in maybe 10 minutes?  You would know enough of the game to be able to play it correctly & according to rules & form.

 

That certainly would NOT make you a winning player....

 

To add strategy on to that would maybe take 1-2 hours.  That would make you good enough to be an overall winning player...

 

To become a better player would take many, many, many more hours of practice & study.

 

Even with the practice & study, it is easy to read about it.  It is slightly more difficult to do it on a simulation for no money....It is entirely a different thing to do it in a casino, for real money, under stress & distractions.

 

Even most of those who fancy themselves experienced & learned players can't be successful in the long run because they lack discipline.  Easy to play right for an hour, especially when you get good cards.  What happens when you have tough opponents and mediocre/bad cards and you've been playing 5-6 hours straight?  Can you still keep discipline and be watchful?  85% of players simple can't do this...

 

I can't tell you how many "pros" I see lose their discipline after several hours...OR go on "tilt" after a setback.  Eventually they walk away after losing all their money.

 

I suspect something similar happens in the investment world.  Easy to read, easy in simulation, very difficult to accomplish in real situations.

 

Poker & investing is a HARD way to make an easy living...

[randomep]

 

 

Poker & investing is a HARD way to make an easy living...

 

DTEJD, I understand what you are saying but the way your are saying it and I wholeheartedly agree.  I once tried investing my mediocre savings full time after I lost a job..... I lasted 3 months. To me it is a miserable way to live.  If I got to that stage again, meaning it isn't really financially worthwhile to work because my savings is great enough AND I hate working.... I would probably become a professional investor for my own money. BUT I would spend 1/2 of my time circling the country playing chess.

 

Investing is just waiting and waiting....... and that is hard!

 

[Gregmal]

Also, perhaps getting a little too much into a philosophical angle here, but who's to say that the entire "its impossible to beat the market" campaign thats been growing the past few years isnt some wonderfully orchestrated marketing campaign promoted by companies like Vangard? Meant to discourage investors from even trying and just toss the money into an index fund?

 

Honestly I don't even know why "beating the market" is even that important to people. To me, I couldn't care less. It is utterly irrelevant how my investments perform compared to the market. As long as I am intimately familiar with what I am invested in and comfortable enough with what I own, I don't care about all the other noise. Its kind of even comparable to the market timing approach. You think Buffett really said "I'm banking on a turnaround at BofA" in 2011? Or do you think he simply found himself a great deal and was comfortable making the investment? I'd compare it to trying to hit a home run in baseball. Typically when you try to hit it out of the park, you grip the bat too tight and swing at bad pitches. Whereas when you just relax and pick your spots, life is a lot easier. I consider trying to beat the market just another unnecessary distraction that complicates one's decision making and adds unneeded stress.

[Hielko]

Hey all:

 

Playing no-limit Texas Hold-Em poker is relatively simple.  I can explain it to anybody with an 85+ IQ in maybe 10 minutes?  You would know enough of the game to be able to play it correctly & according to rules & form.

 

That certainly would NOT make you a winning player....

 

To add strategy on to that would maybe take 1-2 hours.  That would make you good enough to be an overall winning player...

Haha, I wish it was that easy! The only player pool in the world where that would be enough to become a winning player is playing it against the aforementioned 85 IQ players who just got the 10 minute explanation.

[rb]

Also, perhaps getting a little too much into a philosophical angle here, but who's to say that the entire "its impossible to beat the market" campaign thats been growing the past few years isnt some wonderfully orchestrated marketing campaign promoted by companies like Vangard? Meant to discourage investors from even trying and just toss the money into an index fund?

 

Honestly I don't even know why "beating the market" is even that important to people. To me, I couldn't care less. It is utterly irrelevant how my investments perform compared to the market. As long as I am intimately familiar with what I am invested in and comfortable enough with what I own, I don't care about all the other noise. Its kind of even comparable to the market timing approach. You think Buffett really said "I'm banking on a turnaround at BofA" in 2011? Or do you think he simply found himself a great deal and was comfortable making the investment? I'd compare it to trying to hit a home run in baseball. Typically when you try to hit it out of the park, you grip the bat too tight and swing at bad pitches. Whereas when you just relax and pick your spots, life is a lot easier. I consider trying to beat the market just another unnecessary distraction that complicates one's decision making and adds unneeded stress.

This is very much true as well. Beating the index is a very popular topic on CoBF. In real life not so much. People don't really care if their investments beat the index. Mostly they just want their money to be safe and earn a decent/good return. You really find out how much people don't care about beating indexes the moment when you see a client and you tell him that he's down 15% but he should be happy because he's beat the index by 500 basis points. If people's overarching goal in investing is to beat the S&P500 or whatever then no one would buy any bonds.

 

It's also true that investors obsessed with beating indexes will probably fail to do so. Just go out there, do your best, do good work and stop worrying about the index so much.

[DTEJD1997]

Hey all:

 

Playing no-limit Texas Hold-Em poker is relatively simple.  I can explain it to anybody with an 85+ IQ in maybe 10 minutes?  You would know enough of the game to be able to play it correctly & according to rules & form.

 

That certainly would NOT make you a winning player....

 

To add strategy on to that would maybe take 1-2 hours.  That would make you good enough to be an overall winning player...

Haha, I wish it was that easy! The only player pool in the world where that would be enough to become a winning player is playing it against the aforementioned 85 IQ players who just got the 10 minute explanation.

 

Whoops!  I should have correctly completed that thought/sentence!

 

Please add:  "If you followed those rules to a "T" and did not make any deviation from the strategy".

 

I know a guy who plays in local casinos, his street name is "Big Bet Bob".  98% to 99% of the time he simply folds his cards.  When he does bet, he will bet at least $100, to all of his money (most likely).  He may be betting $100+ into a pot of only $5, it does not matter.  He simply waits for super premium hands (A,A K,K Q,Q AsKs) and he fires away.  Incredibly simple strategy...He also picks his room/time well for the promotions.  He claims to win 80%+ of the time and over the long run makes $16 to $18 an hour.  This is an incredibly simple strategy to plan and understand...more difficult in the execution of it.  HOWEVER, it is a winning strategy for him because he has incredible discipline.  He can easily sit for 6+ hours and only play 4-5 hands the whole time. 

 

It is VERY hard to follow correct strategy all of the time and to walk away from money that you have invested in the pot.  I am very good at this, but I also make this mistake from time to time.

 

For example:  I flop a set (3 of a Kind) of Aces...and there is $300 in the pot....The next two cards come out and there is a possibility of somebody having a flush AND the possibility of somebody having a straight.  BOTH of these hands beat 3 Aces....but I simply can't get away from it because:

 

A). I've got a lot of money in the pot already (sunk cost fallacy)

B). 3 Aces is a very, very good hand, and I haven't been getting any good cards all night!

C). I'm not going to let those goofs (my opponents) get one over on me!

D). It is only another $150, so why not?

 

I don't make this type of mistake often, but I admit I've been guilty of it in the past.  How many other people make this same mistake?  Many, many, many players do.  It is easy to see the mistake, easy to write about it, but hard to get away from it in real life situations.

 

Similar mistakes happen in investing all the time....

[KJP]

Also, perhaps getting a little too much into a philosophical angle here, but who's to say that the entire "its impossible to beat the market" campaign thats been growing the past few years isnt some wonderfully orchestrated marketing campaign promoted by companies like Vangard? Meant to discourage investors from even trying and just toss the money into an index fund?

 

Honestly I don't even know why "beating the market" is even that important to people. To me, I couldn't care less. It is utterly irrelevant how my investments perform compared to the market. As long as I am intimately familiar with what I am invested in and comfortable enough with what I own, I don't care about all the other noise. Its kind of even comparable to the market timing approach. You think Buffett really said "I'm banking on a turnaround at BofA" in 2011? Or do you think he simply found himself a great deal and was comfortable making the investment? I'd compare it to trying to hit a home run in baseball. Typically when you try to hit it out of the park, you grip the bat too tight and swing at bad pitches. Whereas when you just relax and pick your spots, life is a lot easier. I consider trying to beat the market just another unnecessary distraction that complicates one's decision making and adds unneeded stress.

This is very much true as well. Beating the index is a very popular topic on CoBF. In real life not so much. People don't really care if their investments beat the index. Mostly they just want their money to be safe and earn a decent/good return. You really find out how much people don't care about beating indexes the moment when you see a client and you tell him that he's down 15% but he should be happy because he's beat the index by 500 basis points. If people's overarching goal in investing is to beat the S&P500 or whatever then no one would buy any bonds.

 

It's also true that investors obsessed with beating indexes will probably fail to do so. Just go out there, do your best, do good work and stop worrying about the index so much.

 

Unless you have no opportunity cost to your time or derive independent enjoyment from the act of investment research, it makes sense to care about whether you are likely to beat an index.  If you can't, why bother to do "good work" on investments?  Wouldn't it be better to do no work, put your money in an index fund, and spend you're time on something else?

[Jurgis]

Unless you have no opportunity cost to your time or derive independent enjoyment from the act of investment research, it makes sense to care about whether you are likely to beat an index.  If you can't, why bother to do "good work" on investments?  Wouldn't it be better to do no work, put your money in an index fund, and spend you're time on something else?

 

What KJP said.

 

Edit: and even if you do something like 60/40 or 70/30, you still can argue that 60-or-70 should be in-index or beating-index.

[SharperDingaan]

Its only the competitive types that care about 'beating an index'.

It is just a number that substitutes for rank; I was 200bp > the index, my neighbor was only 150bp > the index, therefore I'm #1 & my neighbor is #2. The fact that both are well above the index is irrelevant.

 

Most 'owners of that money' don't really care either.

If I have 1M and made 100K versus the 110K some index says I should have made, so what. At 100K/yr (more than most can spend in a year) - how the money was made, starts to become more important than how much. A great many also recognize that 1 year, is an arbitrary measurement period.

 

And anyone with kids will tell you that most returns aren't monetary ...

You are not even measuring your total return, & then you are trying to compare to something that isn't relevant.

And wondering why you're disappointed?

 

SD

 

[CorpRaider]

Because Stevie Cohen, et. al. are out there and their activity is reflected in the price signals?

 

https://www.amazon.com/Black-Edge-Inside-Information-Wanted/dp/0812995805

[scorpioncapital]

It's hard to beat the index because human nature is always seeking a 'gimmick'. But it is also an ignorance issue. It has never been true that cigar butts or low quality that is cheap is likely to be a good long-term investment. So those funds, advisors, individuals seeking these gimmicks, these lures of value are often wasting time to compound and thus even the average beats them. Also human nature is to be jealous of your neighbor so even if they buy a good company , maybe like Google, they tend to buy it too late or after a large upswing, and thus their returns again sink below average. I think it's just human nature and very few are rational enough to keep it in check.

 

 

[alwaysinvert]

Even the very concept of beating the market is completely unknown and/or meaningless to most people. They can't visualize how good a return like 10% p/a is, let alone how good it would be to beat the index with 20 bp. Numbers - not to talk of exponentiality - are completely non-intuitive to human beings. It's expert's curse which makes us forget that.

 

Have you talked to friends who recently entered the stock market, perhaps again? Do you think they have any concept whatsoever of reasonable annual returns in the long term when they consider buying Snapchat? Of course not. The only difference between that and playing slots is that even otherwise reasonable people can fool themselves into thinking they have an edge.