variance in monte carlo results

Every time I run the monte carlo simulation I get different results - I assume this is expected since the random numbers used to generate the results are different on each run. However I find the differences are significant enough that I cant rely on the results for future retirement planning. After 20 years one run generates a 50th percentile living standard of 114K per year, a second rerun of the same inputs generates a 50th percentile living standard of 122K per year. I can imagine that this is not a bug, but is just the way monte carlo simulations work. If so could you add a feature to run the simulation more times (I understand it currently executes the simulation 100 times). I assume with more executions the variance will be less.

Comments

dan royer's picture

I see a little variation in my results each time I run it, so, as you say, that seems by design. The variation I see, however, is very small. That said, my asset allocation is very stable with 70% intermediate bonds and 30% S&P.

Do you have some allocation that would lead you to suspect a lot of variance? Do you use some user-defined assets that have very short (ten years) return history?

It doesn't sound like a "failure to converge" but you could try the "precision" mode and see if that provides more reliable results.

There is (or has been) some work done on the MC that will involve a little different methodology and produce results that might be more intuitive to read etc. I don't know quite when those will appear in a release, perhaps even the next release.

Finally, I'd also suggest that you turn off MC mode and run, say three, reports where one is "bad luck" based on your intuition about your asset allocation--say 2% or 3%. Another at a more expected return rate, and a third at a very happy return rate. I'd adjust each of the three asset pools (reg assets, one partner retirement, two partner retirement). Remember, you'd be setting these based on the assumed average over many years left in life, not on your expectations for next year or even the next decade.

I'm not saying this is a substitute for MC but I think it's smart to look at models from different perspectives just to get a sense of how much play there is in the model.

Dan

I'm running in high res mode. I have 2 portfolios, one with just TIPS and one with just the S&P. I set up my retirement plan to have the TIPS and my wife's to have the S&P. The TIPS portfolio has 100% invested in a user built asset class. The S&P portfolio has 100% invested in the built-in class called large cap stocks. The returns on my retirement plan (TIPS) look to be what I would expect with a variance of less than $10 between the 95th percent and the 5th percent and the variance between runs is even lower. But my wife's plan (with the S&P assets) varies a lot between runs, even for the 50th percentile numbers. First question - does this amount of variance seem reasonable or is there a bug. I know its really hard to debug statistical software, because all the answers sort of look right. Second question - if there are no bugs in the monte carlo calculations, then it looks like more executions are needed to reduce the variance between runs. I'm trying to figure out how much of my retirement portfolio I should put into TIPS vs the S&P. To make sure that I meet my target living standard assuming the the worst case (the 5th percentile) I need numbers that a little more stable. I dont think I should have to run the monte carlo simulation 10 times to get a good estimate of what the 5th percentile number actually is. Attached are the results for two runs.

dan royer's picture

If you are using TIPS and S&P, why then use a user-built asset class? Why not used the canned asset classes there in the program?

We are looking at the retirement asset sheets here and not the distribution on living standard so it's hard to know how much difference this makes in living standard. But that amount of variance from run to run doesn't seem that much to me but I'm not sure what I'd expect really.

I think I'd call either one of those runs "a good estimate" but I'm not in your shoes on that. Given that it's just a guess about the uncertain future based on the past, it seems that much more precision than that could itself be misleading, but I see your point.

I'm jut not expert enough on probability statistics to know what to expect reasonably.

I can ask Kotlikoff to take a look--not sure what he'd say.

1st, to clarify a misunderstanding. I created a TIPS asset class because I am planning on holding the TIPS to maturity. Therefore I know exactly what they will pay, the investment will be zero variance and effectively no risk. The existing canned TIPS class is designed to handle the case where you may sell the TIPS before maturity so the value of the bond will vary over time. However the S&P portfolio I used is based on the canned "large cap stocks" asset class that is built-in to Esplanner.

2nd - all the problems I am reporting are generated by the S&P asset class, which 100% of my wife's retirement account is invested in - There appear to be no problem with the TIPS projections. I am just including information about it to help explain the purpose of the runs - Basically I'm trying to figure out how to balance my assets between TIPS and the stock market (S&P index fund).

Attached are the living standard trajectories from two different runs - one from a run that I previously sent and one from an entirely new run. If my goal is to have a living standard of 62K in every year, one runs looks fine. However the other run comes up short, meaning that to ensure a living standard of 62K (in the worst case ... the 5th percentile case) then I would need to re-balance my portfolio, moving money from riskier stocks to more predictable TIPS.

I understand that the numbers here are not actual guaranteed returns, just statistical estimates based on past experience and its never a good idea to act like these estimate are too accurate, however I do need to decide how much to invest in stocks (the S&P) and how much in TIPS. The 62K living standard number I am using already has a built-in cushion. I dont want to add another cushion because to deal with the fact that the monte carlo results are not accurate and change with every run. It seems like the only thing for me to do is to run the monte carlo simulation multiple times to quantify the variance and get a more accurate result, but it seems like Esplanner should do that for me. I understand that having Esplanner execute the monte carlo simulation 1000 times instead of 100 times would take a while, but given that computers are so much faster today, this seems like a reasonable option for the program to provide.

Hi Sam,

We are doing 500 runs in the Monte Carlo you are using. We are going to introduce a new version in a couple of weeks. The current version is correct, but subject to interpolation bias in the case of households that are borrowing constrained. I don't know if that's your case. There is a lot of variance in returns. I think what we will do in the new release is a) let users chose which version of Monte Carlo they want to use and b) choose their number of draws or return paths to be used in the old version. I would expect the difference across runs between percentile values at a given age to fall with the number of return paths.

The new Monte Carlo won't suffer from any interpolation bias. Indeed, it doesn't use any interpolation. All results will be precisely calculated because we'll be rerunning ESPlanner with out Monte Carlo to produce the Monte Carlo output. ESPlanner without Monte Carlo is accurate to many decimal places. The new method will be to run ESPlanner for 5 sets of paths of returns -- very low, low, medium, high, and very high. Along any path, we rerun ESPlanner each year based on that year's return. This may not be clear, but I'll be sending out an email when we release the new Monte Carlo that's more detailed.
best, Larry