Number of samples needed in Monte Carlo simulation: how good is this approximation?

In Risk Theory Beard, Pentikanen and Pesonen (1969) mention a method of assessing number of samples needed for Monte Carlo simulation as $$ \sigma = \sqrt> \leq \frac \sqrt< \frac> $$ where $F(x) = p$, i.e. it is a probability of observing some value $x$ and $s$ is a number of samples. This shows us that with 99% confidence value we can expect that values observed in simulation study will lie $\pm 2.576 \sigma$ from $p$'s. This is similar to simulation standard error estimation based on observed variance mentioned by Aksakal. The authors seem to suggest that the formula can be used before the simulation to assess number of samples needed ($s$) to obtain simulated results with some precision of interest. How good is this approximation?

$\begingroup$ Well, it depends on the (unknown) value of $p$. If $p$ is close to zero or one then the approximation is poor, but when $p = 1/2$ it's exact. $\endgroup$

$\begingroup$ But no matter what and how you simulate $p$'s. This is quite a rough approximation. (Unless I misunderstood the brief description of the formula that was provided.) $\endgroup$

$\begingroup$ $p$ is not simulated, it's a true but known parameter. You would (presumably, the OP doesn't say) be simulating Bernoulli trials where the event either does or does not happen. The proportion of times it does is an estimate of $p$, and this is a way for determining a Monte Carlo sample size to get a certain margin of error without knowing $p$. $\endgroup$

$\begingroup$ @dsaxton it seems that the original description provided by the authors was misleading. If you could provide a little bit more detailed answer rather than a comment I would be inclined to accept it. $\endgroup$

2 Answers 2

The approximation could be poor when $p$ is close to zero or one, but when $p = 1/2$ it holds exactly.

The idea here is that we want to estimate the probability of an event by using a sample proportion across many Monte Carlo trials, and we want to know how accurate of an estimate that proportion is of the true probability. The standard deviation $\sigma$ of a sample proportion is as the authors note $\sqrt

$ (where $s$ is the number of Monte Carlo simulations), but the problem is we don't know $p$. However, we can maximize $\sigma$ with respect to $p$ and get a conservative "estimate" of this standard error that will always hold no matter what $p$ happens to be. This may end up causing us to run more simulations than we need to, but this won't matter as long as the iterations themselves are computationally cheap.