Combining the stochastic and $\delta N$ formalisms, we derive non perturbative analytical expressions for all correlation functions of scalar perturbations in single-field, slow-roll inflation. The standard, classical formulas are recovered as saddle-point limits of the full results. This yields a classicality criterion that shows that stochastic effects are small only if the potential is sub-Planckian and not too flat. The saddle-point approximation also provides an expansion scheme for calculating stochastic corrections to observable quantities perturbatively in this regime. In the opposite regime, we show that a strong suppression in the power spectrum is generically obtained, and comment on the physical implications of this effect.