We explore the dependence of galaxy stellar population properties that are derived from broad-band spectral energy distribution (SED) fitting - such as age, stellar mass, dust reddening, etc. - on a variety of parameters, such as star formation histories, age grid, metallicity, initial mass function (IMF), dust reddening and reddening law, filter setup and wavelength coverage. Mock galaxies are used as test particles. We confirm our earlier results based on real z = 2 galaxies, that usually adopted τ-models lead to overestimate the star formation rate and to underestimate the stellar mass. Here, we show that - for star-forming galaxies - galaxy ages, masses and reddening, can be well determined simultaneously only when the correct star formation history is identified. This is the case for inverted-τ models at high-z, for which we find that the mass recovery (at fixed IMF) is as good as ~ 0.04 dex. However, since the right star formation history is usually unknown, we quantify the offsets generated by adopting standard fitting setups. Stellar masses are generally underestimated, which results from underestimating ages. For mixed fitting setups with a variety of star formation histories the median mass recovery at z � 2 − 3 is as decent as ~ 0.1 dex (at fixed IMF), albeit with large scatter. The situation worsens towards lower redshifts, because of the variety of possible star formation histories and ages. At z ~ 0.5 the stellar mass can be underestimated by as much as � 0.6 dex (at fixed IMF). A practical trick to improve upon this figure is to exclude reddening from the fitting parameters, as this helps to avoid unrealistically young and dusty solutions. Stellar masses are underestimated by a smaller amount (~ 0.3 dex at z ~ 0.5). Reddening and the star formation rate should then be determined via a separate fitting. As expected, the recovery of properties is better for passive galaxies, for which e.g. the mass can be fully recovered (within ~ 0.01 dex at fixed IMF) when using a fitting setup including metallicity effects. In both cases of star-forming as well as passive galaxies, the recovery of physical parameters is dependent on the spectral range involved in the fitting. We find that a coverage from the rest-frame UV to the rest-frame near-IR appears to be optimal. We also quantify the effect of narrowing the wavelength coverage or adding and removing filter bands, which can be useful for planning observational surveys. Finally, we provide scaling relations that allow the transformation of stellar masses obtained using different template fitting setups and stellar population models.