Bayesian inference is used to extract unknown parameters from gravitational wave signals. Detector noise is typically modelled as stationary, although data from the LIGO and Virgo detectors is not stationary. We demonstrate that the posterior of estimated waveform parameters is no longer valid under the assumption of stationarity. We show that while the posterior is unbiased, the errors will be under- or overestimated compared to the true posterior. A formalism was developed to measure the effect of the mismodelling, and found the effect of any form of non-stationarity has an effect on the results, but are not significant in certain circumstances. We demonstrate the effect of short-duration Gaussian noise bursts and persistent oscillatory modulation of the noise on binary-black-hole-like signals. In the case of short signals, non-stationarity in the data does not have a large effect on the parameter estimation, but the errors from non-stationary data containing signals lasting tens of seconds or longer will be several times worse than if the noise was stationary. Accounting for this limiting factor in parameter sensitivity could be very important for achieving accurate astronomical results, including an estimation of the Hubble parameter. This methodology for handling the non-stationarity will also be invaluable for analysis of waveforms that last minutes or longer, such as those we expect to see with the Einstein Telescope.