It has been shown that the mean shift tracking algorithm can achieve excellent results in the task of pedestrian tracking. It empirically estimates the target position of the current frame by locating the maximum of a density function from the local neighbourhood of the target position of the previous frame. However, this method only considers its past trajectory without taking into account the influence of the pedestrian's environment. In practice, pedestrians always keep a safe distance away from obstacles when programming their paths. To address the issue of obstacle avoidance, this study proposes a novel extended social force model-based mean shift tracking algorithm, in which the pedestrian's environment is taken into full consideration. First, in order to show how the environment impacts pedestrian movements from the viewpoint of force, an extended social force model is presented by considering the interaction between target and obstacle. Furthermore, according to characteristics of pedestrian tracking, directional weights and speed weights are introduced to adjust the strength of the force concerning the difference of individual perspectives and relative velocities. Finally, the initial target position is predicted by Newton's laws of motion and then the mean shift method is integrated to track the target position. Experiment results showed that this algorithm achieved an encouraging performance when an obstacle occurred. An object that moves fast or changes its moving directions quickly can also be robustly tracked in real time by using the proposed algorithm.