Crowd analysis is very important for human behavior analysis, safety science, computational simulation and computer vision applications. One of the most popular applications is video surveillance, which plays an important role in crowd behavior analysis including real-time crowd behavior detection and information retrieval. In the field of video surveillance, many kinds of methods have been proposed for analyzing crowds, such as machine learning, signal processing and a physical model based methods. As a kind of collective movements, crowd behavior contains many physical attributes, such as velocity, direction of motion, interaction force and energy. Therefore, a lot of methods and models derived from physical ideas have been applied in many frameworks of crowd behavior analysis. This survey reviews the development of physical methods of crowd analysis in detail. The physics inspired methods in crowd video analysis are summarized into three categories including fluid dynamics, interaction force and complex crowd motion systems. Furthermore, the existing public databases for crowd video analysis are collated in this paper. Finally, the future research directions of the open issues of crowd video surveillance are also discussed.