The Siamese-based trackers have received much attention due to their great performance in the field of target tracking. However, it ignores the relationships and interdependencies between different features, impeding the robustness under various conditions. In addition, most Siamese-based trackers suffer from multiple special challenges, such as Fast Motion, Occlusion in UAV tracking. In this paper, we propose an anchor-free based object tracking algorithm with multi-scale aggregation Siamese Network. The proposed method consists of three parts: the feature extraction network, Encoder and Decoder. A multi-scale receptive field structure is designed in the encoder to deal with the problem of multi-scale change. The design of adaptive anchor in the decoder effectively reduces the relevant hyper-parameters. Experiments on three challenging UAV tracking benchmarks have demonstrated the robustness and effectiveness of the proposed method.