The rapid growth of big data applications over wireless network such as wireless sensor networks and unman aerial vehicle (UAV) lead to higher requirements for network capacity. One of the advanced solutions is adopting multi-beam smart antenna (MBDA) technology, it enables current transmission and reception through dierent directions. However, it also introduced some new challenges for data and transmission security. In this dissertation, we would like to study the MBSA network from security perspective. Specifically, we investigated the remote data integrity, transmission secure and authentication problems in MBSA network. In Chapter 2, we will design a remote data integrity check (RDIC) scheme for graph structural data collected though MBDA network. These scheme enables verification for both data content and structural integrity and supports verifiable data update. In Chapter 3, we work on a Deep Q-Network (DQN) based defending mechanism for transmission secure in MBDA network. This defender utilizes parameters from dierent network layer to represent the system state and determine the optimal action for monitor nodes to defend the network from intentional quality of service (QoS) and node compromise. In Chapter 4, we propose a hardware testbed with USRP RIO and MBSA for directional neighbor discovery with authentication. We tested directional neighbor discovery, ID based public key authentication and directional neighbor switch based on channel quality and user priority. Also, we have a brief introduction for the background and outline of my dissertation in Chapter 1, and in Chapter 5 we summarized all works during my PhD and some possible future works I may continue.