The security of Internet-of-Things (IoT) is essential for its widespread adoption. The recent advancement in Artificial Intelligence (AI) brings both challenges and opportunities to IoT security. On the one hand, AI enables better security designs. On the other hand, AI-based advanced attacks are more threatening than traditional ones. This dissertation aims to study the dual effects of AI on IoT security, specifically IoT device security and IoT communication security. Particularly, this dissertation investigates three important topics: 1) security of acoustic mobile authentication, 2) Deep Learning (DL)-guided jamming attacks on cross-technology IoT networks, and 3) DL-powered scalable group-key establishment for large IoT networks. Chapter 2 presents a thorough study on the security of acoustic mobile authentication. In particular, this chapter proposes two mobile authentication schemes identifying the user's mobile device with its linear and nonlinear acoustic fingerprints, respectively. Both schemes adopt the Data Mining (DM) techniques to improve their identification accuracy. This chapter identifies a novel fingerprint-emulation attack and proposes the dynamic challenge and response method as an effective defense. A comprehensive comparison between two schemes in terms of security, usability, and deployment is presented at the end of this chapter, which suggests their respective suitable application scenarios. Chapter 3 identifies a novel DL-guided predictive jamming attack named DeepJam. DeepJam targets at cross-technology IoT networks and explores Deep Reinforcement Learning (DRL) to predict the victim's transmissions that are not subject to the Cross-Technology Interference (CTI). This chapter also proposes two effective countermeasures against DeepJam for resource capable and resource constrained IoT networks, respectively. Chapter 4 proposes a drone-aided DL-powered scalable group-key generation scheme, named DroneKey, for large-scale IoT networks. DroneKey is a physical-layer key generation scheme. In particular, DroneKey actively induces correlated changes to the wireless signals received by a group of devices and explores DL techniques to extract a common key from them. DroneKey significantly outperforms existing solutions in terms of the scalability and key-generation rate.