The security of Internet-of-Things (IoT) is essential for its widespread adoption. The recent advancement in Artificial Intelligence (AI) brings both challenges and opportunities to IoT security. On the one hand, AI enables better security designs. On the other hand, AI-based advanced attacks are more threatening than traditional ones. This dissertation aims to study the dual effects of AI on IoT security, specifically IoT device security and IoT communication security. Particularly, this dissertation investigates three important topics: 1) security of acoustic mobile authentication, 2) Deep Learning (DL)-guided jamming attacks on cross-technology IoT networks, and 3) DL-powered scalable group-key establishment for large IoT networks. Chapter 2 presents a thorough study on the security of acoustic mobile authentication. In particular, this chapter proposes two mobile authentication schemes identifying the user's mobile device with its linear and nonlinear acoustic fingerprints, respectively. Both schemes adopt the Data Mining (DM) techniques to improve their identification accuracy. This chapter identifies a novel fingerprint-emulation attack and proposes the dynamic challenge and response method as an effective defense. A comprehensive comparison between two schemes in terms of security, usability, and deployment is presented at the end of this chapter, which suggests their respective suitable application scenarios. Chapter 3 identifies a novel DL-guided predictive jamming attack named DeepJam. DeepJam targets at cross-technology IoT networks and explores Deep Reinforcement Learning (DRL) to predict the victim's transmissions that are not subject to the Cross-Technology Interference (CTI). This chapter also proposes two effective countermeasures against DeepJam for resource capable and resource constrained IoT networks, respectively. Chapter 4 proposes a drone-aided DL-powered scalable group-key generation scheme, named DroneKey, for large-scale IoT networks. DroneKey is a physical-layer key generation scheme. In particular, DroneKey actively induces correlated changes to the wireless signals received by a group of devices and explores DL techniques to extract a common key from them. DroneKey significantly outperforms existing solutions in terms of the scalability and key-generation rate.

Pushing the artificial intelligence frontier to resource-constrained edge nodes for edge intelligence is nontrivial. This dissertation provides a comprehensive study of optimization-based meta-learning algorithms to build a theoretic foundation of edge intelligence, with the focus on two topics: 1) model-based reinforcement learning (RL); 2) distributed edge learning. Under this common theme, this study is broadly organized into two parts. The first part studies meta-learning algorithms for model-based RL. First, the fundamental limit of model learning is explored for linear time-varying systems, using a two-step meta-learning algorithm with an episodic block model. A comprehensive non-asymptotic analysis of the sample complexity is provided, where a two-scale martingale small-ball approach is devised to address the challenges in sample correlation and small sample sizes. Next, policy learning of offline RL in general Markov decision processes is explored. To tackle the challenges therein, e.g., value overestimation and possibly poor quality of offline datasets, a model-based offline Meta-RL approach with regularized policy optimization is proposed, by learning a meta-model for task inference and a meta-policy for safe exploration of out-of-distribution state-actions. The second part investigates meta-learning algorithms for distributed edge learning. First, the general edge supervised learning is considered, where the edge node aims to quickly learn a good model with limited samples. A platform-aided collaborative learning framework is proposed to learn a model initialization via federated meta-learning across multiple nodes, which is transferred to target nodes for fine-tuning. Then, a channel gating module is introduced to select important channels of backbone networks for efficient local computation. A novel federated meta-learning approach is developed to learn meta-initializations for backbone networks and gating modules, from which a task-specific channel gated network is quickly adapted. Taking one step further, the continual edge learning is investigated in the context of online meta-learning, where each node has a sequence of online tasks. A multi-agent online meta-learning framework is developed to accelerate the task-average performance in a single node under limited communication among neighbors, through the lens of distributed online convex optimization. Building on distributed online gradient descent with gradient tracking, the optimal task-average regret is achieved at a faster rate.