Vision-Based Control Using Object Detection and Depth Estimation for Robotic Pick and Place Tasks in Construction Applications

The construction industry holds great promise for improvement through the use of robotic technologies in its workflow. Although this industry was an early adopter of such technologies, growth in construction robotics research and its integration into current construction projects is progressing slowly. Some significant factors that have contributed to the slow pace are high capital costs, low return on investments, and decreasing public infrastructure budgets. Consequently, there is a clear need to reduce the overall costs associated with new construction robotics technologies, which would enable greater dissemination. One solution is to use a swarm robotics approach, in which a large group of relatively low-cost agents are employed to produce a target collective behavior. Given the development of deep learning algorithms for object detection and depth estimation, and novel technologies such as edge computing and augmented reality, it is becoming feasible to engineer low-cost swarm robotic systems that use a vision-only control approach. Toward this end, this thesis develops a vision-based controller for a mobile manipulator robot that relies only on visual feedback from a monocular camera and does not require prior information about the environment. The controller uses deep-learning based methods for object detection and depth estimation to accomplish material retrieval and deposition tasks. The controller is demonstrated in the Gazebo robot simulator for scenarios in which a mobile manipulator must autonomously identify, pick up, transport, and deposit individual blocks with specific colors and shapes. The thesis concludes with a discussion of possible future extensions to the proposed solution, including its scalability to swarm robotic systems.