Cameras have become commonplace with wide-ranging applications of phone photography, computer vision, and medical imaging. With a growing need to reduce size and costs while maintaining image quality, the need to look past traditional style of cameras is becoming more apparent. Several non-traditional cameras have shown to be promising options for size-constraint applications, and while they may offer several advantages, they also usually are limited by image quality degradation due to optical or a need to reconstruct a captured image. In this thesis, we take a look at three of these non-traditional cameras: a pinhole camera, a diffusion-mask lensless camera, and an under-display camera (UDC).

For each of these cases, I present a feasible image restoration pipeline to correct for their particular limitations. For the pinhole camera, I present an early pipeline to allow for practical pinhole photography by reducing noise levels caused by low-light imaging, enhancing exposure levels, and sharpening the blur caused by the pinhole. For lensless cameras, we explore a neural network architecture that performs joint image reconstruction and point spread function (PSF) estimation to robustly recover images captured with multiple PSFs from different cameras. Using adversarial learning, this approach achieves improved reconstruction results that do not require explicit knowledge of the PSF at test-time and shows an added improvement in the reconstruction model’s ability to generalize to variations in the camera’s PSF. This allows lensless cameras to be utilized in a wider range of applications that require multiple cameras without the need to explicitly train a separate model for each new camera. For UDCs, we utilize a multi-stage approach to correct for low light transmission, blur, and haze. This pipeline uses a PyNET deep neural network architecture to perform a majority of the restoration, while additionally using a traditional optimization approach which is then fused in a learned manner in the second stage to improve high-frequency features. I show results from this novel fusion approach that is on-par with the state of the art.

I present my work on a scalable and programmable I/O controller for region-based computing, which will be used in a rhythmic pixel-based camera pipeline. I provide a breakdown of the development and design of the I/O controller and how it fits in to rhythmic pixel regions, along with a studyon memory traffic of rhythmic pixel regions and how this translates to energy efficiency. This rhythmic pixel region-based camera pipeline has been jointly developed through Dr. Robert LiKamWa’s research lab. High spatiotemporal resolutions allow high precision for vision applications, such as for detecting features for augmented reality or face detection. High spatiotemporal resolution also comes with high memory throughput, leading to higher energy usage. This creates a tradeoff between high precision and energy efficiency, which becomes more important in mobile systems. In addition, not all pixels in a frame are necessary for the vision application, such as pixels that make up the background. Rhythmic pixel regions aim to reduce the tradeoff by creating a pipeline that allows an application developer to specify regions to capture at a non-uniform spatiotemporal resolution. This is accomplished by encoding the incoming image, and only sending the pixels within these specified regions. Later these encoded representations will be decoded to a standard frame representation usable by traditional vision applications. My contribution to this effort has been the design, testing and evaluation of the I/O controller.

Semantic image segmentation has been a key topic in applications involving image processing and computer vision. Owing to the success and continuous research in the field of deep learning, there have been plenty of deep learning-based segmentation architectures that have been designed for various tasks. In this thesis, deep-learning architectures for a specific application in material science; namely the segmentation process for the non-destructive study of the microstructure of Aluminum Alloy AA 7075 have been developed. This process requires the use of various imaging tools and methodologies to obtain the ground-truth information. The image dataset obtained using Transmission X-ray microscopy (TXM) consists of raw 2D image specimens captured from the projections at every beam scan. The segmented 2D ground-truth images are obtained by applying reconstruction and filtering algorithms before using a scientific visualization tool for segmentation. These images represent the corrosive behavior caused by the precipitates and inclusions particles on the Aluminum AA 7075 alloy. The study of the tools that work best for X-ray microscopy-based imaging is still in its early stages.

In this thesis, the underlying concepts behind Convolutional Neural Networks (CNNs) and state-of-the-art Semantic Segmentation architectures have been discussed in detail. The data generation and pre-processing process applied to the AA 7075 Data have also been described, along with the experimentation methodologies performed on the baseline and four other state-of-the-art Segmentation architectures that predict the segmented boundaries from the raw 2D images. A performance analysis based on various factors to decide the best techniques and tools to apply Semantic image segmentation for X-ray microscopy-based imaging was also conducted.

Commonly, image processing is handled on a CPU that is connected to the image sensor by a wire. In these far-sensor processing architectures, there is energy loss associated with sending data across an interconnect from the sensor to the CPU. In an effort to increase energy efficiency, near-sensor processing architectures have been developed, in which the sensor and processor are stacked directly on top of each other. This reduces energy loss associated with sending data off-sensor. However, processing near the image sensor causes the sensor to heat up. Reports of thermal noise in near-sensor processing architectures motivated us to study how temperature affects image quality on a commercial image sensor and how thermal noise affects computer vision task accuracy. We analyzed image noise across nine different temperatures and three sensor configurations to determine how image noise responds to an increase in temperature. Ultimately, our team used this information, along with transient analysis of a stacked image sensor’s thermal behavior, to advise thermal management strategies that leverage the benefits of near-sensor processing and prevent accuracy loss at problematic temperatures.

Though a single mode of energy transfer, optical radiation meaningfully interacts with its surrounding environment at over a wide range of physical length scales. For this reason, its reconstruction and measurement are of great importance in remote sensing, as these multi-scale interactions encode a great deal of information about distant objects, surfaces, and physical phenomena. For some remote sensing applications, obtaining a desired quantity of interest does not necessitate the explicit mapping of each point in object space to an image space with lenses or mirrors. Instead, only edge rays or physical boundaries of the sensing instrument are considered, while the spatial intensity distribution of optical energy received from a distant object informs its position, optical characteristics, or physical/chemical state.

Admittedly specialized, the principals and consequences of non-imaging optics are nevertheless applicable to heterogeneous semiconductor integration and automotive light detection and ranging (LiDAR), two important emerging technologies. Indeed, a review of relevant engineering literature finds two under-addressed remote sensing challenges. The semiconductor industry lacks an optical strain metrology with displacement resolution smaller than 100 nanometers capable of measuring strain fields between high-density interconnect lines. Meanwhile, little attention is paid to the per-meter sensing characteristics of scene-illuminating flash LiDAR in the context of automotive applications, despite the technology’s much lower cost. It is here that non-imaging optics offers intriguing instrument design and explanations of observed sensor performance at vastly different length scales.

In this thesis, an effective non-contact technique for mapping nanoscale mechanical strain fields and out-of-plane surface warping via laser diffraction is demonstrated, with application as a novel metrology for next-generation semiconductor packages. Additionally, object detection distance of low-cost automotive flash LiDAR, on the order of tens of meters, is understood though principals of optical energy transfer from the surface of a remote object to an extended multi-segment detector. Such information is of consequence when designing an automotive perception system to recognize various roadway objects in low-light scenarios.

In recent years, the widespread use of deep neural networks (DNNs) has facilitated great improvements in performance for computer vision tasks like image classification and object recognition. In most realistic computer vision applications, an input image undergoes some form of image distortion such as blur and additive noise during image acquisition or transmission. Deep networks trained on pristine images perform poorly when tested on such distortions. DNN predictions have also been shown to be vulnerable to carefully crafted adversarial perturbations. Specifically, so-called universal adversarial perturbations are image-agnostic perturbations that can be added to any image and can fool a target network into making erroneous predictions. This work proposes selective DNN feature regeneration to improve the robustness of existing DNNs to image distortions and universal adversarial perturbations.

In the context of common naturally occurring image distortions, a metric is proposed to identify the most susceptible DNN convolutional filters and rank them in order of the highest gain in classification accuracy upon correction. The proposed approach called DeepCorrect applies small stacks of convolutional layers with residual connections at the output of these ranked filters and trains them to correct the most distortion-affected filter activations, whilst leaving the rest of the pre-trained filter outputs in the network unchanged. Performance results show that applying DeepCorrect models for common vision tasks significantly improves the robustness of DNNs against distorted images and outperforms other alternative approaches.

In the context of universal adversarial perturbations, departing from existing defense strategies that work mostly in the image domain, a novel and effective defense which only operates in the DNN feature domain is presented. This approach identifies pre-trained convolutional features that are most vulnerable to adversarial perturbations and deploys trainable feature regeneration units which transform these DNN filter activations into resilient features that are robust to universal perturbations. Regenerating only the top 50% adversarially susceptible activations in at most 6 DNN layers and leaving all remaining DNN activations unchanged can outperform existing defense strategies across different network architectures and across various universal attacks.

According to the Center for Disease Control and Prevention report around 29,668 United States residents aged greater than 65 years had died as a result of a fall in 2016. Other injuries like wrist fractures, hip fractures, and head injuries occur as a result of a fall. Certain groups of people are more prone to experience falls than others, one of which being individuals with stroke. The two most common issues with individuals with strokes are ankle weakness and foot drop, both of which contribute to falls. To mitigate this issue, the most popular clinical remedy given to these users is thermoplastic Ankle Foot Orthosis. These AFO's help improving gait velocity, stride length, and cadence. However, studies have shown that a continuous restraint on the ankle harms the compensatory stepping response and forward propulsion. It has been shown in previous studies that compensatory stepping and forward propulsion are crucial for the user's ability to recover from postural perturbations. Hence, there is a need for active devices that can supply a plantarflexion during the push-off and dorsiflexion during the swing phase of gait. Although advancements in the orthotic research have shown major improvements in supporting the ankle joint for rehabilitation, there is a lack of available active devices that can help impaired users in daily activities. In this study, our primary focus is to build an unobtrusive, cost-effective, and easy to wear active device for gait rehabilitation and fall prevention in individuals who are at risk. The device will be using a double-acting cylinder that can be easily incorporated into the user's footwear using a novel custom-designed powered ankle brace. The device will use Inertial Measurement Units to measure kinematic parameters of the lower body and a custom control algorithm to actuate the device based on the measurements. The study can be used to advance the field of gait assistance, rehabilitation, and potentially fall prevention of individuals with lower-limb impairments through the use of Active Ankle Foot Orthosis.

The following document describes the hardware implementation and analysis of Temporal Interference Mitigation using High-Level Synthesis. As the problem of spectral congestion becomes more chronic and widespread, Electromagnetic radio frequency (RF) based systems are posing as viable solution to this problem. Among the existing RF methods Cooperation based systems have been a solution to a host of congestion problems. One of the most important elements of RF receiver is the spatially adaptive part of the receiver. Temporal Mitigation is vital technique employed at the receiver for signal recovery and future propagation along the radar chain.

The computationally intensive parts of temporal mitigation are identified and hardware accelerated. The hardware implementation is based on sequential approach with optimizations applied on the individual components for better performance.

An extensive analysis using a range of fixed point data types is performed to find the optimal data type necessary.

Finally a hybrid combination of data types for different components of temporal mitigation is proposed based on results from the above analysis.

Land-use change has arguably been the largest contributor to the emergence of novel zoonotic diseases within the past century. However, the relationship between patterns of land-use change and the resulting landscape configuration on disease spread is poorly understood as current cross-species disease transmission models have not adequately incorporated spatial features of habitats. Furthermore, mathematical-epidemiological studies have not considered the role that land-use change plays in disease transmission throughout an ecosystem.

This dissertation models how a landscape's configuration, examining the amount and shape of habitat overlap, contributes to cross-species disease transmission to determine the role that land-use change has on the spread of infectious diseases. To approach this, an epidemiological model of transmission between a domesticated and a wild species is constructed. Each species is homogeneously mixed in its respective habitat and heterogeneously mixed in the habitat overlap, where cross-species transmission occurs. Habitat overlap is modeled using landscape ecology metrics.

This general framework is then applied to brucellosis transmission between elk and cattle in the Greater Yellowstone Ecosystem. The application of the general framework allows for the exploration of how land-use change has contributed to brucellosis prevalence in these two species, and how land management can be utilized to control disease transmission. This model is then extended to include a third species, bison, in order to provide insight to the indirect consequences of disease transmission for a species that is situated on land that has not been converted. The results of this study can ultimately help stakeholders develop policy for controlling brucellosis transmission between livestock, elk, and bison, and in turn, could lead to less disease prevalence, reduce associated costs, and assist in population management.

This research contributes novelty by combining landscape ecology metrics with theoretical epidemiological models to understand how the shape, size, and distribution of habitat fragments on a landscape affect cross-species disease transmission. The general framework demonstrates how habitat edge in single patch impacts cross-species disease transmission. The application to brucellosis transmission in the Greater Yellowstone Ecosystem between elk, cattle, and bison is original research that enhances understanding of how land conversion is associated with enzootic disease spread.

Object detection is an interesting computer vision area that is concerned with the detection of object instances belonging to specific classes of interest as well as the localization of these instances in images and/or videos. Object detection serves as a vital module in many computer vision based applications. This work focuses on the development of object detection methods that exhibit increased robustness to varying illuminations and image quality. In this work, two methods for robust object detection are presented.

In the context of varying illumination, this work focuses on robust generic obstacle detection and collision warning in Advanced Driver Assistance Systems (ADAS) under varying illumination conditions. The highlight of the first method is the ability to detect all obstacles without prior knowledge and detect partially occluded obstacles including the obstacles that have not completely appeared in the frame (truncated obstacles). It is first shown that the angular distortion in the Inverse Perspective Mapping (IPM) domain belonging to obstacle edges varies as a function of their corresponding 2D location in the camera plane. This information is used to generate object proposals. A novel proposal assessment method based on fusing statistical properties from both the IPM image and the camera image to perform robust outlier elimination and false positive reduction is also proposed.

In the context of image quality, this work focuses on robust multiple-class object detection using deep neural networks for images with varying quality. The use of Generative Adversarial Networks (GANs) is proposed in a novel generative framework to generate features that provide robustness for object detection on reduced quality images. The proposed GAN-based Detection of Objects (GAN-DO) framework is not restricted to any particular architecture and can be generalized to several deep neural network (DNN) based architectures. The resulting deep neural network maintains the exact architecture as the selected baseline model without adding to the model parameter complexity or inference speed. Performance results provided using GAN-DO on object detection datasets establish an improved robustness to varying image quality and a higher object detection and classification accuracy compared to the existing approaches.