In game development, interaction frameworks provide generic functionality for users to engage with virtual worlds and are required to enable this on nonstandard hardware such as extended reality platforms. Currently, no publicly accessible frameworks exist that implement interactive world objects in XR settings, prompting the question: if one were to be made, how and why would it be usable? This thesis explores the properties that make an XR-enabled object interaction framework usable by game developers and game designers. This thesis introduces the basic form of such a framework and the design of a set of user studies centered around this framework’s utilization in a game development workflow. User feedback is gathered for the study’s results, and is evaluated around user perception of framework expressiveness, extensibility, and ease of use. The results of the study found that users primarily judged usability through comparisons to real-world equivalents, utilization of conventional systems, object interactivity, clarity of framework components, usability of framework toolkits and these are discussed in relation to existing research on virtual object interaction.

Recent advancements in computer vision models have largely been driven by supervised training on labeled data. However, the process of labeling datasets remains both costly and time-intensive. This dissertation delves into enhancing the performance of deep neural networks when faced with limited or no labeling information. I address this challenge through four primary methodologies: domain adaptation, self-supervision, input regularization, and label regularization. In situations where labeled data is unavailable but a similar dataset exists, domain adaptation emerges as a valuable strategy for transferring knowledge from the labeled dataset to the target dataset. This dissertation introduces three innovative domain adaptation methods that operate at pixel, feature, and output levels.Another approach to tackle the absence of labels involves a novel self-supervision technique tailored to train Vision Transformers in extracting rich features. The third and fourth approaches focus on scenarios where only a limited amount of labeled data is available. In such cases, I present novel regularization techniques designed to mitigate overfitting by modifying the input data and the target labels, respectively.

3D perception poses a significant challenge in Intelligent Transportation Systems (ITS) due to occlusion and limited field of view. The necessity for real-time processing and alignment with existing traffic infrastructure compounds these limitations. To counter these issues, this work introduces a novel multi-camera Bird-Eye View (BEV) occupancy detection framework. This approach leverages multi-camera setups to overcome occlusion and field-of-view limitations while employing BEV occupancy to simplify the 3D perception task, ensuring critical information is retained. A noble dataset for BEV Occupancy detection, encompassing diverse scenes and varying camera configurations, was created using the CARLA simulator. Subsequent extensive evaluation of various Multiview occupancy detection models showcased the critical roles of scene diversity and occupancy grid resolution in enhancing model performance. A structured framework that complements the generated data is proposed for data collection in the real world. The trained model is validated against real-world conditions to ensure its practical application, demonstrating the influence of robust dataset design in refining ITS perception systems. This contributes to significant advancements in traffic management, safety, and operational efficiency.

In this work, the problem of multi-object tracking (MOT) is studied, particularly the challenges that arise from object occlusions. A solution based on a principled approximate dynamic programming approach called ADPTrack is presented. ADPTrack relies on existing MOT solutions and directly improves them. When matching tracks to objects at a particular frame, the proposed approach simulates executions of these existing solutions into future frames to obtain approximate track extensions, from which a comparison of past and future appearance feature information is leveraged to improve overall robustness to occlusion-based error. The proposed solution when applied to the renowned MOT17 dataset empirically demonstrates a 0.7% improvement in the association accuracy (IDF1 metric) over a state-of-the-art baseline that it builds upon while obtaining minor improvements with respect to all other metrics. Moreover, it is shown that this improvement is even more pronounced in scenarios where the camera maintains a fixed position. This implies that the proposed method is effective in addressing MOT issues pertaining to object occlusions.

This research project seeks to develop an innovative data visualization tool tailored for beginners to enhance their ability to interpret and present data effectively. Central to the approach is creating an intuitive, user-friendly interface that simplifies the data visualization process, making it accessible even to those with no prior background in the field. The tool will introduce users to standard visualization formats and expose them to various alternative chart types, fostering a deeper understanding and broader skill set in data representation. I plan to leverage innovative visualization techniques to ensure the tool is compelling and engaging. An essential aspect of my research will involve conducting comprehensive user studies and surveys to assess the tool's impact on enhancing data visualization competencies among the target audience. Through this, I aim to gather valuable insights into the tool's usability and effectiveness, enabling further refinements. The outcome of this project is a powerful and versatile tool that will be an invaluable asset for students, researchers, and professionals who regularly engage with data. By democratizing data visualization skills, I envisage empowering a broader audience to comprehend and creatively present complex data in a more meaningful and impactful manner.

Recent advances in Artificial Intelligence (AI) have brought AI closer to laypeople than ever before. This leads to a pervasive problem: how would a user ascertain whether an AI system will be safe, reliable, or useful in a given situation? This problem becomes particularly challenging when it is considered that most autonomous systems are not designed by their users; the internal software of these systems may be unavailable or difficult to understand; and the functionality of these systems may even change from initial specifications as a result of learning. To overcome these challenges, this dissertation proposes a paradigm for third-party autonomous assessment of black-box taskable AI systems. The four main desiderata of such assessment systems are: (i) interpretability: generating a description of the AI system's functionality in a language that the target user can understand; (ii) correctness: ensuring that the description of AI system's working is accurate; (iii) generalizability creating a solution approach that works well for different types of AI systems; and (iv) minimal requirements: creating an assessment system that does not place complex requirements on AI systems to support the third-party assessment, otherwise the manufacturers of AI system's might not support such an assessment. To satisfy these properties, this dissertation presents algorithms and requirements that would enable user-aligned autonomous assessment that helps the user understand the limits of a black-box AI system's safe operability. This dissertation proposes a personalized AI assessment module that discovers the high-level ``capabilities'' of an AI system with arbitrary internal planning algorithms/policies and learns an accurate symbolic description of these capabilities in terms of concepts that a user understands. Furthermore, the dissertation includes the associated theoretical results and the empirical evaluations. The results show that (i) a primitive query-response interface can enable the development of autonomous assessment modules that can derive a causally accurate user-interpretable model of the system's capabilities efficiently, and (ii) such descriptions are easier to understand and reason with for the users than the agent's primitive actions.

Animal pose estimation (APE) is utilized in preclinical research settings for various neurological disorders such as Parkinson's disease (PD), Huntington's disease (HD) and multiple sclerosis. The technique includes real-time scoring of impairment in the animals during testing or video recording. This is a time-consuming operation prone to errors due to visual fatigue. To overcome these shortcomings, APE automation by deep learning has been studied. The field of APE has gone through significant development backed by improvements in deep learning techniques. These developments have improved 2D and 3D pose estimation, 3D mesh reconstruction and behavior prediction capabilities. As a result, there are numerous sophisticated tools and datasets available today. Despite these developments, APE still lags behind human observer scoring with respect to accuracy and flexibility under complex scenarios. In this project, two critical challenges are being addressed within the context of neurological research focusing on PD. The first challenge is about the lack of comprehensive diverse datasets necessary for accurate training as well as for fine-tuning deep learning models. This is compounded by the inherent difficulty in working with uncooperative rodent subjects, whose unpredictable behaviors often impede reliable data collection. The second challenge focuses on reduction in variation of scores that result from being scored by different evaluators. This will also involve tackling bias and reducing human error for the purpose of reliable and accurate assessments. In order to address these issues, systematic data collection and deep learning in APE have been utilized to automate manual scoring procedures. This project will contribute to neurological research, particularly in understanding and treating disorders like PD. The goal is to improve methods used in assessing rodent behavior which could aid in developing effective therapeutics. The successful implementation of an automated scoring mechanism could set a new standard in neurological research, offering insights and methodologies that are more accurate and reliable.

Current methods for sequence prediction often fail to account for higher-ordercontinuity. This results in the prediction of sequences that might be continuous but not physically viable as I investigate higher-order smoothness in terms of velocity and acceleration. Hence, I propose a Yuksel Spline-based model that is not only capable of predicting curves that are guaranteed to be C^2 continuous but, also efficient to compute as well. Characteristic properties of the models are demonstrated over toy examples and sequence prediction tasks.

In today's data-driven world, privacy is a significant concern. It is crucial to preserve the privacy of sensitive information while visualizing data. This thesis aims to develop new techniques and software tools that support Vega-Lite visualizations while maintaining privacy. Vega-Lite is a visualization grammar based on Wilkinson's grammar of graphics. The project extends Vega-Lite to incorporate privacy algorithms such as k-anonymity, l-diversity, t-closeness, and differential privacy. This is done by using a unique multi-input loop module logic that generates combinations of attributes as a new anonymization method. Differential privacy is implemented by adding controlled noise (Laplace or Exponential) to the sensitive columns in the dataset. The user defines custom rules in the JSON schema, mentioning the privacy methods and the sensitive column. The schema is validated using Another JSON Validation library, and these rules help identify the anonymization techniques to be performed on the dataset before sending it back to the Vega-Lite visualization server. Multiple datasets satisfying the privacy requirements are generated, and their utility scores are provided so that the user can trade-off between privacy and utility on the datasets based on their requirements. The interface developed is user-friendly and intuitive and guides users in using it. It provides appropriate feedback on the privacy-preserving visualizations generated through various utility metrics. This application is helpful for technical or domain experts across multiple domains where privacy is a big concern, such as medical institutions, traffic and urban planning, financial institutions, educational records, and employer-employee relations. This project is novel as it provides a one-stop solution for privacy-preserving visualization. It works on open-source software, Vega-Lite, which several organizations and users use for business and educational purposes.

Grasping objects in a general household setting is a dexterous task, high compliance is needed to generate a grasp that leads to grasp closure. Standard 6 Degree of Freedom (DoF) manipulators with parallel grippers are naturally incapable of showing such dexterity. This renders many objects in household settings difficult to grasp, as the manipulator cannot access readily available antipodal (planar) grasps. In such scenarios, one must either use a high DoF end effector to learn this compliance or change the initial configuration of the object to find an antipodal grasp. A pipeline that uses the extrinsic forces present in the environment to make up for this lack of compliance is proposed. The proposed method: i) Takes the point cloud input from the environment, and creates a search space with all its available poses. This search space is used to identify the best graspable position for an object with a grasp score network ii) Learn how to approach an object, and generate an appropriate set of motor primitives that converts the current ungraspable pose to a graspable pose. iii) Run a naive grasp detection network to verify the proposed methods and subsequently grasp the initially ungraspable object. By integrating these components, objects that were initially ungraspable, with a standard grasp detection model DexNet, remain no longer ungraspable.