This research evaluates a cyber test-bed, DEXTAR (Defense Exercises for Team Awareness Research), and examines the relationship between good and bad team performance in increasingly difficult scenarios. Twenty-one computer science graduate students (seven three-person teams), with experience in cybersecurity, participated in a team-based cyber defense exercise in the context of DEXTAR, a high fidelity cybersecurity testbed. Performance measures were analyzed in addition to team process, team behavior, and workload to examine the relationship between good and bad teams. Lessons learned are reported that will inform the next generation of DEXTAR.

Intelligence analysts’ work has become progressively complex due to increasing security threats and data availability. In order to study “big” data exploration within the intelligence domain the intelligence analyst task was abstracted and replicated in a laboratory (controlled environment). Participants used a computer interface and movie database to determine the opening weekend gross movie earnings of three pre-selected movies. Data consisted of Twitter tweets and predictive models. These data were displayed in various formats such as graphs, charts, and text. Participants used these data to make their predictions. It was expected that teams (a team is a group with members who have different specialties and who work interdependently) would outperform individuals and groups. That is, teams would be significantly better at predicting “Opening Weekend Gross” than individuals or groups. Results indicated that teams outperformed individuals and groups in the first prediction, under performed in the second prediction, and performed better than individuals in the third prediction (but not better than groups). Insights and future directions are discussed.

Humans and robots need to work together as a team to accomplish certain shared goals due to the limitations of current robot capabilities. Human assistance is required to accomplish the tasks as human capabilities are often better suited for certain tasks and they complement robot capabilities in many situations. Given the necessity of human-robot teams, it has been long assumed that for the robotic agent to be an effective team member, it must be equipped with automated planning technologies that helps in achieving the goals that have been delegated to it by their human teammates as well as in deducing its own goal to proactively support its human counterpart by inferring their goals. However there has not been any systematic evaluation on the accuracy of this claim.

In my thesis, I perform human factors analysis on effectiveness of such automated planning technologies for remote human-robot teaming. In the first part of my study, I perform an investigation on effectiveness of automated planning in remote human-robot teaming scenarios. In the second part of my study, I perform an investigation on effectiveness of a proactive robot assistant in remote human-robot teaming scenarios.

Both investigations are conducted in a simulated urban search and rescue (USAR) scenario where the human-robot teams are deployed during early phases of an emergency response to explore all areas of the disaster scene. I evaluate through both the studies, how effective is automated planning technology in helping the human-robot teams move closer to human-human teams. I utilize both objective measures (like accuracy and time spent on primary and secondary tasks, Robot Attention Demand, etc.) and a set of subjective Likert-scale questions (on situation awareness, immediacy etc.) to investigate the trade-offs between different types of remote human-robot teams. The results from both the studies seem to suggest that intelligent robots with automated planning capability and proactive support ability is welcomed in general.

Node-link diagrams are widely used to visualize the relational structure of real world datasets. As identical data can be visualized in infinite ways by simply changing the spatial arrangement of the nodes, one of the important research topics of the graph drawing community is to visualize the data in the way that can facilitate people's comprehension. The last three decades have witnessed the growth of algorithms for automatic visualization. However, despite the popularity of node-link diagrams and the enthusiasm in improving computational efficiency, little is known about how people read these graphs and what factors (layout, size, density, etc.) have impact on their effectiveness (the usability aspect of the graph, e.g., are they easy to understand?). This thesis is comprehensive research to investigate the factors that affect people's understanding of node-link diagrams using eye-tracking methods. Three experiments were conducted, including 1) a pilot study with 22 participants to explore the layout and size effect; 2) an eye tracking experiment with 43 participants to investigate the layout, size and density effect on people's graph comprehension using abstract node-link diagram and generic tasks; and 3) an eye tracking experiment with the same participants to investigate the same effects using a real visualization analytic application. Results showed that participants' spatial reasoning ability had significant impact on people's graph reading performance. Layout, size, and density were all found to be significant effects under different task circumstances. The applicability of the eye tracking methods on visualization evaluation has been confirmed by providing detailed evidence that demonstrates the cognitive process of participants' graph reading behavior.

The International Standards Organization (ISO) documentation utilizes Fitts’ law to determine the usability of traditional input devices like mouse and touchscreens for one- or two-dimensional operations. To test the hypothesis that Fitts’ Law can be applied to hand/air gesture based computing inputs, Fitts’ multi-directional target acquisition task is applied to three gesture based input devices that utilize different technologies and two baseline devices, mouse and touchscreen. Three target distances and three target sizes were tested six times in a randomized order with a randomized order of the five input technologies. A total of 81 participants’ data were collected for the within subjects design study. Participants were instructed to perform the task as quickly and accurately as possible according to traditional Fitts’ testing procedures. Movement time, error rate, and throughput for each input technology were calculated.

Additionally, no standards exist for equating user experience with Fitts’ measures such as movement time, throughput, and error count. To test the hypothesis that a user’s experience can be predicted using Fitts’ measures of movement time, throughput and error count, an ease of use rating using a 5-point scale for each input type was collected from each participant. The calculated Mean Opinion Scores (MOS) were regressed on Fitts’ measures of movement time, throughput, and error count to understand the extent to which they can predict a user’s subjective rating.

Although current urban search and rescue (USAR) robots are little more than remotely controlled cameras, the end goal is for them to work alongside humans as trusted teammates. Natural language communications and performance data are collected as a team of humans works to carry out a simulated search and rescue task in an uncertain virtual environment. Conditions are tested emulating a remotely controlled robot versus an intelligent one. Differences in performance, situation awareness, trust, workload, and communications are measured. The Intelligent robot condition resulted in higher levels of performance and operator situation awareness (SA).

In this research work, a novel control system strategy for the robust control of an unmanned ground vehicle is proposed. This strategy is motivated by efforts to mitigate the problem for scenarios in which the human operator is unable to properly communicate with the vehicle. This novel control system strategy consisted of three major components: I.) Two independent intelligent controllers, II.) An intelligent navigation system, and III.) An intelligent controller tuning unit. The inner workings of the first two components are based off the Brain Emotional Learning (BEL), which is a mathematical model of the Amygdala-Orbitofrontal, a region in mammalians brain known to be responsible for emotional learning. Simulation results demonstrated the implementation of the BEL model to be very robust, efficient, and adaptable to dynamical changes in its application as controller and as a sensor fusion filter for an unmanned ground vehicle. These results were obtained with significantly less computational cost when compared to traditional methods for control and sensor fusion. For the intelligent controller tuning unit, the implementation of a human emotion recognition system was investigated. This system was utilized for the classification of driving behavior. Results from experiments showed that the affective states of the driver are accurately captured. However, the driver's affective state is not a good indicator of the driver's driving behavior. As a result, an alternative method for classifying driving behavior from the driver's brain activity was explored. This method proved to be successful at classifying the driver's behavior. It obtained results comparable to the common approach through vehicle parameters. This alternative approach has the advantage of directly classifying driving behavior from the driver, which is of particular use in UGV domain because the operator's information is readily available. The classified driving mode was used tune the controllers' performance to a desired mode of operation. Such qualities are required for a contingency control system that would allow the vehicle to operate with no operator inputs.

Cyber threats are growing in number and sophistication making it important to continually study and improve all dimensions of cyber defense. Human teamwork in cyber defense analysis has been overlooked even though it has been identified as an important predictor of cyber defense performance. Also, to detect advanced forms of threats effective information sharing and collaboration between the cyber defense analysts becomes imperative. Therefore, through this dissertation work, I took a cognitive engineering approach to investigate and improve cyber defense teamwork. The approach involved investigating a plausible team-level bias called the information pooling bias in cyber defense analyst teams conducting the detection task that is part of forensics analysis through human-in-the-loop experimentation. The approach also involved developing agent-based models based on the experimental results to explore the cognitive underpinnings of this bias in human analysts. A prototype collaborative visualization tool was developed by considering the plausible cognitive limitations contributing to the bias to investigate whether a cognitive engineering-driven visualization tool can help mitigate the bias in comparison to off-the-shelf tools. It was found that participant teams conducting the collaborative detection tasks as part of forensics analysis, experience the information pooling bias affecting their performance. Results indicate that cognitive friendly visualizations can help mitigate the effect of this bias in cyber defense analysts. Agent-based modeling produced insights on internal cognitive processes that might be contributing to this bias which could be leveraged in building future visualizations. This work has multiple implications including the development of new knowledge about the science of cyber defense teamwork, a demonstration of the advantage of developing tools using a cognitive engineering approach, a demonstration of the advantage of using a hybrid cognitive engineering methodology to study teams in general and finally, a demonstration of the effect of effective teamwork on cyber defense performance.

When discussing human factors and performance, researchers recognize stress as a factor, but overlook mood as contributing factor. To explore the relationship between mood, stress and cognitive performance, a field study was conducted involving fire fighters engaged in a fire response simulation. Firefighter participants completed a stress questionnaire, an emotional state questionnaire, and a cognitive task. Stress and cognitive task performance scores were examined before and after the firefighting simulation for individual cognitive performance depreciation caused by stress or mood. They study revealed that existing stress was a reliable predictor of the pre-simulation cognitive task score, that, as mood becomes more positive, perceived stress scores decrease, and that negative mood and pre-simulation stress are also positively and significantly correlated.

Preoperative team briefings have been suggested to be important for improving team performance in the operating room. Many high risk environments have accepted team briefings; however healthcare has been slower to follow. While applying briefings in the operating room has shown positive benefits including improved communication and perceptions of teamwork, most research has only focused on feasibility of implementation and not on understanding how the quality of briefings can impact subsequent surgical procedures. Thus, there are no formal protocols or methodologies that have been developed.

The goal of this study was to relate specific characteristics of team briefings back to objective measures of team performance. The study employed cognitive interviews, prospective observations, and principle component regression to characterize and model the relationship between team briefing characteristics and non-routine events (NREs) in gynecological surgery. Interviews were conducted with 13 team members representing each role on the surgical team and data were collected for 24 pre-operative team briefings and 45 subsequent surgical cases. The findings revealed that variations within the team briefing are associated with differences in team-related outcomes, namely NREs, during the subsequent surgical procedures. Synthesis of the data highlighted three important trends which include the need to promote team communication during the briefing, the importance of attendance by all surgical team members, and the value of holding a briefing prior to each surgical procedure. These findings have implications for development of formal briefing protocols.

Pre-operative team briefings are beneficial for team performance in the operating room. Future research will be needed to continue understanding this relationship between how briefings are conducted and team performance to establish more consistent approaches and as well as for the continuing assessment of team briefings and other similar team-related events in the operating room.