Developmental Test, Evaluation and Assessment

About the Office of Developmental Test, Evaluation and Assessment

Carnegie Mellon University Software Engineering Institute

About CMU SEI

The Software Engineering Institute (SEI) at Carnegie Mellon University (CMU) is a Federally Funded Research and Development Center (FFRDC)—a nonprofit, public–private partnership that conducts research for the United States government. Sponsored by the U.S. Department of Defense (DoD), the SEI conducts R&D in software engineering, systems engineering, cybersecurity, and many other areas of computing, working to introduce private-sector innovations into government.

Within the SEI’s cybersecurity-focused CERT Division, the Applied Systems Group helps organizations apply best practices, tools, and techniques at the rate needed to keep pace with evolving opportunities, threats, and risks. The goal of the team is to provide expertise in a wide range of skills throughout the software development lifecycle with a focus on holistic cybersecurity and engineering approaches to software intensive systems.

Supporting DTE&A

The Applied Systems Group operates and maintains a computing environment that supports software engineering research and practice. Our team keeps pace with rapid changes to the computing and software landscape; in some cases, the computing has outpaced the methods, and it is incumbent upon us to generate novel approaches to meet specific and unique sponsor needs.

Examples of current, ongoing projects include the following:

Assessing Mission Critical Software Systems

The SEI is collaborating with the office of the Director, Operational Test and Evaluation (DOT&E) to improve its capabilities to assess DoD mission critical software systems for known and unknown vulnerabilities. As part of this processes, we are developing new prototype tools that emulate certain aspects of known cyber threats, transitioning these tools to existing DOT&E testing frameworks, and documenting best practices.

Evaluating and Improving Simulator Realism

The SEI is researching Modeling & Simulation (M&S) corner cases where real-world behavior fails to be captured, such as when simulating new, complex flight behavior or new, complex torpedo behavior. We are collaborating with DOT&E to improve its ability to assess simulator realism in key mission scenarios or operational profiles for selected simulators. The SEI will develop both general recommendations and a methodology toward evaluating and improving simulator realism.

Developing and Documenting Technical and Acquisition for API Requirements

The SEI is currently participating in the API Working Group, to develop and document the technical and acquisition requirements for APIs utilized by the DoD. This involves developing methodologies for evaluating APIs; prototype tools for the methodology to accelerate adoption of API testing and evaluation; and methods for testing and evaluating the functionality, performance, sustainability, and cybersecurity of artificial intelligence systems.