This position is part of the National Institute of Standards and Technology (NIST) Engineering Laboratory's Professional Research Experience Program (PREP). This program recognizes that its research staff may wish to collaborate with researchers at academic institutions on specific projects of mutual interest, thus requires that such institutions must be the recipient of a PREP award. The PREP program requires staff from a wide range of backgrounds to work on scientific research in many areas. Employees in this position will perform technical work that underpins the scientific research of the collaboration.
Research Title:
Verification, Validation, and Uncertainty Quantification (VVUQ) for AI Models and Systems Trained on Synthetic Data in Manufacturing
Candidates must be eligible to obtain a Department of Commerce background check for facility access.
The work will entail:
As part of its AI for Manufacturing project, NIST develops the measurement science — methodologies, metrics, datasets, tools, and benchmark studies — needed to enable trustworthy AI and reliable AI-enabled digital twins across manufacturing system design, operation, and automation. A critical research thrust concerns the verification, validation, and uncertainty quantification (VVUQ) of AI models and systems trained on synthetic data. Synthetic data is rapidly becoming a foundational resource for training AI in manufacturing, robotics, and autonomous systems, yet no widely accepted, systematic VVUQ framework exists for establishing confidence in the resulting models. The Research Associate will develop a measurement-science framework for VVUQ of AI models trained on synthetic data, together with initial methodologies, metrics, benchmark datasets, and reference implementations. The work spans the full synthetic-data lifecycle — from real systems and digital-twin-based data generation, through model training and validation on real-world data, to continuous monitoring for model drift — and positions synthetic-data generation as a capability of AI-enabled digital twins. It draws on statistical uncertainty quantification, machine learning and generative-AI methods, digital-twin modeling, and benchmark design, applied to representative manufacturing use cases such as robotic grasping, defect detection, and predictive maintenance.
Key responsibilities will include but are not limited to:
- Developing a VVUQ framework for synthetic data that defines what must be verified, validated, and quantified across the synthetic-data lifecycle — including taxonomy, terminology, sources of uncertainty, evidence requirements, and a reference workflow spanning real systems, digital-twin-based data generators, synthetic datasets, model training, and validation on real-world data.
- Developing measurement metrics for synthetic-data quality — such as fidelity, representativeness, diversity, completeness, scenario coverage, annotation quality, and provenance — and for AI-model quality, such as generalization, calibration, robustness, uncertainty estimation, confidence, and domain transfer.
- Designing and building benchmark datasets and experimental evaluations for representative manufacturing problems such as robotic grasping, defect detection, and predictive maintenance, comparing models trained on real, synthetic, and mixed data across accuracy, calibration, uncertainty, robustness, and explainability.
- Developing reference implementations and evaluation pipelines, and translating research results into technical guidance — recommended metrics, validation protocols, uncertainty-reporting conventions, and acceptance criteria — to support standards development.
- Presenting results at internal meetings and occasional meetings with external stakeholders, and contributing to publications and relevant standards-development efforts, including ISO 23247-8 (VVUQ for digital twins in manufacturing), ISO/IEC JTC 1/SC 42, ASME and ASTM VVUQ activities, and IEEE AI-evaluation standards.
Qualifications
- A PhD degree (completed or expected) in Computer Science, Statistics, Systems Engineering, Mechanical Engineering, or a related field.
- Research experience in uncertainty quantification, verification and validation, trustworthy AI, or AI-model evaluation.
- Familiarity with synthetic-data generation methods — such as physics-based simulation, digital twins, and generative models — and with metrics for data quality and model evaluation.
- Experience with statistical and machine-learning methods for model calibration, robustness, and uncertainty estimation.
- Familiarity with manufacturing systems, robotics, or digital twins is desirable.
- Ability to develop prototypes of tools, metrics, and evaluation pipelines needed to conduct and evaluate the research.
- Strong oral and written communication skills and a record of research publication.
The university is an Equal Employment Opportunity employer that does not unlawfully discriminate in any of its programs or activities on the basis of race, color, religion, sex, national origin, age, disability, veteran status, sexual orientation, gender identity or expression, or on any other basis prohibited by applicable law.