🔥🔥 Don’t Miss Out on Our End of Autumn Sale. If you have any questions, feel free to click the bottom right corner to contact our customer service..

Concat us[email protected]
Shopping Cart

No products in the cart.

🔍
BSI PD ISO/IEC TR 29119-11:2020

BSI PD ISO/IEC TR 29119-11:2020

$198.66

Software and systems engineering. Software testing – Guidelines on the testing of AI-based systems

Published By Publication Date Number of Pages
BSI 2020 60
PDF Format Searchable Printable Preview Now
Guaranteed Safe Checkout
Category:

If you have any questions, feel free to reach out to our online customer service team by clicking on the bottom right corner. We’re here to assist you 24/7.
Email:[email protected]

This document provides an introduction to AI-based systems. These systems are typically complex (e.g. deep neural nets), are sometimes based on big data, can be poorly specified and can be non-deterministic, which creates new challenges and opportunities for testing them.

This document explains those characteristics which are specific to AI-based systems and explains the corresponding difficulties of specifying the acceptance criteria for such systems.

This document presents the challenges of testing AI-based systems, the main challenge being the test oracle problem, whereby testers find it difficult to determine expected results for testing and therefore whether tests have passed or failed. It covers testing of these systems across the life cycle and gives guidelines on how AI-based systems in general can be tested using black-box approaches and introduces white-box testing specifically for neural networks. It describes options for the test environments and test scenarios used for testing AI-based systems.

In this document an AI-based system is a system that includes at least one AI component.

PDF Catalog

PDF Pages PDF Title
2 National foreword
7 Foreword
8 Introduction
9 1 Scope
2 Normative references
3 Terms, definitions and abbreviated terms
3.1 Terms and definitions
18 3.2 Abbreviated terms
19 4 Introduction to AI and testing
4.1 Overview of AI and testing
4.2 Artificial intelligence (AI)
4.2.1 Definition of ‘artificial intelligence’
4.2.2 AI use cases
20 4.2.3 AI usage and market
4.2.4 AI technologies
23 4.2.5 AI hardware
4.2.6 AI development frameworks
4.2.7 Narrow vs general AI
24 4.3 Testing of AI-based systems
4.3.1 The importance of testing for AI-based systems
4.3.2 Safety-related AI-based systems
4.3.3 Standardization and AI
26 5 AI system characteristics
5.1 AI-specific characteristics
5.1.1 General
27 5.1.2 Flexibility and adaptability
28 5.1.3 Autonomy
5.1.4 Evolution
5.1.5 Bias
29 5.1.6 Complexity
5.1.7 Transparency, interpretability and explainability
30 5.1.8 Non-determinism
5.2 Aligning AI-based systems with human values
5.3 Side-effects
31 5.4 Reward hacking
5.5 Specifying ethical requirements for AI-based systems
32 6 Introduction to the testing of AI-based systems
6.1 Challenges in testing AI-based systems
6.1.1 Introduction to challenges testing AI-based systems
6.1.2 System specifications
33 6.1.3 Test input data
6.1.4 Self-learning systems
6.1.5 Flexibility and adaptability
6.1.6 Autonomy
6.1.7 Evolution
34 6.1.8 Bias
6.1.9 Transparency, interpretability and explainability
6.1.10 Complexity
6.1.11 Probabilistic and non-deterministic systems
6.1.12 The test oracle problem for AI-based systems
35 6.2 Testing AI-based systems across the life cycle
6.2.1 General
6.2.2 Unit/component testing
6.2.3 Integration testing
36 6.2.4 System testing
6.2.5 System integration testing
6.2.6 Acceptance testing
6.2.7 Maintenance testing
7 Testing and QA of ML systems
7.1 Introduction to the testing and QA of ML systems
37 7.2 Review of ML workflow
7.3 Acceptance criteria
7.4 Framework, algorithm/model and hyperparameter selection
7.5 Training data quality
7.6 Test data quality
7.7 Model updates
7.8 Adversarial examples and testing
38 7.9 Benchmarks for machine learning
8 Black-box testing of AI-based systems
8.1 Combinatorial testing
39 8.2 Back-to-back testing
40 8.3 A/B testing
8.4 Metamorphic testing
41 8.5 Exploratory testing
9 White-box testing of neural networks
9.1 Structure of a neural network
43 9.2 Test coverage measures for neural networks
9.2.1 Introduction to test coverage levels
9.2.2 Neuron coverage
9.2.3 Threshold coverage
9.2.4 Sign change coverage
9.2.5 Value change coverage
9.2.6 Sign-sign coverage
44 9.2.7 Layer coverage
9.3 Test effectiveness of the white-box measures
9.4 White-box testing tools for neural networks
45 10 Test environments for AI-based systems
10.1 Test environments for AI-based systems
46 10.2 Test scenario derivation
10.3 Regulatory test scenarios and test environments
47 Annex A Machine learning
56 Bibliography

Related products

BSI PD ISO/IEC TR 29119-11:2020
$198.66