Evidence Generation for Neural Networks in Safety-Critical Embedded Systems

1. Introduction

Neural networks are being integrated into safety-critical embedded systems at a rate that outpaces the maturity of applicable certification guidance. Medical imaging systems, industrial inspection platforms, and autonomous robotic systems all rely on learned models to perform functions that directly affect patient safety, product quality, and physical integrity. Two regulatory frameworks impose concurrent obligations on these deployments: IEC 62304:2006/AMD 1:2015 [1], which governs medical device software lifecycle processes, and the EU AI Act (Regulation 2024/1689) [2], which classifies AI systems used in medical devices, critical infrastructure, and high-consequence industrial automation as high-risk.

IEC 62304 AMD 1 extended the original standard's scope to address Software of Unknown Provenance (SOUP) — a category that encompasses pre-trained neural network models imported from third-party repositories. The amendment requires manufacturers to document SOUP items, assess their functional suitability and known anomalies, and verify that their use does not introduce unacceptable risk. The EU AI Act's Article 10 mandates data governance practices for training, validation, and test datasets; Article 9 requires a documented risk management system; and Annex IV specifies technical documentation content that substantially overlaps with a 62304-compliant technical file [2].

2. Regulatory Overlap and Compliance Strategy

A systematic mapping of IEC 62304 AMD 1 and EU AI Act requirements reveals substantial overlap in four areas: (1) risk management — 62304 Clause 4.2 risk management activities align closely with EU AI Act Article 9; (2) software development planning — 62304 Clause 5.1 maps to EU AI Act Article 17 (quality management system); (3) configuration management and traceability — 62304 Clause 8 maps to EU AI Act Annex IV paragraph 2 (technical documentation); (4) problem resolution — 62304 Clause 9 maps to EU AI Act Article 72 (post-market monitoring and serious incident reporting). Where the two frameworks diverge, the EU AI Act is more prescriptive about data governance and the 62304 is more prescriptive about software architecture classification [1][2].

3. Dataset Governance

Dataset quality is the primary determinant of neural network safety in the absence of formal verification. EU AI Act Article 10 requires that training, validation, and test datasets be: (a) relevant and representative of the intended ODD; (b) free from known errors and complete with respect to the intended purpose; (c) assessed for biases that could affect safety-relevant outputs; (d) governed by documented data management practices. IEEE 2801-2022 [4] provides a recommended practice for medical AI datasets that is substantially compatible with Article 10 and can be adopted as a dual-compliance artefact.

For a medical imaging application (e.g., retinal OCT segmentation for diabetic macular oedema grading), the dataset governance record documents: dataset provenance (clinical site, scanner model, acquisition protocol), demographic distribution (age, sex, disease severity), ground truth labelling protocol (grader qualification, inter-rater agreement), known acquisition artefacts and their prevalence, and train/validation/test split rationale. The record is version-controlled alongside the model weights to maintain traceability throughout the software lifecycle.

4. Verification and Validation Strategy

IEC 62304 AMD 1 Clause 5.7 requires software testing appropriate to the safety class. For Class C software (where failure could result in death or serious injury), comprehensive coverage of software requirements is required. For neural networks, traditional structural coverage criteria (statement, branch, MC/DC) are not meaningful at the source code level — the 'code' is effectively the trained weights. Coverage must instead be defined at the model behaviour level, drawing on neural network coverage criteria from the research literature [6][7].

4.1 Neuron Activation Coverage

Pei et al.'s DeepXplore [6] introduced neuron coverage (NC) as the fraction of neurons activated above a threshold across a test suite. NC has been extended to k-multisection neuron coverage (KMNC), neuron boundary coverage (NBC), and strong neuron activation coverage (SNAC), each targeting different regions of the activation space. For safety-critical applications, a minimum KMNC of 85 % across the test dataset — combined with adversarial testing covering the ODD boundary conditions defined in the safety contract — provides a defensible V&V claim.

4.2 Scenario-Based and Adversarial Testing

Scenario-based testing validates network performance across the ODD by constructing a test matrix covering all combinations of ODD parameters at their boundary values. For a retinal grading model, this matrix includes scanner models not seen in training, image quality grades (excellent / acceptable / degraded), demographic subgroups, and co-morbidity combinations. Adversarial testing using FGSM (Fast Gradient Sign Method) and PGD (Projected Gradient Descent) perturbations at magnitudes calibrated to the sensor noise floor provides evidence that the model is not sensitive to physically plausible input variations.

5. Technical Documentation and Conformity Assessment

EU AI Act Annex IV specifies eight categories of technical documentation required for high-risk AI systems. For a medical device manufacturer also complying with IEC 62304, the existing technical file structure (design history file, software development plan, risk management file, V&V report) provides a foundation that covers approximately 70 % of Annex IV content. The remaining 30 % consists of AI-specific documentation: (a) a general description of the AI system including its intended purpose within the device (Annex IV §1), (b) a description of the elements of the AI system and of the process for its development including algorithm selection rationale (§2), (c) a detailed description of monitoring, functioning, and control of the AI system (§5), and (d) a description of the changes made to the system over its lifecycle (§8) [2].

Conformity assessment for high-risk AI systems in the EU requires involvement of a Notified Body. For medical devices, the Notified Body assessing the Medical Device Regulation (MDR) technical file is the natural choice for the AI Act conformity assessment — reducing duplication and enabling joint audit planning. Manufacturers should initiate pre-submission dialogue with their Notified Body at the architectural design phase, not at submission, to surface AI-specific evidence gaps early.

6. Conclusion

A dual-compliance strategy for IEC 62304 AMD 1 and EU AI Act high-risk requirements is achievable without doubling engineering effort, provided the compliance mapping is established at the project planning phase. The four highest-leverage actions are: (1) adopt IEEE 2801-2022 as the dataset governance standard — it satisfies both 62304 SOUP documentation and EU AI Act Article 10 simultaneously; (2) extend the 62304 V&V plan with neuron coverage and adversarial testing criteria; (3) structure the technical file to include Annex IV sections 1, 2, 5, and 8 as addenda; (4) engage the Notified Body early for joint audit planning [4][2].

As neural networks migrate from Class B to Class C medical device functions — and as the EU AI Act's full enforcement takes effect — the engineering burden of evidence generation will become a competitive differentiator. Organisations that build compliance evidence into their ML development pipeline from the outset — through automated dataset provenance tracking, continuous coverage measurement, and integrated risk management — will certify faster and maintain certification through model updates at a fraction of the cost incurred by organisations that treat compliance as a post-development activity.