When Proven Risk Analyses Reach Their Limits
In classical systems development, there is a reassuring principle: determinism[4]. When writing software, the irrefutable “if-then” principle applies: Input A is precisely and predictably followed by output B.
Safety analyses, such as failure mode and effects analysis (FMEA), or risk analyses according to ISO 14971, have been optimized for decades to examine such clear, logical failure chains in mechanical and electronic systems.
However, this deterministic comfort zone is rapidly dissolving. AI introduces a new dynamic: probability[7]. AI does not operate based on fixed logic but rather on probabilities. It learns from data and may react in unexpected ways in practical applications.
This is precisely where legislation steps in. In May 2024, the European Parliament adopted the EU AI Act, which aims to ensure that artificial intelligence is developed across Europe in a transparent, traceable, and risk-based manner. However, the der EU AI Act in der Industrie momentan eher als reine Drohkulisse. Doch hinter den regulatorischen Daumenschrauben steckt eine fachliche Notwendigkeit. Die eigentliche Herausforderung ist nicht das Ausfüllen neuer Formulare, sondern ein fundamentales Umdenken: Wir müssen ein kontinuierlich lernendes System über seinen gesamten Lebenszyklus hinweg beherrschen.
The New Risk Paradox
Traditional safety standards primarily view risks as the result of predictable physical or logical errors, such as component failure or software bugs. However, the EU AI Act radically expands this perspective. It considers risks arising from learning behavior, data quality, model changes, and statistical drift processes (i.e., gradual changes in real-world data) as relevant.
Examining the practical implications in two regulated industries illustrates what this means in concrete terms.
Example 1: Medical Technology – AI Detects Tumors on MRI Images
- Classical Risk: The MRI image sensor is defective The image is noisy The doctor overlooks the tumor The patient is endangered. (Clearly solvable via hardware redundancy or classical signal error detection).
- AI Risk: The AI’s training data came almost exclusively from older MRI machines. In the field, the model is fed by a state-of-the-art, high-contrast MRI scanner. The statistical distribution shifts (Data Drift[2]). The AI misinterprets the sharper edges and overlooks the tumor—even though the software runs flawlessly from a technical perspective.
Example 2: Mechanical Engineering – Camera Detects Persons in a Machine’s Danger Zone
- Classical Risk: Cable break at the camera No video signal Machine does not stop Person is injured. (Classically secured via the closed-circuit principle or ISO 13849).
- AI Risk: The AI’s training data contained hardly any images of people wearing reflective safety vests. A worker enters the danger zone wearing a brand-new safety vest. The AI does not reliably recognize the person because statistical bias strikes Machine does not stop Person is injured.
How AI Transforms the Engineering World
To make this difference even more tangible in everyday development, a comparison of the working methods helps:
Classical Engineering
System Behavior: Software is deterministic (If-Then).
Requirements:Specifications rarely change after the freeze.
Testing Methods:Test cases remain stable and are regression-tested.
Risk Analysis: One-time/periodic event at release.
AI Engineering
System Behavior: Model is probabilistic (Probabilities).
Requirements:Training and validation data change regularly.
Testing Methods: Test datasets must be continuously expanded.
Risk Analysis: Continuous process over the entire lifecycle.
Three Steps to Concretely Master AI Risks in Engineering
The EU AI Act does not require a separate risk management framework in addition to ISO 14971 or FMEA. Rather, it requires an integrated risk management approach that incorporates the disciplines of data governance[3], model validation, human oversight[6] and post-market monitoring into the existing engineering process. This is also possible with AI support. We’ve examined an example from objectiF RPM in more detail in a blog post.
The following three work instructions will ensure a successful transition:
Step 1: Treat training data for AI models as you would software requirements.
Do not treat data quality as the sole responsibility of data scientists. Data is the new specification.
Implementation: Define the functional and non-functional requirements for your training data, just as you would for traditional systems.
Your checklist for each dataset:
- [ ] Origin: Is the source of the data fully documented and auditable?
- [ ] Representativeness: Does the data cover all real-world use cases (e.g., different types of MRI scans or variations of high-visibility vests)?
- [ ] Versioning: Can you trace exactly which model was trained using which dataset?
- [ ] Bias[1] Assessment: Has the data been checked specifically for statistical biases?
- [ ] Approval: Is there a formal quality gate process for data releases?
Step 2: Clearly define the situations in which the AI is not allowed to make decisions.
Since an AI always provides a statistical answer, even if it is incorrect, the system architecture must set limits. This is the core of the required human oversight. Depending on the risk and application, this can be implemented through human-in-the-loop[5] mechanisms or deterministic safety barriers, for example.
Implementation: Encapsulate the AI component within the SysML or UML model. Implement deterministic safety guards[8].
Rule of thumb: If the development team sets a validated confidence threshold of 90%, for example, and the model falls below this value, the system architecture can block the AI decision and force a manual review.
Step 3: Create a change management process for model updates.
Even with AI systems, the risk management process does not end with product approval. Models can be retrained, updated, or exposed to changed operational data. Each retraining session alters the system’s behavior, which can subsequently alter the risks.
Implementation: Define a strict change management process for model adjustments and subsequent post-market monitoring today. As soon as a model in clinical use is updated, the traceability[9] chain must automatically verify which original risk analyses and validation tests need to be rerun.
5 Questions Your Development Team Should Be Asking Itself Right Now
Before planning your next AI feature, assess your team’s audit readiness using these five questions:
- Do we know exactly which training data was used for which product version?
- Can we seamlessly demonstrate the correct implementation of each AI-specific requirement, down to the corresponding test case and result?
- Do we know precisely what new risks arise when the model is retrained using data from the last three months?
- Can we provide an auditor with evidence proving why the model was released and which metrics were used?
- Could we submit this evidence to an independent audit or certification body at a moment’s notice?
The Tool Landscape as a Compliance Risk
Unfortunately, everyday life in many established companies in the medical technology, mechanical engineering, and automotive sectors still looks like this:
- Requirements are stored in Word documents.
- Risk analyses (FMEAs) are maintained in Excel spreadsheets.
- Test cases are stored in a test management system.
- System architecture is diagrammed using Visio or modeling tools.
- AI training data is scattered across a file server used by data scientists.
For traditional, purely deterministic software, this fragmented tool landscape was cumbersome yet manageable due to the high level of manual effort required. However, for AI systems subject to the EU AI Act, this fragmentation poses an acute compliance and liability risk. Since AI components are constantly changing, ongoing manual synchronization between these tools is necessary.
Tech Glossary: Key Terms at a Glance
Bias[1](Model Distortion): A statistical error in the training data (e.g., one-sidedness) that causes the AI to make systematically flawed or discriminatory decisions in the field.
Data Drift[2]: The phenomenon where real-world data changes over time and no longer matches the AI’s original training data. This can cause the AI’s accuracy to degrade gradually during operation.
Data Governance[3]: The strategic and operational framework for managing data quality, integrity, and security. In an AI context, this means the seamless monitoring and cleaning of training datasets to eliminate errors or distortions from the outset.
Determinism[4]: A programming principle where a system always delivers the exact same output for the same input. The behavior is 100% predictable and logically traceable.
Human-in-the-Loop[5]: A specific form of human oversight. Here, the human is strictly integrated into the active decision-making process of the AI. The system is only permitted to execute an action after a human operator has explicitly approved it.
Human Oversight[6]: The legal mandate of the EU AI Act requiring High-Risk AI systems to be designed in such a way that they can be effectively monitored, understood, and corrected or shut down in an emergency by humans.
Probability[7]: The opposite of determinism. AI systems operate based on statistical probabilities. The same input can result in a slightly different output depending on the system state and data baseline—system behavior is approximately, but not absolutely, predictable.
Safety Guards[8]: These are deterministic safeguards or plausibility checks that are programmed around an AI component. These safeguards intercept probabilistic malfunctions at the system level before any actual damage occurs. For example, they can automatically block an AI decision if the confidence level is too low.
Traceability[9]: The seamless documentation proving which element in the development process (e.g., a test result or a risk) originated from which initial requirement or data baseline.
Conclusion: The Single Source of Truth as a Solution
The solution does not lie in burdening your engineers with more documentation work. It lies in breaking down information silos. End-to-end traceability can only be achieved when requirements, system models (UML/SysML), AI-specific risks, data parameters, and test cases operate on a shared, interconnected data foundation. This supports many of the documentation and verification requirements of the EU AI Act.
This is precisely the area in which an integrated engineering platform like objectiF RPM shines. As a Single Source of Truth (SSOT), objectiF RPM brings all the threads together in a relational repository. If requirements, risks, or other interlinked artifacts change, the affected relationships and dependencies can be traced. This enables targeted analysis of the impacts. Thus, you can navigate the paradigm shift brought by the EU AI Act as a genuine evolutionary leap in quality for your systems, not a bureaucratic burden.
Prepare your risk management for the AI era
Say goodbye to rigid documents and confusing Excel spreadsheets. Learn how objectiF RPM can support your requirements and risk management in an exclusive live presentation. Contact us to schedule a convenient time.