# Notes from AI Risk Seminar (24 Sep 2026. TTSH) Related Notes: - [[AIHGle 2.0]] - [[NIST -Artificial Intelligence Risk Management Framework (AI RMF 1.0)]] - [[Systemic Thinking]] - [[bias]] ____ Pre-Seminar Reading Topics related to AI Risks includes: 1. Wrong Output — Hallucinations etc. 2. Bias —- Human biases and prejudices gets into training data, shaping algorithms 3. Privacy / Security 4. Explainability and Transparency — vs Black box, uncheckable. 5. Human oversight and accountability 6. Monitoring after deployment — e.g drifting. - “Human performances” treated as an unquestioned gold standard when humans themselves are moving, interpretive systems (with bias, cognitive distortion) ___ # First Speaker, Mr Benjamin Chiang - EY ASEAN - How can we use AI with confidence? He offered leaders the following three questions - What are the questions to ask? - What to monitor? - Who is empowered to intervene when there is a problem?” - “Human in the Loop” - But is the human in the loop qualified, skilled, able to spot mistakes? - Risk of human blindly following and trust AI output without question. - Healthcare system globally is under structural strains - different factors from the changes of demographic, and increasing costs. - He cited a study by IBM “Redefining the tech leader’s mandate” - said that 77% of the leaders are adopting AI solutions faster then governance. (Really?) - The point is that, when AI solutions leaves pilot, and entering scaling, there will be increase complexities, variables that are harder to control, and more costly to control. - A useful framework to do risk analysis and management is to categorise AI solutions depending to the domains. - What AI is design to do, what are the risk involved, and then decide on the level of control to be put in place. - To assess the risk of AI solutions, need to look beyond the AI product, to consider the whole ecosystem: input - AI Solution - output - In AI Systems, there are different sources of information, data, Vendors.. Pipelines.. that flows into the AI, Even though the model has not changed, but the output may have drifted. ## Pilot vs Real world - Pilot may have good result but fail in real world setting. The training data are not the same as real world data. - Model performance is only one factor. (Look at the whole process; such as how are workflow, processes, the “conditions” in the real world). Failure Patterns: 1. Context Changes 2. Workflow 3. People adapt to the AI (such as Alert fatigue, human mistakes, bypassing, override) 4. The ecosystem changes 5. Failure may be hidden. **Basically, when something works, we need to understand what is the context that it works** - Reminded me of systemic theories, in relational dynamics, aka what is the social context for that relational pattern to show up. [[Relational Dynamics]] - Also reminded me of psychotherapy fidelity — Therapy approach + other factors = intended effect ## What is the process, or what to check before deploying AI? 1. Use case intake 2. Risk Classficiation 3. Design requirement 4. Independent challenge 5. Go-Live decision 6. Controlled Release (Need to have clear ownership, validation process, monitoring and responses.. and who got the power to stop it if AI goes wrong) ## What are the post deployment monitoring processes? 1. Performance 2. Care Outcome 3. Human intervention 4. Change and Drift 5. Safety signals **Don’t just monitor and don’t do anything with it. Monitor lead to decision to continue, changes and stopping decisions**. Manage the whole eco-system. The goal is not to be afraid and not do anything. It is to manage the uncertainty and have a response plan.