Evaluating AI in Social Programs: Reframing Complex Intervention as Socio-Technical Intervention

There is now prolific interest in Artificial Intelligence (AI) systems in social services and their application in practice is growing apace. However, research on systematic and evidence-based utilization is nascent and existing frameworks are ill-equipped to manage the complex ethical and methodological challenges posed by AI. Intervention development and evaluation must respond to profound uncertainties regarding effectiveness, ethicality and risk mitigation. The UK’s Medical Research Council/National Institute for Health and Care Research (MRC/NIHR) updated framework for developing and evaluating complex interventions offers a promising meta-methodology to address these challenges. Yet, it lacks crucial perspectives on the socio-technical nature of AI systems and their dynamic and emergent properties. Drawing on the Socio-Technical research paradigm, this paper identifies six procedural dimensions to strengthen the framework. These are: (1) “Anticipatory Design” to identify and mitigate uncertain impacts; (2) “Ethical Considerations” to foreground transparency, accountability and equity; (3) “Continuous Impact Evaluation” to monitor emergent and unintended effects; (4) “Participatory Design” to co-produce systems aligned with stakeholder values; (5) “Socio-Material Contingencies of Automated Practice” to understand how AI reshapes professional roles and practices; and (6) “Re-configurations of Intervention Adherence” to capture adaptation, resistance and contextual variability. This conceptual paper advocates to reframe professional practice utilizing AI as “Socio-Technical Intervention” - one that intentionally accounts for the mutual constitution of human and AI systems in the pursuit of ethical, effective, and context-sensitive innovation. This conceptual shift can inform approaches to intervention research and development. Future work should focus on operationalizing it to generate an evidence base for AI utilization in social services.