What counts as harm? Feminist experiments with AI-driven visual content moderation to detect misogynistic violence in mainstream pornography

This paper offers a feminist critique of algorithmic systems of harm-detection by evaluating the visual recognition capabilities of Machine Learning as a Service (MLaaS) tools and their effectiveness in detecting misogynistic violence within online pornography. It asks what counts as harm in commercial artificial intelligence (AI) moderation systems and what happens when these are used to identify misogynistic violence in pornography. While previous research has examined gender bias in AI-based moderation, the detection of male violence against women (MVAW) in visual pornography remains unexplored. This study addresses this gap through an analysis of 100 pornographic thumbnails, combined with visual transitivity analysis. It shows that MLaaS tools miss acts such as strangulation but flag guns as violent, demonstrating that they are ineffective in classifying harm as defined by feminist scholars. From a feminist standpoint, pornography is structured by misogyny. Thus, it functions as a site to test technology's understanding of the structural nature of MVAW, showing that this failure is not a technical oversight but evidence of a broader issue within digital technologies. A combination of training datasets shaped by sexist assumptions, male-centred moderation standards, and biased developers creates a system that sees violence and pornography as mutually exclusive, failing to recognise harm. Without a feminist intervention, the technology defaults to norms that normalise and eroticise pornography as legitimate entertainment. This study contributes to debates on AI bias, technology-facilitated image-based MVAW, and pornography regulation. It provides insights for researchers, policymakers, and platforms seeking an informed basis for adopting a feminist approach to harm-detection.