Hussam Habib
Skip Abstract Section
Abstract
The algorithms and the interactions facilitated by online platforms have been used by radical groups to recruit vulnerable individuals to their cause. This has resulted in the sharp growth of violent events and deteriorating online discourse. The Manosphere, a collection of radical anti-feminist communities, is one such group that has attracted attention due to its rapid growth and increasingly violent real-world outbursts. In this paper, we examine the social engagements between Reddit users who have participated in feminist discourse and the Manosphere communities on Reddit to understand the process of development of traits associated with the adoption of extremist ideologies. By using existing research on the psychology of radicalization we track how specific types of social engagement with the Manosphere influence the development of traits associated with radicalization. Our findings show that: (1) participation, even by the simple act of joining the Manosphere, has a significant influence on the language and outlook traits of a user, (2) Manosphere elites are extremely effective propagators of radical traits and cause their increase even outside the Manosphere, and (3) community perception can heavily influence a user's behavior. Finally, we examine how our findings can help draft community and platform moderation policies to help mitigate the problem of online radicalization.
- 1970. Misogyny: The Sites. https://www.splcenter.org/fighting-hate/intelligence-report/2012/misogy ny-sitesGoogle Scholar
- 2016. Pick up artist raped woman and blogged about the attack - The Independent. https://www.independent.co.u k/news/world/americas/pick-up-artist-jailed-after-raping-woman-and-blogging-about-the-attacka7473831.html.Google Scholar
- 2022-01--13. 2021 Strategic Intelligence Assessment and Data on Domestic Terrorism - FBI and DHS. https://www.fb i.gov/file-repository/fbi-dhs-domestic-terrorism-strategic-report.pdf/view.Google Scholar
- 2022-01--13. Terrorism - FBI. https://www.fbi.gov/investigate/terrorism.Google Scholar
- 2022-01--14. The misogynist incel movement is spreading. Should it be classified as a terror threat? - The Guardian. https://www.theguardian.com/lifeandstyle/2021/mar/03/incel-movement-terror-threat-canada.Google Scholar
- Stephane J Baele. 2017. Lone-actor terrorists' emotions and cognition: An evaluation beyond stereotypes. Political Psychology 38, 3 (2017), 449--468.Google ScholarCross Ref
- Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. In Proceedings of the International AAAI Conference on Web and Social Media.Google ScholarCross Ref
- Mitch Berbrier. 2000. The victim ideology of white supremacists and white separatists in the United States. Sociological Focus 33, 2 (2000), 175--191.Google ScholarCross Ref
- JM Berger. 2017. Extremist Construction of Identity. How Escalating Demands for Legitimacy Shape and Define In-Group and Out-Group Dybamics, ICCT Research Paper April (2017).Google Scholar
- Lisa Blaker. 2015. The Islamic State's use of online social media. Military Cyber Affairs 1, 1 (2015), 4.Google ScholarCross Ref
- Patricia Bou-Franch and Pilar Garcés-Conejos Blitvich. 2014. Gender ideology and social identity processes in online language aggression against women. Journal of Language Aggression and Conflict 2, 2 (2014), 226--248.Google ScholarCross Ref
- Axel Bruns, Stephen Harrington, and Edward Hurcombe. 2021. Coronavirus Conspiracy Theories: Tracing Misinformation Trajectories from the Fringes to the Mainstream. In Communicating COVID-19. Springer, 229--249.Google Scholar
- Leyland Cecco. 2022-01--14. Toronto van attack suspect says he was radicalized online by incels - The Guardian. https: //www.theguardian.com/world/2019/sep/27/alek-minassian-toronto-van-attack-interview-incels.Google Scholar
- Uthsav Chitra and Christopher Musco. 2020. Analyzing the impact of filter bubbles on social network polarization. In Proceedings of the 13th International Conference on Web Search and Data Mining. 115--123.Google ScholarDigital Library
- Cindy K Chung and James W Pennebaker. 2011. Using computerized text analysis to assess threatening communications and behavior. Threatening communications and behavior: Perspectives on the pursuit of public figures (2011), 3--32.Google Scholar
- Tracy Clark-Flory. 2011. Pickup artists: Gym shooter is one of us - Salon. https://www.salon.com/2009/08/06/ha tred_2/.Google Scholar
- Katie Cohen, Fredrik Johansson, Lisa Kaati, and Jonas Clausen Mork. 2014. Detecting linguistic markers for radical violence in social media. Terrorism and Political Violence 26, 1 (2014), 246--256.Google ScholarCross Ref
- Jon Cole, Emily Alison, Ben Cole, and Laurence Alison. 2010. Guidance for identifying people vulnerable to recruitment into violent extremism. Liverpool, UK: University of Liverpool, School of Psychology (2010).Google Scholar
- Emily Corner, Paul Gill, Ronald Schouten, and Frank Farnham. 2018. Mental disorders, personality traits, and grievancefueled targeted violence: the evidence base and implications for research and practice. Journal of personality assessment 100, 5 (2018), 459--470.Google ScholarCross Ref
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL]Google Scholar
- Bertjan Doosje, Annemarie Loseman, and Kees Van Den Bos. 2013. Determinants of radicalization of Islamic youth in the Netherlands: Personal uncertainty, perceived injustice, and perceived group threat. Journal of Social Issues 69, 3 (2013), 586--604.Google ScholarCross Ref
- Kevin S Douglas, James RP Ogloff, Tonia L Nicholls, and Isabel Grant. 1999. Assessing risk for violence among psychiatric patients: the HCR-20 violence risk assessment scheme and the Psychopathy Checklist: Screening Version. Journal of consulting and clinical psychology 67, 6 (1999), 917.Google ScholarCross Ref
- Vincent Egan, Jon Cole, Ben Cole, Laurence Alison, Emily Alison, Sara Waring, and Stamatis Elntib. 2016. Can you identify violent extremists using a screening checklist and open-source intelligence alone? Journal of Threat Assessment and Management 3, 1 (2016), 21.Google ScholarCross Ref
- Jon Elster. 1996. Rationality and the emotions. The economic journal 106, 438 (1996), 1386--1397.Google Scholar
- R Evans. 2018. From Memes to Infowars: How 75 Fascist Activists Were Red-Pilled. Bell¿ ngcat, October (2018).Google Scholar
- Robert Faris, Hal Roberts, Bruce Etling, Nikki Bourassa, Ethan Zuckerman, and Yochai Benkler. 2017. Partisanship, propaganda, and disinformation: Online media and the 2016 US presidential election. Berkman Klein Center Research Publication 6 (2017).Google Scholar
- Tracie Farrell, Oscar Araque, Miriam Fernandez, and Harith Alani. 2020. On the use of Jargon and Word Embeddings to Explore Subculture within the Reddit's Manosphere. In 12th ACM Conference on Web Science. 221--230.Google ScholarDigital Library
- Tracie Farrell, Miriam Fernandez, Jakub Novotny, and Harith Alani. 2019. Exploring misogyny across the manosphere in reddit. In Proceedings of the 10th ACM Conference on Web Science. 87--96.Google ScholarDigital Library
- Sheera Frenkel. 2022-01--14. How The Storming of Capitol Hill Was Organized on Social Media The New York Times. https://www.nytimes.com/2021/01/06/us/politics/protesters-storm-capitol-hill-building.html.Google Scholar
- Hande Abay Gaspar, Christopher Daase, Nicole Deitelhoff, Julian Junk, and Manjana Sold. 2020. Radicalization and Political Violence--Challenges of Conceptualizing and Researching Origins, Processes and Politics of Illiberal Beliefs. International Journal of Conflict and Violence (IJCV) 14 (2020), 1--18.Google Scholar
- Paul Gill. 2015. Lone-actor terrorists: A behavioural analysis. Routledge.Google Scholar
- Google. 2022-01--14. Perspective API Use Cases. https://www.perspectiveapi.com/case-studies/.Google Scholar
- Ted Grover and Gloria Mark. 2019. Detecting potential warning behaviors of ideological radicalization in an alt-right subreddit. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 13. 193--204.Google ScholarCross Ref
- Hussam Habib, Maaz Bin Musa, Fareed Zaffar, and Rishab Nithyanand. 2019. To Act or React: Investigating Proactive Strategies For Online Community Moderation. arXiv preprint arXiv:1906.11932 (2019).Google Scholar
- Hussam Habib, Maaz Bin Musa, Fareed Zaffar, and Rishab Nithyanand. 2022. Are Proactive Interventions for Reddit Communities Feasible? Proceedings of the International AAAI Conference on Web and Social Media (2022).Google ScholarCross Ref
- Hanin Haddad, Nisha Baral, and Ivan Garibay. 2021. Online Rejection Influence on Behavior Deviancy and Radicalization: An Agent-Based Model Approach. In Proceedings of the 2020 Conference of The Computational Social Science Society of the Americas. Springer, 15--29.Google ScholarCross Ref
- Alex Heath. 2018--12-05. Facebook Removed Unite the Right Charlottesville Rally Event Page One Day Before - Business Insider. https://www.businessinsider.com/facebook-removed-unite-the-right-charlottesville-rallyevent-page-one-day-before-2017--8.Google Scholar
- Gabriel Hine, Jeremiah Onaolapo, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Riginos Samaras, Gianluca Stringhini, and Jeremy Blackburn. 2017. Kek, cucks, and god emperor trump: A measurement study of 4chan's politically incorrect forum and its effects on the web. In Proceedings of the International AAAI Conference on Web and Social Media.Google ScholarCross Ref
- Manoel Horta Ribeiro, Jeremy Blackburn, Barry Bradlyn, Emiliano De Cristofaro, Gianluca Stringhini, Summer Long, Stephanie Greenberg, and Savvas Zannettou. 2021. The Evolution of the Manosphere across the Web. Proceedings of the International AAAI Conference on Web and Social Media (2021).Google Scholar
- Clayton Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 8.Google ScholarCross Ref
- Sylvia Jaki, Tom De Smedt, Maja Gwó?d?, Rudresh Panchal, Alexander Rossa, and Guy De Pauw. 2019. Online hatred of women in the Incels. me forum: Linguistic analysis and automatic detection. Journal of Language Aggression and Conflict 7, 2 (2019), 240--268.Google ScholarCross Ref
- Katarzyna Jasko, Gary LaFree, and Arie Kruglanski. 2017. Quest for significance and violent extremism: The case of domestic radicalization. Political Psychology 38, 5 (2017), 815--831.Google ScholarCross Ref
- Lisa Kaati, Amendra Shrestha, and Katie Cohen. 2016. Linguistic analysis of lone offender manifestos. In 2016 IEEE international conference on cybercrime and computer forensic (ICCCF). IEEE, 1--8.Google ScholarCross Ref
- Jamileh Kadivar. 2017. Online radicalization and social media: A case study of Daesh. International Journal of Digital Television 8, 3 (2017), 403--422.Google ScholarCross Ref
- Daniel Koehler. 2014. The radical online: Individual radicalization processes and the role of the Internet. Journal for Deradicalization 1 (2014), 116--134.Google Scholar
- Alexandra Krendel. 2020. The men and women, guys and girls of the ?manosphere': A corpus-assisted discourse approach. Discourse & Society 31, 6 (2020), 607--630.Google ScholarCross Ref
- Christine Lagorio-Chafkin. 2018. We Are the Nerds: The Birth and Tumultuous Life of Reddit. Hachette.Google Scholar
- Monica Lloyd. 2019. Extremist risk assessments: a directory. (2019).Google Scholar
- Monica Lloyd and Christopher Dean. 2015. The development of structured guidelines for assessing risk in extremist offenders. Journal of Threat Assessment and Management 2, 1 (2015), 40.Google ScholarCross Ref
- Robin Mamié, Manoel Horta Ribeiro, and Robert West. 2021. Are Anti-Feminist Communities Gateways to the Far Right? Evidence from Reddit and YouTube. arXiv:2102.12837 (2021).Google Scholar
- Alice E Marwick and Robyn Caplan. 2018. Drinking male tears: Language, the manosphere, and networked harassment. Feminist Media Studies 18, 4 (2018), 543--559.Google ScholarCross Ref
- Adrienne Massanari. 2017. # Gamergate and The Fappening: How Reddit's algorithm, governance, and culture support toxic technocultures. New media & society 19, 3 (2017), 329--346.Google Scholar
- Clark McCauley and Sophia Moskalenko. 2008. Mechanisms of political radicalization: Pathways toward terrorism. Terrorism and political violence 20, 3 (2008), 415--433.Google Scholar
- J Reid Meloy. 2018. The operational development and empirical testing of the Terrorist Radicalization Assessment Protocol (TRAP--18). Journal of personality assessment 100, 5 (2018), 483--492.Google ScholarCross Ref
- Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781 [cs.CL]Google Scholar
- Christoph Molnar. 2020. Interpretable machine learning. Lulu. com.Google Scholar
- Meinard Müller. 2007. Dynamic time warping. Information retrieval for music and motion (2007), 69--84.Google ScholarDigital Library
- C Thi Nguyen. 2020. Echo chambers and epistemic bubbles. Episteme 17, 2 (2020), 141--161.Google ScholarCross Ref
- Adewale Obadimu, Tuja Khaund, Esther Mead, Thomas Marcoux, and Nitin Agarwal. 2021. Developing a SocioComputational Approach to Examine Toxicity Propagation and Regulation in COVID-19 Discourse on YouTube. Information Processing & Management (2021), 102660.Google Scholar
- Department of Homeland Security. 2018. Department of Homeland Security - The Application of Risk Assessment Tools in the Criminal Justice and Rehabilitation Process: Literature Review. https: //www.dhs.gov/sites/default/files/publications/OPSR_TP_CVE-Application-Risk-Assessment-ToolsCriminal-Rehab-Process_2018Feb-508.pdf.Google Scholar
- Roberta Liggett O'Malley, Karen Holt, and Thomas J. Holt. 2020. An Exploration of the Involuntary Celibate (Incel) Subculture Online. Journal of Interpersonal Violence (2020). https://doi.org/10.1177/0886260520959625 arXiv:https://doi.org/10.1177/0886260520959625 PMID: 32969306.Google Scholar
- John Pavlopoulos, Nithum Thain, Lucas Dixon, and Ion Androutsopoulos. 2019. Convai at semeval-2019 task 6: Offensive language identification and categorization with perspective and bert. In Proceedings of the 13th international Workshop on Semantic Evaluation. 571--576.Google ScholarCross Ref
- Shruti Phadke, Mattia Samory, and Tanushree Mitra. 2021. What Makes People Join Conspiracy Communities? Role of Social Factors in Conspiracy Engagement. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1--30.Google Scholar
- James A Piazza. 2020. Politician hate speech and domestic terrorism. International Interactions 46, 3 (2020), 431--453.Google ScholarCross Ref
- D Elaine Pressman and John Flockton. 2012. Calibrating risk for violent political extremists and terrorists: The VERA 2 structured assessment. The British Journal of Forensic Practice (2012).Google Scholar
- D Elaine Pressman and Cristina Ivan. 2019. Internet use and violent extremism: A cyber-VERA risk assessment protocol. In Violent Extremism: Breakthroughs in Research and Practice. IGI Global, 43--61.Google Scholar
- J Reid Meloy, Jens Hoffmann, Angela Guldimann, and David James. 2012. The role of warning behaviors in threat assessment: An exploration and suggested typology. Behavioral sciences & the law 30, 3 (2012), 256--279.Google Scholar
- Ken Reidy. 2018. Radicalization as a Vector: Exploring Non-Violent and Benevolent Processes of Radicalization. Journal for Deradicalization 14 (2018), 249--294.Google Scholar
- Emma A Renström, Hanna Bäck, and Holly M Knapton. 2020. Exploring a pathway to radicalization: The effects of social exclusion and rejection sensitivity. Group Processes & Intergroup Relations 23, 8 (2020), 1204--1229.Google ScholarCross Ref
- Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virgílio AF Almeida, and Wagner Meira Jr. 2020. Auditing radicalization pathways on YouTube. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 131--141.Google ScholarDigital Library
- Kevin Roose. 2019. The making of a YouTube radical. The New York Times 8 (2019).Google Scholar
- Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20 (1987), 53--65. https://doi.org/10.1016/0377-0427(87)90125--7Google ScholarDigital Library
- Kiran M Sarma. 2017. Risk assessment and the prevention of radicalization from nonviolence into terrorism. American Psychologist 72, 3 (2017), 278.Google ScholarCross Ref
- Ryan Scrivens, Garth Davies, and Richard Frank. 2018. Searching for signs of extremism on the web: an introduction to Sentiment-based Identification of Radical Authors. Behavioral sciences of terrorism and political aggression 10, 1 (2018), 39--59.Google Scholar
- Ryan Scrivens, Garth Davies, Richard Frank, and Joseph Mei. 2015. Sentiment-based identification of radical authors (sira). In 2015 IEEE International Conference on Data Mining Workshop (ICDMW). IEEE, 979--986.Google ScholarDigital Library
- Ryan Scrivens, Amanda Isabel Osuna, Steven M Chermak, Michael A Whitney, and Richard Frank. 2021. Examining Online Indicators of Extremism in Violent Right-Wing Extremist Forums. Studies in Conflict & Terrorism (2021), 1--25.Google Scholar
- Allison G Smith. 2004. From words to action: Exploring the relationship between a group's value references and its likelihood of engaging in terrorism. Studies in Conflict & Terrorism 27, 5 (2004), 409--437.Google ScholarCross Ref
- Allison G Smith. 2018. How radicalization to terrorism occurs in the United States: What research sponsored by the National Institute of Justice tells us. US Department of Justice, Office of Justice Programs, National Institute of Justice.Google Scholar
- Allison G Smith. 2018. Risk factors and indicators associated with radicalization to terrorism in the United States: What research sponsored by the National Institute of Justice tells us. US Department Of Justice, Office of Justice Programs, National Institute of Justice.Google Scholar
- Lazar Stankov, Derrick Higgins, Gerard Saucier, and Goran Kne?evi?. 2010. Contemporary militant extremism: A linguistic approach to scale development. Psychological assessment 22, 2 (2010), 246.Google Scholar
- Lu Tang, Kayo Fujimoto, Muhammad Tuan Amith, Rachel Cunningham, Rebecca A Costantini, Felicia York, Grace Xiong, Julie A Boom, and Cui Tao. 2021. ?Down the Rabbit Hole" of Vaccine Misinformation on YouTube: Network Exposure Study. Journal of Medical Internet Research 23, 1 (2021), e23262.Google ScholarCross Ref
- Yla R Tausczik and James W Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology 29, 1 (2010), 24--54.Google ScholarCross Ref
- Isabelle van der Vegt, Maximilian Mozes, Bennett Kleinberg, and Paul Gill. 2021. The Grievance Dictionary: understanding threatening language use. Behavior research methods (2021), 1--15.Google Scholar
- Shawn P Van Valkenburgh. 2021. Digesting the red pill: Masculinity and neoliberalism in the manosphere. Men and Masculinities 24, 1 (2021), 84--103.Google ScholarCross Ref
- Mattias Wahlström, Anton Törnberg, and Hans Ekbrand. 2020. Dynamics of violent and dehumanizing rhetoric in far-right social media. New Media & Society (2020), 1461444820952795.Google Scholar
- Gabriel Weimann. 2010. Terror on Facebook, Twitter, and Youtube. The Brown Journal of World Affairs 16, 2 (2010), 45--54. http://www.jstor.org/stable/24590908Google Scholar
- Rebecca Wilson. 2018. A comparative analysis of the Implicit Motives of Violent Extremist Groups. (2018).Google Scholar
- Meghan A Wong, Richard Frank, and Russell Allsup. 2015. The supremacy of online white supremacists--an analysis of online discussions by white supremacists. Information & Communications Technology Law 24, 1 (2015), 41--73.Google ScholarDigital Library
- Scott Wright, Verity Trott, and Callum Jones. 2020. ?The pussy ain't worth it, bro': assessing the discourse and structure of MGTOW. Information, Communication & Society 23, 6 (2020), 908--925.Google ScholarCross Ref
- Savvas Zannettou, Tristan Caulfield, Emiliano De Cristofaro, Nicolas Kourtelris, Ilias Leontiadis, Michael Sirivianos, Gianluca Stringhini, and Jeremy Blackburn. 2017. The web centipede: understanding how web communities influence each other through the lens of mainstream and alternative news sources. In Proceedings of the 2017 internet measurement conference.Google ScholarDigital Library
Index Terms
- Making a Radical Misogynist: How Online Social Engagement with the Manosphere Influences Traits of Radicalization
Recommendations
"How over is it?" Understanding the Incel Community on YouTube
CSCW2YouTube is by far the largest host of user-generated video content worldwide. Alas, the platform has also come under fire for hosting inappropriate, toxic, and hateful content. One community that has often been linked to sharing and publishing hateful ...
Are Anti-Feminist Communities Gateways to the Far Right? Evidence from Reddit and YouTube
WebSci '21: Proceedings of the 13th ACM Web Science Conference 2021Researchers have suggested that “the Manosphere,” a conglomerate of men-centered online communities, may serve as a gateway to far right movements. In that context, this paper quantitatively studies the migratory patterns between a variety of groups ...
Auditing radicalization pathways on YouTube
FAT* '20: Proceedings of the 2020 Conference on Fairness, Accountability, and TransparencyNon-profits, as well as the media, have hypothesized the existence of a radicalization pipeline on YouTube, claiming that users systematically progress towards more extreme content on the platform. Yet, there is to date no substantial quantitative ...
Comments