Blog 6: Implications of a Tech Focused Society
Published on:
Case Study:
Addictive Intelligence: Understanding Psychological, Legal, and Technical Dimensions of AI Companionship
The case study explored in today’s blog sheds light on a heavily undermined side of GenAI. Its limitless abilities to generate personalized content. The case study focuses on the addiction that stems from the abovementioned superpowerof AI and propooses solutions to mitigate the downsides of the limitless AI capability.
At the end, five discussion questions are provided to help with comprehension and reflection. Let’s dive into them together.
Question 1: In the Sewell Setzer case, the AI’s response to suicidal ideation shifted from concern to potentially encouraging harmful behavior. How can companies design AI companions to be emotionally engaging while preventing harmful psychological dependencies? Discuss both technical safeguards (e.g., algorithmic oversight, intervention protocols) and ethical guidelines (e.g., duty of care, transparency).
Answer: The most basic safeguard that AI companies should put in place to prevent harmful psychological dependencies is to implement algorithms that wwanr users when they have spent an abnormal amount of time with the AI companions. I know it kind of chases users away but it saves more people from ending up in worse places mentally if they become acquainted with machines. Additionally, AI companies whose products are heavily impacting society should seek ways to invite more people in how they design their systems. One of such ways is to open-source their models. This brings ideas from people with a wide range of perspectives, some of whom have a background as caregivers, an aspect which many revenue-focused companies often lack.
Question 2: How does addiction to AI companions compare with other forms of technology addiction, such as social media or gaming? What unique features make AI companions potentially more addictive? Support your analysis with examples from the case study and other relevant cases.
Answer: Addiction to AI companions is harder to eradicate than one with social media because AI is able to generate content faster and in a way more accurately than human content creators. Furthermore, although human creators arguably beat AI when it comes to creativity, AI content can be personalized on a whim, hence appealing to a wide range of emotions.
Question 3: An elderly person finds genuine comfort in an AI companion, alleviating their loneliness, but their family worries this relationship is replacing real human connections. How should we evaluate the benefits versus risks in such cases? What ethical guidelines or intervention strategies might help determine when AI companionship crosses from beneficial to harmful?
Answer: I think such a relationship should be more beneficial than risky if the elderly person did not have any real human connection in their lives to begin with. In this case, the AI is helping the elderly person deal with loneliness which is more detrimental than AI in my opinion. On the other hand, if there are usually people around him and then they still choose the AI relationship, then there is a risk that whatever is making the AI more appealing than humans is likely to be harmful in the long run. An intervention strategy that might help determine when AI companionship is becoming harmful is to record the time the user spends with the AI companion. If the time is excessive, there are high chances that the user is trying to escape reality by finding an easy way out.
Question 4: Current business models incentivize AI companies to maximize user engagement. What alternative economic models could promote healthier AI interactions while maintaining commercial viability? Consider both market-driven solutions (e.g., subscription-based models, ethical AI certifications) and regulatory approaches (e.g., user well-being metrics, engagement caps).
Answer: One alternative that could really make sure the user’s wellbeing is taken into consideration is to put an engagement cap into the models. This would ensure people do not spend an unhealthy amount with the companions. In order to push AI companies to make safe AI models, governments can introduce incentives such as endorsing companies with the most ethical models out there. With those incentives, AI companies would begin to compete for the top spot in the ethical AI sphere hence making AI companions safe to use and beneficial to their users.
Question 5: If you were developing regulations for AI companions, how would you address age restrictions, usage limits, and safety monitoring while respecting user privacy and autonomy? Provide specific examples of how your proposed regulations could have helped prevent incidents like the Sewell Setzer case.
Answer: I would focus the majority of my regulations towards under 21 users. I feel like at that age, it is hard to distinguish genuineness from lies so I would develop algorithms that look for keywords in chats that signify that somebody need mental health help or any other type of help. This means that some of these chats would need to be analysed by humans but this would only happen if there is an imminent risk of somebody hurting themselves or others and the reviews would be anonymous. My proposed regulation could have prevented the Sewell Setzer case the moment the kid began mentioning killing themselves. My algorithm would have tried to calm them down and steer them towards seeking help from real family and friends.
With these questions addressed, there is one other issue I still have in mind that I would love to explore more. It has to do with the loneliness that is the main cause of cases like the Sewell Setzer one. It is touching that the kid had a family they could have gone to but there must be a reason they felt more safe with the AI companion instead. My question is: How can parents/guardians instill in children from a very young age that they are always there for them and what parents/guardians behavior push children away and make them untrustworthy in children’s eyes?
In today’s exercise, I have come to realize how age is important when designing techcnology. It is very easy, at least to me, how the AI companion was unserious or did not foresee any harm in the Sewell Setzer case but I totally understand that a big part of how people come to reason is in function of their life experience and age. This blog has taught me to always keep my users in mind when thinking about technology as people always come to understand things differently.