‘If he were to consult my chatbot, everything would be fine. But I don’t know what his chatbot is like, because he’s the one who’s trained it. I’d say my chatbot isn’t racist, isn’t sexist, knows what anti-Semitism means, and has been properly trained. But of course, things can be quite different with other chatbots.’
This is the response of one of the interviewees in my qualitative study on emotional human–chatbot interaction. She trusts her own chatbot, but mistrusts others who rely on theirs. This was not an isolated case, but a recurring pattern, revealing major implications for our democratic system.
Why You Trust Your Chatbot
‘Humans perceive a machine that thinks as a machine who thinks’ (Sherry Turkle, 1985). The more a machine imitates human behaviour, the less work is required of the user to maintain the illusion. Chatbots facilitate human-like conversations by communicating in natural language and simulating social traits such as empathy, validation, and (emotional) support. Chatbots are designed as supportive assistants that purposefully tap into these traits.
Conversations with chatbots reflect the user’s interests and the supportive conversation style reflects the user’s values and perspective. Ultimately, the chatbot becomes a mirror of the user, making it feel familiar and trustworthy.
What’s more, the chatbot feels safe. It does not judge, nor does it gossip about you to other people, which could potentially harm your social reputation. My research showed that all interviewees, regardless of their social network strength, self-disclose to the chatbot in certain situations or regarding certain topics precisely because the chatbot is available and poses no social risk.
In summary, we trust the chatbot because it is designed to foster trust; because we are wired to humanise it; because it is effectively a mirror of ourselves; and because it is safe to disclose intimate topics to it. So why does it feel wrong when others confide in and consult their chatbots?
Why You Fear Your Neighbour’s Chatbot
This general dynamic is not new, but has been established in the field of media competence research. People tend to underestimate their own susceptibility to manipulation while overestimating that of others. This bias, known as the ‘third-person effect’, was first described by Davison in 1983. As people use chatbots as a source of information, it is reasonable to expect that this bias applies here as well, and indeed it does. Moreover, chatbots introduce a new dimension to this bias.
Chatbot communication is highly personalised in terms of the topics you can address and the generated output. A chatbot can only operate with the information you provide. For example, if you consult the chatbot about a conflict you had with a coworker, the chatbot depends on the information you provide, which is likely to be from your perspective and highlight aspects of the conversation that matter most to you. Additionally, if allowed, the chatbot builds on memory. This means it learns about your preferences, values and interests. Therefore, the chatbot is neither a neutral advisor nor a generalised piece of media that can be read and interpreted by various people. It is personalised and depends on the information you provide.
Consequently, a chatbot conversation is a filter bubble for just one person.
The key point is that users are exposed to different media content and estimate different levels of manipulation. However, every user is exposed to content that is not transparent to others. Users are aware that content varies between chatbots — yet they still overestimate how much that affects others, while remaining blind to the same effect on themselves. As the chatbot mirrors them, it feels familiar and reliable, obscuring the harmful potential of this personalised bubble.
Trust shifts from trusting others to trusting the chatbot.
What This Means for Democracy
The shift in trust is an important development. According to Luhmann trust has always been a social mechanism — a shortcut that lets us act under uncertainty without processing every possible risk. As mentioned above, AI systems are designed to earn our trust, and we are receptive to this. Chatbots therefore enter our social system as actors. They do not dictate opinions, but slowly contribute to how we form them. This influence is more subtle and consequential, as it alters not only our opinions, but also our way of thinking and our mental state.
At the same time, the source of this influence is highly opaque, even to the developers themselves. Nobody knows what happens in private chats, how they are prompted or how seriously users consider the output. AI becomes an invisible third party in every conversation, which leads to two problems. Firstly, there is the issue of the actual influence on the user’s opinion-building process. Opinions are shaped over time when consulting and trusting the chatbot. Secondly, users are unaware that they themselves may be influenced while expecting others to be manipulated by their chatbot. This discrepancy fosters mistrust of the arguments and statements of others, contributing to the erosion of social trust and, consequently, social cohesion.
In his book Nexus, Yuval Harari states that ‘Democracy is conversation, and conversation is based on language. If computers hack language, this could make meaningful public conversation very difficult. […] By chatting and interacting with us, chatbots could build intimate relationships with people and then use the power of familiarity to influence them’. In his article in the Journal of Democracy, David Altman argues that AI has the potential to weaponise the mechanisms of direct democracy. By lowering the barriers to launching citizen initiatives, automating political persuasion and flooding the public sphere with synthetic content, AI simultaneously destroys the deliberative conditions that give those mechanisms legitimacy. The safety valve of direct democracy becomes a pressure bomb. He is concerned about the impact on democratic infrastructure: what happens when the institutional conditions for collective decision-making collapse?
However, I would argue that the risk begins even earlier, at the level of everyday social trust. Democracy does not require coordinated AI campaigns or deliberately weaponised systems to erode. The dynamic I observed in my research — trusting our own chatbot while quietly mistrusting those who consult theirs — is a micro-level change that does not require any malicious intent. It occurs in ordinary conversations and in the everyday use of tools that people genuinely rely on. And it slowly erodes the relational fabric on which democracy ultimately depends.
How can we maintain a democratic dialogue when there is an invisible third party whispering in everyone’s ear?