Think back to your childhood and recall the kind of kid you were. There were likely many days, especially in your teenage years, when you would ‘try on’ entirely new personas to see what they felt like. You’d experiment with clothing, music, language, even friends.

Now ask yourself: are you the same person you were when you were little? Are you the same person you were 10 years ago? How about a year ago? How about a week ago?

The older we get, the harder it becomes for us to summarize ourselves through our current tastes or hobbies alone. We’re sensitive to our environments, to the needs of others, and to the social languages of belongingness and recognition. We’re doing our best to communicate our needs while respecting the boundaries of a constantly evolving and expanding culture.

In short, personal identity is complicated.

Now think of a time when you felt misunderstood; in particular a time when someone misrepresented your intentions. The experience of having one’s purpose, competency, ability, or values called into question can have long-lasting and formative effects on the way we learn to

Quite often, what we need is a little more benefit of the doubt to feel like we can be ourselves.

Anonymity lost

The early internet opened up interaction modalities that addressed identity in novel and important ways. It offered the ability to take on new personas with each chat room encounter, message board discussion, or gaming session; to explore our most private thoughts and questions, often through a simple text box and a list of ten blue text links.

Ten little nonjudgmental blue links. Search engines like Yahoo! and Google became something like a virtual confessional; a transaction that returned reliably useful results for even the most taboo queries, supported by a nearly limitless supply of benefit of the doubt.

The design metaphors of search engines have evolved considerably since their inception. They’ve shifted from research tools, to “answer” engines, to something vaguely resembling the voice interface from Star Trek. The consumer tech market has evolved as well, with advances in ubiquitous connectivity making for an odd-coupling alongside an increasing awareness of issues like data privacy, ethics, and ad targeting. And as the big tech firms have largely succeeded in marketing themselves as mediators of world knowledge, many users have come to hold them to standards closer to journalistic integrity; a stark contrast to the flawed conventional wisdom that big data somehow reflects a ‘mirror of the world’ back to the public (it actually exacerbates the disparities between the mean and the margins).

A false premise

Machine learning is the science of making predictions based on patterns and associations that have been automatically discovered in data. Much the same way one works to develop a hunch or an intuition about something, the job of a machine learning model is to figure out just how wrong it can be in order to be as right as possible as often as possible.

Machine learning models don’t emerge from thin air, though. Every facet of their development is fueled and mediated by human judgment; from the idea to develop the model in the first place; to the sources of data chosen to train from; to the data itself and the methods and labels used to describe it; to the success criteria for the aforementioned wrongness and rightness.

But establishing clear success criteria for personalization models is no simple task, because as we’ve established, identity isn’t easily defined.

A prevailing assumption that accompanies the productization of AI is that it’s possible to know people better than they know themselves. That if algorithms can be exposed to a sufficient amount of human data, inferences can be made about people based solely on their behavior, and those inferences can be used to achieve ideal personalization.

This leaves two troubling paths forward:

1. Presumed accuracy. Classifiers trained on inferences derived from online behaviors will almost always have a higher false positive rate for minority subgroups, in large part because of the sheer statistical challenge of attempting to build predictions from data with unequal representation. This disparate error rate results in a more robust experience for people from majority subgroups because they benefit from a higher resolution graph of their personal traits, thereby exacerbating opportunity gaps — and the prevalence of stereotypes — for people from underrepresented subgroups.

2. “Perfect” accuracy. Sometimes referred to as the “filter bubble” effect, precision-targeted classifiers enable an unprecedented form of psychographic redlining. Even if someone can’t afford to buy that fancy handbag today; even if someone doesn’t agree with the policy position of that politician; even if someone doesn’t understand the beliefs or lifestyle of another; should they not get exposed to their existence because the targeting criteria weren’t aligned? In this model, technologists are emboldening a system where the world of some can become invisible to others; the world of the privileged to the underprivileged; the world of liberals to conservatives; women to men.

Both paths share the same faulty premise: that what people want — and need — is to only experience things they’ll “like” or that have been deemed “right” for them by someone else.

What’s at stake for tech firms

The current AI boom signals as much about the tech industry’s fascination with deep learning as it does its overconfidence in evergreen access to plentiful and genuine user data. Technologists and investors have grown entitled; treating user data like they’re a veritable renewable resource. But consumer trust isn’t going up-and-to-the-right anymore. People are freaked out by the amount that can be known about them. And the rate of publicly visible missteps resulting from automation is ramping up at breakneck speed.

Personal safety

While we may laugh at the frailty of recommender systems that result in shoe ads that follow us around the internet from Amazon to Facebook, the same root causes pose serious risks to human well-being in higher stakes domains such as healthcare, criminal justice, welfare, and education. The trolley problem and the over-heralded coming of the singularity are ultimately distractions from the challenges already in plain sight, as biased data and flawed algorithms are impacting elections, policing, criminal sentencing, finance, jobs, and even dating.

Product quality

Personalization systems in their current state are forcing users to wrestle with the ramifications of their actions (like and subscribe!) without offering any clear model for how to reconcile lurking questions like “who does this system think I am?” or “what am I committing to becoming?”

When we’re overly conscious of what a system will think of us, we alter our behavior, and in serious cases, we regress our uniqueness back to the cultural mean… or we hide in incognito mode… or we stop using the system altogether (assuming we’re privileged enough to be able to do so). More generically, if we’re distrustful of how a system will operate, we’ll take fewer risks, thereby reducing the dimensionality of what these systems can learn from. Either way, the resulting damage to the diversity and representativeness of training data, and thereby useful personalization, would be irreparable.

Brand value

Advertiser trust has already been significantly impacted by inflammatory associations drawn between their brand and the content it’s appeared next to. Skewed, stereotypical, or outright fake results are being discovered by users on an almost-daily basis, and tech journalists are eager to fan the flames. Meanwhile, AI-powered products, from voice recognition to robot assistants, simply can’t keep up with the grandiose claims being made about their capabilities. Tech firms are starting to look pretty stupid, and that’s only going to get worse if they continue to hinge their brand differentiation on an ability to be all-knowing and all-seeing. It’s simply too precipitous a ledge to be constantly balancing on.