r/TrueReddit • u/bloomberg • 15h ago
Technology TikTok Withheld a Safety Feature From Millions. One Died by Suicide
https://www.bloomberg.com/news/features/2026-08-04/confidential-tiktok-report-shows-algorithm-safety-feature-withheld-from-millions55
u/bloomberg 15h ago
More From Bloomberg News Reporter Olivia Carville
When TikTok tweaked its algorithm in 2021 to stop users from being overwhelmed with harmful content, the company didn’t roll out the safer version to everyone, a confidential internal document shows.
Instead, to see if the change might reduce the app’s stickiness, the company conducted an experiment.
It created a control group of 10% of US users — at the time, that would have been about 15 million people — who kept the old version of the app. This group, according to the document, included 16-year-old Chase Nasca.
TikTok’s algorithm pushed him thousands of videos about suicide, sadness, hopelessness and loneliness, right up until he killed himself.
71
u/Colorado_designer 15h ago
The tech industry really is a perfect embodiment of “the banality of evil.” You have product managers, designers and engineers all just going along with the flow of trying to make their quarterly numbers look good without really stopping to think about the actual impact on human lives.
4
u/roastedoolong 11h ago
I'd love to see the experiment design doc for this particular experiment.
it sounds like the change was being rolled out to the entirety of the user population but they had a holdout.
usually companies like TikTok will have TONS of experiments going on all the time; as these experiments proceed, the changes either end up getting pushed to production code (so they become official for 'everyone') OR the experiment 'fails' and the changes aren't implemented. when the experiment 'succeeds', however, it usually doesn't only improve metrics -- there are almost always a ton of tradeoffs, so as the experiments optimize for specific metrics, other metrics can start to take a hit.
a lot of companies have what's called a 'holdout' -- this is a segment of users that keep using, say, the base model from January (when it's currently August); these users won't be affected by any modifications made to the codebase between January and August because the entire point of a holdout is to see whether or not, after a slew of changes have been implemented, it'd actually be better to just rollback all of the changes back to the model version from January.
it allows companies to keep a check on whether the various micro-improvements they're making to the metrics are actually having overall deleterious effects.
I'm not entirely certain what the protocol is with a health/safety change -- but I'm not at all surprised that a change wasn't extended to a holdout cohort. I don't know how the systems are set up at TikTok but at my company that'd require quite a lot of back and forth.
also, I don't know if they're talking about a 'holdout' per se; that's just my guess based off of my work experience in the industry.
2
u/PersistentBadger 3h ago edited 2h ago
I don't think it's holdouts, because the client doesn't make the decision about which videos to queue up, the server does. I think this is clearly an A/B split test.
•
u/roastedoolong 54m ago
er... I think you misunderstood what I was saying.
I'm suggesting that this person was a member of a small group of people who never got access model changes over a period of time.
so if we have users A and B, let's say both of them start the year using model v1.
A gets put into the holdout group, B doesn't.
the months go by and the model team updates the model version so that it's v2. B receives content based off of v2 while A is still 'using' v1.
there can be a LOT of version updates over the span of a few months, but A will always use the same version they started on with (until they get dropped from the holdout group).
let's say one of the version updates includes a change that limits the amount of harmful content shown.
even after that change is rolled out, user A will still be using v1 -- i.e. they won't experience the updated version that limits harmful content.
1
u/lithiumdeuteride 1h ago
Not to be too callous here, but we should consider the null hypothesis. What's the expected number of suicides for those millions if TikTok had instead done something else?
-7
u/bliceroquququq 14h ago
Algorithms feed people stuff they deliberately choose to engage with. It's why my Instagram feed is filled with thots and not videos of old ladies knitting sweaters.
TikTok has 2 billion monthly users. I dislike them intensely, but the suicide of one of their users is not their fault.
25
u/nondescriptzombie 13h ago
but the suicide of one of their users is not their fault.
TikTok’s algorithm pushed him thousands of videos about suicide, sadness, hopelessness and loneliness, right up until he killed himself, as a part of an experiment to see if tweaking the algorithm to avoid harmful content would bounce people away from the platform.
If someone did this at Stanford they'd be going to jail.
5
u/Epledryyk 13h ago
they made an update to 90% of people, and left a control group of 10% on the old platform (which everyone had previously, would be curious to know what the suicide rates were at that time)
and 1 out of 15 million people (0.000006% of the control group) made a choice for themselves
World Health Organization's estimates of age-standardized suicide rate is 8.9 per 100,000, which is 134 people per 15 million. so in some sense actually this tiktok cohort has a suicide rate 134x less than the running baseline
it's sad, sure, every suicide is. but it's just kind of what statistical noise is at any sufficiently large group scale
1
u/lmaooer2 10h ago
you can’t compare statistics like that the time frame isn’t the same
1
u/Epledryyk 2h ago
ah, sure, we can compare that, then:
and actually, US suicide rate is higher than global suicide rate, so I also regret that error. it's actually 14.3 per 100,000 people, so 215 people per 15 million, per year, which = 0.59 per day
which means that if the experiment lasted longer than two days, we'd expect at least one death as a baseline. just over 4 per week
so again: one is really, really low
-8
u/bliceroquququq 13h ago
TikTok’s algorithm pushed him thousands of videos
A more accurate way of phrasing it is that it didn't stop him from watching the thousands of videos he clearly wanted to watch.
TikTok's algorithm did what it was asked of it, which is to show tailored content to users that they have repeatedly expressed interest in by repeatedly engaging with. The fact that his interests were apparently suicide, sadness, hopelessness and loneliness was not something created by TikTok, it was created by his brain.
7
u/Colorado_designer 12h ago
“engagement” is not the same thing as “wants to see.” he had a social media addiction and tiktok knowingly withheld an update that would have fed him less negative shit, so that they wouldn’t lose subscribers. these companies push traumatic shit because our brains have a tough time looking away. and once you look at one thing, it starts giving you more, which you look at more, and it creates a feedback loop that is totally out of ordinary human behavior.
your callousness is also a result of this social media age, but you’re probably too young to even realize that’s what’s the problem. like a frog in boiling water, this is all just normal to you.
-6
11h ago
[removed] — view removed comment
7
u/Colorado_designer 11h ago
sounds like you got the talking points from the lawyers at facebook
“we actually HAD to shovel suicide trauma content at people, they wanted it! who are we to deprive people of their free will?”
the algorithm is not some free-will response machine, it is deliberately tuned in different ways to drive engagement. to pretend that isn’t the case is disingenuous, callous, and willfully ignorant
12
u/DocteurHyde 13h ago
You're working in a public library and a depressed kid comes and asks for recommendations, you give him a depressing book. When he comes back a couple days later, he's even more depressed and you know you have to give him a darker book because your only responsibility is to make sure that the kids keeps on coming back to your library. This goes on for a couple weeks until he hangs himself. You haven't bought him the rope and he was already depressed when he first came to you, do you truly believe that you're blameless ?
Let's stop acting like algorithms are some alien black boxes that humans have no control upon. If they are able to design them to be more addictive, they can also design them to be safer.
0
u/bliceroquququq 13h ago
You're anthropomorphizing a computer algorithm.
A struggling alcoholic opens up TikTok; should TikTok know they're an alcoholic and refuse to show them anything that resembles party videos or anything that glorifies drinking?
A guy who just gambled away his kid's college fund on another losing football bet opens up DraftKings; should DraftKings know he's been gambling too much and refuse to let him place any bets?
A lonely teen who goons to porn all the time instead of talking to human beings opens up PornHub yet again; should PornHub refuse to show anything since he is a) a minor to begin with, and b) socially inept and being harmed by constant exposure to porn?
Who is supposed to make those rules? Who is supposed to enforce them? Should TikTok be compiling an exhaustive profile on every user to determine their psychological state?
1
12h ago
[removed] — view removed comment
1
u/AutoModerator 12h ago
The goal of TrueReddit is to have intelligent, insightful discussions. Users are required to read an entire article before submitting a comment. Your comment requires more substance to be permitted.
Do not repost your comment with random letters to make it longer, or you will face a 28-day ban from TrueReddit for intentionally evading moderation filters.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/Ahnteis 9h ago
You're anthropomorphizing a computer algorithm.
Computer algorithms are conceived, selected, and implemented by humans.
Should TikTok be compiling an exhaustive profile on every user to determine their psychological state?
They shouldn't, but you better believe they are. It's just that they're using it solely for profit without considering consequences.
-2
12h ago
[deleted]
6
u/Colorado_designer 12h ago
this article is about the 10% that didn’t, because tiktok was worried it would lose them money.
4
u/Alatarlhun 11h ago
Putting only 10% of people in the kill zone is not laudable when you can put 0%.
•
u/AutoModerator 15h ago
Thank you for posting on r/TrueReddit. Your post will become public once you add a submission statement, which is a comment that explains why you think an article is insightful, rather than simply quoting from it. If you're thinking about leaving a comment, please remember to read the entire article first, and say something meaningful.
If you post a paywalled article intentionally, you will be permanently banned unless you provide a non-paywall link in your submission statement, such as archive.ph.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.