When it comes to watching television, viewers typically watch everything in the same order — the show will either have a certain broadcast order, or its story will be such that watching it out of order would turn it into nonsense. With its new anthology series Love, Death + Robots, Netflix tried something new: testing out different episode orders to see which performed the best.
For such an anthology series, where each episode stands on their own, you don’t have to worry about making sure that you’re watching things in any particular order — there isn’t one. But how you watch might change how you experience the episodes.
After viewers noticed episodes shifting from viewer to viewer, Netflix explained the discrepancy in a tweet. “We’ve never had a show like Love, Death & Robots before, so we’re trying something completely new,” the official accounts said, “presenting four different episode orders.”
Netflix is well known for its rigorous a/b testing practices, presenting broadly different page layouts and results to different users to see which performs best. That practice is most intense for content recommendations, but also extends to thumbnails and other aspects of the product. Despite some experiments in interactivity, that testing has never extended into the video itself, and this latest ordering experiment is the closest Netflix has come to actually changing the content of a program from viewer to viewer.
Surveying our staff this week, we discovered only two distinct episode orders — one that started with the episode “Sonnie’s Edge,” and another that began with “Beyond the Aquila Rift” — although it’s possible other orders were tested earlier in the release and discarded for poor performance.
Netflix’s experiments have gotten it in trouble in the past. In October, the company faced blowback over concerns that its thumbnail previews were targeting users by race, highlighting black characters in a given program when the user was black. Netflix denied the claim, saying thumbnails were algorithmically generated and that the company does not target users demographically.
The episode reordering ran into similar concerns. When the variation was first discovered, some speculated that the episode order changed based on their gender or sexual orientation — “Sonnie’s Edge” depicts a lesbian relationship, while “Beyond the Aquila Rift” depicts a heterosexual one. Netflix pushed back on those allegations, saying that it doesn’t specifically collect that type of information, and The Verge’s own testing discovered no evidence of targeting by sexuality.