Old cartoons can seem very formulaic and predictable. There are no big surprises, no twists, no cliffhangers. The stories usually fit a similar pattern, and everything is always resolved right before the end of the episode. That systematic predictability makes them the perfect test subject for a machine learning approach to understanding the storylines in film.
Meet the Flintstones Scene Dataset (FSD), created by computer scientists at Ben-Gurion University in Israel. This is a collection of more than 1500 scenes taken from several episodes of the Flintstones, which have been carefully described and labeled.
The tedious work of looking at every scene and tagging elements of the story was done by humans, because computers can’t yet process what’s shown on screen. (Which, incidentally, is why online memes along the lines of “people trained an AI to watch a TV show and this is the script it wrote” are never real.)
The annotated dataset of Flintstones scenes has turned a visual cartoon into neat categories of text describing specific scenes and situations. This is the type of information a computer does understand, and in a recent paper, the researchers describe how they were able to use the dataset to identify recognizable narrative structures. This works well with formulaic cartoons like the Flintstones because the short episodes all have well-defined narrative structures, not unlike the classic three-act structure of set-up, confrontation and resolution that dates all the way back to the works of Aristotle.
If a computer analysis of the Flintstones dataset is able to identify narrative structure, that suggests that it might be possible to do something similar with other TV shows and films, perhaps with more complex narrative structures. This kind of work could for example improve recommendation services, and make it possible to predict what kind of stories people like based on the narrative, rather than on the kind of vague resemblances that now form part of these recommendations. (For example, Netflix has been recommending that I watch baking shows because I watched the show Nailed It, without recognizing that it might be the comedic premise of the show – not the mere presence of cakes – that formed part of the appeal.)
The idea that narrative is something you can analyse with a computer algorithm is not new. Writer Kurt Vonnegut famously described a few different story shapes that he believed would fit all existing stories. He even said “There is no reason why the simple shapes of stories can’t be fed into computers,” and that’s exactly what researchers are doing now.
A few years ago, researchers from the University of Vermont analysed the text of more than 1700 novels and other works of fiction and used machine learning to measure how the sentiment of the stories changed as the work progressed. They were able to identify patterns corresponding to Vonnegut’s story shapes.
This kind of research is a lot harder to do with visual media like TV or film, for reasons mentioned above. To get a computer to analyse these changes in sentiment or specific patterns in the narrative, video first needs to be converted to text – and that’s about much more than simply feeding in a script. The cinematography and music make up a large part of the story as well, and are not easily turned into written descriptions, but a tagged dataset like the Flintstones Scene Dataset does make it possible to run machine analysis.
The paper describing the Flintstones Scene Dataset was published in the journal PLOS One, which has also gathered several other papers about the science of stories in a themed collection. This suggests that it’s a research area that scientists are becoming more interested in.
Even though we tend to think of fiction as being completely unrelated to scientific analysis or machine learning, it looks like Vonnegut was right: You can study stories with computers.
Credit: Google News