Welcome to the age of machine-generated headlines
- How to write news headlines
- Can we write document summaries?
- The simplest summarization model
That is a machine-generated summary of this article. I fed the text through a deep learning model that we’re training here at Primer where we’re building machines that read and write. The three-bullet point summary above took less than a second running on a single 2080Ti GPU.
Now you don’t have to read it. You’re welcome! (To understand the machine-generated title, you’ll have to read to the end.)
The base model that makes this possible is laid out in our recently published paper. We fine-tuned the BERT universal language model on millions of news headlines. The model learned to generate titles for documents as an iterative question-answer task. It takes about 1.5 million news articles for the model to learn how to generate headlines as good or better than the originals in the majority of cases.
Question: Can a neural network write headlines as good as those written by humans?
Answer: Yes, especially when the headlines are clickbait.
Journalists have conflicting motivations when writing headlines. The main task is to inform the reader, giving her a short and meaningful description of what the document contains. But much of the success of online news relies on clicks. Headlines must entice people to read stories that display the ads that pay the bills. Luckily, clickbait headlines are a very small portion of Primer’s news corpus—far less than 1% of our hundreds of millions of English-language news articles. The model learned its writing style from good journalism.
This is what it looks like when a machine re-writes clickbait headlines:
|human||The Obamas Are Getting a Netflix Show, & You Won’t Believe Who It’s For|
|machine||The Obamas are getting a show for kids|
|human||You’ll never guess which U.S. counties grew the fastest|
|machine||McKenzie County in North Dakota had highest population growth in entire US|
|human||The Title Of This Court Filing Will Amaze You!|
|machine||BuzzFeed sued for defamation of Russian hacking defendants|
Here is a more representative sample of news articles. In the majority of cases, the machine-generated headlines are as good or better than the originals, as scored by humans in double-blind trials:
|human||Bad Movie Diaries: A Christmas Prince: The Royal Wedding (2018)|
|machine||Jim Vorel and Kenneth Lowe discuss A Christmas Prince and its sequel, The Royal Wedding|
|human||ABC chairman job: Academic, legal and media executives make the shortlist|
|machine||Former Allens partner Michael Rose among candidates for ABC chairman|
|human||‘It’s been a long 2 years’ for Kelly, Kudlow says|
|machine||John Kelly replacement to be announced in next few days|
|human||Decision in March on 48-team 2022 World Cup, says Infantino|
|machine||FIFA to decide on 2022 World Cup in March|
|human||Khal Drogo Is Alive—Sort of—on This Week’s ‘Saturday Night Live’|
|machine||Khal Drogo is back on SNL|
Ideally, a document’s title should serve as an extremely short but informative summary of its content. Our title generation model is lazy, reading the text of a document only until it finds what seems to be sufficient information to generate that one-line summary.
When the model falls short, it is often because it is confused by longform journalism which is rare in its training data. It focuses on information near the top of the story, simply because that is where most news articles get to the point. Longform journalism has a different structure:
|human||I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here’s How.|
|machine||People lost weight faster if they ate a chocolate bar every day|
|human||Who’s downloading pirated papers? Everyone|
|machine||How to read the scientific papers|
|human||Gamers unravel the secret life of protein|
|machine||Protein chemistry: David Baker|
We could improve our generated titles with rules. But the best way forward is to build a better model by training it on more abundant, higher-quality data. Luckily we have several orders of magnitude more news articles on tap.
Question: Can a title-generating neural network be repurposed to write full document summaries?
Answer: Surprisingly, yes.
For the past few weeks we have been playing with a simplified model. We feed a document into the title generation model and track how deeply it reaches into the text to write a title. Then we feed in the text below that point and generate another. Each “title” becomes the next line in a bullet-point summary. When it reaches the end of the text, the summary is done.
In spite of being completely unsupervised and requiring no additional training, this naive summarization algorithm performs reasonably well right out of the gate.
Here are some human vs. machine-generated summaries, drawing from the CNN/Daily Mail data set:
Police: Teen wanted for killing at Seattle bar hosting Seahawks, fans
Man, 37, arrested in probe of ricin-laced letters found in Washington state
Deep-voiced bosses bring in the big bucks, says study
Remarkably, the model can summarize non-news documents, even though it’s never seen examples. We need this capability at Primer because our customers have their own documents they need to rapidly skim—financial analyst reports, scientific papers, intelligence briefings—before choosing the ones to fully read.
Wikipedia movie plot overviews are a good open-source proxy. This is how our prototype model performs:
The errors that the model makes involve entities. It correctly identifies key events that should be included in the summary, but it incorrectly guesses who is involved. What’s needed is a fact-aware text generation system: a knowledge base that can read and write. By pre-annotating the input text with entity embeddings from Quicksilver, our self-updating knowledge base, our model can learn to keep better track of who’s who.
In Top Gun, Maverick begins a romantic relationship with his flight instructor Charlie, not Goose his copilot. But perhaps we should treat this as a deeper truth and a glimpse at the future. As machine understanding becomes ever more sophisticated, we humans will come to rely on machines that reveal truths deeper than we can see.