Goal: Fine-tune existing Pegasus summarization model for SAMSum Dialogue dataset.
Model (base) : PEGASUS
trained on CNN/DailyMail summaries
to be fine-tuned on SAMSum dialogues
Dataset: SAMSum people dialogues with summaries
Steps:
Compute ROUGE GPT2, T5, BART on single CNN example.
Evaluate Baseline & Pegasus on entire CNN dataset.
Fine-tune Pegasus on SAMSum. Evaluate & conclude.
Joe : Hi, Bob! I baked cookies. Do you want some?
Bob: Hey Joe, sure! Let's have a breakfast together
Joe: OK, I'll bring you cookies tomorrow :)
Bob: Perfect! I'll prepare the rest
Joe: Sounds good Bob! See you then!
Summary: Joe will bring baked cookies for breakfast with Bob
DATASET 1 : CNN / DailyMail dataset
Train examples: 287,133
Validation examples: 13,368
Test examples: 11,490
Example:
Harry Potter star Daniel Radcliffe gains access to a reported £20 million ($41.1 million) fortune as he turns 18 on Monday, but he insists the money won't cast a spell on him. Daniel Radcliffe as Harry Potter in "Harry Potter and the Order of the Phoenix" To the disappointment of gossip columnists around the world, the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties. "I don\'t plan to be one of those people who, as soon as they turn 18, suddenly buy themselves a massive sports car collection or something similar. He told an Australian interviewer earlier this month. "I don\'t think I\'ll be particularly extravagant. "The things I like buying are things that cost about 10 pounds -- books and CDs and DVDs." At 18, Radcliffe will be able to gamble in a casino, buy a drink in a pub or see the horror film "Hostel: Part II," currently six places below his number one movie on the UK box office chart. Details of how he'll mark his landmark birthday are under wraps. His agent and publicist had no comment on his plans. "I\'ll definitely have some sort of party," he said in an interview. "Hopefully none of you will be reading about it." Radcliffe\'s earnings from the first five Potter films have been held in a trust fund which he has not been able to touch. Despite his growing fame and riches, the actor says he is keeping his feet firmly on the ground. "People are always looking to say \'kid star goes off the rails,\'" he told reporters last month. "But I try very hard not to go that way because it would be too easy for them." His latest outing as the boy wizard in "Harry Potter and the Order of the Phoenix" is breaking records on both sides of the Atlantic and he will reprise the role in the last two films. Watch I-Reporter give her review of Potter\'s latest » . There is life beyond Potter, however. The Londoner has filmed a TV movie called "My Boy Jack," about author Rudyard Kipling and his son, due for release later this year. He will also appear in "December Boys," an Australian film about four boys who escape an orphanage. Earlier this year, he made his stage debut playing a tortured teenager in Peter Shaffer\'s "Equus." Meanwhile, he is braced for even closer media scrutiny now that he\'s legally an adult: "I just think I\'m going to be more sort of fair game," he told Reuters. E-mail to a friend . Copyright 2007 Reuters. All rights reserved.This material may not be published, broadcast, rewritten, or redistributed.
Summary:
Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .
Young actor says he has no plans to fritter his cash away .
Radcliffe's earnings from first five Potter films have been held in trust fund
Rouge scores for summarization
🥇 Ground Truth:
Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .
Young actor says he has no plans to fritter his cash away .
Radcliffe's earnings from first five Potter films have been held in trust fund
3️⃣ Baseline (first 3 sentences)
Harry Potter star Daniel Radcliffe gains access to a reported £20 million ($41.1 million) fortune as he turns 18 on Monday, but he insists the money won't cast a spell on him. Daniel Radcliffe as Harry Potter in "Harry Potter and the Order of the Phoenix" To the disappointment of gossip columnists around the world, the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties
☹️ GPT-2 : <Article> + "TL;DR"
Rudyard Kipling's youngest son is coming of age in \"Harry Potter and the Order of the Phoenix.\"Expect Radcliffe's first Potter film to break box office records in the UK, in spite of poor box office and ratings.If Radcliffe can't hit box office records, can the franchise live on in a different form?Thanks
🙂T5 (trained with CNN included)
the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties .he will be able to gamble in a casino, buy a drink or see the horror film \"Hostel: Part II\" his agent and publicist had no comment on his plans .his latest outing as the boy wizard is breaking records on both sides of the Atlantic
🙂 BART (trained on CNN only)
Harry Potter star Daniel Radcliffe turns 18 on Monday.He gains access to a reported £20 million ($41.1 million) fortune.Radcliffe's earnings from the first five Potter films have been held in a trust fund.Details of how he'll mark his landmark birthday are under wraps
🫡 Pegasus (traiend on CNN only)
Harry Potter star Daniel Radcliffe gains access to a reported £20 million fortune.
Young actor says he has no plans to fritter his cash away.
Radcliffe's earnings from the first five Potter films have been held in a trust fund
Rouge (Model vs Ground) for Single "Harry Potter" example
Rouge (Model vs Ground) on entire CNN dataset
Conclusion: We understand CNN dataset. PEGASUS model was trained on it.
DATASET 2: SAMSum
Example:
Hannah: Hey, do you have Betty's number?
Amanda: Lemme check
Hannah: <file_gif>
Amanda: Sorry, can't find it.
Amanda: Ask Larry
Amanda: He called her last time we were at the park together
Hannah: I don't know him well
Hannah: <file_gif>
Amanda: Don't be shy, he's very nice
Hannah: If you say so..
Hannah: I'd rather you texted him
Amanda: Just text him 🙂
Hannah: Urgh.. Alright
Hannah: Bye
Amanda: Bye bye
Ground Truth summary:
Hannah needs Betty's number but Amanda doesn't have it. She needs to contact Larry.
Pegasus (FT:CNN) zero-shot summary:
Amanda: Ask Larry
Amanda: He called her last time we were at the park together.
Hannah: I'd rather you texted him.
Amanda: Just text him .
Rouge Pegasus(FT:CNN) on SAMSum['test']
Remarks:
Model tries to summarize by finding and extracting the key sentences.
This is EXPECTED for CNN/DailyMail summary but NOT for SAMSum
We need to finetune Pegasus on SAMSum: dialog -> abstract summary
Conclusion: PEGASUS itself is not adapted for dialogue summaries, it needs to be fine-tuned
Fine-Tune Pegasus on SAMSum
Pegasus (FT:SAMSum) zero-shot summary:
Amanda can't find Betty's number. Larry called Betty last time they were at the park together. Hannah wants Amanda to text Larry. Amanda will text Larry.
Rouge Pegasus(FT:SAMSum) on SAMSum['test']
Conclusion:
ROUGE of fine-tuned PEGASUS improved considerably
For the same task (summarization), FT helps adapt model to specific domain
Conclusion
Text summarization is a free form token generation task with high risk of garbage output
metrics like accuracy / CE don't reflect the quality of summary. Bleu & Rogue better correlate with human score
Fine-tuning same task models helps adapt them to specific domain (like dialogues)