ROUGE Metric

By Diya Mathew on




The Recall-Oriented Understudy for Gisting Evaluation (ROUGE)(Lin et al., 2004) is a metric, commonly used to measure the accuracy of text summarization models. Summarization is one of the most difficult tasks to measure because there is no straightforward way to do it. The ROUGE score is one of the most used metrics for summarization. This metric compares a generated summary to a set of human-created reference summaries. Using ROUGE, recall is calculated based on how much of the reference summary is represented by the generated summary. ROUGE refers to a set of metrics. Those most likely to be used are ROUGE-N, ROUGE-L, and ROUGE-LSUM. The ROUGE-N metric measures how many ‘n-grams’ match between our model-generated text and the reference text. In simple terms, n-grams are groups of words or tokens. There is only one word in a unigram (1-gram) and two consecutive words in a bigram(2-gram). ROUGE-N uses n-grams to represent the gram. In ROUGE-1, we are measuring the match-rate of unigrams from our model output to the reference. In ROUGE-L, the longest common subsequence (LCS) between output and reference is calculated.
The ROUGE value can be calculated by three methods Recall, Precision and F1 score.
Recall: Using the recall method, the model output and reference are compared for the number of overlapping n-grams, and then this number is divided by the total number of n-grams in the reference.
Recall = Word count in reference summary / Word count of overlapping words in the reference.
Precision: ROUGE uses this measure to determine the relevance of the generated summary. Instead of dividing by the reference n-gram count, we divide by the output n-gram count when calculating precision.
Precision = Word count in generated summary/Word count of overlapping words in the generated summary
F1-Score: It gives a reliable measure of the performance of generated output based on recall and precision values.
F1 score = 2 * ((precision * recall) / (precision + recall))
Reference:
Lin, Chin-Yew., 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Comments

Post a Comment

Popular posts from this blog

Transfer Learning Vs Fine Tuning

By Diya Mathew on
Image
Transfer Learning Transfer Learning involves adapting knowledge attained from one task to resolve other associated tasks we haven't faced before. When it comes to machine learning, it is possible to transfer knowledge from one domain/task to another.   Fine Tuning Transfer learning can be achieved through fine-tuning. The pretraining process starts with random weights and no prior knowledge, and the model is trained from scratch. A large amount of data is usually used for this pretraining. As a result, it requires a large amount of data, and training may  take several weeks. It also causes an environmental impact.  As opposed to pre-trained models, fine-tuned models are trained after they have been pretrained. The fine-tuning process begins by acquiring a pre-trained  language model, followed by additional training with the chosen dataset. The cost of fine-tuning a model is lower in terms of time, data, and finances. Training is also less constraining than full pretraining, so it i
Read more

AutoML

By Diya Mathew on
Image
                      AutoML Automated machine learning(AutoML) is a process to fully automates the end-to-end of the machine learning process and makes it more usable by intended users. AutoML is also related to AI and the introduction of AutoML in Robotics is a boom in technology. AutoML tools will outperform the manual machine learning (ML) modelling, with high robustness then only it has relevance. How to provide robustness while dealing with real-world datasets, with considerable time constraints is a relevant question in this area. TPOT, Auto-sklearn, Auto-keras and H2O-Automl are some of the open-source AutoML tools and Google cloud Automl and Microsoft AzureML are commercial AutoML Tools. To compare the robustness of the open-source AutoML tools many researchers take different datasets from openML and conduct studies based on different criteria. They use quantitative measurement based on time to understand the robustness of these AutoML tools. These experiments show, that diffe
Read more

Pre-trained Language Models (PTLM) in NLP

By Diya Mathew on
Image
  Pre-trained Language Models (PTLM) in NLP The main purpose behind Natural Language Processing(NLP) is to make the computer system capable to understand and communicate the language as a human. NLP tasks are challenging because it is not only important to understand individual words, but also to understand the context in which they are used. According to NLP, each word has a meaning, but the meaning is strongly influenced by the context in which it is used. The Language Model is a machine learning model that predicts the next word based on the parts of a sentence. A pre-trained language model has been trained on a large corpus of data. Through this training, the model learns the language’s general rules for word usage and how it is written. The model is then trained with a task-specific dataset.                                                        Pretrained language model concept Text-to-Text Transfer Transformer Model(T5) T5 (Raffel et al., 2019) is Google’s state-of-the-art (SOTA
Read more