Text Summarization

By Diya Mathew on

 

           Text Summarization



With the rapid increase of textual data over the past few years, a valuable resource for extracting and analyzing data has emerged. In order to retrieve useful knowledge within a reasonable amount of time, it is necessary to summarize this information. Therefore, the demand for automatic text summarization systems is on the rise. Since it is becoming increasingly important to receive information in a concise and easily understood form, summarization is among the prime problems in Natural Language Processing (NLP). Summarization is a complex and challenging task because, as part of summarization, the model needs to identify the context of the given text and ensure that the generated sentences are grammatically correct and readable. A second challenge is determining which information to use for the summary and which to omit. The summarization process is undergoing much research, but the generated summary quality needs improvement. Recent approaches to summarization use Deep Learning-based transformer architectures.

An automatic text summarization keeps the main points and overall meaning while ensuring a concise and fluent summary.

There are two different approaches to automatic text summarization

1. Abstractive Text Summarization (ATS)

2. Extractive Text Summarization (ETS)

Abstractive Text summarization (ATS)

By using this approach, new phrases are generated that capture the meaning of the input sentence. For the summary, an abstractive approach generates grammatically correct sentences. That is why abstractive summarization is a complex process using Deep Learning Recurrent Neural Networks (RNN) architecture. During abstract summarization, words and sentences are constructed and assembled in a clear, comprehensible fashion, and then only the key facts from the source text are added. This way, abstractive summarization approaches are more sophisticated and computationally costly than extractive summarization approaches.

Extractive Text Summarization (ETS)

This approach extracts relevant words/phrases from the input sentence. The summary is created by choosing the most important points from the text input. An extractive approach uses only sentences contained in the input text for the summary. That is, in the extractive summarization, key sentences or phrases from the source documents are extracted and grouped to generate a summary without modifying the original document.

As the Transformer was initially designed and tested for machine translation, it is reasonable to assume that it would also perform well on the abstractive summarization task because of the similarity between machine translation and automatic summarization. The commonly used pre-trained language models for text summarization are Bidirectional Auto-Regressive Transformers(BART), Bidirectional Encoder Representations from Transformers(BERT), Text-to-Text-Transfer-Transformer(T5) and Pre-training with Extracted Gap-sentences for Abstractive SUmmarization Sequence-to-sequence(PEGASUS).

Text Summarization Evolution

Text summarization models’ evolution starts from non-neural statistical models, and then machine learning models and deep sequence-to-sequence models are established. The deep sequence-to-sequence models implement different techniques to improve summarization performance, such as the pointer-generator network and attention mechanisms. Finally, with the implementation of transformer architecture, transfer learning and pre-trained language models, the machine-generated summaries become closer to man-made summaries (Alomari et al., 2022). The figure below shows the evolution of different summarization methods.

                       Evolution of summarization methods (Source: (Alomari et al., 2022)) 

 References:

Alomari, A., Idris, N., Sabri, A.Q.M., Alsmadi, I., 2022. Deep reinforcement and transfer learning for abstractive text summarization: A review. Comput. Speech Lang. 71, 101276. https://doi.org/10.1016/j.csl.2021.101276


Comments

Post a Comment

Popular posts from this blog

Transfer Learning Vs Fine Tuning

By Diya Mathew on
Image
Transfer Learning Transfer Learning involves adapting knowledge attained from one task to resolve other associated tasks we haven't faced before. When it comes to machine learning, it is possible to transfer knowledge from one domain/task to another.   Fine Tuning Transfer learning can be achieved through fine-tuning. The pretraining process starts with random weights and no prior knowledge, and the model is trained from scratch. A large amount of data is usually used for this pretraining. As a result, it requires a large amount of data, and training may  take several weeks. It also causes an environmental impact.  As opposed to pre-trained models, fine-tuned models are trained after they have been pretrained. The fine-tuning process begins by acquiring a pre-trained  language model, followed by additional training with the chosen dataset. The cost of fine-tuning a model is lower in terms of time, data, and finances. Training is also less constraining than full pretraining, so it i
Read more

AutoML

By Diya Mathew on
Image
                      AutoML Automated machine learning(AutoML) is a process to fully automates the end-to-end of the machine learning process and makes it more usable by intended users. AutoML is also related to AI and the introduction of AutoML in Robotics is a boom in technology. AutoML tools will outperform the manual machine learning (ML) modelling, with high robustness then only it has relevance. How to provide robustness while dealing with real-world datasets, with considerable time constraints is a relevant question in this area. TPOT, Auto-sklearn, Auto-keras and H2O-Automl are some of the open-source AutoML tools and Google cloud Automl and Microsoft AzureML are commercial AutoML Tools. To compare the robustness of the open-source AutoML tools many researchers take different datasets from openML and conduct studies based on different criteria. They use quantitative measurement based on time to understand the robustness of these AutoML tools. These experiments show, that diffe
Read more

Pre-trained Language Models (PTLM) in NLP

By Diya Mathew on
Image
  Pre-trained Language Models (PTLM) in NLP The main purpose behind Natural Language Processing(NLP) is to make the computer system capable to understand and communicate the language as a human. NLP tasks are challenging because it is not only important to understand individual words, but also to understand the context in which they are used. According to NLP, each word has a meaning, but the meaning is strongly influenced by the context in which it is used. The Language Model is a machine learning model that predicts the next word based on the parts of a sentence. A pre-trained language model has been trained on a large corpus of data. Through this training, the model learns the language’s general rules for word usage and how it is written. The model is then trained with a task-specific dataset.                                                        Pretrained language model concept Text-to-Text Transfer Transformer Model(T5) T5 (Raffel et al., 2019) is Google’s state-of-the-art (SOTA
Read more