Adaptive Sparse Transformer untuk Meningkatkan ROUGE-1 Score pada Text Summarization Scientific Paper

Andrew Firman Saputra, Liliana Liliana, Djoni Haryadi Setiabudi


Technology advancement and internet causes lots of information that can be accessed at any time. Journal article is one of such many information that’s available that requires time to read thereof in need of automatic summary. Automatic Text Summarization (ATS) basically a process of making a new text that’s smaller than the original text without removing the meanings from the entire input text. The process of making automatic text summarization can be done in extractive and abstractive way. A summary that was made by an extractive method only able to generate a summary with a word that’s included in the original text, whereas summary that was made by an abstractive method can generate a summary that include word that does not exist in the original text. In the previous research in abstractive summarization is found is not optimal thereof need an improvement. The method used in this research is an abstractive summarization with Adaptive Sparse Transformer. Things that will be done in this research are scraping dataset arxiv machine learning, making the dataset, processing the data and trials on hyperparameter configuration in the model to see ROUGE-1 precision performance. The dataset used is Arxiv Scientific Paper dataset and Arxiv Scientific Paper+Machine Learning dataset. The results of this research showed that the method used capable to compete with state of the art methods with average R-1 precision score of 39.4 for Arxiv Scientific Paper+MachineLearning and 42.5 for Arxiv Scientific Paper.


text summarization; deep learning; transformer; encoder; decoder

Full Text:



Cachola, I., Lo, K., Cohan, A. & Weld, D.S. 2020. TLDR:

Extreme Summarization of Scientific Documents. Allen

Institute for AI.


Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S.,

Chang, W. & Goharian, N. 2018. A discourse-aware attention

model for abstractive summarization of long documents.

Association for Computational Linguistics (ACL), 615-621 .


Correia, G. M., Niculae, V. & Martins, A. F. T. 2020.

Adaptively sparse transformers. Association for

Computational Linguistic (ACL), 2174–2184.


El-Kassas, W. S., Salama, C. R., Rafea, A. A., & Mohamed,

H. K. 2020. Automatic Text Summarization: A

Comprehensive Survey. Expert Systems with Applications,

, 113679.


Huang, D., Cui, L., Yang, S. Bao, G., Wang, K. Xie, J. &

Zhang, Y. 2020. What Have We Achieved on Text

Summarization?. School of Engineering, Westlake

University. DOI=

Ju, J., Liu, M., Gao, L. & Pan, S. 2020. SciSummPip: An

Unsupervised Scientific Paper Summarization Pipeline.

Association for Computational Linguistics (ACL), 318–327.


Peters, B., Niculate, V. & Martins, A. F. T. 2019. Sparse

Sequence-to-Sequence Models. In Proceedings of the 57th

Annual Meeting of the Association for Computational

Linguistics, 1504–1519.


Pilault, J., Li, R., Subramanian, S. & Pal, C. 2020. On

Extractive and Abstractive Neural Document Summarization

with Transformer Language Models. Association for

Computational Linguistics (ACL), 9308–9319.


Sun, X. & Zhuge, H. 2018. Summarization of Scientific

Paper Through Reinforcement Ranking on Semantic Link

Network. IEEE Access, 2018-Vol6, 40611-40625.


Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,L.,

Gomez, A. N., Kaiser, Ł., & Polosukhin, I. 2017. Attention

Is All You Need. Advances in Neural Information Processing

Systems, 2017-December, 5999-6009.


Verma, S. & Nidhi, V. 2017. Extractive Summarization using

Deep Learning. Delhi Technological University.



  • There are currently no refbacks.

Jurnal telah terindeks oleh :