Sparse Progressive Neural Networks for Continual Learning

Esra Ergün; Behçet Uğur Töreyin

doi:10.1007/978-3-030-88113-9_58

Sparse Progressive Neural Networks for Continual Learning

Esra Ergün^*, Behçet Uğur Töreyin

^*Corresponding author for this work

Informatics Institute

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Citation (Scopus)

Abstract

Human brain effectively integrates prior knowledge to new skills by transferring experience across tasks without suffering from catastrophic forgetting. In this study, to continuously learn a visual classification task sequence, we employed a neural network model with lateral connections called Progressive Neural Networks (PNN). We sparsified PNNs with sparse group Least Absolute Shrinkage and Selection Operator (LASSO) and trained conventional PNNs with recursive connections. Later, the effect of the task prior on current performance is investigated with various task orders. The proposed approach is evaluated on permutedMNIST and selected subtasks from CIFAR-100 dataset. Results show that sparse Group LASSO regularization effectively sparsifies the progressive neural networks and the task sequence order affects the performance.

Original language	English
Title of host publication	Advances in Computational Collective Intelligence - 13th International Conference, ICCCI 2021, Proceedings
Editors	Krystian Wojtkiewicz, Jan Treur, Elias Pimenidis, Marcin Maleszka
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	715-725
Number of pages	11
ISBN (Print)	9783030881122
DOIs	https://doi.org/10.1007/978-3-030-88113-9_58
Publication status	Published - 2021
Event	13th International Conference on Computational Collective Intelligence, ICCCI 2021 - Virtual, Online Duration: 29 Sept 2021 → 1 Oct 2021

Publication series

Name	Communications in Computer and Information Science
Volume	1463
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	13th International Conference on Computational Collective Intelligence, ICCCI 2021
City	Virtual, Online
Period	29/09/21 → 1/10/21

Bibliographical note

Publisher Copyright:
© 2021, Springer Nature Switzerland AG.

Keywords

Continual learning
Progressive Neural Networks
Sparse group LASSO regularization

Access to Document

10.1007/978-3-030-88113-9_58

Cite this

Ergün, E., & Töreyin, B. U. (2021). Sparse Progressive Neural Networks for Continual Learning. In K. Wojtkiewicz, J. Treur, E. Pimenidis, & M. Maleszka (Eds.), Advances in Computational Collective Intelligence - 13th International Conference, ICCCI 2021, Proceedings (pp. 715-725). (Communications in Computer and Information Science; Vol. 1463). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-88113-9_58

Ergün, Esra ; Töreyin, Behçet Uğur. / Sparse Progressive Neural Networks for Continual Learning. Advances in Computational Collective Intelligence - 13th International Conference, ICCCI 2021, Proceedings. editor / Krystian Wojtkiewicz ; Jan Treur ; Elias Pimenidis ; Marcin Maleszka. Springer Science and Business Media Deutschland GmbH, 2021. pp. 715-725 (Communications in Computer and Information Science).

@inproceedings{794633964d0b45839d8364045502f520,

title = "Sparse Progressive Neural Networks for Continual Learning",

abstract = "Human brain effectively integrates prior knowledge to new skills by transferring experience across tasks without suffering from catastrophic forgetting. In this study, to continuously learn a visual classification task sequence, we employed a neural network model with lateral connections called Progressive Neural Networks (PNN). We sparsified PNNs with sparse group Least Absolute Shrinkage and Selection Operator (LASSO) and trained conventional PNNs with recursive connections. Later, the effect of the task prior on current performance is investigated with various task orders. The proposed approach is evaluated on permutedMNIST and selected subtasks from CIFAR-100 dataset. Results show that sparse Group LASSO regularization effectively sparsifies the progressive neural networks and the task sequence order affects the performance.",

keywords = "Continual learning, Progressive Neural Networks, Sparse group LASSO regularization",

author = "Esra Erg{\"u}n and T{\"o}reyin, {Beh{\c c}et Uğur}",

note = "Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; 13th International Conference on Computational Collective Intelligence, ICCCI 2021 ; Conference date: 29-09-2021 Through 01-10-2021",

year = "2021",

doi = "10.1007/978-3-030-88113-9_58",

language = "English",

isbn = "9783030881122",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "715--725",

editor = "Krystian Wojtkiewicz and Jan Treur and Elias Pimenidis and Marcin Maleszka",

booktitle = "Advances in Computational Collective Intelligence - 13th International Conference, ICCCI 2021, Proceedings",

address = "Germany",

}

Ergün, E & Töreyin, BU 2021, Sparse Progressive Neural Networks for Continual Learning. in K Wojtkiewicz, J Treur, E Pimenidis & M Maleszka (eds), Advances in Computational Collective Intelligence - 13th International Conference, ICCCI 2021, Proceedings. Communications in Computer and Information Science, vol. 1463, Springer Science and Business Media Deutschland GmbH, pp. 715-725, 13th International Conference on Computational Collective Intelligence, ICCCI 2021, Virtual, Online, 29/09/21. https://doi.org/10.1007/978-3-030-88113-9_58

Sparse Progressive Neural Networks for Continual Learning. / Ergün, Esra ; Töreyin, Behçet Uğur.
Advances in Computational Collective Intelligence - 13th International Conference, ICCCI 2021, Proceedings. ed. / Krystian Wojtkiewicz; Jan Treur; Elias Pimenidis; Marcin Maleszka. Springer Science and Business Media Deutschland GmbH, 2021. p. 715-725 (Communications in Computer and Information Science; Vol. 1463).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Sparse Progressive Neural Networks for Continual Learning

AU - Ergün, Esra

AU - Töreyin, Behçet Uğur

PY - 2021

Y1 - 2021

N2 - Human brain effectively integrates prior knowledge to new skills by transferring experience across tasks without suffering from catastrophic forgetting. In this study, to continuously learn a visual classification task sequence, we employed a neural network model with lateral connections called Progressive Neural Networks (PNN). We sparsified PNNs with sparse group Least Absolute Shrinkage and Selection Operator (LASSO) and trained conventional PNNs with recursive connections. Later, the effect of the task prior on current performance is investigated with various task orders. The proposed approach is evaluated on permutedMNIST and selected subtasks from CIFAR-100 dataset. Results show that sparse Group LASSO regularization effectively sparsifies the progressive neural networks and the task sequence order affects the performance.

AB - Human brain effectively integrates prior knowledge to new skills by transferring experience across tasks without suffering from catastrophic forgetting. In this study, to continuously learn a visual classification task sequence, we employed a neural network model with lateral connections called Progressive Neural Networks (PNN). We sparsified PNNs with sparse group Least Absolute Shrinkage and Selection Operator (LASSO) and trained conventional PNNs with recursive connections. Later, the effect of the task prior on current performance is investigated with various task orders. The proposed approach is evaluated on permutedMNIST and selected subtasks from CIFAR-100 dataset. Results show that sparse Group LASSO regularization effectively sparsifies the progressive neural networks and the task sequence order affects the performance.

KW - Continual learning

KW - Progressive Neural Networks

KW - Sparse group LASSO regularization

UR - http://www.scopus.com/inward/record.url?scp=85116865008&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-88113-9_58

DO - 10.1007/978-3-030-88113-9_58

M3 - Conference contribution

AN - SCOPUS:85116865008

SN - 9783030881122

T3 - Communications in Computer and Information Science

SP - 715

EP - 725

BT - Advances in Computational Collective Intelligence - 13th International Conference, ICCCI 2021, Proceedings

A2 - Wojtkiewicz, Krystian

A2 - Treur, Jan

A2 - Pimenidis, Elias

A2 - Maleszka, Marcin

PB - Springer Science and Business Media Deutschland GmbH

T2 - 13th International Conference on Computational Collective Intelligence, ICCCI 2021

Y2 - 29 September 2021 through 1 October 2021

ER -

Ergün E , Töreyin BU. Sparse Progressive Neural Networks for Continual Learning. In Wojtkiewicz K, Treur J, Pimenidis E, Maleszka M, editors, Advances in Computational Collective Intelligence - 13th International Conference, ICCCI 2021, Proceedings. Springer Science and Business Media Deutschland GmbH. 2021. p. 715-725. (Communications in Computer and Information Science). doi: 10.1007/978-3-030-88113-9_58

Sparse Progressive Neural Networks for Continual Learning

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this