Interval type-2 fuzzy systems as deep neural network activation functions

Aykut Beke; Tufan Kumbasar

Interval type-2 fuzzy systems as deep neural network activation functions

Aykut Beke, Tufan Kumbasar

Department of Control and Automation Engineering

Istanbul Technical University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Citations (Scopus)

Abstract

In this paper, we propose a novel activation function, namely, Interval Type-2 (IT2) Fuzzy Rectifying Unit (FRU), to improve the performance of the Deep Neural Networks (DNNs). The IT2-FRU can generate linear or sophisticated activation functions by simply tuning the size of the footprint of uncertainty of the IT2 Fuzzy Sets. The novel IT2-FRU also alleviates vanishing gradient problem and has a fast convergence rate since it pushes the mean activation to zero by allowing the negative outputs. In order to test the performance of the IT2-FRU, comparative experimental studies are performed on the CIFAR-10 dataset. IT2-FRU is compared with widely used conventional activation functions. Experimental results show that IT2-FRU significantly speeds up the learning and has a superior performance compared to other handled activation functions.

Original language	English
Title of host publication	Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019
Editors	Vilem Novak, Vladimir Marik, Martin Stepnicka, Mirko Navara, Petr Hurtik
Publisher	Atlantis Press
Pages	267-273
Number of pages	7
ISBN (Electronic)	9789462527706
Publication status	Published - 2020
Event	11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019 - Prague, Czech Republic Duration: 9 Sept 2019 → 13 Sept 2019

Publication series

Name	Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019

Conference

Conference	11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019
Country/Territory	Czech Republic
City	Prague
Period	9/09/19 → 13/09/19

Bibliographical note

Publisher Copyright:
Copyright © 2019, the Authors. Published by Atlantis Press. This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Funding

This research is supported by the project (118E807) of Scientific and Technological Research Council of Turkey (TUBITAK). All of these supports are appreciated.

Funders	Funder number
TUBITAK
Türkiye Bilimsel ve Teknolojik Araştirma Kurumu

Keywords

Activation unit
Deep learning
Footprint of Uncertainty
Interval type-2 fuzzy system

Cite this

Beke, A., & Kumbasar, T. (2020). Interval type-2 fuzzy systems as deep neural network activation functions. In V. Novak, V. Marik, M. Stepnicka, M. Navara, & P. Hurtik (Eds.), Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019 (pp. 267-273). (Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019). Atlantis Press.

Beke, Aykut ; Kumbasar, Tufan. / Interval type-2 fuzzy systems as deep neural network activation functions. Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019. editor / Vilem Novak ; Vladimir Marik ; Martin Stepnicka ; Mirko Navara ; Petr Hurtik. Atlantis Press, 2020. pp. 267-273 (Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019).

@inproceedings{c2f5a938611545c1b718311a4fa823d9,

title = "Interval type-2 fuzzy systems as deep neural network activation functions",

abstract = "In this paper, we propose a novel activation function, namely, Interval Type-2 (IT2) Fuzzy Rectifying Unit (FRU), to improve the performance of the Deep Neural Networks (DNNs). The IT2-FRU can generate linear or sophisticated activation functions by simply tuning the size of the footprint of uncertainty of the IT2 Fuzzy Sets. The novel IT2-FRU also alleviates vanishing gradient problem and has a fast convergence rate since it pushes the mean activation to zero by allowing the negative outputs. In order to test the performance of the IT2-FRU, comparative experimental studies are performed on the CIFAR-10 dataset. IT2-FRU is compared with widely used conventional activation functions. Experimental results show that IT2-FRU significantly speeds up the learning and has a superior performance compared to other handled activation functions.",

keywords = "Activation unit, Deep learning, Footprint of Uncertainty, Interval type-2 fuzzy system",

author = "Aykut Beke and Tufan Kumbasar",

note = "Publisher Copyright: Copyright {\textcopyright} 2019, the Authors. Published by Atlantis Press. This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).; 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019 ; Conference date: 09-09-2019 Through 13-09-2019",

year = "2020",

language = "English",

series = "Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019",

publisher = "Atlantis Press",

pages = "267--273",

editor = "Vilem Novak and Vladimir Marik and Martin Stepnicka and Mirko Navara and Petr Hurtik",

booktitle = "Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019",

}

Beke, A & Kumbasar, T 2020, Interval type-2 fuzzy systems as deep neural network activation functions. in V Novak, V Marik, M Stepnicka, M Navara & P Hurtik (eds), Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019. Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019, Atlantis Press, pp. 267-273, 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019, Prague, Czech Republic, 9/09/19.

Interval type-2 fuzzy systems as deep neural network activation functions. / Beke, Aykut; Kumbasar, Tufan.
Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019. ed. / Vilem Novak; Vladimir Marik; Martin Stepnicka; Mirko Navara; Petr Hurtik. Atlantis Press, 2020. p. 267-273 (Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Interval type-2 fuzzy systems as deep neural network activation functions

AU - Beke, Aykut

AU - Kumbasar, Tufan

PY - 2020

Y1 - 2020

N2 - In this paper, we propose a novel activation function, namely, Interval Type-2 (IT2) Fuzzy Rectifying Unit (FRU), to improve the performance of the Deep Neural Networks (DNNs). The IT2-FRU can generate linear or sophisticated activation functions by simply tuning the size of the footprint of uncertainty of the IT2 Fuzzy Sets. The novel IT2-FRU also alleviates vanishing gradient problem and has a fast convergence rate since it pushes the mean activation to zero by allowing the negative outputs. In order to test the performance of the IT2-FRU, comparative experimental studies are performed on the CIFAR-10 dataset. IT2-FRU is compared with widely used conventional activation functions. Experimental results show that IT2-FRU significantly speeds up the learning and has a superior performance compared to other handled activation functions.

AB - In this paper, we propose a novel activation function, namely, Interval Type-2 (IT2) Fuzzy Rectifying Unit (FRU), to improve the performance of the Deep Neural Networks (DNNs). The IT2-FRU can generate linear or sophisticated activation functions by simply tuning the size of the footprint of uncertainty of the IT2 Fuzzy Sets. The novel IT2-FRU also alleviates vanishing gradient problem and has a fast convergence rate since it pushes the mean activation to zero by allowing the negative outputs. In order to test the performance of the IT2-FRU, comparative experimental studies are performed on the CIFAR-10 dataset. IT2-FRU is compared with widely used conventional activation functions. Experimental results show that IT2-FRU significantly speeds up the learning and has a superior performance compared to other handled activation functions.

KW - Activation unit

KW - Deep learning

KW - Footprint of Uncertainty

KW - Interval type-2 fuzzy system

UR - http://www.scopus.com/inward/record.url?scp=85090901716&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85090901716

T3 - Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019

SP - 267

EP - 273

BT - Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019

A2 - Novak, Vilem

A2 - Marik, Vladimir

A2 - Stepnicka, Martin

A2 - Navara, Mirko

A2 - Hurtik, Petr

PB - Atlantis Press

T2 - 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019

Y2 - 9 September 2019 through 13 September 2019

ER -

Beke A, Kumbasar T. Interval type-2 fuzzy systems as deep neural network activation functions. In Novak V, Marik V, Stepnicka M, Navara M, Hurtik P, editors, Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019. Atlantis Press. 2020. p. 267-273. (Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology, EUSFLAT 2019).

Interval type-2 fuzzy systems as deep neural network activation functions

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

Other files and links

Fingerprint

Cite this