Improving Neural Network Efficiency Using Piecewise Linear Approximation of Activation Functions

Authors

DOI:

https://doi.org/10.32473/flairs.38.1.139005

Abstract

Activation functions play a pivotal role in Neural Networks by enabling the modeling of complex non-linear relationships within data. However, the computational cost associated with certain activation functions, such as the hyperbolic tangent (tanh) and its gradient, can be substantial. In this study, we demonstrate that a piecewise linear approximation of the tanh function, utilizing pre-calculated slopes, achieves faster computation without significant degradation in performance. Conversely, we show that a piecewise linear approximation of the sigmoid function is computationally slower compared to its continuous counterpart. These findings suggest that the computational efficiency of a piecewise activation function depends on whether the indexing and arithmetic costs of the approximation are lower than those of the continuous function.

Downloads

Published

14-05-2025

How to Cite

Reddy, P., & Gujral, A. S. (2025). Improving Neural Network Efficiency Using Piecewise Linear Approximation of Activation Functions. The International FLAIRS Conference Proceedings, 38(1). https://doi.org/10.32473/flairs.38.1.139005

Issue

Section

Special Track: Neural Networks and Data Mining