Htydjtk

This may seem like a very simple and obvious question, but I haven't actually been able to find a direct answer.

Today in a video explaining deep neural networks I came across the term Squashing function. This is a term that I have never heard used, instead our professor always used the term Activation function. Given the definitions I've been able to find, the two seem to be interchangeable terms.

Are they really synonymous or is there a difference?

An activation function

This the name given to a function, which is applied to a neuron that just had a weight update as a result of new information. It can refer to any of the well known activation funtions, such as the Rectified Linear Unit (ReLU), the hyperbolic tangent function (tanh) or even the identity function! Have a look at somewhere like the Keras documentation for a nice little list of examples.

We usually define the activation function as being a non-linear function, as it is that property, which gives a neural network its ability to approximate any equation (given a few constraints). However, an
activation function can also be linear e.g. the identity function.

A squashing function

This can mean one of two things, as far as I know, in the context of a neural network - the tag you added to the question - and they are close, just differently applied.

The first and most commonplace example, is when people refer to the softmax function, which squashes the final layer's activations/logits into the range [0, 1]. This has the effect of allowing final outputs to be directly interpreted as probabilities (i.e. they must sum to 1).

The second and newest usage of this words in the context of neural networks is from the reletively recent papers (one and two) from Sara Sabour, Geoffrey Hinton and Nicholas Frosst, which presented the idea of Capsule Netoworks. What these are and how they work is beyond the scope of this question; however, the term "squashing function" deserves special mention. Paper number one introduces it followingly:

We want the length of the output vector of a capsule to represent the probability that the entity represented by the capsule is present in the
current input. We therefore use a non-linear "squashing" function to ensure that short vectors get shrunk to almost zero length and long vectors get shrunk to a length slightly below 1.

That description makes it sound vert similar indeed to the softmax!

This squashing function is defined as follows:

$$
v_j = frac{||s_j||^2}{1 + ||s_j||^2} cdot frac{s_j}{||s_j||}
$$

where $v_j$ is the vector output of capsule $j$ and $s_j$ is its total input.

If this is all new to you and you'd like to learn more, I'd recommend having a read of those two papers, as well as perhaps a nice overview blog, like this one.

Activation functions like sigmoid function, hyperbolic tangent function, etc. are also called squashing function because they squash the input into a small range like in sigmoid function output is in range of [-1,1]. But you cannot call ReLU as a squashing function because for a positive input value it returns the output as same.

So there is a formal definition of squashing function used in the paper by Hornik, (1989), see definition 2.3. The paper demonstrates that any neural net with a single layer of sufficient number of nodes where the activation function is a 'squashing' function is a universal approximator. Given the context I think this is what is meant by squashing function.

The definition given there is any function that is non decreasing, $ textrm{lim}_{xrightarrow infty} f(x) = 1$ and $ textrm{lim}_{xrightarrow -infty} f(x) = 0$.
So we have that ReLU is not a squashing function because $ textrm{lim}_{xrightarrow infty} ReLU(x) = infty neq 1$ .

NB. a net with ReLU activation functions is a universal approximator, but the proof in that paper dosn't apply to it.