Activation functions

Choose your learning level

BeginnerIntermediateAdvanced

Start with plain-language explanations. Switch levels at any time.

Beginner explanation

An activation function transforms a neuron’s combined input. Nonlinear activations help a network represent relationships that a straight line cannot. Different functions are useful in different parts of a network.

Intermediate explanation

ReLU returns max(0, z). Sigmoid maps a scalar to (0,1), often for a binary output. Softmax normalizes a vector of scores across classes. A number between zero and one is not automatically a well-calibrated probability.

Advanced explanation

Sigmoid can saturate with small gradients at large absolute inputs. ReLU is piecewise linear and can produce inactive units. Softmax outputs depend on all logits; use numerically stable implementations and a loss appropriate to the task.

PUT IT IN CONTEXT

A worked example

ReLU maps [−2, 0, 3] to [0, 0, 3]. It removes negative values in this calculation, but does not discard “bad information” in a moral or semantic sense. It is simply the chosen mathematical transformation.

TRY IT YOURSELF

Practice the idea

Apply ReLU to −4, 1.5 and 0. Then explain why sigmoid would not produce exactly the same results.

Your working notes (not saved or sent)

Reveal the answer guide

ReLU gives 0, 1.5 and 0. Sigmoid produces values strictly between 0 and 1 for finite inputs, so even its output for zero is 0.5.

Go further: advanced challenge

Explain why softmax is suitable for mutually exclusive classes but independent sigmoid outputs are often used for multiple labels that can coexist.

An open-ended discussion task. Explain your assumptions and evidence; there may be more than one defensible answer.

Continue the conversation

What part of this idea would you like to test or question? Compare your reasoning with the community.

Find related forum discussions ↗

Opens a topic search in the existing AstraCruz forum. Results depend on available posts; signing in may be required. No discussion or activity is fabricated.

Browse all forums