Leaky Integrate-and-Fire Spiking Neural Networks

Standard artificial neurons produce a continuous output on every forward pass. Biological neurons are different — they accumulate input over time and fire a discrete spike only when a threshold is crossed. Leaky Integrate-and-Fire (LIF) neurons model this behavior and produce sparse, event-driven computations.

The LIF Neuron Model

A LIF neuron has a membrane potential V that integrates input current and leaks toward zero over time:

V(t) = decay * V(t-1) + input(t)    # integrate and leak
spike(t) = 1 if V(t) >= threshold else 0
V(t) = V(t) * (1 - spike(t))        # reset on fire

Three key parameters:

decay (0-1): how fast the potential leaks. 0.95 = slow leak, 0.5 = fast leak
threshold: the voltage at which the neuron fires (typically 1.0)
refractory_steps: how many timesteps after firing during which the neuron can't fire again

Implementation

python

import numpy as np
 
class LIFBlock:
    def __init__(
        self,
        input_dim: int,
        n_neurons: int,
        threshold: float = 1.0,
        decay: float = 0.95,
        refractory_steps: int = 1,
    ):
        self.W = np.random.randn(input_dim, n_neurons) * 0.1
        self.b = np.zeros(n_neurons)
        self.threshold = threshold
        self.decay = decay
        self.refractory_steps = refractory_steps
 
        # State (persists across timesteps)
        self._V = np.zeros(n_neurons)
        self._refractory = np.zeros(n_neurons, dtype=int)
        self._spike_count = np.zeros(n_neurons)
        self._spike_times = []
 
    def forward(self, x: np.ndarray, t: int) -> np.ndarray:
        # Active mask: neurons not in refractory period
        active = (self._refractory <= 0).astype(float)
 
        # Integrate
        self._V = self.decay * self._V
        self._V += (x @ self.W + self.b) * active
 
        # Fire
        spike = (self._V >= self.threshold).astype(float)
        fired = spike.nonzero()[0]
 
        # Record
        for idx in fired:
            self._spike_times.append((t, idx))
        self._spike_count += spike
 
        # Reset fired neurons
        self._V *= (1.0 - spike)
 
        # Update refractory
        self._refractory = np.maximum(self._refractory - 1, 0)
        self._refractory[fired] = self.refractory_steps
 
        return spike
 
    @property
    def spike_rates(self) -> np.ndarray:
        """Average spikes per timestep across the simulation."""
        T = max(1, len(self._spike_times))
        return self._spike_count / T
 
    def reset(self):
        self._V[:] = 0
        self._refractory[:] = 0
        self._spike_count[:] = 0
        self._spike_times.clear()

Running a Temporal Simulation

LIF neurons require time — you need to run the same network for T timesteps:

python

def temporal_forward(model_blocks, x: np.ndarray, T: int = 20) -> np.ndarray:
    """Run the network for T timesteps and return final spike rates."""
    for block in model_blocks:
        if hasattr(block, 'reset'):
            block.reset()
 
    spike_outputs = []
    for t in range(T):
        out = x
        for block in model_blocks:
            if isinstance(block, LIFBlock):
                out = block.forward(out, t)
            else:
                out = block.forward(out)
        spike_outputs.append(out)
 
    # Return spike rates from the last LIF layer
    lif_layer = next(b for b in reversed(model_blocks) if isinstance(b, LIFBlock))
    return lif_layer.spike_counts / T

The Surrogate Gradient Problem

The biggest challenge with training spiking networks is that the spike function is non-differentiable. The Heaviside step function has a derivative of zero everywhere except at the threshold, where it's undefined.

The solution is a surrogate gradient — a smooth function that approximates the step function's derivative during backprop:

python

def spike_backward(V: np.ndarray, threshold: float) -> np.ndarray:
    """Fast sigmoid surrogate gradient for LIF backward pass."""
    # Gradient ≈ 0 far from threshold, 1/(4) at threshold
    return 1.0 / (1.0 + np.abs(V - threshold) * 10.0) ** 2

This lets gradients flow through the spike decision. The forward pass uses the hard threshold; the backward pass uses the smooth surrogate. This disconnect is intentional and works well in practice.

Spike Encoder and Decoder

To connect LIF layers to standard dense layers, you need converters:

python

class SpikeEncoder:
    """Rate coding: convert float in [0,1] to binary spike probability."""
    def forward(self, x: np.ndarray) -> np.ndarray:
        x_clamped = np.clip(x, 0, 1)
        return (np.random.rand(*x_clamped.shape) < x_clamped).astype(float)
 
class SpikeDecoder:
    """Convert accumulated spike rates back to continuous values."""
    def __init__(self, lif_block: LIFBlock, output_dim: int):
        self.lif = lif_block
        self.W = np.random.randn(lif_block.n_neurons, output_dim) * 0.1
 
    def forward(self) -> np.ndarray:
        rates = self.lif.spike_rates  # shape: (n_neurons,)
        return rates @ self.W

A complete spiking pipeline: Input → SpikeEncoder → LIFBlock → SpikeDecoder → Output

Why Spiking Networks

For most ML tasks, standard ANNs outperform spiking networks. Spiking networks are interesting for:

Energy efficiency: spikes are sparse events. Neuromorphic hardware (Intel Loihi, IBM TrueNorth) processes spikes directly, achieving orders-of-magnitude efficiency gains over GPU matrix multiply.
Temporal coding: time itself carries information. The precise timing of spike sequences can encode richer signals than rate codes alone.
Biological realism: for computational neuroscience and studying neural dynamics, LIF models are the standard approximation.

The Neural Nexus Spiking dashboard tab lets you set T, run a simulation, and see both the spike raster plot (which neurons fired at which timestep) and the membrane potential trace in real time — the most direct way to develop intuition for what these networks are doing.