Backprop Alternative: Augmented Lagrangian Predictive Coding

In one line

We introduce PC-ALM, a local alternative to backpropagation. PC-ALM trains residual MLPs up to 1000 layers, nearly matching backprop’s performance despite using only layer-local dynamics.

Opening excerpt

We introduce PC-ALM , a local alternative to backpropagation. PC-ALM trains residual MLPs up to 1000 layers , nearly matching backprop’s performance despite using only layer-local dynamics. PC-ALM equips each layer with a feedback control dynamical system that distributes and propagates supervision credit throughout a network.

Standard deep learning relies on backpropagation. The brain, however, cannot implement backpropagation, at least not exactly [1, 2] . How the brain solves the multilayer credit assignment problem without explicit use of backprop remains one of the fundamental unsolved problems in neuroscience (though not without progress [3, 4, 5] ).

There are several reasons the brain can’t implement exact backpropagation. One is “phase locking” [6, 2] . Backpropagation runs in three phases, in strict order: 1) a forward pass, then 2) a backward pass, then 3) a weight update. A weight update is locked until the forward and backward passes have completed—a neuron in an early layer must hold its activation and wait for the error signal to arrive.

(Excerpted from the original; full article via the source link below.)

This story hit the Hacker News front page today (14 points / 1 comments, via pub.sakana.ai). Our Tech Radar aggregates daily signals on AI engineering, backend architecture and DevOps — browse the related services and further reading below, or get in touch with our team.