Backprop Alternative: Augmented Lagrangian Predictive Coding

一句话概要

We introduce PC-ALM, a local alternative to backpropagation. PC-ALM trains residual MLPs up to 1000 layers, nearly matching backprop’s performance despite using only layer-local dynamics.

原文开头节选

We introduce PC-ALM , a local alternative to backpropagation. PC-ALM trains residual MLPs up to 1000 layers , nearly matching backprop’s performance despite using only layer-local dynamics. PC-ALM equips each layer with a feedback control dynamical system that distributes and propagates supervision credit throughout a network.

Standard deep learning relies on backpropagation. The brain, however, cannot implement backpropagation, at least not exactly [1, 2] . How the brain solves the multilayer credit assignment problem without explicit use of backprop remains one of the fundamental unsolved problems in neuroscience (though not without progress [3, 4, 5] ).

There are several reasons the brain can’t implement exact backpropagation. One is “phase locking” [6, 2] . Backpropagation runs in three phases, in strict order: 1) a forward pass, then 2) a backward pass, then 3) a weight update. A weight update is locked until the forward and backward passes have completed—a neuron in an early layer must hold its activation and wait for the error signal to arrive.

(以上为原文节选,完整内容见下方”原文来源”)

这条动态今日登上 Hacker News 首页(14 赞 / 1 评论,来源 pub.sakana.ai)。技术雷达每日自动聚合 AI 工程、后端架构、DevOps 方向的前沿动态;相关工程落地可浏览下方的相关服务与延伸阅读,或直接与我们团队交流。