Overview: We train a tiny ReLU network to output sparse top- distributions over a vocabulary much larger than its residual dimension. The trained network seems to converge to a mechanism closely resembling a Bloom filter: tokens are assigned sparse binary hashes, the hidden layer computes an approximate union indicator, and the output logits are linearly read from this union. Here's what a small network trained on a toy version of the sparse top- distribution task learns to use: Weight matrix of a 1-layer ReLU network trained via gradient descent on the toy -sparse distribution task below, for , , . Truncated at first tokens for visualisation purposes. Plot of the range of values of , it forms a bimodal distribution. That's the input weight matrix of the trained network. Every entry is either or . The network has effectively encoded a binary hash for each token - and as we'll show, this seems to enable the network to approximately simulate a Bloom filter, and so output the correct set of top- tokens with high probability. We provide a theoretical construction showing how to set the weights to exactly implement a Bloom filter. The real network [...] ---Outline:

Overview:

The Task:

Construction:

Formal construction:

Analysis of a single forward pass:

Training:

Behavioural analysis of the trained network:

Mechanistic analysis of the trained network:

Conclusion / Reflections:

Related work: