Judge feedback

Judge feedback is anonymous.

• Interesting work - comparing the compiled model to a learned model is a natural thing to try, but I'm not aware of anyone actually trying it before! The actual analysis is pretty limited and doesn't really engage with understanding what's going on inside the learned model beyond inspecting neuron activations and attention patterns (which ARE the correct things to look at, at least!). I think this work would have been significantly cooler with some attempt to reverse engineer the learned circuit, or at least explore its behaviour on multiple inputs (though this is probably harder than can be achieved in a weekend hackathon!) Hypothesis 1 seems very obviously true (so I'm glad your results bear it out!) Hypothesis 2 seems very likely to be true. Hypothesis 3 is unclear and interesting to me, I don't feel like your work gives strong evidence either way, since I still don't know how things are implement before the final attention layer > We find that the only attention heads that are relevant seem to be in layer four, while the first three layers’ attention appear redundant. I don't think this statement is supported? Unless I'm missing something (eg, you investigating what happens if you delete those attention layers), all you show is that they aren't OBVIOUSLY doing anything relevant. Did you vary the length of the input list when training? Reversing a fixed length list is very, very easy, it just needs to match up the positional embeddings, so this is an important detail re what to learn from your project Some broader feedback - I think interpreting the learned model is significantly harder because it's so big! I think it could be much smaller while still able to do the task (1L might work, 2L attn-only is also worth a shot). I was pleasantly surprised by how sparse the learned MLP layer was, and it's totally worth trying to interpret the neurons, especially the one that are sharply on for some neurons, and off for others!