Take a look at the stones. If you rotate them by 90 degrees they fit with the levers.
AHA the problem was that I didn't see the levers. The coloration did look a bit odd to me, and I was pressing E all along that wall, but I didn't see any change or realize they were levers to push/pull in a pattern. It would be much better if, e.g. there was red at the tips of the handles. But thank you, that worked!