129 - Transformers and Hierarchical Structure, with Shunyu Yao

In this episode, we talk to Shunyu Yao about recent insights into how transformers can represent hierarchical structure in language. Bounded-depth hierarchical structure is thought to be a key feature of natural languages, motivating Shunyu and his coauthors to show that transformers can efficiently represent bounded-depth Dyck languages, which can be thought of as a formal model of the structure of natural languages. We went on to discuss some of the intuitive ideas that emerge from the proofs, connections to RNNs, and insights about positional encodings that may have practical implications. More broadly, we also touched on the role of formal languages and other theoretical tools in modern NLP. Papers discussed in this episode: - Self-Attention Networks Can Process Bounded Hierarchical Languages ( - Theoretical Limitations of Self-Attention in Neural Sequence Models ( - RNNs can generate bounded hierarchical languages with optimal memory ( - On the Practical Computational Power of Finite Precision RNNs for Language Recognition ( Shunyu Yao's webpage: The hosts for this episode are William Merrill and Matt Gardner.

You can listen to 129 - Transformers and Hierarchical Structure, with Shunyu Yao online on Radio and Podcast. Open the player on this page to stream the available audio.

129 - Transformers and Hierarchical Structure, with Shunyu Yao is an episode from NLP Highlights by NLP Highlights.

This episode is 00:35:43 long.

This episode was published on Jul 2, 2021.

Yes. Use the heart button on the episode page to add it to your favorite episodes list.

Yes. This page shows related episodes from NLP Highlights when more episodes are available from the podcast feed.