Mapping the Timescale Organization of Neural Language Models Article Swipe

MAKE IMPACT

Hsiang-Yun Sherry Chien , Jinhan Zhang , Christopher J. Honey · YOU? · · 2020 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2012.06717

In the human brain, sequences of language input are processed within a distributed and hierarchical architecture, in which higher stages of processing encode contextual information over longer timescales. In contrast, in recurrent neural networks which perform natural language processing, we know little about how the multiple timescales of contextual information are functionally organized. Therefore, we applied tools developed in neuroscience to map the "processing timescales" of individual units within a word-level LSTM language model. This timescale-mapping method assigned long timescales to units previously found to track long-range syntactic dependencies. Additionally, the mapping revealed a small subset of the network (less than 15% of units) with long timescales and whose function had not previously been explored. We next probed the functional organization of the network by examining the relationship between the processing timescale of units and their network connectivity. We identified two classes of long-timescale units: "controller" units composed a densely interconnected subnetwork and strongly projected to the rest of the network, while "integrator" units showed the longest timescales in the network, and expressed projection profiles closer to the mean projection profile. Ablating integrator and controller units affected model performance at different positions within a sentence, suggesting distinctive functions of these two sets of units. Finally, we tested the generalization of these results to a character-level LSTM model and models with different architectures. In summary, we demonstrated a model-free technique for mapping the timescale organization in recurrent neural networks, and we applied this method to reveal the timescale and functional organization of neural language models.

Type

article

Language

en

Landing Page

http://arxiv.org/abs/2012.06717

PDF

https://arxiv.org/pdf/2012.06717

OA Status

green

Cited By

1

References

16

Related Works

20

OpenAlex ID

https://openalex.org/W3112986129

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W3112986129

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2012.06717

Digital Object Identifier

Title

Mapping the Timescale Organization of Neural Language Models

Work title

Type

article

OpenAlex work type

Language

en

Primary language

Publication year

2020

Year of publication

Publication date

2020-12-12

Full publication date if available

Authors

Hsiang-Yun Sherry Chien, Jinhan Zhang, Christopher J. Honey

List of authors in order

Landing page

https://arxiv.org/abs/2012.06717

Publisher landing page

PDF URL

https://arxiv.org/pdf/2012.06717

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2012.06717

Direct OA link when available

Concepts

Computer science, Sentence, Subnetwork, Language model, Artificial intelligence, Projection (relational algebra), ENCODE, Artificial neural network, Natural language processing, Generalization, Word (group theory), Algorithm, Mathematics, Philosophy, Chemistry, Linguistics, Mathematical analysis, Gene, Computer security, Biochemistry

Top concepts (fields/topics) attached by OpenAlex

Cited by

1

Total citation count in OpenAlex

Citations by year (recent)

2023: 1

Per-year citation counts (last 5 years)

References (count)

16

Number of works referenced by this work

Related works (count)

20

Other works algorithmically related by OpenAlex

Full payload