Interpreting BERT-based Text Similarity via Activation and Saliency Maps Article Swipe
Itzik Malkiel
,
Dvir Ginzburg
,
Oren Barkan
,
Avi Caciularu
,
Jonathan Weill
,
Noam Koenigstein
·
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2208.06612
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2208.06612
Recently, there has been growing interest in the ability of Transformer-based models to produce meaningful embeddings of text with several applications, such as text similarity. Despite significant progress in the field, the explanations for similarity predictions remain challenging, especially in unsupervised settings. In this work, we present an unsupervised technique for explaining paragraph similarities inferred by pre-trained BERT models. By looking at a pair of paragraphs, our technique identifies important words that dictate each paragraph's semantics, matches between the words in both paragraphs, and retrieves the most important pairs that explain the similarity between the two. The method, which has been assessed by extensive human evaluations and demonstrated on datasets comprising long and complex paragraphs, has shown great promise, providing accurate interpretations that correlate better with human perceptions.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2208.06612
- https://arxiv.org/pdf/2208.06612
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4292107361
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4292107361Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2208.06612Digital Object Identifier
- Title
-
Interpreting BERT-based Text Similarity via Activation and Saliency MapsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2022Year of publication
- Publication date
-
2022-08-13Full publication date if available
- Authors
-
Itzik Malkiel, Dvir Ginzburg, Oren Barkan, Avi Caciularu, Jonathan Weill, Noam KoenigsteinList of authors in order
- Landing page
-
https://arxiv.org/abs/2208.06612Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2208.06612Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2208.06612Direct OA link when available
- Concepts
-
Paragraph, Similarity (geometry), Computer science, Artificial intelligence, Natural language processing, Semantic similarity, Transformer, World Wide Web, Physics, Image (mathematics), Quantum mechanics, VoltageTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4292107361 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2208.06612 |
| ids.doi | https://doi.org/10.48550/arxiv.2208.06612 |
| ids.openalex | https://openalex.org/W4292107361 |
| fwci | |
| type | preprint |
| title | Interpreting BERT-based Text Similarity via Activation and Saliency Maps |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9993000030517578 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T10181 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.992900013923645 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Natural Language Processing Techniques |
| topics[2].id | https://openalex.org/T13083 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.977400004863739 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Advanced Text Analysis Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2777206241 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8960026502609253 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q194431 |
| concepts[0].display_name | Paragraph |
| concepts[1].id | https://openalex.org/C103278499 |
| concepts[1].level | 3 |
| concepts[1].score | 0.6975338459014893 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q254465 |
| concepts[1].display_name | Similarity (geometry) |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.6506611108779907 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.5961034297943115 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C204321447 |
| concepts[4].level | 1 |
| concepts[4].score | 0.5930919647216797 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[4].display_name | Natural language processing |
| concepts[5].id | https://openalex.org/C130318100 |
| concepts[5].level | 2 |
| concepts[5].score | 0.48167240619659424 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q2268914 |
| concepts[5].display_name | Semantic similarity |
| concepts[6].id | https://openalex.org/C66322947 |
| concepts[6].level | 3 |
| concepts[6].score | 0.4588533043861389 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q11658 |
| concepts[6].display_name | Transformer |
| concepts[7].id | https://openalex.org/C136764020 |
| concepts[7].level | 1 |
| concepts[7].score | 0.05181005597114563 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q466 |
| concepts[7].display_name | World Wide Web |
| concepts[8].id | https://openalex.org/C121332964 |
| concepts[8].level | 0 |
| concepts[8].score | 0.0 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q413 |
| concepts[8].display_name | Physics |
| concepts[9].id | https://openalex.org/C115961682 |
| concepts[9].level | 2 |
| concepts[9].score | 0.0 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q860623 |
| concepts[9].display_name | Image (mathematics) |
| concepts[10].id | https://openalex.org/C62520636 |
| concepts[10].level | 1 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q944 |
| concepts[10].display_name | Quantum mechanics |
| concepts[11].id | https://openalex.org/C165801399 |
| concepts[11].level | 2 |
| concepts[11].score | 0.0 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q25428 |
| concepts[11].display_name | Voltage |
| keywords[0].id | https://openalex.org/keywords/paragraph |
| keywords[0].score | 0.8960026502609253 |
| keywords[0].display_name | Paragraph |
| keywords[1].id | https://openalex.org/keywords/similarity |
| keywords[1].score | 0.6975338459014893 |
| keywords[1].display_name | Similarity (geometry) |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.6506611108779907 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.5961034297943115 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/natural-language-processing |
| keywords[4].score | 0.5930919647216797 |
| keywords[4].display_name | Natural language processing |
| keywords[5].id | https://openalex.org/keywords/semantic-similarity |
| keywords[5].score | 0.48167240619659424 |
| keywords[5].display_name | Semantic similarity |
| keywords[6].id | https://openalex.org/keywords/transformer |
| keywords[6].score | 0.4588533043861389 |
| keywords[6].display_name | Transformer |
| keywords[7].id | https://openalex.org/keywords/world-wide-web |
| keywords[7].score | 0.05181005597114563 |
| keywords[7].display_name | World Wide Web |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2208.06612 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2208.06612 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2208.06612 |
| locations[1].id | doi:10.48550/arxiv.2208.06612 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2208.06612 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5067773841 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-4151-9119 |
| authorships[0].author.display_name | Itzik Malkiel |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Malkiel, Itzik |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5102868612 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-3410-8453 |
| authorships[1].author.display_name | Dvir Ginzburg |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Ginzburg, Dvir |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5077269072 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-5059-0905 |
| authorships[2].author.display_name | Oren Barkan |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Barkan, Oren |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5077903013 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-0573-1075 |
| authorships[3].author.display_name | Avi Caciularu |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Caciularu, Avi |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5057343912 |
| authorships[4].author.orcid | https://orcid.org/0009-0009-4582-9406 |
| authorships[4].author.display_name | Jonathan Weill |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Weill, Jonathan |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5083002103 |
| authorships[5].author.orcid | https://orcid.org/0000-0001-8219-4512 |
| authorships[5].author.display_name | Noam Koenigstein |
| authorships[5].author_position | last |
| authorships[5].raw_author_name | Koenigstein, Noam |
| authorships[5].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2208.06612 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Interpreting BERT-based Text Similarity via Activation and Saliency Maps |
| has_fulltext | True |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9993000030517578 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W4376653378, https://openalex.org/W2114797768, https://openalex.org/W2380654781, https://openalex.org/W2176214140, https://openalex.org/W2516873349, https://openalex.org/W2898077673, https://openalex.org/W4385239468, https://openalex.org/W4200476258, https://openalex.org/W1990601549, https://openalex.org/W2022470916 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2208.06612 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2208.06612 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2208.06612 |
| primary_location.id | pmh:oai:arXiv.org:2208.06612 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2208.06612 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2208.06612 |
| publication_date | 2022-08-13 |
| publication_year | 2022 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 62 |
| abstract_inverted_index.By | 59 |
| abstract_inverted_index.In | 42 |
| abstract_inverted_index.an | 47 |
| abstract_inverted_index.as | 22 |
| abstract_inverted_index.at | 61 |
| abstract_inverted_index.by | 55, 102 |
| abstract_inverted_index.in | 6, 28, 39, 80 |
| abstract_inverted_index.of | 9, 16, 64 |
| abstract_inverted_index.on | 108 |
| abstract_inverted_index.to | 12 |
| abstract_inverted_index.we | 45 |
| abstract_inverted_index.The | 96 |
| abstract_inverted_index.and | 83, 106, 112 |
| abstract_inverted_index.for | 33, 50 |
| abstract_inverted_index.has | 2, 99, 115 |
| abstract_inverted_index.our | 66 |
| abstract_inverted_index.the | 7, 29, 31, 78, 85, 91, 94 |
| abstract_inverted_index.BERT | 57 |
| abstract_inverted_index.been | 3, 100 |
| abstract_inverted_index.both | 81 |
| abstract_inverted_index.each | 73 |
| abstract_inverted_index.long | 111 |
| abstract_inverted_index.most | 86 |
| abstract_inverted_index.pair | 63 |
| abstract_inverted_index.such | 21 |
| abstract_inverted_index.text | 17, 23 |
| abstract_inverted_index.that | 71, 89, 122 |
| abstract_inverted_index.this | 43 |
| abstract_inverted_index.two. | 95 |
| abstract_inverted_index.with | 18, 125 |
| abstract_inverted_index.great | 117 |
| abstract_inverted_index.human | 104, 126 |
| abstract_inverted_index.pairs | 88 |
| abstract_inverted_index.shown | 116 |
| abstract_inverted_index.there | 1 |
| abstract_inverted_index.which | 98 |
| abstract_inverted_index.words | 70, 79 |
| abstract_inverted_index.work, | 44 |
| abstract_inverted_index.better | 124 |
| abstract_inverted_index.field, | 30 |
| abstract_inverted_index.models | 11 |
| abstract_inverted_index.remain | 36 |
| abstract_inverted_index.Despite | 25 |
| abstract_inverted_index.ability | 8 |
| abstract_inverted_index.between | 77, 93 |
| abstract_inverted_index.complex | 113 |
| abstract_inverted_index.dictate | 72 |
| abstract_inverted_index.explain | 90 |
| abstract_inverted_index.growing | 4 |
| abstract_inverted_index.looking | 60 |
| abstract_inverted_index.matches | 76 |
| abstract_inverted_index.method, | 97 |
| abstract_inverted_index.models. | 58 |
| abstract_inverted_index.present | 46 |
| abstract_inverted_index.produce | 13 |
| abstract_inverted_index.several | 19 |
| abstract_inverted_index.accurate | 120 |
| abstract_inverted_index.assessed | 101 |
| abstract_inverted_index.datasets | 109 |
| abstract_inverted_index.inferred | 54 |
| abstract_inverted_index.interest | 5 |
| abstract_inverted_index.progress | 27 |
| abstract_inverted_index.promise, | 118 |
| abstract_inverted_index.Recently, | 0 |
| abstract_inverted_index.correlate | 123 |
| abstract_inverted_index.extensive | 103 |
| abstract_inverted_index.important | 69, 87 |
| abstract_inverted_index.paragraph | 52 |
| abstract_inverted_index.providing | 119 |
| abstract_inverted_index.retrieves | 84 |
| abstract_inverted_index.settings. | 41 |
| abstract_inverted_index.technique | 49, 67 |
| abstract_inverted_index.comprising | 110 |
| abstract_inverted_index.embeddings | 15 |
| abstract_inverted_index.especially | 38 |
| abstract_inverted_index.explaining | 51 |
| abstract_inverted_index.identifies | 68 |
| abstract_inverted_index.meaningful | 14 |
| abstract_inverted_index.semantics, | 75 |
| abstract_inverted_index.similarity | 34, 92 |
| abstract_inverted_index.evaluations | 105 |
| abstract_inverted_index.paragraph's | 74 |
| abstract_inverted_index.paragraphs, | 65, 82, 114 |
| abstract_inverted_index.pre-trained | 56 |
| abstract_inverted_index.predictions | 35 |
| abstract_inverted_index.significant | 26 |
| abstract_inverted_index.similarity. | 24 |
| abstract_inverted_index.challenging, | 37 |
| abstract_inverted_index.demonstrated | 107 |
| abstract_inverted_index.explanations | 32 |
| abstract_inverted_index.perceptions. | 127 |
| abstract_inverted_index.similarities | 53 |
| abstract_inverted_index.unsupervised | 40, 48 |
| abstract_inverted_index.applications, | 20 |
| abstract_inverted_index.interpretations | 121 |
| abstract_inverted_index.Transformer-based | 10 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 6 |
| citation_normalized_percentile |