Uncovering Latent Chain of Thought Vectors in Language Models Article Swipe
Jason Zhang
,
Scott Viteri
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2409.14026
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2409.14026
In this work, we examine how targeted perturbations in the activation space of Language Models (LMs) can encode complex reasoning patterns. We inject steering vectors, derived from LM activations, into LMs during inference time and study whether these vectors can induce Chain-of-Thought (CoT) reasoning in LMs without the need for natural language prompting. We demonstrate this approach on Llama3 8B Instruct and Mistral 7B v0.2 Instruct and show that activation-space interventions achieve competitive, if not superior, performance compared to traditional CoT prompting across multiple reasoning benchmarks, including GSM8k, MMLU, AGI Eval, and ARC AI2. These findings suggest that neural network activations can encode reasoning patterns, offering a new application of activation space manipulation as a tool for tuning model behavior.
Related Topics
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2409.14026
- https://arxiv.org/pdf/2409.14026
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4403752927
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4403752927Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2409.14026Digital Object Identifier
- Title
-
Uncovering Latent Chain of Thought Vectors in Language ModelsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-09-21Full publication date if available
- Authors
-
Jason Zhang, Scott ViteriList of authors in order
- Landing page
-
https://arxiv.org/abs/2409.14026Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2409.14026Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2409.14026Direct OA link when available
- Concepts
-
Computer science, Linguistics, Natural language processing, Psychology, PhilosophyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4403752927 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2409.14026 |
| ids.doi | https://doi.org/10.48550/arxiv.2409.14026 |
| ids.openalex | https://openalex.org/W4403752927 |
| fwci | |
| type | preprint |
| title | Uncovering Latent Chain of Thought Vectors in Language Models |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9607999920845032 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.4298114776611328 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C41895202 |
| concepts[1].level | 1 |
| concepts[1].score | 0.39652836322784424 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q8162 |
| concepts[1].display_name | Linguistics |
| concepts[2].id | https://openalex.org/C204321447 |
| concepts[2].level | 1 |
| concepts[2].score | 0.3868448734283447 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[2].display_name | Natural language processing |
| concepts[3].id | https://openalex.org/C15744967 |
| concepts[3].level | 0 |
| concepts[3].score | 0.3554390072822571 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q9418 |
| concepts[3].display_name | Psychology |
| concepts[4].id | https://openalex.org/C138885662 |
| concepts[4].level | 0 |
| concepts[4].score | 0.22924408316612244 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[4].display_name | Philosophy |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.4298114776611328 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/linguistics |
| keywords[1].score | 0.39652836322784424 |
| keywords[1].display_name | Linguistics |
| keywords[2].id | https://openalex.org/keywords/natural-language-processing |
| keywords[2].score | 0.3868448734283447 |
| keywords[2].display_name | Natural language processing |
| keywords[3].id | https://openalex.org/keywords/psychology |
| keywords[3].score | 0.3554390072822571 |
| keywords[3].display_name | Psychology |
| keywords[4].id | https://openalex.org/keywords/philosophy |
| keywords[4].score | 0.22924408316612244 |
| keywords[4].display_name | Philosophy |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2409.14026 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2409.14026 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2409.14026 |
| locations[1].id | doi:10.48550/arxiv.2409.14026 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2409.14026 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5101875766 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-4632-7730 |
| authorships[0].author.display_name | Jason Zhang |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Zhang, Jason |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5114402638 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Scott Viteri |
| authorships[1].author_position | last |
| authorships[1].raw_author_name | Viteri, Scott |
| authorships[1].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2409.14026 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Uncovering Latent Chain of Thought Vectors in Language Models |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9607999920845032 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W4396696052 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2409.14026 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2409.14026 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2409.14026 |
| primary_location.id | pmh:oai:arXiv.org:2409.14026 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2409.14026 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2409.14026 |
| publication_date | 2024-09-21 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 106, 114 |
| abstract_inverted_index.7B | 63 |
| abstract_inverted_index.8B | 59 |
| abstract_inverted_index.In | 0 |
| abstract_inverted_index.LM | 27 |
| abstract_inverted_index.We | 21, 53 |
| abstract_inverted_index.as | 113 |
| abstract_inverted_index.if | 73 |
| abstract_inverted_index.in | 8, 44 |
| abstract_inverted_index.of | 12, 109 |
| abstract_inverted_index.on | 57 |
| abstract_inverted_index.to | 78 |
| abstract_inverted_index.we | 3 |
| abstract_inverted_index.AGI | 89 |
| abstract_inverted_index.ARC | 92 |
| abstract_inverted_index.CoT | 80 |
| abstract_inverted_index.LMs | 30, 45 |
| abstract_inverted_index.and | 34, 61, 66, 91 |
| abstract_inverted_index.can | 16, 39, 101 |
| abstract_inverted_index.for | 49, 116 |
| abstract_inverted_index.how | 5 |
| abstract_inverted_index.new | 107 |
| abstract_inverted_index.not | 74 |
| abstract_inverted_index.the | 9, 47 |
| abstract_inverted_index.AI2. | 93 |
| abstract_inverted_index.from | 26 |
| abstract_inverted_index.into | 29 |
| abstract_inverted_index.need | 48 |
| abstract_inverted_index.show | 67 |
| abstract_inverted_index.that | 68, 97 |
| abstract_inverted_index.this | 1, 55 |
| abstract_inverted_index.time | 33 |
| abstract_inverted_index.tool | 115 |
| abstract_inverted_index.v0.2 | 64 |
| abstract_inverted_index.(CoT) | 42 |
| abstract_inverted_index.(LMs) | 15 |
| abstract_inverted_index.Eval, | 90 |
| abstract_inverted_index.MMLU, | 88 |
| abstract_inverted_index.These | 94 |
| abstract_inverted_index.model | 118 |
| abstract_inverted_index.space | 11, 111 |
| abstract_inverted_index.study | 35 |
| abstract_inverted_index.these | 37 |
| abstract_inverted_index.work, | 2 |
| abstract_inverted_index.GSM8k, | 87 |
| abstract_inverted_index.Llama3 | 58 |
| abstract_inverted_index.Models | 14 |
| abstract_inverted_index.across | 82 |
| abstract_inverted_index.during | 31 |
| abstract_inverted_index.encode | 17, 102 |
| abstract_inverted_index.induce | 40 |
| abstract_inverted_index.inject | 22 |
| abstract_inverted_index.neural | 98 |
| abstract_inverted_index.tuning | 117 |
| abstract_inverted_index.Mistral | 62 |
| abstract_inverted_index.achieve | 71 |
| abstract_inverted_index.complex | 18 |
| abstract_inverted_index.derived | 25 |
| abstract_inverted_index.examine | 4 |
| abstract_inverted_index.natural | 50 |
| abstract_inverted_index.network | 99 |
| abstract_inverted_index.suggest | 96 |
| abstract_inverted_index.vectors | 38 |
| abstract_inverted_index.whether | 36 |
| abstract_inverted_index.without | 46 |
| abstract_inverted_index.Instruct | 60, 65 |
| abstract_inverted_index.Language | 13 |
| abstract_inverted_index.approach | 56 |
| abstract_inverted_index.compared | 77 |
| abstract_inverted_index.findings | 95 |
| abstract_inverted_index.language | 51 |
| abstract_inverted_index.multiple | 83 |
| abstract_inverted_index.offering | 105 |
| abstract_inverted_index.steering | 23 |
| abstract_inverted_index.targeted | 6 |
| abstract_inverted_index.vectors, | 24 |
| abstract_inverted_index.behavior. | 119 |
| abstract_inverted_index.including | 86 |
| abstract_inverted_index.inference | 32 |
| abstract_inverted_index.patterns, | 104 |
| abstract_inverted_index.patterns. | 20 |
| abstract_inverted_index.prompting | 81 |
| abstract_inverted_index.reasoning | 19, 43, 84, 103 |
| abstract_inverted_index.superior, | 75 |
| abstract_inverted_index.activation | 10, 110 |
| abstract_inverted_index.prompting. | 52 |
| abstract_inverted_index.activations | 100 |
| abstract_inverted_index.application | 108 |
| abstract_inverted_index.benchmarks, | 85 |
| abstract_inverted_index.demonstrate | 54 |
| abstract_inverted_index.performance | 76 |
| abstract_inverted_index.traditional | 79 |
| abstract_inverted_index.activations, | 28 |
| abstract_inverted_index.competitive, | 72 |
| abstract_inverted_index.manipulation | 112 |
| abstract_inverted_index.interventions | 70 |
| abstract_inverted_index.perturbations | 7 |
| abstract_inverted_index.Chain-of-Thought | 41 |
| abstract_inverted_index.activation-space | 69 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 2 |
| citation_normalized_percentile |