Instruction-Guided Scene Text Recognition Article Swipe
Yongkun Du
,
Zhineng Chen
,
Yuchen Su
,
Caiyan Jia
,
Yu–Gang Jiang
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2401.17851
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2401.17851
Multi-modal models have shown appealing performance in visual recognition tasks, as free-form text-guided training evokes the ability to understand fine-grained visual content. However, current models cannot be trivially applied to scene text recognition (STR) due to the compositional difference between natural and text images. We propose a novel instruction-guided scene text recognition (IGTR) paradigm that formulates STR as an instruction learning problem and understands text images by predicting character attributes, e.g., character frequency, position, etc. IGTR first devises $\left \langle condition,question,answer\right \rangle$ instruction triplets, providing rich and diverse descriptions of character attributes. To effectively learn these attributes through question-answering, IGTR develops a lightweight instruction encoder, a cross-modal feature fusion module and a multi-task answer head, which guides nuanced text image understanding. Furthermore, IGTR realizes different recognition pipelines simply by using different instructions, enabling a character-understanding-based text reasoning paradigm that differs from current methods considerably. Experiments on English and Chinese benchmarks show that IGTR outperforms existing models by significant margins, while maintaining a small model size and fast inference speed. Moreover, by adjusting the sampling of instructions, IGTR offers an elegant way to tackle the recognition of rarely appearing and morphologically similar characters, which were previous challenges. Code: https://github.com/Topdu/OpenOCR.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2401.17851
- https://arxiv.org/pdf/2401.17851
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4391462917
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4391462917Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2401.17851Digital Object Identifier
- Title
-
Instruction-Guided Scene Text RecognitionWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-01-31Full publication date if available
- Authors
-
Yongkun Du, Zhineng Chen, Yuchen Su, Caiyan Jia, Yu–Gang JiangList of authors in order
- Landing page
-
https://arxiv.org/abs/2401.17851Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2401.17851Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2401.17851Direct OA link when available
- Concepts
-
Computer science, Artificial intelligence, Computer vision, Speech recognition, Natural language processing, Pattern recognition (psychology)Top concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4391462917 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2401.17851 |
| ids.doi | https://doi.org/10.48550/arxiv.2401.17851 |
| ids.openalex | https://openalex.org/W4391462917 |
| fwci | |
| type | preprint |
| title | Instruction-Guided Scene Text Recognition |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10601 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9965999722480774 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1707 |
| topics[0].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[0].display_name | Handwritten Text Recognition Techniques |
| topics[1].id | https://openalex.org/T10181 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.963699996471405 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Natural Language Processing Techniques |
| topics[2].id | https://openalex.org/T13523 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9478999972343445 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1703 |
| topics[2].subfield.display_name | Computational Theory and Mathematics |
| topics[2].display_name | Mathematics, Computing, and Information Processing |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.6013590097427368 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C154945302 |
| concepts[1].level | 1 |
| concepts[1].score | 0.3998432457447052 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[1].display_name | Artificial intelligence |
| concepts[2].id | https://openalex.org/C31972630 |
| concepts[2].level | 1 |
| concepts[2].score | 0.345506489276886 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q844240 |
| concepts[2].display_name | Computer vision |
| concepts[3].id | https://openalex.org/C28490314 |
| concepts[3].level | 1 |
| concepts[3].score | 0.34202706813812256 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q189436 |
| concepts[3].display_name | Speech recognition |
| concepts[4].id | https://openalex.org/C204321447 |
| concepts[4].level | 1 |
| concepts[4].score | 0.3371826410293579 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[4].display_name | Natural language processing |
| concepts[5].id | https://openalex.org/C153180895 |
| concepts[5].level | 2 |
| concepts[5].score | 0.32188305258750916 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q7148389 |
| concepts[5].display_name | Pattern recognition (psychology) |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.6013590097427368 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[1].score | 0.3998432457447052 |
| keywords[1].display_name | Artificial intelligence |
| keywords[2].id | https://openalex.org/keywords/computer-vision |
| keywords[2].score | 0.345506489276886 |
| keywords[2].display_name | Computer vision |
| keywords[3].id | https://openalex.org/keywords/speech-recognition |
| keywords[3].score | 0.34202706813812256 |
| keywords[3].display_name | Speech recognition |
| keywords[4].id | https://openalex.org/keywords/natural-language-processing |
| keywords[4].score | 0.3371826410293579 |
| keywords[4].display_name | Natural language processing |
| keywords[5].id | https://openalex.org/keywords/pattern-recognition |
| keywords[5].score | 0.32188305258750916 |
| keywords[5].display_name | Pattern recognition (psychology) |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2401.17851 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://arxiv.org/pdf/2401.17851 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2401.17851 |
| locations[1].id | doi:10.48550/arxiv.2401.17851 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2401.17851 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5102915298 |
| authorships[0].author.orcid | https://orcid.org/0009-0000-9859-721X |
| authorships[0].author.display_name | Yongkun Du |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Du, Yongkun |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5080463909 |
| authorships[1].author.orcid | https://orcid.org/0000-0003-1543-6889 |
| authorships[1].author.display_name | Zhineng Chen |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Chen, Zhineng |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5101940296 |
| authorships[2].author.orcid | https://orcid.org/0009-0009-4034-5883 |
| authorships[2].author.display_name | Yuchen Su |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Su, Yuchen |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5085282915 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-0650-9564 |
| authorships[3].author.display_name | Caiyan Jia |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Jia, Caiyan |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5047962986 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-1907-8567 |
| authorships[4].author.display_name | Yu–Gang Jiang |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Jiang, Yu-Gang |
| authorships[4].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2401.17851 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Instruction-Guided Scene Text Recognition |
| has_fulltext | True |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10601 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9965999722480774 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1707 |
| primary_topic.subfield.display_name | Computer Vision and Pattern Recognition |
| primary_topic.display_name | Handwritten Text Recognition Techniques |
| related_works | https://openalex.org/W2755342338, https://openalex.org/W2058170566, https://openalex.org/W2036807459, https://openalex.org/W2775347418, https://openalex.org/W1969923398, https://openalex.org/W2166024367, https://openalex.org/W2772917594, https://openalex.org/W3116076068, https://openalex.org/W2229312674, https://openalex.org/W2079911747 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2401.17851 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2401.17851 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2401.17851 |
| primary_location.id | pmh:oai:arXiv.org:2401.17851 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://arxiv.org/pdf/2401.17851 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2401.17851 |
| publication_date | 2024-01-31 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 46, 101, 105, 111, 133, 161 |
| abstract_inverted_index.To | 92 |
| abstract_inverted_index.We | 44 |
| abstract_inverted_index.an | 58, 178 |
| abstract_inverted_index.as | 10, 57 |
| abstract_inverted_index.be | 26 |
| abstract_inverted_index.by | 66, 128, 156, 170 |
| abstract_inverted_index.in | 6 |
| abstract_inverted_index.of | 89, 174, 185 |
| abstract_inverted_index.on | 145 |
| abstract_inverted_index.to | 17, 29, 35, 181 |
| abstract_inverted_index.STR | 56 |
| abstract_inverted_index.and | 41, 62, 86, 110, 147, 165, 188 |
| abstract_inverted_index.due | 34 |
| abstract_inverted_index.the | 15, 36, 172, 183 |
| abstract_inverted_index.way | 180 |
| abstract_inverted_index.IGTR | 75, 99, 122, 152, 176 |
| abstract_inverted_index.etc. | 74 |
| abstract_inverted_index.fast | 166 |
| abstract_inverted_index.from | 140 |
| abstract_inverted_index.have | 2 |
| abstract_inverted_index.rich | 85 |
| abstract_inverted_index.show | 150 |
| abstract_inverted_index.size | 164 |
| abstract_inverted_index.text | 31, 42, 50, 64, 118, 135 |
| abstract_inverted_index.that | 54, 138, 151 |
| abstract_inverted_index.were | 193 |
| abstract_inverted_index.(STR) | 33 |
| abstract_inverted_index.Code: | 196 |
| abstract_inverted_index.e.g., | 70 |
| abstract_inverted_index.first | 76 |
| abstract_inverted_index.head, | 114 |
| abstract_inverted_index.image | 119 |
| abstract_inverted_index.learn | 94 |
| abstract_inverted_index.model | 163 |
| abstract_inverted_index.novel | 47 |
| abstract_inverted_index.scene | 30, 49 |
| abstract_inverted_index.shown | 3 |
| abstract_inverted_index.small | 162 |
| abstract_inverted_index.these | 95 |
| abstract_inverted_index.using | 129 |
| abstract_inverted_index.which | 115, 192 |
| abstract_inverted_index.while | 159 |
| abstract_inverted_index.$\left | 78 |
| abstract_inverted_index.(IGTR) | 52 |
| abstract_inverted_index.answer | 113 |
| abstract_inverted_index.cannot | 25 |
| abstract_inverted_index.evokes | 14 |
| abstract_inverted_index.fusion | 108 |
| abstract_inverted_index.guides | 116 |
| abstract_inverted_index.images | 65 |
| abstract_inverted_index.models | 1, 24, 155 |
| abstract_inverted_index.module | 109 |
| abstract_inverted_index.offers | 177 |
| abstract_inverted_index.rarely | 186 |
| abstract_inverted_index.simply | 127 |
| abstract_inverted_index.speed. | 168 |
| abstract_inverted_index.tackle | 182 |
| abstract_inverted_index.tasks, | 9 |
| abstract_inverted_index.visual | 7, 20 |
| abstract_inverted_index.Chinese | 148 |
| abstract_inverted_index.English | 146 |
| abstract_inverted_index.\langle | 79 |
| abstract_inverted_index.ability | 16 |
| abstract_inverted_index.applied | 28 |
| abstract_inverted_index.between | 39 |
| abstract_inverted_index.current | 23, 141 |
| abstract_inverted_index.devises | 77 |
| abstract_inverted_index.differs | 139 |
| abstract_inverted_index.diverse | 87 |
| abstract_inverted_index.elegant | 179 |
| abstract_inverted_index.feature | 107 |
| abstract_inverted_index.images. | 43 |
| abstract_inverted_index.methods | 142 |
| abstract_inverted_index.natural | 40 |
| abstract_inverted_index.nuanced | 117 |
| abstract_inverted_index.problem | 61 |
| abstract_inverted_index.propose | 45 |
| abstract_inverted_index.similar | 190 |
| abstract_inverted_index.through | 97 |
| abstract_inverted_index.However, | 22 |
| abstract_inverted_index.\rangle$ | 81 |
| abstract_inverted_index.content. | 21 |
| abstract_inverted_index.develops | 100 |
| abstract_inverted_index.enabling | 132 |
| abstract_inverted_index.encoder, | 104 |
| abstract_inverted_index.existing | 154 |
| abstract_inverted_index.learning | 60 |
| abstract_inverted_index.margins, | 158 |
| abstract_inverted_index.paradigm | 53, 137 |
| abstract_inverted_index.previous | 194 |
| abstract_inverted_index.realizes | 123 |
| abstract_inverted_index.sampling | 173 |
| abstract_inverted_index.training | 13 |
| abstract_inverted_index.Moreover, | 169 |
| abstract_inverted_index.adjusting | 171 |
| abstract_inverted_index.appealing | 4 |
| abstract_inverted_index.appearing | 187 |
| abstract_inverted_index.character | 68, 71, 90 |
| abstract_inverted_index.different | 124, 130 |
| abstract_inverted_index.free-form | 11 |
| abstract_inverted_index.inference | 167 |
| abstract_inverted_index.pipelines | 126 |
| abstract_inverted_index.position, | 73 |
| abstract_inverted_index.providing | 84 |
| abstract_inverted_index.reasoning | 136 |
| abstract_inverted_index.triplets, | 83 |
| abstract_inverted_index.trivially | 27 |
| abstract_inverted_index.attributes | 96 |
| abstract_inverted_index.benchmarks | 149 |
| abstract_inverted_index.difference | 38 |
| abstract_inverted_index.formulates | 55 |
| abstract_inverted_index.frequency, | 72 |
| abstract_inverted_index.multi-task | 112 |
| abstract_inverted_index.predicting | 67 |
| abstract_inverted_index.understand | 18 |
| abstract_inverted_index.Experiments | 144 |
| abstract_inverted_index.Multi-modal | 0 |
| abstract_inverted_index.attributes, | 69 |
| abstract_inverted_index.attributes. | 91 |
| abstract_inverted_index.challenges. | 195 |
| abstract_inverted_index.characters, | 191 |
| abstract_inverted_index.cross-modal | 106 |
| abstract_inverted_index.effectively | 93 |
| abstract_inverted_index.instruction | 59, 82, 103 |
| abstract_inverted_index.lightweight | 102 |
| abstract_inverted_index.maintaining | 160 |
| abstract_inverted_index.outperforms | 153 |
| abstract_inverted_index.performance | 5 |
| abstract_inverted_index.recognition | 8, 32, 51, 125, 184 |
| abstract_inverted_index.significant | 157 |
| abstract_inverted_index.text-guided | 12 |
| abstract_inverted_index.understands | 63 |
| abstract_inverted_index.Furthermore, | 121 |
| abstract_inverted_index.descriptions | 88 |
| abstract_inverted_index.fine-grained | 19 |
| abstract_inverted_index.compositional | 37 |
| abstract_inverted_index.considerably. | 143 |
| abstract_inverted_index.instructions, | 131, 175 |
| abstract_inverted_index.understanding. | 120 |
| abstract_inverted_index.morphologically | 189 |
| abstract_inverted_index.instruction-guided | 48 |
| abstract_inverted_index.question-answering, | 98 |
| abstract_inverted_index.character-understanding-based | 134 |
| abstract_inverted_index.condition,question,answer\right | 80 |
| abstract_inverted_index.https://github.com/Topdu/OpenOCR. | 197 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile |