Human-Free Automated Prompting for Vision-Language Anomaly Detection: Prompt Optimization with Meta-guiding Prompt Scheme Article Swipe
P. C. Chen
,
Jerry Chun‐Wei Lin
,
Jia Ji
,
Feng-Hao Yeh
,
Chao‐Chun Chen
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2406.18197
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2406.18197
Pre-trained vision-language models (VLMs) are highly adaptable to various downstream tasks through few-shot learning, making prompt-based anomaly detection a promising approach. Traditional methods depend on human-crafted prompts that require prior knowledge of specific anomaly types. Our goal is to develop a human-free prompt-based anomaly detection framework that optimally learns prompts through data-driven methods, eliminating the need for human intervention. The primary challenge in this approach is the lack of anomalous samples during the training phase. Additionally, the Vision Transformer (ViT)-based image encoder in VLMs is not ideal for pixel-wise anomaly segmentation due to a locality feature mismatch between the original image and the output feature map. To tackle the first challenge, we have developed the Object-Attention Anomaly Generation Module (OAGM) to synthesize anomaly samples for training. Furthermore, our Meta-Guiding Prompt-Tuning Scheme (MPTS) iteratively adjusts the gradient-based optimization direction of learnable prompts to avoid overfitting to the synthesized anomalies. For the second challenge, we propose Locality-Aware Attention, which ensures that each local patch feature attends only to nearby patch features, preserving the locality features corresponding to their original locations. This framework allows for the optimal prompt embeddings by searching in the continuous latent space via backpropagation, free from human semantic constraints. Additionally, the modified locality-aware attention improves the precision of pixel-wise anomaly segmentation.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2406.18197
- https://arxiv.org/pdf/2406.18197
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4400104535
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4400104535Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2406.18197Digital Object Identifier
- Title
-
Human-Free Automated Prompting for Vision-Language Anomaly Detection: Prompt Optimization with Meta-guiding Prompt SchemeWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-06-26Full publication date if available
- Authors
-
P. C. Chen, Jerry Chun‐Wei Lin, Jia Ji, Feng-Hao Yeh, Chao‐Chun ChenList of authors in order
- Landing page
-
https://arxiv.org/abs/2406.18197Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2406.18197Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2406.18197Direct OA link when available
- Concepts
-
Scheme (mathematics), Anomaly (physics), Anomaly detection, Computer science, Artificial intelligence, Physics, Mathematics, Mathematical analysis, Condensed matter physicsTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4400104535 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2406.18197 |
| ids.doi | https://doi.org/10.48550/arxiv.2406.18197 |
| ids.openalex | https://openalex.org/W4400104535 |
| fwci | |
| type | preprint |
| title | Human-Free Automated Prompting for Vision-Language Anomaly Detection: Prompt Optimization with Meta-guiding Prompt Scheme |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11512 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9948999881744385 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Anomaly Detection Techniques and Applications |
| topics[1].id | https://openalex.org/T11241 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9751999974250793 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1711 |
| topics[1].subfield.display_name | Signal Processing |
| topics[1].display_name | Advanced Malware Detection Techniques |
| topics[2].id | https://openalex.org/T10400 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9746000170707703 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1705 |
| topics[2].subfield.display_name | Computer Networks and Communications |
| topics[2].display_name | Network Security and Intrusion Detection |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C77618280 |
| concepts[0].level | 2 |
| concepts[0].score | 0.6669002771377563 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q1155772 |
| concepts[0].display_name | Scheme (mathematics) |
| concepts[1].id | https://openalex.org/C12997251 |
| concepts[1].level | 2 |
| concepts[1].score | 0.5266389846801758 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q567560 |
| concepts[1].display_name | Anomaly (physics) |
| concepts[2].id | https://openalex.org/C739882 |
| concepts[2].level | 2 |
| concepts[2].score | 0.4768748879432678 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q3560506 |
| concepts[2].display_name | Anomaly detection |
| concepts[3].id | https://openalex.org/C41008148 |
| concepts[3].level | 0 |
| concepts[3].score | 0.464506596326828 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[3].display_name | Computer science |
| concepts[4].id | https://openalex.org/C154945302 |
| concepts[4].level | 1 |
| concepts[4].score | 0.18207314610481262 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[4].display_name | Artificial intelligence |
| concepts[5].id | https://openalex.org/C121332964 |
| concepts[5].level | 0 |
| concepts[5].score | 0.1810457706451416 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q413 |
| concepts[5].display_name | Physics |
| concepts[6].id | https://openalex.org/C33923547 |
| concepts[6].level | 0 |
| concepts[6].score | 0.17640027403831482 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[6].display_name | Mathematics |
| concepts[7].id | https://openalex.org/C134306372 |
| concepts[7].level | 1 |
| concepts[7].score | 0.0 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q7754 |
| concepts[7].display_name | Mathematical analysis |
| concepts[8].id | https://openalex.org/C26873012 |
| concepts[8].level | 1 |
| concepts[8].score | 0.0 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q214781 |
| concepts[8].display_name | Condensed matter physics |
| keywords[0].id | https://openalex.org/keywords/scheme |
| keywords[0].score | 0.6669002771377563 |
| keywords[0].display_name | Scheme (mathematics) |
| keywords[1].id | https://openalex.org/keywords/anomaly |
| keywords[1].score | 0.5266389846801758 |
| keywords[1].display_name | Anomaly (physics) |
| keywords[2].id | https://openalex.org/keywords/anomaly-detection |
| keywords[2].score | 0.4768748879432678 |
| keywords[2].display_name | Anomaly detection |
| keywords[3].id | https://openalex.org/keywords/computer-science |
| keywords[3].score | 0.464506596326828 |
| keywords[3].display_name | Computer science |
| keywords[4].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[4].score | 0.18207314610481262 |
| keywords[4].display_name | Artificial intelligence |
| keywords[5].id | https://openalex.org/keywords/physics |
| keywords[5].score | 0.1810457706451416 |
| keywords[5].display_name | Physics |
| keywords[6].id | https://openalex.org/keywords/mathematics |
| keywords[6].score | 0.17640027403831482 |
| keywords[6].display_name | Mathematics |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2406.18197 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2406.18197 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2406.18197 |
| locations[1].id | doi:10.48550/arxiv.2406.18197 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2406.18197 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5025799171 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | P. C. Chen |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Chen, Pi-Wei |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5000640263 |
| authorships[1].author.orcid | https://orcid.org/0000-0001-8768-9709 |
| authorships[1].author.display_name | Jerry Chun‐Wei Lin |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Lin, Jerry Chun-Wei |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5104858239 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Jia Ji |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Ji, Jia |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5114183560 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Feng-Hao Yeh |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Yeh, Feng-Hao |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5085618113 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-7028-5609 |
| authorships[4].author.display_name | Chao‐Chun Chen |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Chen, Chao-Chun |
| authorships[4].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2406.18197 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2024-06-28T00:00:00 |
| display_name | Human-Free Automated Prompting for Vision-Language Anomaly Detection: Prompt Optimization with Meta-guiding Prompt Scheme |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T11512 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9948999881744385 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Anomaly Detection Techniques and Applications |
| related_works | https://openalex.org/W2806741695, https://openalex.org/W4290647774, https://openalex.org/W3189286258, https://openalex.org/W3207797160, https://openalex.org/W3210364259, https://openalex.org/W4300558037, https://openalex.org/W2912112202, https://openalex.org/W2667207928, https://openalex.org/W4377864969, https://openalex.org/W2972971679 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2406.18197 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2406.18197 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2406.18197 |
| primary_location.id | pmh:oai:arXiv.org:2406.18197 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2406.18197 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2406.18197 |
| publication_date | 2024-06-26 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 18, 40, 93 |
| abstract_inverted_index.To | 106 |
| abstract_inverted_index.by | 186 |
| abstract_inverted_index.in | 62, 82, 188 |
| abstract_inverted_index.is | 37, 65, 84 |
| abstract_inverted_index.of | 31, 68, 138, 208 |
| abstract_inverted_index.on | 24 |
| abstract_inverted_index.to | 7, 38, 92, 120, 141, 144, 165, 174 |
| abstract_inverted_index.we | 111, 152 |
| abstract_inverted_index.For | 148 |
| abstract_inverted_index.Our | 35 |
| abstract_inverted_index.The | 59 |
| abstract_inverted_index.and | 101 |
| abstract_inverted_index.are | 4 |
| abstract_inverted_index.due | 91 |
| abstract_inverted_index.for | 56, 87, 124, 181 |
| abstract_inverted_index.not | 85 |
| abstract_inverted_index.our | 127 |
| abstract_inverted_index.the | 54, 66, 72, 76, 98, 102, 108, 114, 134, 145, 149, 170, 182, 189, 201, 206 |
| abstract_inverted_index.via | 193 |
| abstract_inverted_index.This | 178 |
| abstract_inverted_index.VLMs | 83 |
| abstract_inverted_index.each | 159 |
| abstract_inverted_index.free | 195 |
| abstract_inverted_index.from | 196 |
| abstract_inverted_index.goal | 36 |
| abstract_inverted_index.have | 112 |
| abstract_inverted_index.lack | 67 |
| abstract_inverted_index.map. | 105 |
| abstract_inverted_index.need | 55 |
| abstract_inverted_index.only | 164 |
| abstract_inverted_index.that | 27, 46, 158 |
| abstract_inverted_index.this | 63 |
| abstract_inverted_index.avoid | 142 |
| abstract_inverted_index.first | 109 |
| abstract_inverted_index.human | 57, 197 |
| abstract_inverted_index.ideal | 86 |
| abstract_inverted_index.image | 80, 100 |
| abstract_inverted_index.local | 160 |
| abstract_inverted_index.patch | 161, 167 |
| abstract_inverted_index.prior | 29 |
| abstract_inverted_index.space | 192 |
| abstract_inverted_index.tasks | 10 |
| abstract_inverted_index.their | 175 |
| abstract_inverted_index.which | 156 |
| abstract_inverted_index.(MPTS) | 131 |
| abstract_inverted_index.(OAGM) | 119 |
| abstract_inverted_index.(VLMs) | 3 |
| abstract_inverted_index.Module | 118 |
| abstract_inverted_index.Scheme | 130 |
| abstract_inverted_index.Vision | 77 |
| abstract_inverted_index.allows | 180 |
| abstract_inverted_index.depend | 23 |
| abstract_inverted_index.during | 71 |
| abstract_inverted_index.highly | 5 |
| abstract_inverted_index.latent | 191 |
| abstract_inverted_index.learns | 48 |
| abstract_inverted_index.making | 14 |
| abstract_inverted_index.models | 2 |
| abstract_inverted_index.nearby | 166 |
| abstract_inverted_index.output | 103 |
| abstract_inverted_index.phase. | 74 |
| abstract_inverted_index.prompt | 184 |
| abstract_inverted_index.second | 150 |
| abstract_inverted_index.tackle | 107 |
| abstract_inverted_index.types. | 34 |
| abstract_inverted_index.Anomaly | 116 |
| abstract_inverted_index.adjusts | 133 |
| abstract_inverted_index.anomaly | 16, 33, 43, 89, 122, 210 |
| abstract_inverted_index.attends | 163 |
| abstract_inverted_index.between | 97 |
| abstract_inverted_index.develop | 39 |
| abstract_inverted_index.encoder | 81 |
| abstract_inverted_index.ensures | 157 |
| abstract_inverted_index.feature | 95, 104, 162 |
| abstract_inverted_index.methods | 22 |
| abstract_inverted_index.optimal | 183 |
| abstract_inverted_index.primary | 60 |
| abstract_inverted_index.prompts | 26, 49, 140 |
| abstract_inverted_index.propose | 153 |
| abstract_inverted_index.require | 28 |
| abstract_inverted_index.samples | 70, 123 |
| abstract_inverted_index.through | 11, 50 |
| abstract_inverted_index.various | 8 |
| abstract_inverted_index.approach | 64 |
| abstract_inverted_index.features | 172 |
| abstract_inverted_index.few-shot | 12 |
| abstract_inverted_index.improves | 205 |
| abstract_inverted_index.locality | 94, 171 |
| abstract_inverted_index.methods, | 52 |
| abstract_inverted_index.mismatch | 96 |
| abstract_inverted_index.modified | 202 |
| abstract_inverted_index.original | 99, 176 |
| abstract_inverted_index.semantic | 198 |
| abstract_inverted_index.specific | 32 |
| abstract_inverted_index.training | 73 |
| abstract_inverted_index.adaptable | 6 |
| abstract_inverted_index.anomalous | 69 |
| abstract_inverted_index.approach. | 20 |
| abstract_inverted_index.attention | 204 |
| abstract_inverted_index.challenge | 61 |
| abstract_inverted_index.detection | 17, 44 |
| abstract_inverted_index.developed | 113 |
| abstract_inverted_index.direction | 137 |
| abstract_inverted_index.features, | 168 |
| abstract_inverted_index.framework | 45, 179 |
| abstract_inverted_index.knowledge | 30 |
| abstract_inverted_index.learnable | 139 |
| abstract_inverted_index.learning, | 13 |
| abstract_inverted_index.optimally | 47 |
| abstract_inverted_index.precision | 207 |
| abstract_inverted_index.promising | 19 |
| abstract_inverted_index.searching | 187 |
| abstract_inverted_index.training. | 125 |
| abstract_inverted_index.Attention, | 155 |
| abstract_inverted_index.Generation | 117 |
| abstract_inverted_index.anomalies. | 147 |
| abstract_inverted_index.challenge, | 110, 151 |
| abstract_inverted_index.continuous | 190 |
| abstract_inverted_index.downstream | 9 |
| abstract_inverted_index.embeddings | 185 |
| abstract_inverted_index.human-free | 41 |
| abstract_inverted_index.locations. | 177 |
| abstract_inverted_index.pixel-wise | 88, 209 |
| abstract_inverted_index.preserving | 169 |
| abstract_inverted_index.synthesize | 121 |
| abstract_inverted_index.(ViT)-based | 79 |
| abstract_inverted_index.Pre-trained | 0 |
| abstract_inverted_index.Traditional | 21 |
| abstract_inverted_index.Transformer | 78 |
| abstract_inverted_index.data-driven | 51 |
| abstract_inverted_index.eliminating | 53 |
| abstract_inverted_index.iteratively | 132 |
| abstract_inverted_index.overfitting | 143 |
| abstract_inverted_index.synthesized | 146 |
| abstract_inverted_index.Furthermore, | 126 |
| abstract_inverted_index.Meta-Guiding | 128 |
| abstract_inverted_index.constraints. | 199 |
| abstract_inverted_index.optimization | 136 |
| abstract_inverted_index.prompt-based | 15, 42 |
| abstract_inverted_index.segmentation | 90 |
| abstract_inverted_index.Additionally, | 75, 200 |
| abstract_inverted_index.Prompt-Tuning | 129 |
| abstract_inverted_index.corresponding | 173 |
| abstract_inverted_index.human-crafted | 25 |
| abstract_inverted_index.intervention. | 58 |
| abstract_inverted_index.segmentation. | 211 |
| abstract_inverted_index.Locality-Aware | 154 |
| abstract_inverted_index.gradient-based | 135 |
| abstract_inverted_index.locality-aware | 203 |
| abstract_inverted_index.vision-language | 1 |
| abstract_inverted_index.Object-Attention | 115 |
| abstract_inverted_index.backpropagation, | 194 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile |