Incremental IVF Index Maintenance for Streaming Vector Search Article Swipe
Jason Mohoney
,
Anil Pacaci
,
Shihabur Rahman Chowdhury
,
Umar Farooq Minhas
,
Jeffery Pound
,
Cédric Renggli
,
Nima Reyhani
,
Ihab F. Ilyas
,
Theodoros Rekatsinas
,
Shivaram Venkataraman
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2411.00970
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2411.00970
The prevalence of vector similarity search in modern machine learning applications and the continuously changing nature of data processed by these applications necessitate efficient and effective index maintenance techniques for vector search indexes. Designed primarily for static workloads, existing vector search indexes degrade in search quality and performance as the underlying data is updated unless costly index reconstruction is performed. To address this, we introduce Ada-IVF, an incremental indexing methodology for Inverted File (IVF) indexes. Ada-IVF consists of 1) an adaptive maintenance policy that decides which index partitions are problematic for performance and should be repartitioned and 2) a local re-clustering mechanism that determines how to repartition them. Compared with state-of-the-art dynamic IVF index maintenance strategies, Ada-IVF achieves an average of 2x and up to 5x higher update throughput across a range of benchmark workloads.
Related Topics
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2411.00970
- https://arxiv.org/pdf/2411.00970
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4404350843
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4404350843Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2411.00970Digital Object Identifier
- Title
-
Incremental IVF Index Maintenance for Streaming Vector SearchWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-11-01Full publication date if available
- Authors
-
Jason Mohoney, Anil Pacaci, Shihabur Rahman Chowdhury, Umar Farooq Minhas, Jeffery Pound, Cédric Renggli, Nima Reyhani, Ihab F. Ilyas, Theodoros Rekatsinas, Shivaram VenkataramanList of authors in order
- Landing page
-
https://arxiv.org/abs/2411.00970Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2411.00970Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2411.00970Direct OA link when available
- Concepts
-
Index (typography), Computer science, Statistics, Mathematics, World Wide WebTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4404350843 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2411.00970 |
| ids.doi | https://doi.org/10.48550/arxiv.2411.00970 |
| ids.openalex | https://openalex.org/W4404350843 |
| fwci | |
| type | preprint |
| title | Incremental IVF Index Maintenance for Streaming Vector Search |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11269 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9563999772071838 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Algorithms and Data Compression |
| topics[1].id | https://openalex.org/T12101 |
| topics[1].field.id | https://openalex.org/fields/18 |
| topics[1].field.display_name | Decision Sciences |
| topics[1].score | 0.9433000087738037 |
| topics[1].domain.id | https://openalex.org/domains/2 |
| topics[1].domain.display_name | Social Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1803 |
| topics[1].subfield.display_name | Management Science and Operations Research |
| topics[1].display_name | Advanced Bandit Algorithms Research |
| topics[2].id | https://openalex.org/T12761 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.919700026512146 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Data Stream Mining Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2777382242 |
| concepts[0].level | 2 |
| concepts[0].score | 0.6511908769607544 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q6017816 |
| concepts[0].display_name | Index (typography) |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.42560267448425293 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C105795698 |
| concepts[2].level | 1 |
| concepts[2].score | 0.37057656049728394 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q12483 |
| concepts[2].display_name | Statistics |
| concepts[3].id | https://openalex.org/C33923547 |
| concepts[3].level | 0 |
| concepts[3].score | 0.3208468556404114 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[3].display_name | Mathematics |
| concepts[4].id | https://openalex.org/C136764020 |
| concepts[4].level | 1 |
| concepts[4].score | 0.16532132029533386 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q466 |
| concepts[4].display_name | World Wide Web |
| keywords[0].id | https://openalex.org/keywords/index |
| keywords[0].score | 0.6511908769607544 |
| keywords[0].display_name | Index (typography) |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.42560267448425293 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/statistics |
| keywords[2].score | 0.37057656049728394 |
| keywords[2].display_name | Statistics |
| keywords[3].id | https://openalex.org/keywords/mathematics |
| keywords[3].score | 0.3208468556404114 |
| keywords[3].display_name | Mathematics |
| keywords[4].id | https://openalex.org/keywords/world-wide-web |
| keywords[4].score | 0.16532132029533386 |
| keywords[4].display_name | World Wide Web |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2411.00970 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2411.00970 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2411.00970 |
| locations[1].id | doi:10.48550/arxiv.2411.00970 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2411.00970 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5041665675 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-5497-0481 |
| authorships[0].author.display_name | Jason Mohoney |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Mohoney, Jason |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5052360032 |
| authorships[1].author.orcid | https://orcid.org/0000-0003-4994-8014 |
| authorships[1].author.display_name | Anil Pacaci |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Pacaci, Anil |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5055538536 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-6232-2027 |
| authorships[2].author.display_name | Shihabur Rahman Chowdhury |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Chowdhury, Shihabur Rahman |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5032253873 |
| authorships[3].author.orcid | https://orcid.org/0009-0005-6520-3794 |
| authorships[3].author.display_name | Umar Farooq Minhas |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Minhas, Umar Farooq |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5114637044 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | Jeffery Pound |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Pound, Jeffery |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5063262952 |
| authorships[5].author.orcid | https://orcid.org/0000-0003-3271-3059 |
| authorships[5].author.display_name | Cédric Renggli |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Renggli, Cedric |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5074375305 |
| authorships[6].author.orcid | |
| authorships[6].author.display_name | Nima Reyhani |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Reyhani, Nima |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5000141065 |
| authorships[7].author.orcid | https://orcid.org/0000-0001-9052-9714 |
| authorships[7].author.display_name | Ihab F. Ilyas |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Ilyas, Ihab F. |
| authorships[7].is_corresponding | False |
| authorships[8].author.id | https://openalex.org/A5002060759 |
| authorships[8].author.orcid | https://orcid.org/0000-0001-6148-1854 |
| authorships[8].author.display_name | Theodoros Rekatsinas |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Rekatsinas, Theodoros |
| authorships[8].is_corresponding | False |
| authorships[9].author.id | https://openalex.org/A5082710354 |
| authorships[9].author.orcid | https://orcid.org/0000-0001-9575-7935 |
| authorships[9].author.display_name | Shivaram Venkataraman |
| authorships[9].author_position | last |
| authorships[9].raw_author_name | Venkataraman, Shivaram |
| authorships[9].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2411.00970 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Incremental IVF Index Maintenance for Streaming Vector Search |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T11269 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9563999772071838 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Algorithms and Data Compression |
| related_works | https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W4391375266, https://openalex.org/W1979597421, https://openalex.org/W2007980826, https://openalex.org/W2061531152, https://openalex.org/W3002753104, https://openalex.org/W2077600819, https://openalex.org/W2142036596, https://openalex.org/W2072657027 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2411.00970 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2411.00970 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2411.00970 |
| primary_location.id | pmh:oai:arXiv.org:2411.00970 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2411.00970 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2411.00970 |
| publication_date | 2024-11-01 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 98, 130 |
| abstract_inverted_index.1) | 78 |
| abstract_inverted_index.2) | 97 |
| abstract_inverted_index.2x | 121 |
| abstract_inverted_index.5x | 125 |
| abstract_inverted_index.To | 60 |
| abstract_inverted_index.an | 66, 79, 118 |
| abstract_inverted_index.as | 48 |
| abstract_inverted_index.be | 94 |
| abstract_inverted_index.by | 19 |
| abstract_inverted_index.in | 6, 43 |
| abstract_inverted_index.is | 52, 58 |
| abstract_inverted_index.of | 2, 16, 77, 120, 132 |
| abstract_inverted_index.to | 105, 124 |
| abstract_inverted_index.up | 123 |
| abstract_inverted_index.we | 63 |
| abstract_inverted_index.IVF | 112 |
| abstract_inverted_index.The | 0 |
| abstract_inverted_index.and | 11, 24, 46, 92, 96, 122 |
| abstract_inverted_index.are | 88 |
| abstract_inverted_index.for | 29, 35, 70, 90 |
| abstract_inverted_index.how | 104 |
| abstract_inverted_index.the | 12, 49 |
| abstract_inverted_index.File | 72 |
| abstract_inverted_index.data | 17, 51 |
| abstract_inverted_index.that | 83, 102 |
| abstract_inverted_index.with | 109 |
| abstract_inverted_index.(IVF) | 73 |
| abstract_inverted_index.index | 26, 56, 86, 113 |
| abstract_inverted_index.local | 99 |
| abstract_inverted_index.range | 131 |
| abstract_inverted_index.them. | 107 |
| abstract_inverted_index.these | 20 |
| abstract_inverted_index.this, | 62 |
| abstract_inverted_index.which | 85 |
| abstract_inverted_index.across | 129 |
| abstract_inverted_index.costly | 55 |
| abstract_inverted_index.higher | 126 |
| abstract_inverted_index.modern | 7 |
| abstract_inverted_index.nature | 15 |
| abstract_inverted_index.policy | 82 |
| abstract_inverted_index.search | 5, 31, 40, 44 |
| abstract_inverted_index.should | 93 |
| abstract_inverted_index.static | 36 |
| abstract_inverted_index.unless | 54 |
| abstract_inverted_index.update | 127 |
| abstract_inverted_index.vector | 3, 30, 39 |
| abstract_inverted_index.Ada-IVF | 75, 116 |
| abstract_inverted_index.address | 61 |
| abstract_inverted_index.average | 119 |
| abstract_inverted_index.decides | 84 |
| abstract_inverted_index.degrade | 42 |
| abstract_inverted_index.dynamic | 111 |
| abstract_inverted_index.indexes | 41 |
| abstract_inverted_index.machine | 8 |
| abstract_inverted_index.quality | 45 |
| abstract_inverted_index.updated | 53 |
| abstract_inverted_index.Ada-IVF, | 65 |
| abstract_inverted_index.Compared | 108 |
| abstract_inverted_index.Designed | 33 |
| abstract_inverted_index.Inverted | 71 |
| abstract_inverted_index.achieves | 117 |
| abstract_inverted_index.adaptive | 80 |
| abstract_inverted_index.changing | 14 |
| abstract_inverted_index.consists | 76 |
| abstract_inverted_index.existing | 38 |
| abstract_inverted_index.indexes. | 32, 74 |
| abstract_inverted_index.indexing | 68 |
| abstract_inverted_index.learning | 9 |
| abstract_inverted_index.benchmark | 133 |
| abstract_inverted_index.effective | 25 |
| abstract_inverted_index.efficient | 23 |
| abstract_inverted_index.introduce | 64 |
| abstract_inverted_index.mechanism | 101 |
| abstract_inverted_index.primarily | 34 |
| abstract_inverted_index.processed | 18 |
| abstract_inverted_index.determines | 103 |
| abstract_inverted_index.partitions | 87 |
| abstract_inverted_index.performed. | 59 |
| abstract_inverted_index.prevalence | 1 |
| abstract_inverted_index.similarity | 4 |
| abstract_inverted_index.techniques | 28 |
| abstract_inverted_index.throughput | 128 |
| abstract_inverted_index.underlying | 50 |
| abstract_inverted_index.workloads, | 37 |
| abstract_inverted_index.workloads. | 134 |
| abstract_inverted_index.incremental | 67 |
| abstract_inverted_index.maintenance | 27, 81, 114 |
| abstract_inverted_index.methodology | 69 |
| abstract_inverted_index.necessitate | 22 |
| abstract_inverted_index.performance | 47, 91 |
| abstract_inverted_index.problematic | 89 |
| abstract_inverted_index.repartition | 106 |
| abstract_inverted_index.strategies, | 115 |
| abstract_inverted_index.applications | 10, 21 |
| abstract_inverted_index.continuously | 13 |
| abstract_inverted_index.re-clustering | 100 |
| abstract_inverted_index.repartitioned | 95 |
| abstract_inverted_index.reconstruction | 57 |
| abstract_inverted_index.state-of-the-art | 110 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 10 |
| citation_normalized_percentile |