Distributing User Code with the CernVM FileSystem Article Swipe
The CernVM FileSystem (CVMFS) is widely used in High Throughput Computing to efficiently distributed experiment code. However, the standard CVMFS publishing tools are designed for a small group of people from each experiment to maintain common software, and the tools don't work well for the majority of users that submit jobs related to each experiment. As a result, most user code, such as code to do specific physics analyses, is still sent with jobs. That process is relatively inefficient, especially when the user code is large. To overcome these limitations, we have built a CVMFS user code publication system. Users submit code as a tarball with their jobs, and the job submission client sends the tarball to one of a pair of user code publishing servers. The code is published in one of a small number of shared CVMFS repositories, and updates are expedited through the standard CVMFS infrastructure to be available on worker nodes worldwide within several minutes. A job wrapper automatically passes the location of the code to the jobs. The user code is automatically deleted from CVMFS after a period of no use. Most of the software for the system is available as a single self-contained open source rpm called cvmfs-user-pub, and is available for other deployments.
Related Topics
Concepts
Metadata
- Type
- paratext
- Language
- en
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4288027060
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4288027060Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.5281/zenodo.3599253Digital Object Identifier
- Title
-
Distributing User Code with the CernVM FileSystemWork title
- Type
-
paratextOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2019Year of publication
- Publication date
-
2019-11-07Full publication date if available
- Authors
-
Dave DykstraList of authors in order
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- Concepts
-
Computer science, Code (set theory), Programming language, Operating system, Parallel computing, Set (abstract data type)Top concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4288027060 |
|---|---|
| doi | https://doi.org/10.5281/zenodo.3599253 |
| ids.doi | https://doi.org/10.5281/zenodo.3599253 |
| ids.openalex | https://openalex.org/W4288027060 |
| fwci | 0.0 |
| type | paratext |
| title | Distributing User Code with the CernVM FileSystem |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11181 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9815000295639038 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1705 |
| topics[0].subfield.display_name | Computer Networks and Communications |
| topics[0].display_name | Advanced Data Storage Technologies |
| topics[1].id | https://openalex.org/T10715 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9610000252723694 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1705 |
| topics[1].subfield.display_name | Computer Networks and Communications |
| topics[1].display_name | Distributed and Parallel Computing Systems |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.8318749666213989 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C2776760102 |
| concepts[1].level | 3 |
| concepts[1].score | 0.5846952199935913 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q5139990 |
| concepts[1].display_name | Code (set theory) |
| concepts[2].id | https://openalex.org/C199360897 |
| concepts[2].level | 1 |
| concepts[2].score | 0.5168857574462891 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[2].display_name | Programming language |
| concepts[3].id | https://openalex.org/C111919701 |
| concepts[3].level | 1 |
| concepts[3].score | 0.43598806858062744 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q9135 |
| concepts[3].display_name | Operating system |
| concepts[4].id | https://openalex.org/C173608175 |
| concepts[4].level | 1 |
| concepts[4].score | 0.3236832022666931 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q232661 |
| concepts[4].display_name | Parallel computing |
| concepts[5].id | https://openalex.org/C177264268 |
| concepts[5].level | 2 |
| concepts[5].score | 0.0 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q1514741 |
| concepts[5].display_name | Set (abstract data type) |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.8318749666213989 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/code |
| keywords[1].score | 0.5846952199935913 |
| keywords[1].display_name | Code (set theory) |
| keywords[2].id | https://openalex.org/keywords/programming-language |
| keywords[2].score | 0.5168857574462891 |
| keywords[2].display_name | Programming language |
| keywords[3].id | https://openalex.org/keywords/operating-system |
| keywords[3].score | 0.43598806858062744 |
| keywords[3].display_name | Operating system |
| keywords[4].id | https://openalex.org/keywords/parallel-computing |
| keywords[4].score | 0.3236832022666931 |
| keywords[4].display_name | Parallel computing |
| language | en |
| locations[0].id | pmh:oai:figshare.com:article/11532525 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400572 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | False |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | OPAL (Open@LaTrobe) (La Trobe University) |
| locations[0].source.host_organization | https://openalex.org/I196829312 |
| locations[0].source.host_organization_name | La Trobe University |
| locations[0].source.host_organization_lineage | https://openalex.org/I196829312 |
| locations[0].license | cc-by |
| locations[0].pdf_url | |
| locations[0].version | submittedVersion |
| locations[0].raw_type | Text |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | |
| locations[1].id | pmh:oai:osti.gov:1630708 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306402487 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | False |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) |
| locations[1].source.host_organization | https://openalex.org/I139351228 |
| locations[1].source.host_organization_name | Office of Scientific and Technical Information |
| locations[1].source.host_organization_lineage | https://openalex.org/I139351228 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | submittedVersion |
| locations[1].raw_type | |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | False |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://www.osti.gov/biblio/1630708 |
| locations[2].id | pmh:oai:zenodo.org:3599253 |
| locations[2].is_oa | True |
| locations[2].source.id | https://openalex.org/S4306400562 |
| locations[2].source.issn | |
| locations[2].source.type | repository |
| locations[2].source.is_oa | True |
| locations[2].source.issn_l | |
| locations[2].source.is_core | False |
| locations[2].source.is_in_doaj | False |
| locations[2].source.display_name | Zenodo (CERN European Organization for Nuclear Research) |
| locations[2].source.host_organization | https://openalex.org/I67311998 |
| locations[2].source.host_organization_name | European Organization for Nuclear Research |
| locations[2].source.host_organization_lineage | https://openalex.org/I67311998 |
| locations[2].license | cc-by |
| locations[2].pdf_url | |
| locations[2].version | submittedVersion |
| locations[2].raw_type | info:eu-repo/semantics/lecture |
| locations[2].license_id | https://openalex.org/licenses/cc-by |
| locations[2].is_accepted | False |
| locations[2].is_published | False |
| locations[2].raw_source_name | |
| locations[2].landing_page_url | https://zenodo.org/record/3599253 |
| locations[3].id | doi:10.5281/zenodo.3599253 |
| locations[3].is_oa | True |
| locations[3].source.id | https://openalex.org/S4306400562 |
| locations[3].source.issn | |
| locations[3].source.type | repository |
| locations[3].source.is_oa | True |
| locations[3].source.issn_l | |
| locations[3].source.is_core | False |
| locations[3].source.is_in_doaj | False |
| locations[3].source.display_name | Zenodo (CERN European Organization for Nuclear Research) |
| locations[3].source.host_organization | https://openalex.org/I67311998 |
| locations[3].source.host_organization_name | European Organization for Nuclear Research |
| locations[3].source.host_organization_lineage | https://openalex.org/I67311998 |
| locations[3].license | cc-by |
| locations[3].pdf_url | |
| locations[3].version | |
| locations[3].raw_type | article-journal |
| locations[3].license_id | https://openalex.org/licenses/cc-by |
| locations[3].is_accepted | False |
| locations[3].is_published | |
| locations[3].raw_source_name | |
| locations[3].landing_page_url | https://doi.org/10.5281/zenodo.3599253 |
| indexed_in | datacite |
| authorships[0].author.id | https://openalex.org/A5082448170 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Dave Dykstra |
| authorships[0].countries | US |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I1314696892 |
| authorships[0].affiliations[0].raw_affiliation_string | Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States) |
| authorships[0].institutions[0].id | https://openalex.org/I1314696892 |
| authorships[0].institutions[0].ror | https://ror.org/020hgte69 |
| authorships[0].institutions[0].type | facility |
| authorships[0].institutions[0].lineage | https://openalex.org/I1314696892, https://openalex.org/I1330989302, https://openalex.org/I39565521, https://openalex.org/I4210114836 |
| authorships[0].institutions[0].country_code | US |
| authorships[0].institutions[0].display_name | Fermi National Accelerator Laboratory |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Dykstra, Dave |
| authorships[0].is_corresponding | True |
| authorships[0].raw_affiliation_strings | Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States) |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Distributing User Code with the CernVM FileSystem |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T11181 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9815000295639038 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1705 |
| primary_topic.subfield.display_name | Computer Networks and Communications |
| primary_topic.display_name | Advanced Data Storage Technologies |
| related_works | https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W2382290278, https://openalex.org/W2350741829, https://openalex.org/W2130043461, https://openalex.org/W2056396287 |
| cited_by_count | 0 |
| locations_count | 4 |
| best_oa_location.id | pmh:oai:figshare.com:article/11532525 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400572 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | False |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | OPAL (Open@LaTrobe) (La Trobe University) |
| best_oa_location.source.host_organization | https://openalex.org/I196829312 |
| best_oa_location.source.host_organization_name | La Trobe University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I196829312 |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | Text |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | |
| primary_location.id | pmh:oai:figshare.com:article/11532525 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400572 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | False |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | OPAL (Open@LaTrobe) (La Trobe University) |
| primary_location.source.host_organization | https://openalex.org/I196829312 |
| primary_location.source.host_organization_name | La Trobe University |
| primary_location.source.host_organization_lineage | https://openalex.org/I196829312 |
| primary_location.license | cc-by |
| primary_location.pdf_url | |
| primary_location.version | submittedVersion |
| primary_location.raw_type | Text |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | |
| publication_date | 2019-11-07 |
| publication_year | 2019 |
| referenced_works_count | 0 |
| abstract_inverted_index.A | 159 |
| abstract_inverted_index.a | 25, 56, 93, 103, 119, 133, 181, 196 |
| abstract_inverted_index.As | 55 |
| abstract_inverted_index.To | 86 |
| abstract_inverted_index.as | 62, 102, 195 |
| abstract_inverted_index.be | 150 |
| abstract_inverted_index.do | 65 |
| abstract_inverted_index.in | 7, 130 |
| abstract_inverted_index.is | 4, 69, 76, 84, 128, 175, 193, 205 |
| abstract_inverted_index.no | 184 |
| abstract_inverted_index.of | 28, 46, 118, 121, 132, 136, 166, 183, 187 |
| abstract_inverted_index.on | 152 |
| abstract_inverted_index.to | 11, 33, 52, 64, 116, 149, 169 |
| abstract_inverted_index.we | 90 |
| abstract_inverted_index.The | 0, 126, 172 |
| abstract_inverted_index.and | 37, 108, 140, 204 |
| abstract_inverted_index.are | 22, 142 |
| abstract_inverted_index.for | 24, 43, 190, 207 |
| abstract_inverted_index.job | 110, 160 |
| abstract_inverted_index.one | 117, 131 |
| abstract_inverted_index.rpm | 201 |
| abstract_inverted_index.the | 17, 38, 44, 81, 109, 114, 145, 164, 167, 170, 188, 191 |
| abstract_inverted_index.High | 8 |
| abstract_inverted_index.Most | 186 |
| abstract_inverted_index.That | 74 |
| abstract_inverted_index.code | 63, 83, 96, 101, 123, 127, 168, 174 |
| abstract_inverted_index.each | 31, 53 |
| abstract_inverted_index.from | 30, 178 |
| abstract_inverted_index.have | 91 |
| abstract_inverted_index.jobs | 50 |
| abstract_inverted_index.most | 58 |
| abstract_inverted_index.open | 199 |
| abstract_inverted_index.pair | 120 |
| abstract_inverted_index.sent | 71 |
| abstract_inverted_index.such | 61 |
| abstract_inverted_index.that | 48 |
| abstract_inverted_index.use. | 185 |
| abstract_inverted_index.used | 6 |
| abstract_inverted_index.user | 59, 82, 95, 122, 173 |
| abstract_inverted_index.well | 42 |
| abstract_inverted_index.when | 80 |
| abstract_inverted_index.with | 72, 105 |
| abstract_inverted_index.work | 41 |
| abstract_inverted_index.CVMFS | 19, 94, 138, 147, 179 |
| abstract_inverted_index.Users | 99 |
| abstract_inverted_index.after | 180 |
| abstract_inverted_index.built | 92 |
| abstract_inverted_index.code, | 60 |
| abstract_inverted_index.code. | 15 |
| abstract_inverted_index.don't | 40 |
| abstract_inverted_index.group | 27 |
| abstract_inverted_index.jobs, | 107 |
| abstract_inverted_index.jobs. | 73, 171 |
| abstract_inverted_index.nodes | 154 |
| abstract_inverted_index.other | 208 |
| abstract_inverted_index.sends | 113 |
| abstract_inverted_index.small | 26, 134 |
| abstract_inverted_index.still | 70 |
| abstract_inverted_index.their | 106 |
| abstract_inverted_index.these | 88 |
| abstract_inverted_index.tools | 21, 39 |
| abstract_inverted_index.users | 47 |
| abstract_inverted_index.CernVM | 1 |
| abstract_inverted_index.called | 202 |
| abstract_inverted_index.client | 112 |
| abstract_inverted_index.common | 35 |
| abstract_inverted_index.large. | 85 |
| abstract_inverted_index.number | 135 |
| abstract_inverted_index.passes | 163 |
| abstract_inverted_index.people | 29 |
| abstract_inverted_index.period | 182 |
| abstract_inverted_index.shared | 137 |
| abstract_inverted_index.single | 197 |
| abstract_inverted_index.source | 200 |
| abstract_inverted_index.submit | 49, 100 |
| abstract_inverted_index.system | 192 |
| abstract_inverted_index.widely | 5 |
| abstract_inverted_index.within | 156 |
| abstract_inverted_index.worker | 153 |
| abstract_inverted_index.(CVMFS) | 3 |
| abstract_inverted_index.deleted | 177 |
| abstract_inverted_index.physics | 67 |
| abstract_inverted_index.process | 75 |
| abstract_inverted_index.related | 51 |
| abstract_inverted_index.result, | 57 |
| abstract_inverted_index.several | 157 |
| abstract_inverted_index.system. | 98 |
| abstract_inverted_index.tarball | 104, 115 |
| abstract_inverted_index.through | 144 |
| abstract_inverted_index.updates | 141 |
| abstract_inverted_index.wrapper | 161 |
| abstract_inverted_index.However, | 16 |
| abstract_inverted_index.designed | 23 |
| abstract_inverted_index.location | 165 |
| abstract_inverted_index.maintain | 34 |
| abstract_inverted_index.majority | 45 |
| abstract_inverted_index.minutes. | 158 |
| abstract_inverted_index.overcome | 87 |
| abstract_inverted_index.servers. | 125 |
| abstract_inverted_index.software | 189 |
| abstract_inverted_index.specific | 66 |
| abstract_inverted_index.standard | 18, 146 |
| abstract_inverted_index.Computing | 10 |
| abstract_inverted_index.analyses, | 68 |
| abstract_inverted_index.available | 151, 194, 206 |
| abstract_inverted_index.expedited | 143 |
| abstract_inverted_index.published | 129 |
| abstract_inverted_index.software, | 36 |
| abstract_inverted_index.worldwide | 155 |
| abstract_inverted_index.FileSystem | 2 |
| abstract_inverted_index.Throughput | 9 |
| abstract_inverted_index.especially | 79 |
| abstract_inverted_index.experiment | 14, 32 |
| abstract_inverted_index.publishing | 20, 124 |
| abstract_inverted_index.relatively | 77 |
| abstract_inverted_index.submission | 111 |
| abstract_inverted_index.distributed | 13 |
| abstract_inverted_index.efficiently | 12 |
| abstract_inverted_index.experiment. | 54 |
| abstract_inverted_index.publication | 97 |
| abstract_inverted_index.deployments. | 209 |
| abstract_inverted_index.inefficient, | 78 |
| abstract_inverted_index.limitations, | 89 |
| abstract_inverted_index.automatically | 162, 176 |
| abstract_inverted_index.repositories, | 139 |
| abstract_inverted_index.infrastructure | 148 |
| abstract_inverted_index.self-contained | 198 |
| abstract_inverted_index.cvmfs-user-pub, | 203 |
| cited_by_percentile_year | |
| corresponding_author_ids | https://openalex.org/A5082448170 |
| countries_distinct_count | 1 |
| institutions_distinct_count | 1 |
| corresponding_institution_ids | https://openalex.org/I1314696892 |
| citation_normalized_percentile.value | 0.36865608 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |