Evaluating Generative Models for Graph-to-Text Generation Article Swipe
Shuzhou Yuan
,
Michael Färber
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2307.14712
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2307.14712
Large language models (LLMs) have been widely employed for graph-to-text generation tasks. However, the process of finetuning LLMs requires significant training resources and annotation work. In this paper, we explore the capability of generative models to generate descriptive text from graph data in a zero-shot setting. Specifically, we evaluate GPT-3 and ChatGPT on two graph-to-text datasets and compare their performance with that of finetuned LLM models such as T5 and BART. Our results demonstrate that generative models are capable of generating fluent and coherent text, achieving BLEU scores of 10.57 and 11.08 for the AGENDA and WebNLG datasets, respectively. However, our error analysis reveals that generative models still struggle with understanding the semantic relations between entities, and they also tend to generate text with hallucinations or irrelevant information. As a part of error analysis, we utilize BERT to detect machine-generated text and achieve high macro-F1 scores. We have made the text generated by generative models publicly available.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2307.14712
- https://arxiv.org/pdf/2307.14712
- OA Status
- green
- Cited By
- 3
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4385374140
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4385374140Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2307.14712Digital Object Identifier
- Title
-
Evaluating Generative Models for Graph-to-Text GenerationWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-07-27Full publication date if available
- Authors
-
Shuzhou Yuan, Michael FärberList of authors in order
- Landing page
-
https://arxiv.org/abs/2307.14712Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2307.14712Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2307.14712Direct OA link when available
- Concepts
-
Generative grammar, Computer science, Graph, Generative model, Text generation, Natural language processing, Artificial intelligence, Macro, Information retrieval, Machine learning, Theoretical computer science, Programming languageTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
3Total citation count in OpenAlex
- Citations by year (recent)
-
2023: 3Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4385374140 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2307.14712 |
| ids.doi | https://doi.org/10.48550/arxiv.2307.14712 |
| ids.openalex | https://openalex.org/W4385374140 |
| fwci | |
| type | preprint |
| title | Evaluating Generative Models for Graph-to-Text Generation |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9983999729156494 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T11273 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.954800009727478 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Advanced Graph Neural Networks |
| topics[2].id | https://openalex.org/T11995 |
| topics[2].field.id | https://openalex.org/fields/14 |
| topics[2].field.display_name | Business, Management and Accounting |
| topics[2].score | 0.9366000294685364 |
| topics[2].domain.id | https://openalex.org/domains/2 |
| topics[2].domain.display_name | Social Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1404 |
| topics[2].subfield.display_name | Management Information Systems |
| topics[2].display_name | FinTech, Crowdfunding, Digital Finance |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C39890363 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8390142321586609 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q36108 |
| concepts[0].display_name | Generative grammar |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.793360710144043 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C132525143 |
| concepts[2].level | 2 |
| concepts[2].score | 0.6287534236907959 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q141488 |
| concepts[2].display_name | Graph |
| concepts[3].id | https://openalex.org/C167966045 |
| concepts[3].level | 3 |
| concepts[3].score | 0.6208494901657104 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q5532625 |
| concepts[3].display_name | Generative model |
| concepts[4].id | https://openalex.org/C2985684807 |
| concepts[4].level | 2 |
| concepts[4].score | 0.6107141971588135 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q1513879 |
| concepts[4].display_name | Text generation |
| concepts[5].id | https://openalex.org/C204321447 |
| concepts[5].level | 1 |
| concepts[5].score | 0.595000147819519 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[5].display_name | Natural language processing |
| concepts[6].id | https://openalex.org/C154945302 |
| concepts[6].level | 1 |
| concepts[6].score | 0.5800105929374695 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[6].display_name | Artificial intelligence |
| concepts[7].id | https://openalex.org/C166955791 |
| concepts[7].level | 2 |
| concepts[7].score | 0.5330226421356201 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q629579 |
| concepts[7].display_name | Macro |
| concepts[8].id | https://openalex.org/C23123220 |
| concepts[8].level | 1 |
| concepts[8].score | 0.39664363861083984 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q816826 |
| concepts[8].display_name | Information retrieval |
| concepts[9].id | https://openalex.org/C119857082 |
| concepts[9].level | 1 |
| concepts[9].score | 0.3695894479751587 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[9].display_name | Machine learning |
| concepts[10].id | https://openalex.org/C80444323 |
| concepts[10].level | 1 |
| concepts[10].score | 0.20408865809440613 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q2878974 |
| concepts[10].display_name | Theoretical computer science |
| concepts[11].id | https://openalex.org/C199360897 |
| concepts[11].level | 1 |
| concepts[11].score | 0.08585762977600098 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[11].display_name | Programming language |
| keywords[0].id | https://openalex.org/keywords/generative-grammar |
| keywords[0].score | 0.8390142321586609 |
| keywords[0].display_name | Generative grammar |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.793360710144043 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/graph |
| keywords[2].score | 0.6287534236907959 |
| keywords[2].display_name | Graph |
| keywords[3].id | https://openalex.org/keywords/generative-model |
| keywords[3].score | 0.6208494901657104 |
| keywords[3].display_name | Generative model |
| keywords[4].id | https://openalex.org/keywords/text-generation |
| keywords[4].score | 0.6107141971588135 |
| keywords[4].display_name | Text generation |
| keywords[5].id | https://openalex.org/keywords/natural-language-processing |
| keywords[5].score | 0.595000147819519 |
| keywords[5].display_name | Natural language processing |
| keywords[6].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[6].score | 0.5800105929374695 |
| keywords[6].display_name | Artificial intelligence |
| keywords[7].id | https://openalex.org/keywords/macro |
| keywords[7].score | 0.5330226421356201 |
| keywords[7].display_name | Macro |
| keywords[8].id | https://openalex.org/keywords/information-retrieval |
| keywords[8].score | 0.39664363861083984 |
| keywords[8].display_name | Information retrieval |
| keywords[9].id | https://openalex.org/keywords/machine-learning |
| keywords[9].score | 0.3695894479751587 |
| keywords[9].display_name | Machine learning |
| keywords[10].id | https://openalex.org/keywords/theoretical-computer-science |
| keywords[10].score | 0.20408865809440613 |
| keywords[10].display_name | Theoretical computer science |
| keywords[11].id | https://openalex.org/keywords/programming-language |
| keywords[11].score | 0.08585762977600098 |
| keywords[11].display_name | Programming language |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2307.14712 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2307.14712 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2307.14712 |
| locations[1].id | doi:10.48550/arxiv.2307.14712 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2307.14712 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5008824325 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Shuzhou Yuan |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Yuan, Shuzhou |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5031600582 |
| authorships[1].author.orcid | https://orcid.org/0000-0001-5458-8645 |
| authorships[1].author.display_name | Michael Färber |
| authorships[1].author_position | last |
| authorships[1].raw_author_name | Färber, Michael |
| authorships[1].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2307.14712 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Evaluating Generative Models for Graph-to-Text Generation |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9983999729156494 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W4365211920, https://openalex.org/W3014948380, https://openalex.org/W4380551139, https://openalex.org/W4317695495, https://openalex.org/W4387506531, https://openalex.org/W4238433571, https://openalex.org/W3174044702, https://openalex.org/W2967848559, https://openalex.org/W4299831724, https://openalex.org/W4283803360 |
| cited_by_count | 3 |
| counts_by_year[0].year | 2023 |
| counts_by_year[0].cited_by_count | 3 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2307.14712 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2307.14712 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2307.14712 |
| primary_location.id | pmh:oai:arXiv.org:2307.14712 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2307.14712 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2307.14712 |
| publication_date | 2023-07-27 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 43, 129 |
| abstract_inverted_index.As | 128 |
| abstract_inverted_index.In | 25 |
| abstract_inverted_index.T5 | 68 |
| abstract_inverted_index.We | 146 |
| abstract_inverted_index.as | 67 |
| abstract_inverted_index.by | 152 |
| abstract_inverted_index.in | 42 |
| abstract_inverted_index.of | 15, 32, 62, 79, 88, 131 |
| abstract_inverted_index.on | 52 |
| abstract_inverted_index.or | 125 |
| abstract_inverted_index.to | 35, 120, 137 |
| abstract_inverted_index.we | 28, 47, 134 |
| abstract_inverted_index.LLM | 64 |
| abstract_inverted_index.Our | 71 |
| abstract_inverted_index.and | 22, 50, 56, 69, 82, 90, 95, 116, 141 |
| abstract_inverted_index.are | 77 |
| abstract_inverted_index.for | 8, 92 |
| abstract_inverted_index.our | 100 |
| abstract_inverted_index.the | 13, 30, 93, 111, 149 |
| abstract_inverted_index.two | 53 |
| abstract_inverted_index.BERT | 136 |
| abstract_inverted_index.BLEU | 86 |
| abstract_inverted_index.LLMs | 17 |
| abstract_inverted_index.also | 118 |
| abstract_inverted_index.been | 5 |
| abstract_inverted_index.data | 41 |
| abstract_inverted_index.from | 39 |
| abstract_inverted_index.have | 4, 147 |
| abstract_inverted_index.high | 143 |
| abstract_inverted_index.made | 148 |
| abstract_inverted_index.part | 130 |
| abstract_inverted_index.such | 66 |
| abstract_inverted_index.tend | 119 |
| abstract_inverted_index.text | 38, 122, 140, 150 |
| abstract_inverted_index.that | 61, 74, 104 |
| abstract_inverted_index.they | 117 |
| abstract_inverted_index.this | 26 |
| abstract_inverted_index.with | 60, 109, 123 |
| abstract_inverted_index.10.57 | 89 |
| abstract_inverted_index.11.08 | 91 |
| abstract_inverted_index.BART. | 70 |
| abstract_inverted_index.GPT-3 | 49 |
| abstract_inverted_index.Large | 0 |
| abstract_inverted_index.error | 101, 132 |
| abstract_inverted_index.graph | 40 |
| abstract_inverted_index.still | 107 |
| abstract_inverted_index.text, | 84 |
| abstract_inverted_index.their | 58 |
| abstract_inverted_index.work. | 24 |
| abstract_inverted_index.(LLMs) | 3 |
| abstract_inverted_index.AGENDA | 94 |
| abstract_inverted_index.WebNLG | 96 |
| abstract_inverted_index.detect | 138 |
| abstract_inverted_index.fluent | 81 |
| abstract_inverted_index.models | 2, 34, 65, 76, 106, 154 |
| abstract_inverted_index.paper, | 27 |
| abstract_inverted_index.scores | 87 |
| abstract_inverted_index.tasks. | 11 |
| abstract_inverted_index.widely | 6 |
| abstract_inverted_index.ChatGPT | 51 |
| abstract_inverted_index.achieve | 142 |
| abstract_inverted_index.between | 114 |
| abstract_inverted_index.capable | 78 |
| abstract_inverted_index.compare | 57 |
| abstract_inverted_index.explore | 29 |
| abstract_inverted_index.process | 14 |
| abstract_inverted_index.results | 72 |
| abstract_inverted_index.reveals | 103 |
| abstract_inverted_index.scores. | 145 |
| abstract_inverted_index.utilize | 135 |
| abstract_inverted_index.However, | 12, 99 |
| abstract_inverted_index.analysis | 102 |
| abstract_inverted_index.coherent | 83 |
| abstract_inverted_index.datasets | 55 |
| abstract_inverted_index.employed | 7 |
| abstract_inverted_index.evaluate | 48 |
| abstract_inverted_index.generate | 36, 121 |
| abstract_inverted_index.language | 1 |
| abstract_inverted_index.macro-F1 | 144 |
| abstract_inverted_index.publicly | 155 |
| abstract_inverted_index.requires | 18 |
| abstract_inverted_index.semantic | 112 |
| abstract_inverted_index.setting. | 45 |
| abstract_inverted_index.struggle | 108 |
| abstract_inverted_index.training | 20 |
| abstract_inverted_index.achieving | 85 |
| abstract_inverted_index.analysis, | 133 |
| abstract_inverted_index.datasets, | 97 |
| abstract_inverted_index.entities, | 115 |
| abstract_inverted_index.finetuned | 63 |
| abstract_inverted_index.generated | 151 |
| abstract_inverted_index.relations | 113 |
| abstract_inverted_index.resources | 21 |
| abstract_inverted_index.zero-shot | 44 |
| abstract_inverted_index.annotation | 23 |
| abstract_inverted_index.available. | 156 |
| abstract_inverted_index.capability | 31 |
| abstract_inverted_index.finetuning | 16 |
| abstract_inverted_index.generating | 80 |
| abstract_inverted_index.generation | 10 |
| abstract_inverted_index.generative | 33, 75, 105, 153 |
| abstract_inverted_index.irrelevant | 126 |
| abstract_inverted_index.demonstrate | 73 |
| abstract_inverted_index.descriptive | 37 |
| abstract_inverted_index.performance | 59 |
| abstract_inverted_index.significant | 19 |
| abstract_inverted_index.information. | 127 |
| abstract_inverted_index.Specifically, | 46 |
| abstract_inverted_index.graph-to-text | 9, 54 |
| abstract_inverted_index.respectively. | 98 |
| abstract_inverted_index.understanding | 110 |
| abstract_inverted_index.hallucinations | 124 |
| abstract_inverted_index.machine-generated | 139 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 2 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/4 |
| sustainable_development_goals[0].score | 0.8600000143051147 |
| sustainable_development_goals[0].display_name | Quality Education |
| citation_normalized_percentile |