Kenneth Holstein
YOU?
Author Swipe
View article: WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI
WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI Open
There has been growing interest from both practitioners and researchers in engaging end users in AI auditing, to draw upon users' unique knowledge and lived experiences. However, we know little about how to effectively scaffold end users i…
View article: Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks
Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks Open
Data scientists often formulate predictive modeling tasks involving fuzzy, hard-to-define concepts, such as the ''authenticity'' of student writing or the ''healthcare need'' of a patient. Yet the process by which data scientists translate…
View article: Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz
Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz Open
Recent advancements in multimodal generative AI (GenAI) enable the creation of personal context-aware real-time agents that, for example, can augment user workflows by following their on-screen activities and providing contextual assistanc…
View article: Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas
Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas Open
As Generative AI (GenAI) systems see growing adoption, a key concern involves the external validity of evaluations, or the extent to which they generalize from lab-based to real-world deployment conditions. Threats to the external validity…
View article: Funding AI for Good: A Call for Meaningful Engagement
Funding AI for Good: A Call for Meaningful Engagement Open
Artificial Intelligence for Social Good (AI4SG) is a growing area that explores AI's potential to address social issues, such as public health. Yet prior work has shown limited evidence of its tangible benefits for intended communities, an…
View article: Making the Right Thing: Bridging HCI and Responsible AI in Early-Stage AI Concept Selection
Making the Right Thing: Bridging HCI and Responsible AI in Early-Stage AI Concept Selection Open
AI projects often fail due to financial, technical, ethical, or user acceptance challenges -- failures frequently rooted in early-stage decisions. While HCI and Responsible AI (RAI) research emphasize this, practical approaches for identif…
View article: Exploring the Potential of Metacognitive Support Agents for Human-AI Co-Creation
Exploring the Potential of Metacognitive Support Agents for Human-AI Co-Creation Open
Despite the potential of generative AI (GenAI) design tools to enhance design processes, professionals often struggle to integrate AI into their workflows. Fundamental cognitive challenges include the need to specify all design criteria as…
View article: Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflows
Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflows Open
Despite Generative AI (GenAI) systems' potential for enhancing content creation, users often struggle to effectively integrate GenAI into their creative workflows. Core challenges include misalignment of AI-generated content with user inte…
View article: AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development
AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development Open
AI systems are often introduced with high expectations, yet many fail to deliver, resulting in unintended harm and missed opportunities for benefit. We frequently observe significant "AI Mismatches", where the system's actual performance f…
View article: WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI
WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI Open
There has been growing interest from both practitioners and researchers in engaging end users in AI auditing, to draw upon users’ unique knowledge and lived experiences. However, we know little about how to effectively scaffold end users i…
View article: WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI
WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI Open
There has been growing interest from both practitioners and researchers in engaging end users in AI auditing, to draw upon users' unique knowledge and lived experiences. However, we know little about how to effectively scaffold end users i…
View article: Studying Up Public Sector AI: How Networks of Power Relations Shape Agency Decisions Around AI Design and Use
Studying Up Public Sector AI: How Networks of Power Relations Shape Agency Decisions Around AI Design and Use Open
As public sector agencies rapidly introduce new AI tools in high-stakes domains like social services, it becomes critical to understand how decisions to adopt these tools are made in practice. We borrow from the anthropological practice to…
View article: Responsible Crowdsourcing for Responsible Generative AI: Engaging Crowds in AI Auditing and Evaluation
Responsible Crowdsourcing for Responsible Generative AI: Engaging Crowds in AI Auditing and Evaluation Open
With the rise of generative AI (GenAI), there has been an increased need for participation by large and diverse user bases in AI evaluation and auditing. GenAI developers are increasingly adopting crowdsourcing approaches to test and audit…
View article: Investigating What Factors Influence Users’ Rating of Harmful Algorithmic Bias and Discrimination
Investigating What Factors Influence Users’ Rating of Harmful Algorithmic Bias and Discrimination Open
There has been growing recognition of the crucial role users, especially those from marginalized groups, play in uncovering harmful algorithmic biases. However, it remains unclear how users’ identities and experiences might impact their ra…
View article: PolicyCraft: Supporting Collaborative and Participatory Policy Design through Case-Grounded Deliberation
PolicyCraft: Supporting Collaborative and Participatory Policy Design through Case-Grounded Deliberation Open
Community and organizational policies are typically designed in a top-down, centralized fashion, with limited input from impacted stakeholders. This can result in policies that are misaligned with community needs or perceived as illegitima…
View article: Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration
Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration Open
Across various applications, humans increasingly use black-box artificial intelligence (AI) systems without insight into these systems' reasoning. To counter this opacity, explainable AI (XAI) methods promise enhanced transparency and inte…
View article: Carefully Unmaking the “Marginalized User”: A Diffractive Analysis of a Gay Online Community
Carefully Unmaking the “Marginalized User”: A Diffractive Analysis of a Gay Online Community Open
HCI scholars are increasingly engaging in research about “marginalized groups,” such as LGBTQ+ people. While normative habitual readings of marginalized people in HCI often highlight real problems, this work has been criticized for flatten…
View article: AI Failure Cards: Understanding and Supporting Grassroots Efforts to Mitigate AI Failures in Homeless Services
AI Failure Cards: Understanding and Supporting Grassroots Efforts to Mitigate AI Failures in Homeless Services Open
AI-based decision support tools have been used in a wide range of high-stakes settings. However, many of them have failed. Past literature in FAccT contributes important insights into how to detect and mitigate AI failures from a technical…
View article: Building, Shifting, & Employing Power: A Taxonomy of Responses From Below to Algorithmic Harm
Building, Shifting, & Employing Power: A Taxonomy of Responses From Below to Algorithmic Harm Open
A large body of research has attempted to ensure that algorithmic systems adhere to notions of fairness and transparency. Increasingly, researchers have highlighted that mitigating algorithmic harms requires explicitly taking power structu…
View article: Studying Up Public Sector AI: How Networks of Power Relations Shape Agency Decisions Around AI Design and Use
Studying Up Public Sector AI: How Networks of Power Relations Shape Agency Decisions Around AI Design and Use Open
As public sector agencies rapidly introduce new AI tools in high-stakes domains like social services, it becomes critical to understand how decisions to adopt these tools are made in practice. We borrow from the anthropological practice to…
View article: Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia
Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia Open
AI tools are increasingly deployed in community contexts. However, datasets used to evaluate AI are typically created by developers and annotators outside a given community, which can yield misleading conclusions about AI performance. How …
View article: Trust and Reliance in Evolving Human-AI Workflows (TREW)
Trust and Reliance in Evolving Human-AI Workflows (TREW) Open
State-of-the-art AIs, including Large Language Models (LLMs) like GPT-4, now possess capabilities once unique to humans, such as coding, idea generation, and planning. Advanced AIs are now integrated into a plethora of platforms and tools,…
View article: The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder, Early-stage Deliberations Around Public Sector AI Proposals
The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder, Early-stage Deliberations Around Public Sector AI Proposals Open
Public sector agencies are rapidly deploying AI systems to augment or automate critical decisions in real-world contexts like child welfare, criminal justice, and public health. A growing body of work documents how these AI systems often f…
View article: An Evidence-based Workflow for Studying and Designing Learning Supports for Human-AI Co-creation
An Evidence-based Workflow for Studying and Designing Learning Supports for Human-AI Co-creation Open
Generative artificial intelligence (GenAI) systems introduce new possibilities for enhancing professionals' workflows, enabling novel forms of human–AI co-creation. However, professionals often struggle to learn to work with GenAI systems …
View article: Predictive Performance Comparison of Decision Policies Under Confounding
Predictive Performance Comparison of Decision Policies Under Confounding Open
Predictive models are often introduced to decision-making tasks under the rationale that they improve performance over an existing decision-making policy. However, it is challenging to compare predictive performance against an existing dec…