LipCoder: Voice-Enabled Coding Toolkit

Hayoon Kim; Sungho Lee; Juhwi Kim; Bongwon Suh; Kyogu Lee

cs.HCAug 31, 2026

LipCoder: Voice-Enabled Coding Toolkit

Authors: Hayoon Kim, Sungho Lee, Juhwi Kim, Bongwon Suh, Kyogu Lee

Organizations: Music and Audio Research Group · Department of Intelligence and Information, Seoul National University · Independent Researcher · Human-Centered Computing Lab ·

Abstract

AI-assisted programming environments have accelerated software development, giving rise to new paradigms like vibe coding. However, their benefits remain largely inaccessible to visually impaired programmers, as existing screen readers and assistive tools offer limited support for these emerging workflows. We introduce LipCoder, a voice-centric programming toolkit designed to deliver editor-level functionality through auditory and speech-based interfaces. LipCoder offers features comprising speech feedback and earcon cues for comprehension and validation, as well as natural language input for navigation and modification. In an exploratory evaluation, 5 visually impaired programmers performed a series of coding tasks comparing LipCoder with a baseline of VSCode, Copilot, and VoiceOver. Quantitative trends and qualitative feedback point to directions for auditory-first design that may broaden accessibility in speech-driven coding environments.

Figures & tables

	Metric	F1	F2	F3	F4	F5	F6	F7	F8	F9	F10	F11	F12	F13	F14	F15	F16	F17
VI	PU $\uparrow$	3.40	5.20	4.20	5.80	6.60	6.40	6.40	6.20	6.20	7.00	6.80	6.80	6.80	6.40	6.60	7.00	7.00
	PEOU $\uparrow$	4.60	4.60	6.40	6.60	6.20	6.80	6.40	6.40	6.60	6.60	6.40	6.60	6.00	6.40	6.60	7.00	7.00
	BI $\uparrow$	3.40	5.40	4.60	5.20	6.80	6.40	6.00	6.60	6.20	6.80	7.00	7.00	6.80	6.40	7.00	7.00	6.80

Table 3. Perceived acceptance measures across LipCoder features for visually impaired participants (

n=5

), on a 7-point Likert scale (1 = strongly disagree, 7 = strongly agree). PU: Perceived Usefulness

\uparrow

, PEOU: Perceived Ease of Use

\uparrow

, BI: Behavioral Intention to use

\uparrow

Task	System	Visually Impaired ( $n=5$ )
Task	System	SUS $\uparrow$	TLX $\downarrow$	Time $\downarrow$	Acc. $\uparrow$
Basic	Baseline	37.0 ± 12.2	61.7 ± 25.1	13.1 ± 4.2	52.0 ± 48.2
Basic	LipCoder	64.5 ± 13.6	33.9 ± 12.3	10.9 ± 4.0	96.0 ± 8.9
Boost	Baseline	41.5 ± 17.0	42.2 ± 20.2	12.9 ± 2.1	100.0 ± 0.0
Boost	LipCoder	70.5 ± 14.7	29.4 ± 12.8	10.7 ± 4.3	100.0 ± 0.0
Debug	Baseline	45.0 ± 15.4	36.7 ± 21.5	7.1 ± 3.8	100.0 ± 0.0

Table 1. Outcomes by task condition for visually impaired participants (

n=5

), reported as mean ± standard deviation . SUS: System Usability Scale

\uparrow

(0–100), TLX: NASA Task Load Index

\downarrow

(0–100), Time

\downarrow

(min), and Accuracy

\uparrow

(%).

Task	SUS			TLX			Time			Acc.
Task	p	q	$d_{z}$	p	q	$d_{z}$	p	q	$d_{z}$	p	q	$d_{z}$
Basic	0.06	0.27	4.49	0.13	0.40	-1.15	0.59	0.76	-0.30	0.19	0.42	0.82
Boost	0.06	0.27	5.59	0.13	0.40	-1.08	0.28	0.52	-0.60	1.00	1.00	0.00
Debug	0.06	0.27	6.33	0.19	0.42	-0.78	0.63	0.77	0.09	1.00	1.00	0.00
Overall	0.06	0.27	10.15	0.13	0.40	-1.37	0.63	0.77	-0.26	0.19	0.42	0.82

Table 2. Statistical significance test between the baseline and LipCoder for visually impaired participants (

n=5

p

-values (paired t-test or Wilcoxon as appropriate), corrected

q

-values, and effect sizes (Cohen’s

d_{z}

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1. The quadrant model of programming paradigms. The

x

axis corresponds to interaction mode (traditional visual vs. voice-based), and the

y

axis corresponds to the abstraction level of coding (manual utility coding vs. AI-assisted coding). A two-by-two quadrant diagram. The horizontal axis runs from "Unaccessible with Voice" on the left to "Accessible with Voice" on the right; the vertical axis runs from "Low Programming Abstraction" at the bottom to "High Programming Abstraction" at the top. Quadrant A, bottom left in green, is Vision-Centric Utility Coding. Quadrant B, bottom right in blue, is Audio-Centric Utility Coding. Quadrant C, top left in yellow, is Vision-Centric Vibe Coding. Quadrant D, top right in pink, is Audio-Centric Vibe Coding.

Environment & Tool	Main purposes	Functionality	Accessibility	Q
StructJumper (Emacs extension) ( Baker et al., 2015 )	Structure navigation, AST exploration	Navigation	Navigability	A
CodeTalk (Visual Studio plugin) ( Potluri et al., 2018 )	Token reading & summary, structure navigation	Comprehension	Glanceability	B
Tactile Code Skimmer (Hardware) ( Falase et al., 2019 )	Rendering of indentation profiles	Comprehension	Glanceability	B
Emacspeak (Built into Emacs) ( Raman, 1996 )	Menu & buffer event narration	Navigation	Navigability	B
JavaSpeak (Java IDE integration) ( Smith et al., 2000 )	Java project navigation	Navigation	Navigability	B
Sodbeans (NetBeans fork) ( Stefik et al., 2009 )	GUI menu navigation, button/event announcements	Navigation	Navigability	B

Appendix

Table 4. Comparison of non-visual development environments across purposes, programming and accessibility aspects, and quadrant classification.

Figure 2. Overview of the LipCoder system. Illustration of a programmer with closed eyes coding by voice at a keyboard. Numbered speech bubbles show a spoken dialogue with LipCoder: the user says "Open training.py" and LipCoder replies "In editor, training.py"; the user says "Run this code to start training" and LipCoder replies "Running main.py to start training" and then plays an earcon with "Error occurred in line 27, training.py. Seems like a syntax error"; the user says "Go to the problematic line" and then "Fix the syntax error and save", and LipCoder replies "Added missing bracket. Saved training.py"; the user says "Run the code again" and LipCoder replies "Running main.py" and "Successfully ran main.py. Logs and results saved at run 251106". Arrows show that spoken commands pass through an ASR icon into Parse Intent and LipCoder Action blocks, and responses are rendered through a TTS icon; a monitor on the right displays the code editor.

Label	Feature	Modules	Functionality	Accessibility	Q
Screenreader
F1	Diverse timbre vocal per token type	AS + TTS	Comprehension	Glanceability	B
F2	Earcons for matching delimiters	AS	Comprehension	Glanceability	B
F3	Depth level earcons for structure hierarchy	AS	Comprehension	Glanceability	B
F4	Spatial cursor panning	AS + TTS	Navigation	Glanceability	B
Project Overview Readouts

Appendix

Table 5. Mapping of LipCoder features across modules, programming aspects, accessible programming aspects, and quadrant classification. All features can be executed with ASR, text input, or shortcuts. AS refers to Audio scheduler. ASR could be an input for executing all commands, but could also be accessed by shortcuts and typing in the command palette.

Figure 3. Implementation architecture of LipCoder . Some modules are executed on the server side, while others are on the client side; see the legend (bottom of the figure). LipCoder supports four input types: voice, chat, code, and shortcuts. The latter three are keyboard inputs, and immediate auditory feedback is provided. The voice input is routed to the voice analysis stage, comprising Voice Activity Detection (VAD) and Automatic Speech Recognition (ASR) submodules, transcribing the speech into commands. The command text, either given directly by chat or transcribed from voice, is passed to the intention analysis module. In this stage, a dictionary-based command search is first performed, then falls back to a Large Language Model (LLM) if the search gives no results. LipCoder also maintains the current and past status of the project, including the files (and their changes), suggestions, and commands. These status data are passed for the feature execution, along with the code (preprocessed with the syntax analysis module), command, or shortcut inputs. The execution results in various outputs: interaction with UI, visual feedback in the VSCode UI, text responses (both in code and chat), and audio outputs. The text responses are generated with LLM, and audio outputs are controlled by the audio scheduling module, communicating with speech synthesis modules on the server side if necessary. Note that each LipCoder feature executes only the necessary part of the entire system. Block diagram of the LipCoder architecture. On the left, four pink input blocks: Voice, Chat, Code from VSCode, and Shortcuts. Voice feeds a Voice Analysis block (voice activity detection and ASR) whose transcription goes to Intention Analysis, a two-stage block combining dictionary-based command search with fallback to an LLM, supported by a Command Dictionary and a Status and History store of files, suggestions, and commands. Code feeds a Syntax Analysis block (LSP listener and AST parser). All streams converge on a central Feature Execution block. Downstream, Feature Execution fans out to UI Interaction (panel change and navigation), UI Rendering (visual feedback such as git diff and syntax error list), Response Generation (LLM-based text responses for code and chat), and Audio Scheduling (keyed speech and earcons), which drives Speech Synthesis (concatenative or neural TTS) to produce audio outputs of speech and earcons. A legend indicates pink blocks are inputs, green blocks are client modules, yellow blocks are server modules, and blue blocks are outputs.

ID	General Info.		Development Setup			Coding Experience
ID	Type of VI	Gender	Preferred IDE	Screen Reader(s)	LLM / Vibe Coding Tools	Main Languages	Python	Total
P1	Low Vision	Female	PyCharm	SR	ChatGPT	Java, Python	1 yr	1.5 yrs
P2	Blind	Male	Nano	VO, SR, NVDA	ChatGPT / Windsurf, Gemini-cli	Python, JavaScript	5 yrs	6 yrs
P3	Low Vision	Male	VSCode	VO, SR, JAWS	Gemini / Windsurf	Python, Swift	20 yrs	27 yrs
P4	Blind	Male	VSCode	VO, SR, NVDA	ChatGPT, Gemini / Copilot	JavaScript, Python	1 yr	6 yrs
P5	Blind	Male	XCode	VO, SR, JAWS	ChatGPT	Objective-C, Python	6 yrs	10 yrs

Appendix

Table 6. Demographics of visually impaired programmers. For each participant, screen readers, LLM/vibe coding tools, and primary programming languages are listed in order of preference (left to right). Abbreviations: SR = Sense Reader, VO = VoiceOver.

Figure 4. Experiment Procedure. Number units are minutes. Horizontal timeline of the experiment session with durations in minutes. After a pre-survey, participants complete a 15-minute training session. Then a 100-minute block, repeated three times (once per task type), contains two 15-minute programming tasks (version A or B, one with each system), each followed by a post-survey. The session ends with a final survey and a 15-minute interview.

Task	System	Sighted ( $n=20$ )
Task	System	SUS $\uparrow$	TLX $\downarrow$	Time $\downarrow$	Acc. $\uparrow$
Basic	Baseline	57.6 ± 14.4	45.4 ± 12.1	14.2 ± 2.2	57.0 ± 33.3
Basic	LipCoder	63.5 ± 13.4	41.2 ± 14.4	14.2 ± 1.3	82.0 ± 28.2
Boost	Baseline	62.6 ± 16.5	41.4 ± 15.8	12.7 ± 3.6	75.0 ± 33.6
Boost	LipCoder	65.3 ± 14.7	40.9 ± 14.6	14.0 ± 2.3	71.0 ± 29.4
Debug	Baseline	67.8 ± 14.5	34.7 ± 13.8	12.2 ± 4.2	91.0 ± 10.2

Appendix

Table 7. Outcomes by task condition for sighted participants (

n=20

). Each metric is reported as “ mean ± standard deviation ”. Boldface numbers indicate better results. SUS: System Usability Scale

\uparrow

(0–100), TLX: NASA Task Load Index

\downarrow

(0–100), Time

\downarrow

(min), and Accuracy

\uparrow

(%).

Task	SUS			NASA–TLX			Time			Accuracy
Task	p	q	$d_{z}$	p	q	$d_{z}$	p	q	$d_{z}$	p	q	$d_{z}$
Basic	0.145	0.419	0.38	0.350	0.579	-0.26	0.388	0.600	-0.01	0.042	0.273	0.63
Boost	0.420	0.611	0.18	0.794	0.886	-0.00	0.192	0.419	0.28	0.412	0.611	-0.11
Debug	0.302	0.537	0.28	0.513	0.704	-0.02	0.535	0.714	-0.14	1.000	1.000	0.00
Overall	0.245	0.491	0.34	0.701	0.838	-0.17	0.793	0.886	0.06	0.190	0.419	0.38

Appendix

Table 8. Statistical significance test between the baseline and LipCoder for sighted participants (

n=20

). Reported are

p

-values (paired t-test or Wilcoxon as appropriate), FDR-corrected

q

-values, and effect sizes (Cohen’s

d_{z}

). Shaded results indicate that LipCoder showed better results than the baseline.

	Metric	F1	F2	F3	F4	F5	F6	F7	F8	F9	F10	F11	F12	F13	F14	F15	F16	F17
Sighted	PU $\uparrow$	2.40	3.60	2.10	3.21	4.68	3.79	4.26	5.11	4.16	5.53	5.42	4.45	4.79	4.40	4.11	5.95	4.79
	PEOU $\uparrow$	6.40	6.60	6.05	6.26	6.26	6.05	6.11	6.42	6.32	6.26	6.37	6.20	6.26	6.00	5.33	6.37	6.26
	BI $\uparrow$	2.65	4.00	2.50	3.89	4.89	3.89	4.58	5.26	3.74	5.84	5.74	4.40	5.16	4.50	4.50	6.37	5.32

Appendix

Table 9. Perceived acceptance measures across LipCoder features for sighted participants (

n=20

). Each metric is reported on a 7-point Likert scale (1 = strongly disagree, 7 = strongly agree). PU: Perceived Usefulness

\uparrow

, PEOU: Perceived Ease of Use

\uparrow

, BI: Behavioral Intention to use

\uparrow

Human-AI Interaction

Human-AI interaction studies how people and AI systems communicate, collaborate, make decisions, and shape one another over time.

See all 333 papers

Insight: Enhancing Mobile Accessibility for Blind and Visually Impaired Users with LLMs
May 10, 2026Joshua Owusu Ansah, Anuj Kapoor, Ayush Khanna +3Assistive Technology for Blind and Low-Vision People
Arizona State University · Tempe, Arizona, USA
Choose Your Own Adventure: Non-Linear AI-Assisted Programming with EvoGraph
Apr 20, 2026Vassilios Exarhakos, Jinghui Cheng, Jin L. C. GuoHuman-AI Interaction Human-AI Collaboration
McGill University Montreal, Quebec, Canada · Polytechnique Montreal Montreal, Quebec, Canada
Vibe Coding Ate My Homework: An evaluation of AI approaches to greenfield software engineering and programming
Jun 15, 2026Callum BarbourSoftware Engineering Code Generation Evaluation

May 10, 2026J/K move · Enter open · S save

Insight: Enhancing Mobile Accessibility for Blind and Visually Impaired Users with LLMs

Joshua Owusu Ansah, Anuj Kapoor, Ayush Khanna, Manvika Vinod+2

Arizona State University · Tempe, Arizona, USA

Assistive Technology for Blind and Low-Vision People

Open paper AskPDF arXiv

This research paper addresses the limitations of current mobile accessibility services like TalkBack, which provide manual gesture-based sequential feedback to BVI users. Motivated by the promise of large language models (LLMs), this paper introduces Insight, an Android accessibility service that provides natural language interaction and real-time summarization of the screen. The paper performs a within-subject experimental study with users to compare Insight and TalkBack on usability factors. Results show Insight reduced mental effort and task time, and was preferred because of its dialogue interface, but users felt the need for interruption management. Results show LLM-based interfaces can significantly improve mobile accessibility, and describe the potential of hybrid solutions combining gesture and dialogue modalities towards more inclusive design.

Abstract

Human-AI Interaction

Explore similar work

Insight: Enhancing Mobile Accessibility for Blind and Visually Impaired Users with LLMs

Choose Your Own Adventure: Non-Linear AI-Assisted Programming with EvoGraph

Vibe Coding Ate My Homework: An evaluation of AI approaches to greenfield software engineering and programming