1. Gesture-Efficient Global Language Input Methods
Speaker notes
Thank you to the members of the committee: Dr. Wu, Dr. Liang, Dr. Gao, Dr. Wang, and Dr. Huang.
October 9, 2026 dissertation defense
Download the 59-slide narrated PowerPoint with notes (65.4 MB)
These are the presenter’s edited notes for the October 9 defense. For what was said, read the presentation transcript. The public-session Q&A transcript is separate.
Slides 26–37 were hidden reference slides with no individual notes. Slide 25 introduces them as a group. The links below navigate the text, independent of recording time.
Thank you to the members of the committee: Dr. Wu, Dr. Liang, Dr. Gao, Dr. Wang, and Dr. Huang.
I’m going to talk about my dissertation on gesture-efficient global language input methods. First, I’ll explain why this project started. Then I’ll define the problem we are trying to solve and briefly touch on the literature review—all the articles I had to read to do this thing. I’ll describe how the program works and the results of the study, then indicate what still needs to be done and the conclusions based on the study.
There are languages that are hard to type on a QWERTY keyboard. I’ve used the existing input software and haven’t been very happy with it. Being a software engineer, I thought I could do better. Spanish, Japanese, and Chinese were languages I had tried to type. The software works, which is good, but I think it could be better.
Being a scientist as well, I turned the idea of better software into a hypothesis that could be tested. The hypothesis for this engineering project is that we can use fewer gestures for any language when typing individual characters or common words and phrases.
Rather than trying to prove it logically, I build something that uses fewer gestures, if I can. I build an IME—an input method—and compare its gesture counts with existing IMEs. If it has fewer gestures, that supports the hypothesis. If I can’t achieve fewer gestures, the hypothesis is not disproven, but it is not proven either.
We can’t test every language; there are tens of thousands of them. So we pick a few—in this case, English, Spanish, Vietnamese, and Chinese—and test the hypothesis on those.
The first case is Spanish, the easiest one. The normal way to enter these characters is to type Alt codes.
For instance, if I’m in Windows, typing in Word, and need a Spanish character, I hold down Alt and type 0225 for an a with an acute accent. For the capitalized version, I hold down Alt and type 0193. The character appears in Microsoft Word, Excel, or whatever application I’m using.
If you’re typing Spanish, that becomes a pain because you have to look up or memorize the codes. Then you have to type five keys to get a single character. I think we can do better.
It takes five keystrokes to type n with a tilde, and four to type the peseta indicator, the Spanish unit of currency. In Word, it takes six keys. Five keystrokes for one character is not desirable.
Can an IME—an input method—reduce that to one keyboard gesture?
If so, can we make it easily learned? The design for μLex, or μTeclas, the keyboard-only Spanish case study, starts by noticing that n with a tilde is the same base letter as n. There is a one-to-one mapping.
The upside-down question mark resembles the right-side-up question mark. There is a graphical similarity between Spanish quotation marks and the greater-than and less-than signs, and between the peseta sign and capital P. For the euro mark, we rely on a semantic association rather than a visual one.
Using these keys, we can create a one-to-one interface. Hold the key down for an extra beat and you get the target: n with a tilde, or the upside-down question mark.
Spanish is not the only motivation. Other languages also have a single character that resembles one of the QWERTY characters. There are 51 other languages for which we can use the same strategy.
Once we write the software for, say, Basque or Albanian, users can download it, hold down the appropriate key for an extra beat, and go on about their business. Here we have taken six keystrokes and reduced them to one in a program operated using only the keyboard.
Not everybody wants to use only a keyboard. For all those languages, we can also provide a mouse-operated display. You click on what you want and get that as well.
The next class of languages also uses letters, but we can’t use the same one-to-one approach for Vietnamese. It has two times three times six versions of the letter a.
We need a system that presents those possibilities logically, so they can be walked through. We can’t put up two hundred and some odd characters and expect someone to find the one they want easily.
Systems already exist. Windows Vietnamese Telex reduces finding those characters to a memorized set of keys. The number of gestures is basically the number of characters plus the number of marks.
A simple a or n takes one key. A vowel with a form mark, but no tone mark, takes two keys. For an a with the upside-down smile, type aw. For the hat, type aa. For o with the horn, type ow.
That reduces what would normally be six keystrokes with code entry to two or three, using memorized extra keys for the marks on the vowels.
In trying to do better, we really couldn’t. We do the same thing, but use a visual interface. For Vietnamese characters, the hypothesis is disproven, at least so far, because we use the same gestures.
I think the interface is better, but that is for somebody else to decide. These are the same gestures as the Microsoft program, presented visually. So here we can’t do better for Vietnamese characters.
There are 67 other languages in this class. We could write similar software for each of them to make it easier for their users to type more fluidly.
I didn’t discover this until about midway through the study, but concentrating on words and phrases lets us make an improvement for English as well.
If we write the software so that users input words and phrases instead of entering them letter by letter, we can reduce the number of gestures. Using μWords, one of the case studies, “understand” takes two gestures rather than the ten it would take on a normal QWERTY keyboard. I’ll demonstrate that later.
These are the statistics for that. I’ll go over them later.
There are fifty-some languages that are basically QWERTY with no marks. Once we have a corpus to analyze and determine the most frequent words and phrases, we can present the same kind of interface that we use for English. As we’ll see, we can do the same thing for Chinese.
So this applies to English and 56 other languages if we take just QWERTY into account.
There are also about 150 other languages that put marks, sometimes multiple marks, on their letters. If we compile corpora and do frequency analyses for those languages, we can build word-by-word interfaces for them as well.
Presumably, we can reduce the number of gestures from somewhere around ten to somewhere around four once we’ve done the corpus analysis.
For Chinese, the software I got frustrated with is illustrated here: the Microsoft input method. To enter the Chinese word for father-in-law, I type y-u-e. The software supplies the mark indicating that I’ve finished a syllable, then I type f-u.
That takes five keystrokes, then one more—the number 1, or a click on the blue area—to send the word to Word, Notepad, or a spreadsheet.
Can we reduce the number of gestures needed for Chinese words and phrases? That is the question.
There are about a dozen other languages in this class, listed here. I did an interface for Cherokee some years ago and could probably do a better job now. I’m also partway through one for Japanese, but those are not part of this study.
We already have many IMEs. I think there is an organization in the central Chinese government that keeps track of proposed IMEs, and I’m getting ready to add to that.
The goals are to make input methods that are immediately usable and easily learned; compatible with ordinary applications, operating systems, and input devices; and able to increase user productivity over time. Compatibility is actually the hardest part of writing the software.
What do those goals mean? For immediate usability, these interfaces are basically what you see is what you get. Click something and you get it. They are designed to be immediately accessible to native speakers of the particular language—easily learned and immediately usable.
Ordinary applications include Microsoft Word, Excel, Notepad, and other productivity applications. Operating systems include Windows, Linux, OS X on Apple computers, and Android.
Each time we add a language, we would also need a version of its software for each operating system. Technically, that is the hardest part of getting these into users’ hands.
For keyboards, the longpress—the extra beat while holding down a key—keeps us from interfering with Alt, Shift, Control, and function-key combinations to which operating systems and applications assign special meanings. The goal is compatibility with applications, operating systems, and input devices.
One of the most enjoyable parts was figuring out how people could get better at using the software while they were using it. We do that by arranging the grids in particular ways.
First, having grids at all—the last item, display guidance—makes it easy to click on what you see and know that it will be sent to your application.
The grids remain consistent: the same sequence of mouse or keyboard cues always selects the same target. This is cue–target consistency.
Finally, we arrange targets so that the most common ones are easy to find in a left-to-right, top-to-bottom scan. The most common are at the upper left; the least common are at the lower right. You should be able to find what you want in the top few rows.
Let’s walk through the software so you can see what I mean. μTeclas—teclas is Spanish for “keys”—is keyboard-only, letter-by-letter input where targets map one-to-one to keyboard keys. μLetras adds a display. μVietChu introduces branching for further choices. μWords and μCiYu handle English and Chinese words and phrases. The English prototype includes 3,000 of the 330,557 available words and phrases. The Chinese prototype includes 5,000 of the 436,409 available words and phrases. These are available on my website.
In μTeclas, the display is optional. It tells you which key you’ve hit and when the extra beat has been crossed. Type a normally and you get a. Hold it down for an extra beat and you get a with an acute accent. The same works with Shift for uppercase. The keyboard supplies visual cues for the special characters. This is useful for languages with one-to-one mappings between target characters and familiar keyboard cues, and there are several dozen of those.
Not everyone always wants to use a keyboard, so the next class is μLetras. We are still doing Spanish letter by letter, and the keyboard still works. An ordinary g repeats because it has no variation. We can enter capital N with a tilde using shortpresses and longpresses on keys. We can also shortpress or longpress the letters on the display. A shortpress gives a; a longpress gives a with an acute accent. The other mouse button gives capital A. The display tells you when you have crossed the longpress threshold. This class has one-to-one cue–target mappings, with or without a display.
μVietChu shows how multiple targets can share one cue. The a family contains 36 forms: a, ă, and â, each with six tones in lowercase and uppercase. Black indicates what you can insert immediately. Red, accessible by longpress, brings up another grid. Variations with one mark are available with one further gesture; variations with two marks use another gesture on another display. It is essentially one gesture per mark—the same as the existing Windows Telex IME. The important point is that branching leads to new displays, which offer further choices.
We carry that strategy into the English words-and-phrases interface, μWords. What appears in black is accessible immediately by clicking. Red with dots offers more choices on a different grid. In this example, “quarter” is accessible in two gestures. For convenience, we hide what is not available. Hovering over one half of the cell offers a grid of further choices; hovering over the other offers the lexeme to send to the productivity application.
The same strategy works with Mandarin Chinese. Common words and phrases are immediately accessible. If you don’t see the word you want but know its first sound, you can choose D and get a grid of common words and phrases beginning with that sound. Dōu, meaning “all” or “both,” is immediately accessible. There are also choices reached using D-B. One begins with D and the next with B. Here, débùdào means “unable to get.” In two—perhaps three—gestures, we enter three characters that normally take perhaps seven gestures with a pinyin-based interface.
The same simple strategy works all the way down: insert something directly, or find a grid offering further inputs. It is like a card trick that looks very hard but has fairly simple mechanics.
Back to the slides: the prototypes include 3,000 of the 330,557 available English words and phrases, and 5,000 of the 436,409 available Chinese words and phrases. The final versions would cover those inventories.
I read a lot of articles to support and write this dissertation. Rather than go through them individually, I’ll describe the categories.
Several, perhaps a dozen, concern languages and Unicode. Others concern usability and performance. Some discuss other devices, such as one-handed input devices and voice input in particular cases. Those are less central to this dissertation.
To produce word-and-phrase grids, I needed corpus studies and frequency lists. Other articles concern Mandarin input and the Mandarin language, or input software and platforms.
You can see the list of articles I went through. Some were my own: I had a patent on an early version of this earlier in this century. A lot of articles.
Hidden reference slide. See slide 25 for the literature review.
Hidden reference slide. See slide 25 for the literature review.
Hidden reference slide. See slide 25 for the literature review.
Hidden reference slide. See slide 25 for the literature review.
Hidden reference slide. See slide 25 for the literature review.
Hidden reference slide. See slide 25 for the literature review.
Hidden reference slide. See slide 25 for the literature review.
Hidden reference slide. See slide 25 for the literature review.
Hidden reference slide. See slide 25 for the literature review.
Hidden reference slide. See slide 25 for the literature review.
Hidden reference slide. See slide 25 for the literature review.
Hidden reference slide. See slide 25 for the literature review.
Now, how does the software work? It is simpler than it might seem. At its core, it runs a finite state automaton, or finite state machine, using tables to say how we get from one state to another.
I’ll discuss finite state machines after showing how simple the software is at its core, using the tables that drive it.
For μTeclas, the keyboard-only interface, we need to know the cue, what to do on a shortpress, and what to do on a longpress.
The table has rows like this. The cue is n. A shortpress gives n; a longpress gives n with a tilde. Hold Shift and shortpress to get capital N. Longpress with Shift held to get capital N with a tilde (Ñ).
One shared event loop—the finite state machine—handles every record. This simple table drives the software.
Adding the display makes the relationship clearer. The cue appears in the upper-left corner. The primary target is essentially the cue, so it doesn’t need to be shown separately. The secondary target appears in the middle because that is the special character that is otherwise hard to enter.
A shortpress selects n; a longpress selects n with a tilde.
The Vietnamese application has two separate targets. The primary target is the letter a. A label indicates the secondary target, which opens a further grid—a child grid.
The indicator appears in the cell at the left center. Hold and release to select it, and an updated grid appears. Once you make a selection, you return to the home grid.
Similarly for μWords, this table drives that cell. A shortpress sends “have,” H-A-V-E, to the productivity application. A longpress opens a grid of common words and phrases beginning with u.
We use letters similarly in the two systems. For English, they are letters of the spelling. For Chinese, they are letters in the pinyin pronunciation system.
For Chinese words and phrases, the actions are the same: send something to the productivity application and return to the home grid, or open a secondary grid.
This cell shows yào and the letter s. A shortpress gives the character yào. A longpress opens a further grid of common words and phrases that begin with the sound, or pinyin letter, s.
The table is on the left; what it looks like when implemented by the software is on the right.
Those are the tables that run the software. How does the software itself run? A finite state machine, or FSM, is a basic construct in computer science. The software maintains an internal state and reacts to events. Each time it reacts, it may or may not do something.
Starting at the bottom, there are three possible actions: change the way a cell looks; send input and return to the home grid; or display a child grid.
It performs those actions in response to eight things that can happen while you manipulate the software. With a mouse, you can hover over a cell, leave the cell, press on it, reach the timer threshold where shortpress becomes longpress, or release the mouse button or keyboard key. On the keyboard, there are five things: press a key, release it, press with Shift, and release with Shift.
Given those actions and events, the states remember what the user has done. If I’m the program, I know I’m Ready when nothing is happening. I may know the mouse is hovering over a cell, or that a key has been pressed without Shift, preparing the primary target. If Shift is pressed, the user is preparing capital N.
Holding down the key moves us from Ready to Primary. Holding Shift moves us to Shifted. Longpressing moves us to the corresponding secondary state. Releasing returns us to Ready for the next selection. All of that is represented in the finite state diagram.
The finite state diagram accounts for the explanation I just gave. It says what to do in response to what the user is doing.
This is why I said compatibility is the hard part. We do something different for a keyboard, a two-button mouse, or a one-button mouse.
Even with a mouse, there is another variation: the split-cell interface used in the word-and-phrase IMEs, or input methods.
Now we move from implementation to coverage statistics. The core question is: how much of the represented language use can be covered in how few gestures?
These gesture budgets exceed 99% frequency-weighted coverage within each stated corpus inventory. Coverage is the cumulative share of the inventory’s frequency weight contributed by entries selectable within the budget. The listed numbers of lexemes describe the inventories being evaluated. Some less frequent entries require more gestures.
For character input, twelve gestures exceed 99% weighted coverage in both the English BNC-all inventory and the Spanish inventory. The Vietnamese character-input curve reaches that threshold at thirteen gestures. Twelve is therefore a coverage threshold, not the maximum English entry length: the longest retained English entry requires 86 gestures.
Whole-lexeme selection reaches more than 99% weighted coverage in four gestures for both the 3,000-item English prototype and the 5,000-target Chinese prototype. Their complete-coverage budgets are five and seven gestures respectively. The preliminary Vietnamese lexeme model reaches more than 99% in five gestures and every modeled target within ten.
The larger English and Mandarin lexeme models exceed 99% weighted coverage in six gestures. Complete coverage takes fifteen gestures for the English model and twelve for the Mandarin model. The gesture savings come from selecting words and phrases as units; these counts describe the evaluated inventories and modeled paths.
For character-by-character input over the 460,681 retained BNC-all entries, twelve gestures reach 99.04% frequency-weighted coverage. The longest retained entry requires 86 gestures, so complete coverage extends beyond the first fifteen budgets shown here.
At four gestures, the curve reaches about 57.04% of the retained occurrence weight. The percentages describe frequency-weighted coverage within this inventory.
For Spanish character input, twelve gestures reach 99.17% frequency-weighted coverage over 621,606 retained entries. Both Spanish IMEs use one gesture per supported character and share this curve.
Six gestures cover about 78.11% of the occurrence weight, and seven cover about 85.74%. Complete coverage requires 483 gestures because the retained source contains a very long string; that tail is outside the first fifteen budgets plotted here.
For Vietnamese character input over the 23,985 retained forms, thirteen gestures reach 99.31% frequency-weighted coverage. Complete coverage requires 38 gestures.
Marked vowels can require two or three gestures per character. The graph shows the cumulative occurrence weight covered within each gesture budget, including the cost of composing internal spaces.
For the 3,000-lexeme English prototype, whole-lexeme selection reaches 99.76% frequency-weighted coverage within four gestures. Every retained lexeme is reachable within five.
Two gestures already cover about 78.43% of the occurrence weight in this inventory. This measures how much of the represented usage is reachable within the budget.
For the Chinese prototype, three gestures reach about 95.93% of the modeled frequency weight, and four reach 99.17%. Complete coverage requires seven gestures.
The prototype has 5,000 written targets represented by 5,155 Lex+TP entries. The percentages use the modeled weights assigned to those reading-specific entries.
The preliminary Vietnamese lexeme model reaches 99.69% frequency-weighted coverage within five gestures over 23,985 retained forms. All modeled targets are reachable within ten gestures.
The preliminary Vietnamese lexeme model selects fully marked forms through unmarked spelling cues. It first exceeds 99% frequency-weighted coverage within five gestures, compared with thirteen for character-by-character input over the same inventory. Diacritics are included in the selected lexeme, without separate mark-entry gestures.
For the larger English whole-lexeme model, six gestures reach 99.75824% frequency-weighted coverage over 460,681 retained entries and 96,334,201 occurrences. Every modeled entry is reachable within fifteen gestures.
Six gestures is the budget that first exceeds 99% coverage; less frequent entries account for the longer tail.
For the larger Mandarin whole-lexeme model, six gestures reach about 99.15% of the modeled frequency weight. The inventory contains 438,181 Lex+TP entries representing 436,408 written targets. Complete coverage requires twelve gestures.
The cue-to-target strategy matters: a different arrangement could require more gestures. With this pinyin-initial strategy, six gestures cover more than 99% of the modeled weight. Values at ten and eleven gestures round to 100%, but some entries still require twelve.
Those are the statistics. We did well in reducing gestures, especially for words and phrases, and less well when entering text letter by letter.
For μLetras, I would like to split the cells as we did in μWords and μCiYu. For μWords, I would like capitalization through secondary gestures, an interface for spaces included with a word or phrase, and punctuation following it. I would also consider alphabetizing grids for easier lookup.
For the Chinese interface, I would like secondary gestures to select traditional characters and primary gestures to select simplified characters. I would also add access to an entire pinyin syllable and to individual characters.
Human language diversity makes it impossible to do all the hypothesis testing. These case studies suggest that cue systems can be engineered so that word-and-phrase IMEs use substantially fewer gestures than letter-by-letter IMEs.
Thank you very much.
That’s it. That’s all I have.