LNCS 4555 - An Accessible and Usable Soft Keyboard - Springer Link
An Accessible and Usable Soft Keyboard Alexandros Mourouzis1, Evangelos Boutsakis1, Stavroula Ntoa1, Margherita Antona1, and Constantine Stephanidis1,2 1
Foundation for Research and Technology – Hellas (FORTH), Institute of Computer Science, Heraklion, GR-70013, Greece {mourouzi,boutsak,stant,antona,cs}@ics.forth.gr 2 University of Crete, Department of Computer Science
Abstract. AUK is a 3x3 multi-tier onscreen keyboard. It supports various entry modes, including 1 to 10-key and joystick modes, allowing text entry with a remarkable range of devices. This paper presents the menu structure of AUK, the alternative entry modes, and several layouts for novice, moderate and expert users. The potential of AUK, as a text entry solution both for disabled and ablebodied users, is discussed. Overall, the work presented here is considered as a contribution to Universal Access and towards ambient text entry. Keywords: Text entry, soft keyboard, Universal Access, Ambient Intelligence.
1 Introduction Manual text entry, as the human process of communicating text to computer-based systems, is, in essence, an alternative way of writing. The main difference is that, in text entry, the storage medium is a computer memory rather than paper. Naturally, the medium itself poses certain limitations. Computers require users to select among a set of encoded characters1, whereas paper, as a flat surface, implies the use of 2D symbols. However, the new medium brought along new options that changed writing forever. For instance, it enabled authors to act on transcribed texts without imposing restrictions for modifications, and to copy-and-paste text sections easily. In these terms, text entry is merely a selection procedure [1], and can be defined as follows (adapted from [2]): T(i, C, M) is the human process of selecting from a set of encoded characters C2 and modifying options M (backspace, delete, etc.), by means of a total number i of discrete and detectable user actions (e.g., tap a key). According to [3], there have been two major waves of research on text entry, focussing on mainstream desktop computers and on mobile devices respectively. However, today a number of reasons call for practical solutions for further widening the potential user base and context of use of text entry systems. Such reasons include the need for inclusion into the information society [4] and the emergence of ambient intelligence environments [5]. This paper presents AUK, a multi-device soft keyboard for accessible, usable and efficient text entry across multiple contexts of use. 1 2
Graphemes, symbols, etc., or a set of encoded strings of characters, i.e., words or sentences. E.g., the 95 printable ASCII characters based on English, numbered 32-126.
C. Stephanidis (Ed.): Universal Access in HCI, Part II, HCII 2007, LNCS 4555, pp. 961–970, 2007. © Springer-Verlag Berlin Heidelberg 2007
962
A. Mourouzis et al.
2 Background Typing, originally introduced by mechanical typewriters, has been a diachronic entry method since the early emergence of computers and command line interfaces. In this method, there is a correspondence between characters and a set of keys spread across a device. Keyboards with different numbers of keys exist, but the 101-keys keyboard is predominant. In these terms, typing can be considered a T(i, C, M), where i equals 101+, C includes 95+ characters2, and M includes numerous options for modifying text. Such options are mainly of two types: navigation / selection (back, forward, home, select all, etc.) and modifying / processing functions (delete, backspace, copypaste, etc.). Clearly, with a 101+ keys keyboard, C and M together exceed the number of input controls (i.e., of keys), thus synchronous and asynchronous key combinations (e.g., though the SHIFT and CAPS LOCK key) are employed to allow associating more than one character (or modifying option) to a single key. With the emergence of graphical interfaces, menu-based text entry was introduced as a supplement of typing, allowing users to enter additional characters3 not found on the keyboard through menus. In this T(i, C, M) case, if navigation across a characters set is made through the keys back, forward, down, up, and enter, i equals 5, C includes, for example, 256 characters4, and M is an empty set. Such menus can be perceived as a primitive form of soft or onscreen keyboards. These are features of a program or operating system that generate a graphical keyboard operated through mouse, stylus, etc. Soft keyboards are particularly useful in two cases: (a) for making text entry accessible to people with manual dexterities - e.g., it can be combined with scanning techniques [6] and allow operation through few special switches; (b) for removing keypads from mobile devices and reducing their physical size – e.g., characters can be selected through tapping on a touch display. In both cases, soft keyboards are usually a representation of a full-size QWERTY keyboard. With the emergence of mobile devices, typing and menu-based methods were combined to allow text entry through few-keys [3]. The success of SMS messaging on mobile phones raised the attention on 12-key keypads, with letters A-Z traditionally encoded on eight keys5. A number of 12-key entry methods exist so that more than one letters can be assigned to each key [7], but the most common are multi-tap and dictionary-based disambiguation6 (for descriptions see [8, 9]). Finally, other widely explored entry techniques include speech, handwriting or sign language recognition [1, 3, 9] and virtual keyboards7 [10]. These techniques, up today, have failed to enter mainstream technologies, mainly because they do not reach acceptable levels of speed and accuracy, two primary performance metrics for text entry [3]. For details on text entry issues and methods, see [1, 3, 8, 9, and 11]. 3
4 5 6 7
Over the years, the volume of encoded characters increased dramatically and, ultimately, included characters from numerous writing systems around the world. From ISO 8859-1 (the most popular encoding in the Western world with 256 characters). See ITU E.161 international recommendation for telephone keypads at http://www.itu.int. The most common application is T9 by Tegic Communications (http://www.t9.com). Although the term virtual keyboards (VKs) is often used interchangeably with soft keyboards, VKs are defined as touch-typing keyboards that do not have physical manifestation of the sensing areas [10], and thus should be perceived as different from onscreen keyboards.
An Accessible and Usable Soft Keyboard
963
3 The AUK Concept The work presented here was originally motivated from previous experiences with scanning techniques for users with limited motor functionality of upper limbs. The evaluation of an accessible Web browser [12], which included a QWERTY-like soft keyboard operated via three special switches, indicated frustrating rates of words per minute (wpm)8. To overcome this barrier, the idea was put forward to use the layouts and current practices in mobile phones and implement a 12-key soft keyboard for offering movement minimisation and high learnability thanks to familiarity. Such a keyboard was anticipated to allow entry with various devices for typing (i.e., keypads), 2D indication and selection (e.g., a mouse), or simple selection (e.g., a single switch) if a scanning technique is built in. Then, having in mind that the letters A-Z on a 12-key keypad are typically encoded on eight keys, an 8-direction method9 was explored for text entry also by means of a joystick: In that case, if one takes into account the joystick’s default state (centre), there are 9 possible positions that can be mapped to a 3x3 grid, which leads us to the basic concept of AUK. That is, a multitier 3x3 menu system, consisting of a primary menu, where eight cells are used for entering letters and one cell is reserved for entering into additional, again 3x3, menus that accommodate extra characters and options. Initially, letters were placed alphabetically, similarly to phone keypads (see Fig.1), and the multi-tap technique was considered for character disambiguation, employing system timeout to segment consecutive letters on the same key. The main benefit of such an approach is clear: familiarity. However, there are also some disadvantages recorded: (a) timeouts slow down users; (b) with an alphabetic layout, some frequent letters require more keystrokes than some less frequently needed letters [1]. Thus, as for people with reduced manual dexterity movement minimisation is a primary requirement, less keystrokes per character (KSPC [8]) were further pursued. AUK AUK
Foot pedals
Switch
AUK stands for Accessible & Usable Keyboard
Hello world! 1-key
Special switches
3-key
TV control
This is John typing his first text using the AUK system.
2-key
Wheelchair control
Multidevice text entry
This is NIC
«enter
;
L
abc jkl
Joystick
Telephone
PDA
pqrs
tuv
9 wxyz
Stylus 10-key
def ’ mno
ghi 5-key
AUK Logo
U
L Dictionary nice nicely nickname nicotine nictitate nice and soft niceness and over… nice kettle of fish
Botoni MT ABC
Transcribed text display Dictionary-based word prediction or disambiguation Visual 3x3, multi-tier keyboard System status bar
Fig. 1. Overview of AUK10 soft keyboard for multi-device text entry (stand-alone version) 8
The main reason is that the QWERTY layout has been designed for ten-fingers typing and not for linear or tree-based navigation as in scanning techniques. 9 With one direction N, E, S, W, NE, ES, SW, WN used for each of the eight keys. 10 About the AUK logo: According to wikipedia.com, auks are birds that look like, but aren’t penguins; they are considered as a product of “moderate convergent evolution”.
964
A. Mourouzis et al.
In summary, AUK supports (a) several different entry modes (1 to10-key, and joystick), rendering it appropriate for the widest possible range of input devices; (b) various layout configurations for different performance levels; and (c) the potential to integrate additional techniques, such as text prediction and dictionary-based or prefixbased disambiguation, as these ensure even lower KSPC values [ 3, 8, 10].
4 A Multi-tier 3x3 Menu Structure AUK is similar to a multi-tier 3x3 menu system. Each 3x3 grid has eight virtual character keys and a menu key for entering or exiting alternative menus. Five basic menus can be interchanged sequentially11 (Fig. 2). In the primary menu (1), the letters A-Z, the apostrophe, SPACE (SP), BACK (BK), and SHIFT (SH)12 are placed on the character keys, and the menu key is used for timeout kill, swapping to menu (2), or entering a linear menu (D) for navigations, e.g., within a word prediction list13. Additionally, there are five secondary menus: numerals (6), special characters (7), brackets (8), formatting options (9), and numeric operators (10). Finally, menu (5) has a cell for selecting extra characters through menu (D), and an empty cell to allow space for adding new menus as necessary, thus rendering the structure extensible.
5 Entry Modes and Menu Layouts The presented menu structure can be employed with various input methods. In particular, given T(i, C, M), AUK supports text entry for: −
−
−
11
i=9. This is the basic mode of AUK, in which there is a correspondence between the nine AUK menu options (see Fig. 2) and nine keys (e.g., phone keypads). This mode has two main variations: (a) the 0-key mode for command without click [15] e.g., by means of eye-tracking; and (b) the joystick mode. i9. In this category focus is mainly on 10-key modes for number pads or 10thumbs typing (see section 8). In the first case, the tenth key (zero key) is used for
E.g., in the 9-key mode, starting from the menu (1), the key combination 1-1-1-1-1 will result returning to menu (1), after visiting the text navigation menu (2), the punctuation menu (3), the mode menu (4), and the extend menu (5). If the user selects any character key in one of these menus, then the menu key changes to the exit key for returning directly to (1). 12 SP, BK, and SH are very common [14]; in AUK, SP, being the most common key, is always the most easy to reach; the apostrophe, indicates missing letters, and thus is included here. 13 If dictionary-based disambiguation is activated, then a list of alternative words is displayed. 14 An automatic scanning technique is employed in this case. 15 Two methods are used for the 3-key mode: the date stamp method [see 985], and one proposed for AUK, through the buttons DOWN, RIGHT and SELECT, as this provides less KSPC.
An Accessible and Usable Soft Keyboard
965
timeout kill (single tap) and for the SH modifier (hold down). A second approach, mainly for 10-thumbs-typing, is to place the SP key alone, and use the tenth key for timeout kill (first tap) and BK (additional taps). Furthermore, in this second case, the space key may also be used for the CTRL modifier (hold down) and the tenth key for SH, whereas SH+SP is for entering menu (2) - see Fig. 4. Indicatively, the arrangement of keys for the 3-, 9-, 10-key and joystick modes are presented in Fig. 3, along with the indicative entry scenarios presented in Table 1. (1b)
Menu (1)
(1a)
;
« enter L
« enter
def ’
abc
2
mno
ghi
jkl
pqrs
tuv
L
tuv
.
?
2*
“
,
!
“
-
:
;
-
.
?
,
!
:
;
12
3
More options
Pg Up
123
4,
5.
6
Home
Caps Lock
End
7
80
9
)
Pg Dn
alt
77
7
(10)
(D’)
+
3
1
jkl
nice nickel nicely nicotine
select 6
tuv
9
5
b
9
8
8
1
(9)
(7)
9,
c
tab
exit
#
~
exit
e
« enter
X
`
@
&
B
delete
d
ctrl
_
|
\
exit
exit
×
insert symbol
{
(
)
-
+
=
}
[
]
,
÷
√
F1-12
+
@
U
8
7
1
9
(D’)
(10)
(1)
(9)
8
U
I change fonts
(10)
(8)
exit
1
L Dictionary
Turn dictionary off
2
3
(5)
exit
1
a
H1
wxyz 7
1
(4)
(D)
def ’ mno
5
(2)
Caps Lock
2
pqrs
wxyz
(6) (1)
4
(3)
exit
; abc
ghi
9
wxyz
1
L
1
jkl
pqrs
(3a) (1)
2
mno
ghi
9
« enter
def ’
; abc
%
Fig. 2. The basic menu structure of AUK (for i=9) Joystick
9-key
« enter L
1
Menu (1)
;
2
def ’
jkl 5
pqrs
6
tuv
9
wxyz
7
Menu (1)
« enter
;
Ã
iuy
srj ’ É
L
twgv
9
9
(minTap)
dcx
tsw
L
(R-lessTap)
eiu ’
hmfk
hfj
nmv
;
rgk
.
?
Caps Lock
.
“
“
,
!
“
Caps Lock
!
,
?
!
-
:
;
-
:
;
-
:
;
« enter L
;i
u y
a o q
e l z
9d c x
tw gv
edf ’ 9
onm lkj 6
5
srpq
tuv
9
wyxz 1
,
acb 8
ihg
«enter
?
Horizontal Key-bar (minTimeout)
L
4
lpq
Æ
« enter 7
odyx npb
acbz
.
Caps Lock
Menu (3)
9
8
elz
3
mno
ghi 4
abc
aoq
10-key
3-key:
(minTimeout)
(R-standard)
3
2
.
?
“
,
!
-
:
;
Caps Lock
s r j ’
hm fk
; exit 0
ctrl
exit 0
n pb
Fig. 3. Keys arrangement for modes 3-, 9-, 10-key and joystick. The bottom part shows the AUK menu as a horizontal bar for PDAs (see section 6).
966
A. Mourouzis et al.
6 Layouts for Novice, Moderate and Expert Performance A significant amount of research has been conducted on 12-key text entry methods for mobile phones. Most efforts aim at achieving the upper limits of typing speed with a particular design, often at the expenses of an acceptable experience for novice users [3]. Yet, the ease of use for first-time users is critical for the success of any method. For this purpose, in addition to the entry mode described above, different A-Z letters layouts have been designed for novice, moderate and expert users. The following are layouts for multi-tap and involve the use of system timeout. Table 1. Examples of user input, resulting transcribed text and menu transitions User Input (with layouts in Fig. 3) 9-key stan)
(R-
Joystick (minTout )
3-key ( minTap)
10-key (R-Lesstap)
Menus onscreen ‡
Text (transcribed)
1
3
(1)
_
2,3
+,
,,, ,
0+9
(1)
E_
3,3,4,4,4
,
,,, ,
9,4
(1)
ei_
2,2,1,2,2,2
,, ,
,,, ,,,
8,0,8,8
(1)
ac_
3,3,8
,
,,T, ,
9,2
(1)
et
3,3,9
,
,,T,
9,3
(1)
e_
(1>2>3>1)
e._
3,3,1,1,1,2
,,,
,,,, ,, ,,,
9,7,7,2
1+5
H,
H,
1+5
(1>D>1)
1st word_
7,0+1
(1>2>1)
(ctrl+s)
1,1,9,7,7,7,7
,, ,
T: system timeout
,,, ,,, ,,,, _: text cursor’s position
: SPACE
H: Hold down key until system timeou t
R-standard. This layout is the standard phone layout5 adapted to include SP, BK, SH and apostrophe to the character keys with the lowest overall frequency (see the example layout for 9-key in Fig. 3). This is a familiar and easy to remember layout. R-lessTap. This layout is an adaptation of the LessTap [13], in which letters within each key of the standard layout are rearranged according to their frequency in the English Language16 and thus achieving lower KSPC. R-LessTap simply includes the four extra options mentioned above (see the layout for 10-key in Fig. 3). Taking into account that several studies [e.g., 16] have indicated that an alphabetic order is not critical for user experience, more effective layouts can be created by disregarding the alphabetical order all together, as described below. minTap. In this layout, the more common an input is, the easier it is to enter it. This is achieved through the following algorithm: (a) inputs are sorted by frequency order17; (b) all menu positions are coded, e.g., from 1-001 (representing the 1st position, on key 1, in menu 1) up to n-900 (representing the third position, on key 9, in menu n), and sorted by their reachability order (e.g., the number of keystrokes (KS) required to select each position); (c) the most frequent input (i.e., SP) is assigned to the position with the best reachability, and so on; (d) if an option can be assigned to 16 17
E.g., see http://en.wikipedia.org/wiki/Letter_frequencies (used here for KSPC calculations). Frequency order used here: SP BS eta SH oinshrdlcumwfgypbv apostrophe kjxqz.
An Accessible and Usable Soft Keyboard
967
more than one positions that share equal reachability values, then it is placed on the key that currently has the lowest overall frequency18. An example appears in Fig. 3. minTimout. In this layout, (a) letters on each key have the lowest possible digraph frequencies [11] in an English text corpus in order to have consecutive letters on the same key (and thus timeouts) less frequently19, and (b) the letters on each key are arranged in decreasing letter frequency order – thus ensuring less ‘taps’ for frequent letters (see the layout used for the joystick mode in Fig. 3). R-qwerty. In this layout, characters are grouped and arranged in a way that is related to the QWERTY layout. Such relation can be vision-based (e.g., having the letters “qwer” grouped together on the same key) or motor-based (i.e., with keys that are assigned to a specific finger20 being grouped together (see Fig. 4). In modes with i=9 for automatically rearranging the letters within each key, so that the first letter is always the most probable. In addition, AUK may have the following general modes: (a) multi-tap; (b) dictionary based disambiguation; (c) prefix based disambiguation [18]; (d) with or without text prediction [19] displayed though the menu (D); (e) synthetic speech output enabled for blind users. Finally, the following display options have been considered: (a) 3x3 keypad; (b) vertical or horizontal bar (e.g., for PDAs – see Fig. 3); and (c) cursor view (e.g., see Fig. 4).
7 Discussion: Universal Access to Text Entry The proposed 3x3 structure shows a number of benefits. First of all, it complies with the fact that visual search involves “combination of gestalting the area surrounding the current center of interest –3x3 matrix of letters– and using linear search … as a back up strategy” [17]. Yet, this structure offers access to a large and extensible input set, and thus AUK can operate as a stand-alone word processor or even as a plug-in for other applications. Furthermore, it is potentially easy to learn and use for users with different expertise, since: (a) functions can be placed as close to the user expectations, e.g., for improving novice experience [9], or according to their (individual or related) probabilities for expert performance; (b) onscreen visual21 and aural feedback is provided for character entry and menu transitions; and (c) the backspace key is always the second most reachable option, thus allowing fast error recovery [20]. In addition, since i
Foundation for Research and Technology – Hellas (FORTH), Institute of Computer Science, Heraklion, GR-70013, Greece {mourouzi,boutsak,stant,antona,cs}@ics.forth.gr 2 University of Crete, Department of Computer Science
Abstract. AUK is a 3x3 multi-tier onscreen keyboard. It supports various entry modes, including 1 to 10-key and joystick modes, allowing text entry with a remarkable range of devices. This paper presents the menu structure of AUK, the alternative entry modes, and several layouts for novice, moderate and expert users. The potential of AUK, as a text entry solution both for disabled and ablebodied users, is discussed. Overall, the work presented here is considered as a contribution to Universal Access and towards ambient text entry. Keywords: Text entry, soft keyboard, Universal Access, Ambient Intelligence.
1 Introduction Manual text entry, as the human process of communicating text to computer-based systems, is, in essence, an alternative way of writing. The main difference is that, in text entry, the storage medium is a computer memory rather than paper. Naturally, the medium itself poses certain limitations. Computers require users to select among a set of encoded characters1, whereas paper, as a flat surface, implies the use of 2D symbols. However, the new medium brought along new options that changed writing forever. For instance, it enabled authors to act on transcribed texts without imposing restrictions for modifications, and to copy-and-paste text sections easily. In these terms, text entry is merely a selection procedure [1], and can be defined as follows (adapted from [2]): T(i, C, M) is the human process of selecting from a set of encoded characters C2 and modifying options M (backspace, delete, etc.), by means of a total number i of discrete and detectable user actions (e.g., tap a key). According to [3], there have been two major waves of research on text entry, focussing on mainstream desktop computers and on mobile devices respectively. However, today a number of reasons call for practical solutions for further widening the potential user base and context of use of text entry systems. Such reasons include the need for inclusion into the information society [4] and the emergence of ambient intelligence environments [5]. This paper presents AUK, a multi-device soft keyboard for accessible, usable and efficient text entry across multiple contexts of use. 1 2
Graphemes, symbols, etc., or a set of encoded strings of characters, i.e., words or sentences. E.g., the 95 printable ASCII characters based on English, numbered 32-126.
C. Stephanidis (Ed.): Universal Access in HCI, Part II, HCII 2007, LNCS 4555, pp. 961–970, 2007. © Springer-Verlag Berlin Heidelberg 2007
962
A. Mourouzis et al.
2 Background Typing, originally introduced by mechanical typewriters, has been a diachronic entry method since the early emergence of computers and command line interfaces. In this method, there is a correspondence between characters and a set of keys spread across a device. Keyboards with different numbers of keys exist, but the 101-keys keyboard is predominant. In these terms, typing can be considered a T(i, C, M), where i equals 101+, C includes 95+ characters2, and M includes numerous options for modifying text. Such options are mainly of two types: navigation / selection (back, forward, home, select all, etc.) and modifying / processing functions (delete, backspace, copypaste, etc.). Clearly, with a 101+ keys keyboard, C and M together exceed the number of input controls (i.e., of keys), thus synchronous and asynchronous key combinations (e.g., though the SHIFT and CAPS LOCK key) are employed to allow associating more than one character (or modifying option) to a single key. With the emergence of graphical interfaces, menu-based text entry was introduced as a supplement of typing, allowing users to enter additional characters3 not found on the keyboard through menus. In this T(i, C, M) case, if navigation across a characters set is made through the keys back, forward, down, up, and enter, i equals 5, C includes, for example, 256 characters4, and M is an empty set. Such menus can be perceived as a primitive form of soft or onscreen keyboards. These are features of a program or operating system that generate a graphical keyboard operated through mouse, stylus, etc. Soft keyboards are particularly useful in two cases: (a) for making text entry accessible to people with manual dexterities - e.g., it can be combined with scanning techniques [6] and allow operation through few special switches; (b) for removing keypads from mobile devices and reducing their physical size – e.g., characters can be selected through tapping on a touch display. In both cases, soft keyboards are usually a representation of a full-size QWERTY keyboard. With the emergence of mobile devices, typing and menu-based methods were combined to allow text entry through few-keys [3]. The success of SMS messaging on mobile phones raised the attention on 12-key keypads, with letters A-Z traditionally encoded on eight keys5. A number of 12-key entry methods exist so that more than one letters can be assigned to each key [7], but the most common are multi-tap and dictionary-based disambiguation6 (for descriptions see [8, 9]). Finally, other widely explored entry techniques include speech, handwriting or sign language recognition [1, 3, 9] and virtual keyboards7 [10]. These techniques, up today, have failed to enter mainstream technologies, mainly because they do not reach acceptable levels of speed and accuracy, two primary performance metrics for text entry [3]. For details on text entry issues and methods, see [1, 3, 8, 9, and 11]. 3
4 5 6 7
Over the years, the volume of encoded characters increased dramatically and, ultimately, included characters from numerous writing systems around the world. From ISO 8859-1 (the most popular encoding in the Western world with 256 characters). See ITU E.161 international recommendation for telephone keypads at http://www.itu.int. The most common application is T9 by Tegic Communications (http://www.t9.com). Although the term virtual keyboards (VKs) is often used interchangeably with soft keyboards, VKs are defined as touch-typing keyboards that do not have physical manifestation of the sensing areas [10], and thus should be perceived as different from onscreen keyboards.
An Accessible and Usable Soft Keyboard
963
3 The AUK Concept The work presented here was originally motivated from previous experiences with scanning techniques for users with limited motor functionality of upper limbs. The evaluation of an accessible Web browser [12], which included a QWERTY-like soft keyboard operated via three special switches, indicated frustrating rates of words per minute (wpm)8. To overcome this barrier, the idea was put forward to use the layouts and current practices in mobile phones and implement a 12-key soft keyboard for offering movement minimisation and high learnability thanks to familiarity. Such a keyboard was anticipated to allow entry with various devices for typing (i.e., keypads), 2D indication and selection (e.g., a mouse), or simple selection (e.g., a single switch) if a scanning technique is built in. Then, having in mind that the letters A-Z on a 12-key keypad are typically encoded on eight keys, an 8-direction method9 was explored for text entry also by means of a joystick: In that case, if one takes into account the joystick’s default state (centre), there are 9 possible positions that can be mapped to a 3x3 grid, which leads us to the basic concept of AUK. That is, a multitier 3x3 menu system, consisting of a primary menu, where eight cells are used for entering letters and one cell is reserved for entering into additional, again 3x3, menus that accommodate extra characters and options. Initially, letters were placed alphabetically, similarly to phone keypads (see Fig.1), and the multi-tap technique was considered for character disambiguation, employing system timeout to segment consecutive letters on the same key. The main benefit of such an approach is clear: familiarity. However, there are also some disadvantages recorded: (a) timeouts slow down users; (b) with an alphabetic layout, some frequent letters require more keystrokes than some less frequently needed letters [1]. Thus, as for people with reduced manual dexterity movement minimisation is a primary requirement, less keystrokes per character (KSPC [8]) were further pursued. AUK AUK
Foot pedals
Switch
AUK stands for Accessible & Usable Keyboard
Hello world! 1-key
Special switches
3-key
TV control
This is John typing his first text using the AUK system.
2-key
Wheelchair control
Multidevice text entry
This is NIC
«enter
;
L
abc jkl
Joystick
Telephone
PDA
pqrs
tuv
9 wxyz
Stylus 10-key
def ’ mno
ghi 5-key
AUK Logo
U
L Dictionary nice nicely nickname nicotine nictitate nice and soft niceness and over… nice kettle of fish
Botoni MT ABC
Transcribed text display Dictionary-based word prediction or disambiguation Visual 3x3, multi-tier keyboard System status bar
Fig. 1. Overview of AUK10 soft keyboard for multi-device text entry (stand-alone version) 8
The main reason is that the QWERTY layout has been designed for ten-fingers typing and not for linear or tree-based navigation as in scanning techniques. 9 With one direction N, E, S, W, NE, ES, SW, WN used for each of the eight keys. 10 About the AUK logo: According to wikipedia.com, auks are birds that look like, but aren’t penguins; they are considered as a product of “moderate convergent evolution”.
964
A. Mourouzis et al.
In summary, AUK supports (a) several different entry modes (1 to10-key, and joystick), rendering it appropriate for the widest possible range of input devices; (b) various layout configurations for different performance levels; and (c) the potential to integrate additional techniques, such as text prediction and dictionary-based or prefixbased disambiguation, as these ensure even lower KSPC values [ 3, 8, 10].
4 A Multi-tier 3x3 Menu Structure AUK is similar to a multi-tier 3x3 menu system. Each 3x3 grid has eight virtual character keys and a menu key for entering or exiting alternative menus. Five basic menus can be interchanged sequentially11 (Fig. 2). In the primary menu (1), the letters A-Z, the apostrophe, SPACE (SP), BACK (BK), and SHIFT (SH)12 are placed on the character keys, and the menu key is used for timeout kill, swapping to menu (2), or entering a linear menu (D) for navigations, e.g., within a word prediction list13. Additionally, there are five secondary menus: numerals (6), special characters (7), brackets (8), formatting options (9), and numeric operators (10). Finally, menu (5) has a cell for selecting extra characters through menu (D), and an empty cell to allow space for adding new menus as necessary, thus rendering the structure extensible.
5 Entry Modes and Menu Layouts The presented menu structure can be employed with various input methods. In particular, given T(i, C, M), AUK supports text entry for: −
−
−
11
i=9. This is the basic mode of AUK, in which there is a correspondence between the nine AUK menu options (see Fig. 2) and nine keys (e.g., phone keypads). This mode has two main variations: (a) the 0-key mode for command without click [15] e.g., by means of eye-tracking; and (b) the joystick mode. i9. In this category focus is mainly on 10-key modes for number pads or 10thumbs typing (see section 8). In the first case, the tenth key (zero key) is used for
E.g., in the 9-key mode, starting from the menu (1), the key combination 1-1-1-1-1 will result returning to menu (1), after visiting the text navigation menu (2), the punctuation menu (3), the mode menu (4), and the extend menu (5). If the user selects any character key in one of these menus, then the menu key changes to the exit key for returning directly to (1). 12 SP, BK, and SH are very common [14]; in AUK, SP, being the most common key, is always the most easy to reach; the apostrophe, indicates missing letters, and thus is included here. 13 If dictionary-based disambiguation is activated, then a list of alternative words is displayed. 14 An automatic scanning technique is employed in this case. 15 Two methods are used for the 3-key mode: the date stamp method [see 985], and one proposed for AUK, through the buttons DOWN, RIGHT and SELECT, as this provides less KSPC.
An Accessible and Usable Soft Keyboard
965
timeout kill (single tap) and for the SH modifier (hold down). A second approach, mainly for 10-thumbs-typing, is to place the SP key alone, and use the tenth key for timeout kill (first tap) and BK (additional taps). Furthermore, in this second case, the space key may also be used for the CTRL modifier (hold down) and the tenth key for SH, whereas SH+SP is for entering menu (2) - see Fig. 4. Indicatively, the arrangement of keys for the 3-, 9-, 10-key and joystick modes are presented in Fig. 3, along with the indicative entry scenarios presented in Table 1. (1b)
Menu (1)
(1a)
;
« enter L
« enter
def ’
abc
2
mno
ghi
jkl
pqrs
tuv
L
tuv
.
?
2*
“
,
!
“
-
:
;
-
.
?
,
!
:
;
12
3
More options
Pg Up
123
4,
5.
6
Home
Caps Lock
End
7
80
9
)
Pg Dn
alt
77
7
(10)
(D’)
+
3
1
jkl
nice nickel nicely nicotine
select 6
tuv
9
5
b
9
8
8
1
(9)
(7)
9,
c
tab
exit
#
~
exit
e
« enter
X
`
@
&
B
delete
d
ctrl
_
|
\
exit
exit
×
insert symbol
{
(
)
-
+
=
}
[
]
,
÷
√
F1-12
+
@
U
8
7
1
9
(D’)
(10)
(1)
(9)
8
U
I change fonts
(10)
(8)
exit
1
L Dictionary
Turn dictionary off
2
3
(5)
exit
1
a
H1
wxyz 7
1
(4)
(D)
def ’ mno
5
(2)
Caps Lock
2
pqrs
wxyz
(6) (1)
4
(3)
exit
; abc
ghi
9
wxyz
1
L
1
jkl
pqrs
(3a) (1)
2
mno
ghi
9
« enter
def ’
; abc
%
Fig. 2. The basic menu structure of AUK (for i=9) Joystick
9-key
« enter L
1
Menu (1)
;
2
def ’
jkl 5
pqrs
6
tuv
9
wxyz
7
Menu (1)
« enter
;
Ã
iuy
srj ’ É
L
twgv
9
9
(minTap)
dcx
tsw
L
(R-lessTap)
eiu ’
hmfk
hfj
nmv
;
rgk
.
?
Caps Lock
.
“
“
,
!
“
Caps Lock
!
,
?
!
-
:
;
-
:
;
-
:
;
« enter L
;i
u y
a o q
e l z
9d c x
tw gv
edf ’ 9
onm lkj 6
5
srpq
tuv
9
wyxz 1
,
acb 8
ihg
«enter
?
Horizontal Key-bar (minTimeout)
L
4
lpq
Æ
« enter 7
odyx npb
acbz
.
Caps Lock
Menu (3)
9
8
elz
3
mno
ghi 4
abc
aoq
10-key
3-key:
(minTimeout)
(R-standard)
3
2
.
?
“
,
!
-
:
;
Caps Lock
s r j ’
hm fk
; exit 0
ctrl
exit 0
n pb
Fig. 3. Keys arrangement for modes 3-, 9-, 10-key and joystick. The bottom part shows the AUK menu as a horizontal bar for PDAs (see section 6).
966
A. Mourouzis et al.
6 Layouts for Novice, Moderate and Expert Performance A significant amount of research has been conducted on 12-key text entry methods for mobile phones. Most efforts aim at achieving the upper limits of typing speed with a particular design, often at the expenses of an acceptable experience for novice users [3]. Yet, the ease of use for first-time users is critical for the success of any method. For this purpose, in addition to the entry mode described above, different A-Z letters layouts have been designed for novice, moderate and expert users. The following are layouts for multi-tap and involve the use of system timeout. Table 1. Examples of user input, resulting transcribed text and menu transitions User Input (with layouts in Fig. 3) 9-key stan)
(R-
Joystick (minTout )
3-key ( minTap)
10-key (R-Lesstap)
Menus onscreen ‡
Text (transcribed)
1
3
(1)
_
2,3
+,
,,, ,
0+9
(1)
E_
3,3,4,4,4
,
,,, ,
9,4
(1)
ei_
2,2,1,2,2,2
,, ,
,,, ,,,
8,0,8,8
(1)
ac_
3,3,8
,
,,T, ,
9,2
(1)
et
3,3,9
,
,,T,
9,3
(1)
e_
(1>2>3>1)
e._
3,3,1,1,1,2
,,,
,,,, ,, ,,,
9,7,7,2
1+5
H,
H,
1+5
(1>D>1)
1st word_
7,0+1
(1>2>1)
(ctrl+s)
1,1,9,7,7,7,7
,, ,
T: system timeout
,,, ,,, ,,,, _: text cursor’s position
: SPACE
H: Hold down key until system timeou t
R-standard. This layout is the standard phone layout5 adapted to include SP, BK, SH and apostrophe to the character keys with the lowest overall frequency (see the example layout for 9-key in Fig. 3). This is a familiar and easy to remember layout. R-lessTap. This layout is an adaptation of the LessTap [13], in which letters within each key of the standard layout are rearranged according to their frequency in the English Language16 and thus achieving lower KSPC. R-LessTap simply includes the four extra options mentioned above (see the layout for 10-key in Fig. 3). Taking into account that several studies [e.g., 16] have indicated that an alphabetic order is not critical for user experience, more effective layouts can be created by disregarding the alphabetical order all together, as described below. minTap. In this layout, the more common an input is, the easier it is to enter it. This is achieved through the following algorithm: (a) inputs are sorted by frequency order17; (b) all menu positions are coded, e.g., from 1-001 (representing the 1st position, on key 1, in menu 1) up to n-900 (representing the third position, on key 9, in menu n), and sorted by their reachability order (e.g., the number of keystrokes (KS) required to select each position); (c) the most frequent input (i.e., SP) is assigned to the position with the best reachability, and so on; (d) if an option can be assigned to 16 17
E.g., see http://en.wikipedia.org/wiki/Letter_frequencies (used here for KSPC calculations). Frequency order used here: SP BS eta SH oinshrdlcumwfgypbv apostrophe kjxqz.
An Accessible and Usable Soft Keyboard
967
more than one positions that share equal reachability values, then it is placed on the key that currently has the lowest overall frequency18. An example appears in Fig. 3. minTimout. In this layout, (a) letters on each key have the lowest possible digraph frequencies [11] in an English text corpus in order to have consecutive letters on the same key (and thus timeouts) less frequently19, and (b) the letters on each key are arranged in decreasing letter frequency order – thus ensuring less ‘taps’ for frequent letters (see the layout used for the joystick mode in Fig. 3). R-qwerty. In this layout, characters are grouped and arranged in a way that is related to the QWERTY layout. Such relation can be vision-based (e.g., having the letters “qwer” grouped together on the same key) or motor-based (i.e., with keys that are assigned to a specific finger20 being grouped together (see Fig. 4). In modes with i=9 for automatically rearranging the letters within each key, so that the first letter is always the most probable. In addition, AUK may have the following general modes: (a) multi-tap; (b) dictionary based disambiguation; (c) prefix based disambiguation [18]; (d) with or without text prediction [19] displayed though the menu (D); (e) synthetic speech output enabled for blind users. Finally, the following display options have been considered: (a) 3x3 keypad; (b) vertical or horizontal bar (e.g., for PDAs – see Fig. 3); and (c) cursor view (e.g., see Fig. 4).
7 Discussion: Universal Access to Text Entry The proposed 3x3 structure shows a number of benefits. First of all, it complies with the fact that visual search involves “combination of gestalting the area surrounding the current center of interest –3x3 matrix of letters– and using linear search … as a back up strategy” [17]. Yet, this structure offers access to a large and extensible input set, and thus AUK can operate as a stand-alone word processor or even as a plug-in for other applications. Furthermore, it is potentially easy to learn and use for users with different expertise, since: (a) functions can be placed as close to the user expectations, e.g., for improving novice experience [9], or according to their (individual or related) probabilities for expert performance; (b) onscreen visual21 and aural feedback is provided for character entry and menu transitions; and (c) the backspace key is always the second most reachable option, thus allowing fast error recovery [20]. In addition, since i
