Math Braille UI

For sighted users, Microsoft Office applications like Word, PowerPoint and OneNote have user interface (UI) cues that reveal math zones, selected text, the insertion point (IP) if no text is selected, and the argument of the innermost math object (fraction, subscript, integral, matrix, …) that contains the IP. Math speech also reveals these properties. These sighted and speech math UI’s enable unambiguous reading and editing of mathematics. Naturally, it’s desirable to reveal these properties in refreshable braille displays as well. This post describes a promising methodology. We represent math using the Nemeth math code since it is the most efficient for the math zones used in modern applications such as Microsoft Word and LaTeX.

First here’s how these features are revealed to sighted users. In all math-enabled Office apps, the innermost math argument containing the IP is lightly shaded and selected text has the same selection background color as text not in math zones. In PowerPoint and OneNote, the math object containing the IP is shaded a bit more lightly than the argument and if the IP isn’t in a math object, the whole math zone has this lighter shading. In Word, the math zone is enclosed in a boundary and the object containing the IP doesn’t have the lighter shading. The user always knows what kind of an argument is involved just by looking at the built-up (Professional) display. This information is also conveyed in math fine-grained speech.

Refreshable braille display

A refreshable braille display typically has a row of 40 or 80 8-dot cells with the dots represented by small rounded pins that are raised by solenoids. The dots are arranged in two columns of four dots. The left column is numbered starting at the top 1, 2, 3, 7 and the right column is numbered starting from the top 4, 5, 6, 8. Like most braille codes, the Nemeth math code uses the dots 1 through 6. This leaves dots 7 and 8 for UI purposes, although dot 7 is occasionally used to indicate upper case. The Nemeth code precedes a letter with the capitalization indicator “⠠” (lone dot 6) to get upper-case letters, e.g., “⠠⠁” for “A” since “⠁” is the braille code for the letter a. So, we don’t use dot 7 to indicate upper case, at least in math zones.

The regular math braille display shows the whole math zone in braille, limited only by the number of display cells. This gives a lot of context to math braille, significantly more than math speech provides, but not as much as screen or paper.

Selection and insertion point (IP)

Typically, selected text appears with both dots 7 and 8. So if “a” is selected, it appears as “⣁”. This approach seems well suited to math expressions as well.

We’re left with needing ways to identify a math zone and the insertion point and to highlight the innermost argument containing the IP if any. Braille displays don’t have multiple shading levels, only two extra dots! They also have hot keys.

The IP needs a cell by itself to stand out. As described in the post Text Insertion Point, the IP is in between two characters in rich text, although for plain text one can get away with thinking of the IP as being on top of the character that actually follows the IP. Built-up (Professional) math text is rich text notably because it has special display constructs, such as stacked fractions, multilevel subscripts and superscripts, integrals, matrices, etc. For this purpose (and perhaps others), dots 7-8 “⣀” comprise a simple, effective IP. Admittedly this is the same as a lone selected space, but it seems to be readily distinguishable since the user usually knows when something is selected versus having an IP and s/he can easily move the IP (or hit the IP-identification hot key coming up) to check if in doubt.

Math-object arguments

To reveal the innermost argument containing the IP, one can turn on dot 8 for the characters in that argument. This is similar to the argument shading used in regular displays. To illustrate this approach, consider the fraction 1/2π, which in built-up form is given by the Nemeth braille string “⠹⠂⠌⠆⠨⠏⠼”. If the IP precedes the 2 in the denominator, the braille display would have “⠹⠂⠌⣀⢆⢨⢏⠼”.

At first the dot 8 in the denominator cells here might be confusing, but it resolves ambiguities as to whether the IP “⣀” is inside or outside of a math object. This isn’t a serious problem with fractions since the fraction start, fraction bar, and fraction end appear as the explicit braille codes ⠹,⠌,⠼, respectively, although it’s always helpful to know when the IP is in a math argument. But consider the quantity a², which is given in Nemeth braille by “⠁⠃⠆”. In Office apps and MathML, superscripts are represented by two arguments, the base and the superscript. If the IP precedes the base, is the IP at the start of the base or at the start of the superscript object? That position is ambiguous without the dot 8 option. With dot 8, you can tell the difference: in “⣀⠁⠃⠆” the IP precedes the superscript object, while in “⣀⢁⠃⠆” the IP is inside the base in front of the “a”. Distinguishing these positions is essential for unambiguous editing of mathematical text.

IP-identification hot key

Dot-8 highlighting reveals when the IP is at the start or end of an argument or somewhere in between. But it doesn’t define what kind of argument. To get this kind of information on a braille display, it’s handy to have an IP-identification hot key that flashes the name of the argument containing the IP (or “math zone” if the IP isn’t inside an argument) onto the braille display. This name needs to be localized to the current user language, while the regular braille for the math zone is globalized by nature. For example in English, depending on where the IP is in a denominator, the hot key displays “start of denominator” (⠎⠞⠁⠗⠞⠀⠕⠋⠀⠙⠑⠝⠕⠍⠊⠝⠁⠞⠕⠗), “end of denominator” or just “denominator”. This is more informative than the corresponding math speech, which only announces the kind of argument when the IP is at the end of an argument, or the kind of math object when the IP is at the start of an object. This difference occurs because fine-grained speech needs to say the character at the IP, whereas the math braille display continuously shows the characters around the IP, limited only by the number of display cells.

It might be worth having options to enable/disable dot-8 highlighting according to user preference. Even without the dot-8 highlighting, the user can resolve ambiguities by hitting the IP-identification hot key so some users might prefer to work with the simpler braille display.

Math zone identification

Lastly, how do you reveal a math zone? If the IP is inside a math-object argument, the presence of dot 8 is a good indicator. As described in the post Braille for Math Zones, math zones start with “⠸⠩” and end with “⠸⠱”. So, the start and end of a math zone are not ambiguous in math braille. In the Microsoft Office math representation, whether the IP at the start of a math zone is inside the math zone or outside is revealed by shading or the Word math-zone border, since the character position is the same for both cases. Ditto for the end of a math zone. I tried setting dot 8 for all cells in a math zone when the IP is in a math zone, but not inside an argument, but it seems too messy. So hopefully the math zone start and end delimiters will suffice; the user can hit the IP-identification hot key to find out whether the IP is in a math zone.

With these uses of dots 7 and 8 and the IP-identification hot key, you can edit virtually all levels of mathematics using a refreshable braille display in an interoperable way with sighted users. Pretty cool, eh?!