Too much fun at CSUN! Part 2: Herbi WriteAbout - Handwriting development (Windows Store app, XAML/C#)
I started working on an app to help with handwriting development back in 2006, when an Occupational Therapist (OT) asked me if I knew of a Tablet PC app which could help her students, (who had a range of disabilities,) develop their handwriting skills. I couldn’t find any such app, so I set about building one.
By the way, I must say that over the coming years I learnt that I can’t build useful apps without expert input from OTs, teachers and parents, as they can help me focus on what’s most important to the students. These days I don’t work on any new features in my apps unless some expert input has prompted the work.
When I first built the Herbi WriteAbout app, I leveraged the ink support and handwriting recognition that’s available to any Windows app.
Try it yourself - Build your own Windows Store app with ink and handwriting recognition! Download the “Input: Simplified ink” sample at https://code.msdn.microsoft.com/windowsapps/Input-simplified-ink-sample-11614bbf and see how easy it is to add ink and handwriting recognition to your app. The sample specifically chooses to only allow inking with the pen or mouse, but my Herbi WriteAbout app also allows inking with a finger.
So while it was easy for me to add handwriting recognition to my app, one OT told me that it’s not only important what her student’s ink looks like, but also how it was formed. So I built my own ink stroke comparison algorithm. I get the array of points making up the student’s ink strokes, and get a rough idea of where evenly distributed points along the strokes would be. I then do the same with a sample letter, and then compare these points with those from the student’s ink strokes. Once the points diverge beyond a certain threshold, (based on the user’s current settings,) then the student’s letter is not considered to be correct. I’m sure my algorithm could be made more accurate, but the feedback I got suggested that it was working well for the students.
Figure 1: The Herbi WriteAbout app showing ink representing a sample letter.
A handy side effect of using my own ink stroke comparison algorithm, was that I could help a teacher who said her students needed to practice drawing shapes, as they hadn’t yet progressed to letters. My app now only cares about ink strokes, not specifically about whether the ink looks like a letter.
You can even add your own set of ink shapes, be it the alphabet in a particular style, or as I demo’d at CSUN, the cool Windows logo…
Figure 2: Custom sample ink with a student’s ink partially added, all being shown with colors from the active high contrast theme.
My original handwriting development app ended up being used by OTs and teachers all around the US, and another important lesson for me was that an app doesn’t have to do much to be useful, it just has to do exactly what’s needed. For example, I was asked to add exciting positive feedback when the student’s letter was evaluated as correct. But I wasn’t at all confident that the exciting positive feedback that I ended up showing was in any way exciting. Basically, this is it, and it’s not exactly like something out of Halo…
Figure 3: Visuals changing in the Herbi WriteAbout app to provide positive reinforcement to the student.
But I was blown away when I saw how a couple of students were reacting to the simple change in visuals. The visuals were having the desired effect and were indeed encouraging the students to use the app. It was then that I realized I could never stop building this sort of stuff.
By basing all my work that followed on feedback from the experts, I could feel confident that I was concentrating on the most valuable features for the students. And it during this time that I fell in love with the Windows Text-to-Speech (TTS) API. I was asked if the app could output speech to provide positive audio feedback when the student’s letter was evaluated as correct. This turned out to be trivial for me, and it’s amazing to see so much power added to the app with so little code. It’s just a few lines of code, and it adds something so beneficial to the student. And over the years I’ve added TTS to a bunch more apps because it has so many applications, whether that’s positive reinforcement, communication, or education.
A while later, the Windows Store came along and things really got interesting for me. My own web site was fine for making the app available to anyone who wanted it, but somehow people had to find the app. What I really needed was to have anyone using Windows to search for “handwriting development” at the Windows Store, and find my app. This could get my app in front of millions of people, and would fix my nagging feeling that the app just wasn’t reaching people who might find it useful.
Figure 4: The Herbi WriteAbout app being shown as the first result for handwriting development apps at the Windows Store.
So I created a new Windows Store app, and copied across the C# from my original app for the fundamental features of the app. It’s now up at the Store at https://apps.microsoft.com/windows/en-us/app/herbi-writeabout/a34ed781-7d11-4663-bfda-09f9ebdf1216.
While I built a Windows Store version of the app in order to get it in front of millions of people, I also found the reporting features available through the Store to be a refreshing change from the complex reporting mechanism I can theoretically get through my own site. For example, it’s a piece of cake for me to add translations to my app, such that Windows will automatically present localized text in the app based on my user’s current settings. All I need is to get the translations. A colleague at work kindly gave me Spanish translations, and someone in Germany gave me German translations, and now it’s easy for me to see where the app’s being installed.
Figure 5: A chart showing regional download data for the Herbi WriteAbout Windows Store app.
I want my apps to reach people of all abilities, and all over the world. The Windows Store is helping me achieve that.
A note from CSUN
After demo’ing the Herbi WriteAbout app at CSUN, an attendee who works in the education system mentioned another possible application for the app. In some educational situations, a student is shown a letter for a while, and then the letter is removed. After some period the student proceeds to write the letter, based on their recollection of what the letter looked like. This means the student’s memory is also involved with the letter writing process. The person at CSUN suggested that perhaps my Herbi WriteAbout app could be updated to support that activity.
In many ways, the app would seem ideal here. It would be easy for me to update it to show sample ink for some period and then remove it. Then after some configurable period, ink would become enabled at the app, and the user could then input their own ink. But while this would all be straightforward for the app, I do wonder how my current ink stroke comparison algorithm would fare. Perhaps in this situation, the size of the student’s ink relative to the original sample might diverge more than when the student’s tracing over the sample. And perhaps it’s still ok in that case for the letter to be evaluated as correct.
So this is certainly an interesting new application for my app, and I’ll investigate this further if I get more thoughts from experts on what’s best for the students here.
Another note from CSUN
I also got a question about whether the app works with words. So for example, if the OT, teacher or parent had added custom ink to show a word of particular interest to the student, how well would that work for the student?
Technically, the app does work with words, as after all, it’s all about comparing ink strokes. How well the app works in practice with words, I don’t know. Whether the word’s cursive or printed, the student’s going to have to maintain the required level of accuracy throughout the word, and that’s going to be more of a challenge than for a single letter. (The app provides three levels of evaluation, to account for students having different levels of handwriting skills.)
So again, this is something I’ll explore more if I get further input from experts.
Further posts in this series:
Too much fun at CSUN! Part 1: Demo’ing four assistive technology apps for Windows at CSUN 2015
Too much fun at CSUN! Part 2: Herbi WriteAbout - Handwriting development
Too much fun at CSUN! Part 4: My Way Speaker - Using a switch device to efficiently browse the web
Too much fun at CSUN! Part 5: Herbi Speaks - Creating boards for education or communication