Posts
Snippet Pixie 1.3.1 Released
This week I released Snippet Pixie 1.3, with zero new features, but it took many many hours of development.
Umm, what? How could you bump Snippet Pixie up a version, and claim to have spent "many many hours" on development, but not have any new features?
And isn't this post titled "Snippet Pixie 1.3.1 Released"?
Well, take a look at the gif attached to this post, that's a snippet expanding in Standard Notes, an Electron based application. ๐ฑ๐
Demo of Snippet Pixie working with Standard Notes (Electron) [IMG]
![Demo of Snippet Pixie working with Standard Notes (Electron) [IMG]](/media/2020/02/snippet-pixie-working-with-standard-notes-electron.gif){kind=link}
Ok, to some people that might not mean anything, but believe me, it's a major relief to have Snippet Pixie able to work with Electron based apps, and of course, Chrome and Chromium too.
Before v1.3, Snippet Pixie relied on apps using the ATK library, or at the very least, doing what's needed to work with the accessibility DBus interfaces that AT-SPI interacts with. Further still, the user had to be typing in an EditableText and Text interface compliant control. There are many Linux applications that are ATK/AT-SPI compliant, but unfortunately a few very popular ones are not.
The biggest trouble maker for Snippet Pixie was Chrome, turns out it's quite a popular browser. ๐
And many companies build cross platform apps with Electron, a technology that basically uses Chrome under the hood, so they weren't working well with Snippet Pixie.
I tried all kinds of things to try and get Chrome and Electron apps to play nice, and there was the odd occasion when a control here and there would actually work with the accessibility framework, but in the end I had to throw out the accessibility stuff and go "old skool".
Well, I say "throw out the accessibility stuff", but that's not quite true, and is the reason for v1.3.1, but we'll come to that in a minute.
Snippet Pixie still uses AT-SPI for monitoring keystrokes, but it's using a method that is a little more greedy in its scope, monitoring "all windows" for the current app rather than just the currently focused text control. With this method, Snippet Pixie can see when a key is pressed that matches the last character of some snippet's abbreviation, and then go look to see whether previously entered text matches an abbreviation.
This may sound less performant than monitoring the currently focused text control, and technically it likely is, but, it turns out that with this method I was able to eliminate a huge performance problem that earlier versions of Snippet Pixie suffered.
Snippet Pixie used to have to notice when the current app changed, then go look through all the widgets in that app to try and find the currently focused one, which hopefully was an editable text control. When that app happens to be a spreadsheet with thousands of widgets, that search for the currently focused control (cell) could take a noticeable amount of time. Now, we're not talking minutes here, not even seconds, but if it took even anything near half a second, you'd notice it.
Because Snippet Pixie is now monitoring at a higher level by default, it does not need to go look for a widget to attach monitoring code to. That's a big boost in performance, it no longer feels like Snippet Pixie is slowing down app switching for some apps.
Hang on a minute, if you no longer know where the text is being entered, how are you checking whether an abbreviation precedes that key stroke that matches the end of one or more abbreviations?
Blimey, you do ask some very intelligent questions don't you? ๐
So this is where a huge amount of time was spent in developing this release. When I first thought of building Snippet Pixie, one of the initial ideas I had for how to get the just entered text was to see if there was a way to walk back through the text with some sort of selection method, and then check the contents of the selection. I didn't get too far in that investigation before finding the accessibility interfaces, which seemed like a much saner approach.
Recently I found out about another open source text expander style app called Espanso. There was an issue raised on the project about the clipboard being overwritten when an expansion happened, or something like that, can't quite remember. But anyway, seeing that made me think that maybe I should take another look at my mad idea of selecting text, stuffing it into the clipboard and then checking the contents of the clipboard.
After much research and experimentation, I eventually managed to get Snippet Pixie to start a text selection by "pressing" the shift key and then "pressing and releasing" the left cursor key, then fake a copy shortcut, check the contents of the clipboard, and then keep hitting the left cursor key until an abbreviation was found, or the max length of known abbreviations was exceeded.
This kind of worked, and once I had an abbreviation match it was a cinch to then stuff the expanded text into the clipboard and fake a paste hotkey. Thanks for the initial code Clipped!
Grabbing the selected text into the clipboard wasn't ideal though. It was very flakey, using the copy shortcut for every extra character was hit and miss, and there was some weirdness with the clipboard not always updating.
I learnt a lot about the async mechanisms that the Linux desktop (GTK+ based in this case) clipboard uses, and that there's such a thing as a clipboard that most people think about, but also a second selection clipboard that auto populates when text is selected.
With that knowledge to hand, and after much experimentation with threads and strategic yielding and microscopic usleeps (I hate handling threads, but who doesn't?), I finally got a nice mechanism that recognises the abbreviation via selection, and then expands via the clipboard.
Over time I fine tuned the mechanism, and added a bunch of speed ups and error correcting retries to the way abbreviations are searched for, and ensured that the current clipboard is saved away and restored before and after the expansion respectively. The attached gif is intentionally slow so you can see the walk back working, even error correcting, but in general the mechanism is way quicker.
It worked really well with Chrome, Chromium and even Electron apps, but when I tested LibreOffice Writer, which had worked fine with the previous version of Snippet Pixie, it did the most weird things and lost text selection all the time, making abbreviation recognition impossible.
So I brought back the core abbreviation recognition engine of the old accessibility framework dependent mechanism, and sure enough, that still worked with LibreOffice Writer. Well, it wasn't quite that simple, I also had to add back a mechanism for being notified of when an accessible control was focused, and do a bunch of work to dynamically swap between the "all windows" monitoring method and the control based one as appropriate when apps are switched.
I did not need to re-add the old widget lookup code though, and I was able to improve the accessibility based mechanism with the abbreviation lookup speedups and threading from the new text selection mechanism.
In the end I got to a place where the previously supported applications still worked with the faster accessibility based text editing, and added support for a much wider set of applications via the new text selection mechanism.
When I applied those speed ups to the old mechanism that I created while developing the new one, I did miss one problem that necessitated v1.3.1. I used a slightly different monitoring mode with the re-implemented accessibility based code, async rather than blocking, kinda needed as part of the performance improvements and thread usage. Sometimes the control where the text was being entered would report the caret position incorrectly, especially near the beginning of the control, and after initial switch into the app. Turns out I needed to yield the abbreviation checking thread for a split second to let the app get itself together.
There are unfortunately a few apps that remain incompatible with Snippet Pixie 1.3.1.
The one that hurts the most for me is Firefox, my preferred web browser, which at some point in its last few releases started to misbehave when Snippet Pixie was in use. The old accessible method that Snippet Pixie used stopped working for some reason, something must have changed in Firefox to stop keystroke monitoring. I wouldn't be surprised if this was an intentional security measure, but I bet it also stops screen readers from working now.
When forcing use of the new text selection mechanism with Firefox, it keeps losing focus while you're typing, which breaks everything, text stops being entered. To get around this problem temporarily, until I can find a proper solution, or Firefox gets fixed, Snippet Pixie now blacklists Firefox so at least Firefox is usable, even if snippets don't expand.
Similarly, Snippet Pixie has never worked with terminal emulators, their text handling mechanism is fundamentally different. To avoid any potential issues with the new text selection mechanism that Snippet Pixie uses, a bunch of popular terminal emulators are now blacklisted by Snippet Pixie and it turns off abbreviation checking while they're the active application.
This new blacklisting mechanism actually brought to the fore another performance improvement for Snippet Pixie, as turning off the abbreviation checking speeds things up in incompatible apps. And along with this change, Snippet Pixie also monitors less keystrokes in general now too anyway, as if you're using either the Ctrl or left Alt key, as far as I can tell you can't be entering a character. The right Alt Gr key is still monitored in combination with normal keys though as that's how you generally enter extended characters such as โฌ and ยข.
The final incompatibility with Snippet Pixie 1.3.1 is Wayland compositors. This too hurts as I'm a fan of Sway, but alas the way I've had to implement active window / app checking is via libwnck, which unfortunately only supports X11.
I'm actively trying to find either an alternative robust method of tracking the current application that works in X11 and Wayland, or an additional Wayland only method that I can use when Wayland is detected.
So I guess you could maybe class the expanded compatibility and vastly improved performance as new features, but even if you don't, I'm sure you can see why I bumped the version from 1.2.x to 1.3.x.
Anyway, that was a bit of a rambling retrospective into what went into Snippet Pixie 1.3.1. Hope you enjoy this release, please submit bugs and feature requests in the GitHub repo.
https://appcenter.elementary.io/com.github.bytepixie.snippetpixie
https://snapcraft.io/snippetpixie
---
"Snippet Pixie 1.3.1 Released" was published on March 1, 2020.