Upgraded the flatpak to qt 5.14 (hoping to get my hands on the new markdown support), which resulted in discovering a regression for QSet properties.
The flatpak now employs a patch so the pinentry tool just uses libsecret as cache, which means if you run gnome-keyring you get password-less logins (and if somebody is going to finish ksecretservice that would of course work too). I have also looked into getting access to the host gpg-agent (which seems like the better solution), but that effort is currently stuck due to a missing feature in bubblewrap and because it’s not entirely clear if this will really be the way to go forward. Feel free to weigh in.
Fixed a bunch of rendering issues in invitations and calendar.
“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.” For more info, head over to: kube-project.com
HiDPI support. After a bit of massaging, Kube now scales correctly on HiDPI displays, so we no longer end up with tiny icons everywhere.
Fixed HTML view resizing. Resizing the WebengineView to fit the contents has been a constant source of problems and continues to break every now and then.
Improved keyboard navigation that allows for switching between views.
Improved IMAP sync which fixes a couple of corner cases and improves performance as well as responsiveness while a sync is executed.
Experimented with zstd compression for things like mime messages, which reduced the database size, but doesn’t seem to affect performance much.
The account password is now protected using your gpg key as well, so if there is a gpg-key set up, you will only have to unlock your keyring to unlock kube. This effectively turns the gpg-keyring into your Kube keyring as well (assuming you have a gpg-key for every account).
Implemented fulltext indexing of encrypted emails as well as basic support for protected headers. Read more
Shaved off another 2000 SLOC from the messageparser library as I moved it over to sink for encrypted email indexing.
The todo view
The todo view’s goal is to have a small personal list of todos, acknowledging that you can only accomplish so much during a day, and there is no intention of turning this into a project management suite down the road.
The idea is that you have a couple of lists as backlogs, and that you then pick a reasonable amount of items (<10 probably?) from those lists as currently in progress (that’s also how it’s stored in iCal). This then gives you a nice little list of things during the day/week/whatever suits you, that you can tick off.
New items can quickly be entered using keyboard shortcuts (press “?”) and that’s about it for the time being.
I think sub-todos might find their way eventually in there, but the rest should rather be quality of life improvements and eventually taking other sources of “things you need to act on” into account, such as emails that you should probably be answering or events that need to be prepared.
The todo view was the last officially missing piece, so with that we are view-complete (feature complete may be a bit a stretch still).
The keyring
Having to enter your account password for every account whenever you start Kube doesn’t make for a great user-experience, so this was fairly high on the nuisance list.
Naturally the first thought was that we would just use your platform’s keyring, but regrettably there still isn’t a solution that works on multiple platforms (not even on Linux, libsecret was never implemented for KDE), so that results in a lot of effort for implementation and maintenance.
Fortunately, there is an alternative.
We already rely on GPG for end-to-end encryption, so why not use your GPG-key to also secure your account related secrets?
We already had an experimental feature that stores the account password encrypted using the key as a POC, so the next was to build this into the core and improve the experience somewhat.
The result is that you will no longer have to enter any account passwords after the initial entry, but will instead be prompted to unlock your GPG-keyring (if not already unlocked), and just like that we’ll gain a keyring and sidestep the keyring mess. With gpg-agent we can at least reuse something that we rely on anyways, and that we have available on all supported platforms.
The one fly in the ointment is that we currently have to start a gpg-agent inside the flatpak, so we can’t reuse an already unlocked keyring on the system.
“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.” For more info, head over to: kube-project.com
When most of your email traffic is encrypted there are two things that really start to become a problem:
You can’t search for encrypted emails
If the sender is using a client that supports encrypted headers, such as thunderbird, your emails will be completely unrecognisable because the subject no longer contains anything useful.
To fix this we’re going to start decrypting encrypted emails when syncing and indexing the encrypted content. That way we can make sure encrypted emails are just as usable as non-encrypted emails, at least as long as you’re using Kube.
This means that in the future you will not only be able to search through all your email, it also means you get a more useful subject displayed than “…” or some other nonsense.
Who doesn’t love a conversation list full of descriptive subjects like “…”
This is of course a trade-off in terms of security; you will have at least parts of your encrypted email in decrypted form on your disk, and so an attacker that has access to your system will have access to the encrypted contents in plain text. However, it’s a reasonable trade-off as an attacker that has access to your system, which contains your private GPG key as well, will have have good chances of obtaining what he looks for anyway. It’s also a necessary trade-off to make encrypted communication usable enough so it can be used across the board.
Of course we could attempt to protect the index, but this is best left to the same tools that protect the rest of your system, such as full-disk encryption.
“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.” For more info, head over to: kube-project.com
After a waaaaaay to long “break” I have finally tagged another release.
The largest change in this release is the addition of the calendar view, which is not only useful, but also marks an important milestone in our development roadmap; We finally have all the pieces together from a technology perspective.
The calendar’s week view
The calendar was a major undertaking due to a couple of challenges:
It’s synchronized over its own protocol (CalDAV)
It’s for once not a list, so it’s visually a completely different beast than everything else.
It has lots of fun special cases such as recurring events, timezones, overlapping events, multiday vs. single day events, …
We wanted to avoid loading your complete calendar (including the past 10 years) into memory, while making sure you get recurring events displayed even if they started 10 years ago.
The work done so far solves most of the important challenges, but there are also definitely a couple of holes in it still, such as no drag and drop support.
While the calendar is certainly the biggest new feature, there’s also a bunch of other improvements in there:
A new editor view, providing a much cleaner look than what we used to have.
Basic support for scheduling via iTip (both for sending an invitation to attendees, and for interacting with invitations you have recieved).
Builds and runs on MacOS and Windows (it admittedly is not getting a lot of testing on those platforms, especially on Windows, but the baseline is there).
A fastmail account configuration dialog.
It’s now possible to create and modify contacts.
We no longer default to displaying HTML email
… and a bunch of other stuff in a little over 226 commits to sink and 561 commits to kube.
“Kube is a modern communication and collaboration client built with
QtQuick on top of a high performance, low resource usage core. It
provides online and offline access to all your mail, contacts,
calendars, notes, todo’s and more. With a strong focus on usability, the
team works with designers and UX experts from the ground up, to build a
product that is not only visually appealing but also a joy to use.”
Kube is still alive! I got distracted for a while, both professionally and privately, and writing blog posts is unfortunately always the first thing that ends up on the chopping block. Anyways, lot’s of progress has been made (103 commits in sink, ~130 commits in the kube codebase):
For sink we had a variety of bugfixes and performance improvements, especially on the more recent CalDAV/CardDAV backends.
For CalDAV/CardDAV we now do basic autodiscovery using the .well-known url’s. The DNS part of the spec has not been implemented so far.
This means for a properly set-up server you only have to specify the base url, and everything else will be discovered automatically from there.
On the more user-facing front we have:
Sent emails are now collapsed by default
Plain text is now the preferred method of viewing emails. You can still view the HTML variant if available by clicking a button.
The Addressbook is no longer read-only and you can now create contacts as well.
A visually reworked composer that avoids becoming too wide and removes a lot of the visual clutter.
The calendar can now render recurring-events.
It is now possible to create events as well.
Work on a tasks view has started
Conversations
Composer and drafts
Composer and recipients
Addressbook
Calendar
Tasks
Releases
You may have noticed that it’s been a while since the last release. This is not only because releases are additional work, but also because we already have a continuous delivery method with the nightly flatpak.
It’s clear that releases do provide value, both as a communication tool which version should be packaged, and if they would be maintained. With the current manpower we cannot maintain releases though, which makes it significantly less interesting.
With that said, the 0.8 release with the calendar is now long overdue and should be coming out soonish.
Experimental flatpak
Just to put it out there; Additionally to the usual “master” branch of the flatpak, there is also an “experimental” branch, containing, surprise, various experimental bits and pieces.
This currently entails:
A plugin that stores the accounts password encrypted by the accounts gpg key (blindly assuming there is one with a matching email address).
A search view
The upcoming calendar view (which we’ll move over in the next release)
The above todo view (which will take a little longer to move to master)
A “File as expense” plugin (a showcase how we could do extensions in the mail view).
The Inbox crusher view (an experiment for a view to go through your inbox one-by-one).
It typically serves as a staging ground for new components, and is the version that I’m running day-to-day. flatpak makes it easy to switch back and forth between the branches on top of the same dataset, so you can try it and switch back if you don’t like what you see.
To give it a shot use the following command to install and switch to the experimental flatpak branch:
“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.” For more info, head over to: kube-project.com
As we’re closing in on a simple but functional calendar for Kube, I’d like to share our progress with you.
We’ve decided to start with a week view, as that seems to be a good compromise between information density and enough information for day-to-day use.
We will eventually complement that with a month view, which is probably all we need for the time being.
An agenda view will probably rather become part of a separate view, showing you upcoming events, tasks that need to be taken care of, important emails, ….
Anyhow, here’s the week view:
This view is based on two models, one for full-day events (on top) and one for the rest below. The models each query for all events that overlap with the time-range shown, which we can do efficiently due to the work of Rémi on the calendar queries. That means we really only have to deal with a few events in memory, which we can easily do on the spot. The models then do their magic to calculate the position and sizes of all the necessary boxes, so all we have to do in QML is paint them. This now also includes recurring events, although we’re not dealing with exceptions just yet.
The colors of the events are taken from the calendar, which gets synchronized via CalDAV. This is a tradeoff design wise because you can bet that those colors will not match Kube in any way. But we decided that Kube should work well with other devices and clients, and the color is a major factor to recognize immediately what belongs where. Perhaps we can still make the colors a bit easier on the eye by desaturating them a bit, we’ll have to see.
Otherwise there is dimming of the past, and a blue line indicating the current time. Simple and without distractions.
The next steps are going to be adding a detail view, as well as a simple editor, and then we should already have the basics for your daily needs.
Near the middle / end of my internship, I got to modify parts of the storage system in Sink, the library handling all the data of Kube.
The goal was to both to speed up the storage and reducing disk space. These two goals often go hands in hand in databases, since smaller data means faster disk lookup, and more data put in memory, available for direct usage.
Reducing key sizes
The first important modification was for keys to be stored in a binary format instead of the displayable format. This allowed data retrieval to be faster, because the binary format would be much smaller than the display format.
On the memory usage side, this is a bit more awkward to measure: on the one hand, the since the size of keys is smaller in memory, but on the other hand, LMDB (the database system we are using) will try to put more data in memory.
But on the whole, either we are using less memory, or Kube will be faster altogether. The reality is probably a mix of the two.
As for the numbers, these are the results of two different benchmarks:
Develop branch
This patch
Current Rss usage [kb]: 40700
Current Rss usage [kb]: 39112
On disk [kb]: 10788
On disk [kb]: 8836
Write amplification: 12.0075
Write amplification: 9.83485
Develop branch
This patch
Total pages: 760
Total pages: 603
Used size: 1425408
Used size: 1191936
Total on disk: 3293184
Total on disk: 2650112
Write amplification: 3.63268
Write amplification: 3.51402
As we can see on both tables, we use less disk space after the patch, but memory usage has not gone down in all places.
Overall we use approximately 20% less disk space.
Separating UIDs and revisions
Another important modification on the storage system was separating UIDs and revisions. Before this patch, we used to store the UID and the revision as keys inside the database.
What’s more is that the key was in the format “{this-is-a-uid}0000000000042”. The reason for padding the number with zeroes was that we need the data to be ordered, and we needed the keys to be stored as a string for the UID.
However, every revision is unique, so we can store only the revision as a key. This also allows us to store the key as a number, use the “integer key” feature of LMDB so the sorting stays correct and save a nice amout of space.
But, since we can keep old revisions, we need to track them. Therefore we need another table for mapping UIDs and revisions.
The rationale behind this patch is that since the “uid to revisions” database will be much smaller than the main database, it can be put into memory, leading to faster lookups.
Even though the results were not as consequent as the “reducing key sizes” patch, it seems we still got a small improvement performance wise. We will better see how different the performance is in the near future, though.
In the meantime, we can use the benchmarks graphics to see how performance, memory and disk usage was impacted (courtesy of our CI):
Disk usage with dummy writesWrite amplificationMemory usage with dummy writesInitial query timePipeline performance
