Last week in Kube

You can now view “flagged”/”starred” messages per account. This is a short-cut to getting that functionality eventually folded into the todo view (I think…), and allows you to quickly show a list of messages that you have marked as important so you can deal with them. The view works best if there is a small number of marked messages, and you unflag them once you have dealt with them.
The experimental flatpak now contains spellcheck-highlighting support based on sonata. There is no configuration (language is autodetected per sentence), and there is also no UI to get to corrections, so it’s primarily useful to spot typos (which is good enough for the time being).
Quotes in plaintext emails are now highlighted, which makes for a better reading experiece.
Lot’s of enhancements to the todo view, which is now very functional as a day-to-day todo manager.
- The todo-view now has an Inbox that shows all tasks that are neither done, nor in doing. This allows to go through open tasks from all task-lists, to pick them for the “Doing” list (which represents the set you’re currently working on).
Various fixes to the invtation-management code to work correctly with exceptions to recurrences.
Kube is now a single-instance application when run with the –lockfile option (as used in the flatpak). This was necessary to to deal with the fact that we can’t start multiple instances of kube in separate sandboxes, because storage and lockfiles rely on PID uniqueness. Starting kube again will now simply hide and show the window of the current instance, which results in the window showing up on the workspace you currently are (which also seems to be more consitent with how Gnome and MacOS behave.
The IMAP synchronization got more efficient, primarily enhancing the case where nothing changed on folders.
Adapted to source changes in KCalendarCore (so the Applications/19.08 release is now required)
It’s now possible to create/remove calendars and tasklists in Kube, and sink can modify the associated color (but there’s no UI yet).
Recipients in the email composer can now be moved between To/CC/BCC using drag and drop.
Listviews now have subtle fade-in animations, which helps spotting new items.

More information on the Kolab Now blog!

“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.” For more info, head over to: kube-project.com

Last week in Kube

Upgraded the flatpak to qt 5.14 (hoping to get my hands on the new markdown support), which resulted in discovering a regression for QSet properties.
The flatpak now employs a patch so the pinentry tool just uses libsecret as cache, which means if you run gnome-keyring you get password-less logins (and if somebody is going to finish ksecretservice that would of course work too). I have also looked into getting access to the host gpg-agent (which seems like the better solution), but that effort is currently stuck due to a missing feature in bubblewrap and because it’s not entirely clear if this will really be the way to go forward. Feel free to weigh in.
Fixed a bunch of rendering issues in invitations and calendar.

Kube Commits, Sink Commits

Previous updates

More information on the Kolab Now blog!

“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.” For more info, head over to: kube-project.com

Last months in Kube

Kube got a basic but functional todo view.
HiDPI support. After a bit of massaging, Kube now scales correctly on HiDPI displays, so we no longer end up with tiny icons everywhere.
Fixed HTML view resizing. Resizing the WebengineView to fit the contents has been a constant source of problems and continues to break every now and then.
Improved keyboard navigation that allows for switching between views.
Improved IMAP sync which fixes a couple of corner cases and improves performance as well as responsiveness while a sync is executed.
Fixed free-page accumulation in lmdb database, which results in ~10x smaller db files.
Experimented with zstd compression for things like mime messages, which reduced the database size, but doesn’t seem to affect performance much.
The account password is now protected using your gpg key as well, so if there is a gpg-key set up, you will only have to unlock your keyring to unlock
kube. This effectively turns the gpg-keyring into your Kube keyring as well (assuming you have a gpg-key for every account).
Implemented fulltext indexing of encrypted emails as well as basic support for protected headers. Read more
Shaved off another 2000 SLOC from the messageparser library as I moved it over to sink for encrypted email indexing.

The todo view

The todo view’s goal is to have a small personal list of todos, acknowledging that you can only accomplish so much during a day, and there is no intention of turning this into a project management suite down the road.

The idea is that you have a couple of lists as backlogs, and that you then pick a reasonable amount of items (<10 probably?) from those lists as currently in progress (that’s also how it’s stored in iCal). This then gives you a nice little list of things during the day/week/whatever suits you, that you can tick off.

New items can quickly be entered using keyboard shortcuts (press “?”) and that’s about it for the time being.

I think sub-todos might find their way eventually in there, but the rest should rather be quality of life improvements and eventually taking other sources of “things you need to act on” into account, such as emails that you should probably be answering or events that need to be prepared.

The todo view was the last officially missing piece, so with that we are view-complete (feature complete may be a bit a stretch still).

The keyring

Having to enter your account password for every account whenever you start Kube doesn’t make for a great user-experience, so this was fairly high on the nuisance list.

Naturally the first thought was that we would just use your platform’s keyring, but regrettably there still isn’t a solution that works on multiple platforms (not even on Linux, libsecret was never implemented for KDE),
so that results in a lot of effort for implementation and maintenance.

Fortunately, there is an alternative.

We already rely on GPG for end-to-end encryption, so why not use your GPG-key to also secure your account related secrets?

We already had an experimental feature that stores the account password encrypted using the key as a POC, so the next was to build this into the core and improve the experience somewhat.

The result is that you will no longer have to enter any account passwords after the initial entry, but will instead be prompted to unlock your GPG-keyring (if not already unlocked), and just like that we’ll gain a keyring and sidestep the keyring mess. With gpg-agent we can at least reuse something that we rely on anyways, and that we have available on all supported platforms.

The one fly in the ointment is that we currently have to start a gpg-agent inside the flatpak, so we can’t reuse an already unlocked keyring on the system.

Kube Commits, Sink Commits

Previous updates

More information on the Kolab Now blog!

“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.” For more info, head over to: kube-project.com

Search in encrypted content and support for encrypted headers

When most of your email traffic is encrypted there are two things that really start to become a problem:

You can’t search for encrypted emails
If the sender is using a client that supports encrypted headers, such as thunderbird, your emails will be completely unrecognisable because the subject no longer contains anything useful.

To fix this we’re going to start decrypting encrypted emails when syncing and indexing the encrypted content. That way we can make sure encrypted emails are just as usable as non-encrypted emails, at least as long as you’re using Kube.

This means that in the future you will not only be able to search through all your email, it also means you get a more useful subject displayed than “…” or some other nonsense.

Who doesn’t love a conversation list full of descriptive subjects like “…”

This is of course a trade-off in terms of security; you will have at least parts of your encrypted email in decrypted form on your disk, and so an attacker that has access to your system will have access to the encrypted contents in plain text. However, it’s a reasonable trade-off as an attacker that has access to your system, which contains your private GPG key as well, will have have good chances of obtaining what he looks for anyway. It’s also a necessary trade-off to make encrypted communication usable enough so it can be used across the board.

Of course we could attempt to protect the index, but this is best left to the same tools that protect the rest of your system, such as full-disk encryption.

“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.” For more info, head over to: kube-project.com

Kube 0.8.0 is out!

After a waaaaaay to long “break” I have finally tagged another release.

The largest change in this release is the addition of the calendar view, which is not only useful, but also marks an important milestone in our development roadmap; We finally have all the pieces together from a technology perspective.

The calendar was a major undertaking due to a couple of challenges:

It’s synchronized over its own protocol (CalDAV)
It’s for once not a list, so it’s visually a completely different beast than everything else.
It has lots of fun special cases such as recurring events, timezones, overlapping events, multiday vs. single day events, …
We wanted to avoid loading your complete calendar (including the past 10 years) into memory, while making sure you get recurring events displayed even if they started 10 years ago.

The work done so far solves most of the important challenges, but there are also definitely a couple of holes in it still, such as no drag and drop support.

While the calendar is certainly the biggest new feature, there’s also a bunch of other improvements in there:

A new editor view, providing a much cleaner look than what we used to have.
Basic support for scheduling via iTip (both for sending an invitation to attendees, and for interacting with invitations you have recieved).
Autodiscovery support for CalDAV and CardDAV servers (So https://example.com instead of https://caldav.example.com/calendars/user@example.com/ is enough to configure your account).
Builds and runs on MacOS and Windows (it admittedly is not getting a lot of testing on those platforms, especially on Windows, but the baseline is there).
A fastmail account configuration dialog.
It’s now possible to create and modify contacts.
We no longer default to displaying HTML email
… and a bunch of other stuff in a little over 226 commits to sink and 561 commits to kube.

Tarballs

Tarballs are available at the usual locations:

Get It!

Of course the release is already outdated, so you may want to try a flatpak or some distro provided package instead:

https://kube.kde.org/getit.html

“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.”
For more info, head over to: kube.kde.org

Last months in Kube

Kube is still alive! I got distracted for a while, both professionally and privately, and writing blog posts is unfortunately always the first thing that ends up on the chopping block. Anyways, lot’s of progress has been made (103 commits in sink, ~130 commits in the kube codebase):

For sink we had a variety of bugfixes and performance improvements, especially on the more recent CalDAV/CardDAV backends.
For CalDAV/CardDAV we now do basic autodiscovery using the .well-known url’s. The DNS part of the spec has not been implemented so far.
This means for a properly set-up server you only have to specify the base url, and everything else will be discovered automatically from there.
On the more user-facing front we have:
- Sent emails are now collapsed by default
- Plain text is now the preferred method of viewing emails. You can still view the HTML variant if available by clicking a button.
  - The Addressbook is no longer read-only and you can now create contacts as well.
- A visually reworked composer that avoids becoming too wide and removes a lot of the visual clutter.
- The calendar can now render recurring-events.
- It is now possible to create events as well.
- Work on a tasks view has started

Conversations

Composer and drafts

Composer and recipients

Addressbook

Calendar

Tasks

Releases

You may have noticed that it’s been a while since the last release. This is not only because releases are additional work, but also because we already have a continuous delivery method with the nightly flatpak.
It’s clear that releases do provide value, both as a communication tool which version should be packaged, and if they would be maintained. With the current manpower we cannot maintain releases though, which makes it significantly less interesting.

With that said, the 0.8 release with the calendar is now long overdue and should be coming out soonish.

Experimental flatpak

Just to put it out there; Additionally to the usual “master” branch of the flatpak, there is also an “experimental” branch, containing, surprise, various experimental bits and pieces.

This currently entails:

A plugin that stores the accounts password encrypted by the accounts gpg key (blindly assuming there is one with a matching email address).
A search view
The upcoming calendar view (which we’ll move over in the next release)
The above todo view (which will take a little longer to move to master)
A “File as expense” plugin (a showcase how we could do extensions in the mail view).
The Inbox crusher view (an experiment for a view to go through your inbox one-by-one).

It typically serves as a staging ground for new components, and is the version that I’m running day-to-day. flatpak makes it easy to switch back and forth between the branches on top of the same dataset, so you can try it and switch back if you don’t like what you see.

To give it a shot use the following command to install and switch to the experimental flatpak branch:

flatpak -y --user install --from https://files.kube-project.com/flatpak/com.kubeproject.kube.experimental.flatpakref
flatpak --user make-current com.kubeproject.kube experimental

To switch back simply issue:

flatpak --user make-current com.kubeproject.kube master

Kube Commits, Sink Commits

Previous updates

More information on the Kolab Now blog!

“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.” For more info, head over to: kube-project.com

Calendar progress

As we’re closing in on a simple but functional calendar for Kube, I’d like to share our progress with you.

We’ve decided to start with a week view, as that seems to be a good compromise between information density and enough information for day-to-day use.
We will eventually complement that with a month view, which is probably all we need for the time being.

An agenda view will probably rather become part of a separate view, showing you upcoming events, tasks that need to be taken care of, important emails, ….

Anyhow, here’s the week view:

Screenshot_20180823_103304

This view is based on two models, one for full-day events (on top) and one for the rest below. The models each query for all events that overlap with the time-range shown, which we can do efficiently due to the work of Rémi on the calendar queries. That means we really only have to deal with a few events in memory, which we can easily do on the spot. The models then do their magic to calculate the position and sizes of all the necessary boxes, so all we have to do in QML is paint them. This now also includes recurring events, although we’re not dealing with exceptions just yet.

The colors of the events are taken from the calendar, which gets synchronized via CalDAV. This is a tradeoff design wise because you can bet that those colors will not match Kube in any way. But we decided that Kube should work well with other devices and clients, and the color is a major factor to recognize immediately what belongs where. Perhaps we can still make the colors a bit easier on the eye by desaturating them a bit, we’ll have to see.

Otherwise there is dimming of the past, and a blue line indicating the current time. Simple and without distractions.

The next steps are going to be adding a detail view, as well as a simple editor, and then we should already have the basics for your daily needs.

We’re still running a Kolab Now promotion for Kube! For more information head over to: blogs.kolabnow.com/2018/08/29/kube-campaign-last-chance

Optimizing Kube’s storage

Near the middle / end of my internship, I got to modify parts of the storage system in Sink, the library handling all the data of Kube.

The goal was to both to speed up the storage and reducing disk space. These two goals often go hands in hand in databases, since smaller data means faster disk lookup, and more data put in memory, available for direct usage.

Reducing key sizes

The first important modification was for keys to be stored in a binary format instead of the displayable format. This allowed data retrieval to be faster, because the binary format would be much smaller than the display format.

On the memory usage side, this is a bit more awkward to measure: on the one hand, the since the size of keys is smaller in memory, but on the other hand, LMDB (the database system we are using) will try to put more data in memory.

But on the whole, either we are using less memory, or Kube will be faster altogether. The reality is probably a mix of the two.

As for the numbers, these are the results of two different benchmarks:

Develop branch	This patch
Current Rss usage [kb]: 40700	Current Rss usage [kb]: 39112
On disk [kb]: 10788	On disk [kb]: 8836
Write amplification: 12.0075	Write amplification: 9.83485

Develop branch	This patch
Total pages: 760	Total pages: 603
Used size: 1425408	Used size: 1191936
Total on disk: 3293184	Total on disk: 2650112
Write amplification: 3.63268	Write amplification: 3.51402

As we can see on both tables, we use less disk space after the patch, but memory usage has not gone down in all places.

Overall we use approximately 20% less disk space.

Separating UIDs and revisions

Another important modification on the storage system was separating UIDs and revisions. Before this patch, we used to store the UID and the revision as keys inside the database.

What’s more is that the key was in the format “{this-is-a-uid}0000000000042”. The reason for padding the number with zeroes was that we need the data to be ordered, and we needed the keys to be stored as a string for the UID.

However, every revision is unique, so we can store only the revision as a key. This also allows us to store the key as a number, use the “integer key” feature of LMDB so the sorting stays correct and save a nice amout of space.

But, since we can keep old revisions, we need to track them. Therefore we need another table for mapping UIDs and revisions.

The rationale behind this patch is that since the “uid to revisions” database will be much smaller than the main database, it can be put into memory, leading to faster lookups.

Even though the results were not as consequent as the “reducing key sizes” patch, it seems we still got a small improvement performance wise. We will better see how different the performance is in the near future, though.

In the meantime, we can use the benchmarks graphics to see how performance, memory and disk usage was impacted (courtesy of our CI):

Continue reading “Optimizing Kube’s storage”

Last week in Kube

Lot’s of progress on the calendar. We now have a nice little read-only calendar. There’s still important stuff missing, like recurrences, but we’re getting there and I’m already using it with my actual calendar data.
- A date switcher to move between weeks.
- We now show the date for each day and the week number for the week view.
- The events get the proper colors from the calendar.
- The view can be filtered by calendar using the checkbox.
- The calendar checkbox now indicates the color, to save some precious space in that area.
- A tooltip provides the full calendar name if there is not enough space anyways.
- The calendar list is now scrollable with many calendars.
Fixed the name in the menu and dock as well as the mounted image name on Mac OS.
Fixed attachment dialogs on Mac OS.
The CI now checks the intended download paths for availability.
The website now has build status badges from the CI.
The “New Email” button (and similar green buttons) now has improved focus indication.
We’re now fixing the minimum font size to 11 if there is no QT_QPA_PLATFORMTHEME defined on linux. Qt hardcodes it to 9 otherwise, which is tiny.
Because flatpak uses pid-namespaces and lmdb uses pid’s for locking, starting two flatpaks results in a crash (the db is shared but the pid’s are guaranteed to be the same). It doesn’t seem like there is a good solution to this short of communicating with the original flatpak and ensuring all related processes are running in the same container. For now this is circumvented by simply not starting the second instance using a lockfile.
We removed the IMAP and similar protocol specific accounts and replaced it with a single “Custom” account. The account concept is supposed to encapsulate multiple procols, such as IMAP, SMTP, CalDAV and CardDAV as one logical unit. It thus doesn’t make much sense to have an IMAP specific account, and with the new “Custom” account you can still set-up IMAP only, or combine it with CalDAV and CardDAV so you can use all aspects of Kube.

Kube Commits, Sink Commits

Previous updates

More information on the Kolab Now blog!

“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.”

For more info, head over to: kube-project.com

Last week in Kube

We have a new filter overlay. We opted to avoid the inline searchbar entirely, and are using the overlay to display it instead.
We try harder to display all search results and not artifically limit it. This was especially problematic because we don’t sort the search results before the limit, meaning we could end up loosing very relevant search results (such as recent mails).
The composers html switch is gone. Instead we simply always offer the buttons to apply formatting, together with a button to remove all formatting for plaintext instead.
Rémi landed his storage improvement patch which shaves off a cool ~15% of storage requirements. This also helps performance because smaller db mean less data to load into memory. This was followed up by fix for a performance regression caught by the CI.
The Kube icon is now available in all necessary sizes to make it look good on Mac.
Important emails are now indicated in the maillist.
The default settings of the GMail account have been fixed.
There is finally a separate login and email address field in the IMAP account configuration.
We have a new website at kube-project.com where we now also offer the nighlies. These are now real nightlies that are automatically updated if all CI checks pass.
Prevented multiple flatpak instances at the same time as this does not currently work properly.
Last but not least, there’s progress on the Calendar.

Screenshot_20180816_165028 — Never mind the colors, they are coming from the CalDAV backend.

Kube Commits, Sink Commits

Previous updates

More information on the Kolab Now blog!

“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.”

For more info, head over to: kube-project.com