Release of Kube 0.1.0

It’s finally done! Kube 0.1.0 is out the door.

First off, this is a tech preview really and not meant for production use.

However, this marks a very important step for us, as it lifts us out of a rather long stretch of doing the ground work to get regular development up and running. With that out of the way we can now move in a steadier fashion, milestone by milestone.

That said, it’s also the perfect time to get involved!
We’re planning our milestones on phabricator, at least the ones within reach, so that’s the place to follow development along and where you can contribute, be it with ideas, feedback, packaging, builds on new platforms or, last but not least, code.

So what is there yet?

You can setup an IMAP account, you can read your mail (even encrypted), you can move messages around or delete them, and you can even write some mails.

kube_main

BUT there are of course a lot of missing bits:

  • GMail support is not great (it needs some extra treatment because GMail IMAP doesn’t really behave like IMAP), so you’ll see some duplicated messages.
  • We don’t offer an upgrade path between versions yet. You’ll have to nuke your local cache from time to time and resync.
  • User feedback in the UI is limited.
  • A lot of commonly expected functions are not existing yet.
  • ….

As you see… tech preview =)

What’s next?

We’ll focus on getting a solid mail client together first, so that’s what the next few milestones are all about.

The next milestone will focus on getting an addressbook ready, and after that we’ll focus on search for a bit.

I hope we can scope the milestones approximately ~1 month, but we’ll have to see how well that works. In any case releases will be done only once the milestone is reached, and if that takes a little longer, so be it.

Packaging

This also marks the point where it starts to make sense to package Kube.
I’ve built some packages on copr already which might help packagers as a start. I’ll also maintain a .spec file in the dist/ subdirectory for the kube and sink repositories (that you are welcome to use).

Please note that the codebase is not yet prepared for translations, so please wait with any translation efforts (of course patches to get translation going are very welcome).

In order to release Kube a couple of other dependencies are released with it (see also their separate release announcements):

  • sink-0.1.0: Being the heart of Kube, it will also see regular releases in the near future.
  • kimap2-0.1.0: The brushed up imap library that we use in sink.
  • kasync-0.1.0: Heavily used in sink for writing asynchronous code.

Tarballs

Release of KAsync 0.1.0

I’m pleased to announce KAsync 0.1.0.

KAsync is a library to write composable asynchronous code using lambda-based continuations.

In a nutshell:

Instead of:

class Test : public QObject {
    Q_OBJECT
public:
    void start() {
        step1();
    }
public signals:
    void complete();
private:

    void step1() {
        .... execute step one
        QObject::connect(step1, Step1::result, this, Test::step2);
    }

    void step2() {
        .... execute step two
        QObject::connect(step1, Step1::result, this, Test::done);
    }

    void done() {
        emit complete();
    }

};

you write:

KAsync::Job step1() {
    return KAsync::start([] {
        //execute step one
    });
}

KAsync::Job step2() {
    return KAsync::start([] {
        //execute step two
    });
}

KAsync::Job test() {
    return step1().then(step2());
}

The first approach is the typical “job” object (e.g. KJob), using the object for encapsulation but otherwise just chaining various slots together.

This is however very verbose (because what typically would be a function now has to be a class), resulting in huge functional components implemented in a single class, which is really the same as having a huge function.

The problem get’s even worse with asynchronous for-loops and other constructs, because at that point member variables have to be used as the “stack” of your “function” and the chaining of slots resembles a function with lot’s and lot’s of goto statements (It becomes really hard to decipher what’s going on.

KAsync allows you to write such code in a much more compact fashion and also brings the necessary tools to write things like asynchronous for loops.

There’s a multitude of benefits with this:

  • The individual steps become composable functions again (that can also have input and output). Just like with regular functions.
  • The full assembly of steps (the test() function), becomes composable as well. Just like with regular functions.
  • Your function’s stack doesn’t leak to class member variables.

Additional features:

  • The job execution handles error propagation. Errors just bubble up through all error handlers unless a job reconciles the error (in which case the normal execution continues).
  • Each job has a “context” that can be used to manage the lifetime of objects that need to be available for the whole duration of the execution.

Please note that for the time being KAsync doesn’t offer any API/ABI guarantees.
The implementation heavily relies on templates and is thus mostly implemented in the header, thus most changes will require recompilation of your source code.

If you’d like to help out with KAsync or have feedback or comments, please use the comment section, or the phabricator page.

If you’d like to see KAsync in action, please see Sink.

Thanks go to Daniel Vrátil who did most of the initial implementation.

Tarball

Release of KIMAP2 0.1.0

I’m pleased to announce the release of KIMAP2 0.1.0.

KIMAP2 is a KJob based IMAP protocol implementation.

KIMAP2 received a variety of improvements since it’s KIMAP times, among others:

  • A vastly reduced dependency chain as we no longer depend on KIO. KIMAP2 depends only on Qt, KF5CoreAddons, KF5Codecs, KF5Mime and the cyrus SASL library.
  • A completely overhauled parser. The old parser performed poorly in situations where data arrived faster at the socket than we could process it, resulting in a lot of time wasted memcpying buffers around and an unbounded memory usage. To fix this a dedicated thread was used for every socket, resulting in a lot of additional complexity and threading issues. The new parser uses a ringbuffer with fixed memory usage and doesn’t require any extra threads, allowing the socket to fill up if the application can’t process fast enough, and thus keeping the server from sending more data eventually.
    It also minimizes memcopies and other parsing overhead, so the cost of parsing is roughly reading the data once.
  • Minor API improvements for the fetch and list jobs.
  • Various fixes for the login process.
  • No more KI18N dependencies. KIMAP2 is a protocol implementation and doesn’t require translations.

For more information please refer to the README.

While KIMAP2 the successor of KIMAP, KIMAP will stick around for the time being for kdepim.
KIMAP and KIMAP2 are completely coinstallable: library names, namespaces, envirionment variables etc. have all been adjusted accordingly.

KIMAP2 is actively used in sink and is fully functional, but we do not yet guarantee API or ABI stability (this will mark the 1.0 release).
If you need API stability rather sooner than later, please do get in touch with me.

If you’d like to help out with KIMAP2 or have feedback or comments, please use the comment section, the kde-pim mailinglist or the phabricator page.

Tarball

Kube: Accounts

Kube is a next generation communication and collaboration client, built with QtQuick on top of a high performance, low resource usage core called Sink.
It provides online and offline access to all your mail, contacts, calendars, notes, todo’s etc.
Kube has a strong focus on usability and the team works with designers and Ux experts from the ground up, to build a product that is not only visually appealing but also a joy to use.

To learn more about Kube, please see here.

Kube’s Account System

Data ownership

Kube is a network application at its core. That doesn’t mean you can’t use it without network (even permanently), but you’d severely limit its capabilities given that it’s meant to be a communication and collaboration tool.

Since network communication typically happens over a variety of services where you have a personal account, an account provides a good starting point for our domain model. If you have a system with large amounts of data that are constantly changing it’s vital to have a clear understanding of data ownership within the system. In Kube, this is always an account.

By putting the account front and center we ensure that we don’t have any data that just belongs to the system as a whole. This is important because it becomes very complex to work with data that “belongs to everyone” once we try to synchronize that data with various backends. If we modify a dataset should that replicate to all copies of it? What if one backend already deleted that record? Would that mean we also have to remove it from the other services?
And what if we have a second client that has a different set of account connected?
If we ensure that we always only have a single owner, we can avoid all those issues and build a more reliable and predictable system.

The various views can of course still correlate data across accounts where useful, e.g. to show a single person entry instead of one contact per addressbook, but they then also have to make sure that it is clear what happens if you go and modfiy e.g. the address of that person (Do we modify all copies in all accounts? What happens if one copy goes out of sync again because you used the webinterface?).

Last but not least we ensure this way that we have a clear path to synchronize all data to a backend eventually, even if we can’t do so immediately. E.g. because the backend in use does not support that data type yet.

The only bit of data that is stored outside of the account is data specific to the device in use, such as configuration data for the application itself. Data that isn’t hard to recreate, is easy to migrate and backup, and very little data in the first place.

Account backends

Most services provide you with a variety of data for an individual account. Whether you use Kolabnow, Google or a set of local Maildirs and ICal files,
you typically have access to Contact, Mails, Events, Todos and many more. Fortunately most services provide access to most data through open protocols,
but unfortunately we often end up in a situation where we need a variety of protocols to get to all data.

Within Sink we call each backend a “Resource”. A resource typically has a process to synchronize data to an offline cache, and then makes that data accessible through a standardized interface. This ensures that even if one resource synchronizes email over IMAP and another just gathers it from a local Maildir,
the data is accessible to the application through the same interface.

Because various accounts use various combinations of protocols, accounts can mix and match various resources to provide access to all data they have.
A Kolab account for instance, could combine an IMAP resource for email, a CALDAV resource for calendars and CARDDAV resource for contacts, plus any additional resources for instant messaging, notes, … you get the idea. Alternatively we could decide to get to all data over JMAP (a potential IMAP successor with support for more datatypes than just email) and thus implement a JMAP resource instead (which again could be reused by other accounts with the same requirements).

diagram

 

Specialized accounts

While accounts within Sink are mostly an assembly of some resources with some extra configuration, on the Kube side a QML plugin is used (we’re using KPackage for that) to define the configuration UI for the account. Because accounts are ideally just an assembly of a couple of existing Sink resources with a QML file to define the configuration UI, it becomes very cheap to create account plugins specific to a service. So while a generic IMAP account settings page could look like this:

imapaccount

… a Kolabnow setup page could look like this (and this already includes the setup of all resources including IMAP, CALDAV, CARDDAV, etc.):

kolabaccount

Because we can build all we know about the service directly into that UI, the user is optimally supported and all that is left ideally, are the credentials.

Conclusion

In the end the aim of this setup is that a user first starting Kube selects the service(s) he uses, enters his credentials and he’s good to go.
In a corporate setup, login and service can of course be preconfigured, so all that is left is whatever is used for authentication (such as a password).

By ensuring all data lives under the account we ensure no data ends up in limbo with unclear ownership, so all your devices have the same dataset available, and connecting a new devices is a matter of entering credentials.

This also helps simplifying backup, migration and various deployment scenarios.

Kube going cross-platform in Randa

I’m on my way back home from the cross-platform sprint in Randa. The four days of hacking, discussing and hiking that I spent there, allowed me to get a much clearer picture of how the cross-platform story for Kube can work, and what effort we will require to get there.

We intend to ship Kube eventually on not only Linux, Windows and Mac, but also on mobile platforms like Android, so it is vital that we figure out blockers as early as possible, and keep the whole stack portable.

Flatpak

The first experiment was a new distribution mechanism instead of a full new platform. Fortunately Aleix Pol already learned the ropes and quickly whipped up a Flatpak definition file that resultet in a self contained Kube distribution that actually worked.

Given that we already use homegrown docker containers to achieve similar results, we will likely switch to Flatpak to build git snapshots for early adopters and people participating in the development process (such as designers).

Android

Anreas Cord-Landwehr perpared a docker image that brings the complete cross-compiler toolchain with it, so that makes for a much smoother setup process than doing everything manually. I mostly just followed the KDE-Android documentation .

After resolving some initial issues with the Qt-Installer with Andreas (Qt has to be installed using the Gui-Installer from console as well, using some arcane configuration script that changes variables with every release…. WTF Qt), this got me quickly set up to compile the first dependencies.

Thanks to the work of Andreas, most frameworks already compile flawlessly, some other dependencies like LMDB, flatbuffers and KIMAP (which currently still depends on KIO), will require some more work though. However, I have a pretty good idea by now what will be required to get everything to build on Android, which was the point of the exercise.

Windows

I postponed the actual building of anything on Windows until I get back to my workstation, but I’ve had some good discussions about the various possibilities that we have to build for Windows.

While I was initially intrigued by using MXE to cross-compile the whole stack (I’d love not having to have a Windows VM to build packages), the simplicity of the external CMake project that Kåre Särs setup for Kate is tempting as well. The downside of MXE would be of course that we don’t get to use the native compiler, which may or may not result in performance impacts, but definitely doesn’t allow developers on Windows to work with MSVC (should we get any at some point….).

I guess some experimentation will be in order to see what works best.

Mac

Lacking an OS X machine this is also still in the theoretical realm, but we also discussed the different building techniques and how the aim must be to produce end-user installable application bundles.

 

As you can see there is still plenty to do, so if you feel like trying to build the stack on your favorite platform, that help would be greatly appreciated! Feel free to contact me directly, grab a task on Phabricator, or join our weekly meetings on meet.jit.si/kube (currently every Wednesday 12:00).

The time I spent in Randa showed once more how tremendously useful these sprints are to exchange knowledge. I would have had a much harder time figuring out all the various processes and issues without the help of the various experts at the sprint, so this was a nice kick-start of the cross-platform effort for Kube. So thank you Mario and team that you organized this excellent event once more, and if you can, please help keeping these Sprints happening.

So what is Kube? (and who is Sink?)

Michael first blogged about Kube, but we apparently missed to properly introduce the Project. Let me fix that for you 😉

Kube is a modern groupware client, built to be effective and efficient on a variety of platforms and form-factors. It is built on top of a high-performance data access layer and Qt Quick to provide an exceptional user experience with minimal resource usage. Kube is based on the lessons learned from KDE Kontact and Akonadi, building on the strengths and replacing the weak points.

Kube is further developed in coordination with Roundcube Next, to achieve a consistent user experience across the two interfaces and to ensure that we can collaborate while building the UX.

A roadmap has been available for some time for the first release here, but in the long run we of course want to go beyond a simple email application. The central aspects of the the problem space that we want to address is communication and collaboration as well as organization. I know this is till a bit fuzzy, but there is a lot of work to be done before we can specify this clearly.

To ensure that we can move fast once the basic framework is ready, the architecture is very modular to enable component reuse and make it as easy as possible to create new ones. This way we can shift our focus over time from building the technology stack to evolving the UX.

Sink

Sink is a high-performance data access layer that provides a plugin mechanism for various backends (remote servers e.g. imap, local maildir, …) an editable offline cache that can replay changes to the server, a query system for efficient data-access and a unified API for groupware types such as events, mails, todos, etc.

It is built on top of LMDB (a key-value store) and Qt to be fast and efficient.

Sink is built for reliability, speed and maintainability.

What Kube & Sink aren’t

It is not a rename of Kontact and Akonadi.
Kontact and Akonadi will continue to be maintained by the KDEPIM team and Kube is a separate project (altough we share bits and pieces under the hood).
It is not a rewrite of Kontact
There is no intention of replicating Kontact. We’re not interested in providing every feature that Kontact has, but rather focus on a set that is useful for the usecases we try to solve (which is WIP).

Development

Development planning happens on phabricator, and the kdepim mailinglist. Our next sprint is in Toulouse together with the rest of the KDEPIM team.

We also have a weekly meeting on Wednesday, 16:00 CET with notes sent to the ML. If you would like to participate in those meetings just let me know, you’re more than welcome.

Current state

Kube is under heavy development and in an early stage, but we’re making good progress and starting to see the first results (you can read mail from maildir and even reply to mails). However, it is not yet ready for general consumption (though installable installable).

If you want to follow the development closely it is also possible to build Kube inside a docker container, or just use the container that contains a built version of Kube (it’s not yet updated automatically, so let me know if you want further information on that).

I hope that makes it a bit clearer what Kube and Sink is and isn’t, and where we’re going with it. If something is still unclear, please let me know in the comments section, and if you want to participate, by all means, join us =)