Progress on the prototype for a possible next version of akonadi

Ever since we introduced our ideas the next version of akonadi, we’ve been working on a proof of concept implementation, but we haven’t talked a lot about it. I’d therefore like to give a short progress report.

By choosing decentralized storage and a key-value store as the underlying technology, we first need to prove that this approach can deliver the desired performance with all pieces of the infrastructure in place. I think we have mostly reached that milestone by now. The new architecture is very flexible and looks promising so far. We managed IMO quite well to keep the levels of abstraction to a necessary minimum, which results in a system that is easily adjusted as new problems need to be solved and feels very controllable from a developer perspective.

We’ve started off with implementing the full stack for a single resource and a single domain type. For this we developed a simple dummy-resource that currently has an in-memory hash map as backend, and can only store events. This is a sufficient first step, as turning that into the full solution is a matter of adding further flatbuffer schemas for other types and defining the relevant indexes necessary to query what we want to query. By only working on a single type we can first carve out the necessary interfaces and make sure that we make the effort required to add new types minimal and thus maximize code reuse.

The design we’re pursuing, as presented during the pim sprint, consists of:

  • A set of resource processes
  • A store per resource, maintained by the individual resources (there is no central store)
  • A set of indexes maintained by the individual resources
  • A clientapi that knows how to access the store and how to talk to the resources through a plugin provided by the resource implementation.

By now we can write to the dummyresource through the client api, the resource internally queues the new entity, updates it’s indexes and writes the entity to storage. On the reading part we can execute simple queries against the indexes and retrieve the found entities. The synchronizer process can meanwhile generate also new entities, so client and synchronizer can write concurrently to the store. We therefore can do the full write/read roundtrip meaning we have most fundamental requirements covered. Missing are other operations than creating new entities (removal and modifications), and the writeback to the source by the synchronizer. But that’s just a matter of completing the implementation (we have the design).

To the numbers: Writing from the client is currently implemented in a very inefficient way and it’s trivial to drastically improve this, but in my latest test I could already write ~240 (small) entities per second. Reading works around 40k entities per second (in a single query) including the lookup on the secondary index. The upper limit of what the storage itself can achieve (on my laptop) is at 30k entities per second to write, and 250k entities per second to read, so there is room for improvement =)

Given that design and performance look promising so far, the next milestone will be to refactor the codebase sufficiently to ensure new resources can be added with sufficient ease, and making sure all the necessary facilities (such as a proper logging system), or at least stubs thereof, are available.

I’m writing this on a plane to Singapore which we’re using as gateway to Indonesia to chase after waves and volcanoes for the next few weeks, but after that I’m  looking forward to go full steam ahead with what we started here. I think it’s going to be something cool =)

Advertisements

Author: cmollekopf

Christian Mollekopf is an open source software enthusiast with a special interest in personal organization tools. He started to contribute actively to KDE in 2008 and currently works for Kolab Systems leading the development for the next generation desktop client.

19 thoughts on “Progress on the prototype for a possible next version of akonadi”

  1. How parallelizable will this new architecture be?

    If I recall correctly in kmail if an operation (like opening a large folder) took ages, you couldn’t do anything else in the meantime. It’s nice that the UI stays responsive, but it would have been better if it let you load another folder/email while you wait.

    I presume this was a limitation in akonadi. If that’s the case, how will this new version address that?

    1. It’s not a fundamental design limitation in akonadi, but the current implementation does in some cases have this behaviour. Reading is in-process and each query is executed in a thread (at least at the moment), so that is never blocked. The resource itself will separate the synchronization from other modifications, so i.e. writing to a resource while a large sync is in progress works and stays responsive. On demand-loading of data, which is probably the most critical part in that regard, can also be implemented in a separate thread, so we can avoid having to wait for a sync to finish first.

      The processing of messages (modifications) is as parallelizable as we want, but we have to first figure out what actually makes sense.

      The shouldn’t be any cross-resource interference because data is directly aggregated in the client process (the application) and so resources can only affect each other by shared resources such as cpu time and disk access.

      The most important change that we want to achieve is that opening a large folder will no longer result in all items of that folder getting loaded, but that we can i.e. load items as we scroll in the view. This should drastically reduce the perceived loading times. Of course that means we have to have the necessary information to filter and sort as required.

  2. One thing I’m actually curious about is regarding the choice of Flatbuffers. The presentation touched on this (“capnproto, protobufs also considered”), but I’d be interested in more detail as to the reasoning and tradeoffs.

    1. all three are generally interesting as they all solve the problem loading data from a schemaless storage with a schema. AFAIR we ruled protobufs out because it doesn’t provide zero-copy loading and flatbuffers seems to be a more lightweight successor for it. capnproto is very similar to flatbuffers and for our purpose there is little difference, except that capnproto isn’t readily available on windows.

      zero-copy is interesting for us because it allows us, together with the memory-mapped loading provided by lmdb, to read only parts of the buffer from disk, and thus to minimize unnecessary copying and potentially parsing of data (although the corresponding pages will still be loaded).

  3. Sounds like it is still too memory hungry. I can’t work with KMail at all since launching akonadi alone fills my 1G memory. Nothing much else is possible after that, especially since akonadi/KMail people still outright refuse to stop the service when the last client stops using it. I long for the good old KMail 1 days when it was usable and resource friendly.

    1. This is actually a lot less memory hungry than current Akonadi. No MySQL process, for once, and only 1 external process per resource (which is kept minimal to the dataset in use). Far less inter-process communication means far less serialization/deserialization of messages, and flatbuffers give us extremely efficient read out of the store. That all means far fewer data copies, which easily lead to data fragmentation (given all the smallish messages that generally makes up groupware).

      Do you have any particular reason for your skepticism, or is it just “it doesn’t meet my standards no, so it is unlikely to ever do so?” type expectations?

      1. Honestly, this sounds a little bit too good to be true. But then again, so did baloo when it first got announced. I gave them an honest try after heaving read their claims and they actually managed to fix the whole mess that was nepomuk. I actually never notice the baloo indexer staying around and doing its business all time.
        If you claims turn out to be true, I might return back to KMail. You might also want to consider rebranding to get a clean slate in that case. But if you can’t get it that low on resource usage when it is idling around, please, for the love of all those who can’t buy the latest and shiniest systems, consider implementing an exit-on-idle feature, or donating shiny new hardware to all those who can’t buy themselves ones. 😀

  4. Will it make Kmail open folders without having to wait for a couple of minutes because Akonadi is doing who knows what in its databases? Will it stop wiping messages and notes that are in the databases but not in the app’s message and note storage folder?

    Akonadi is such a great idea, and when it workd it’s wonderful, but the pice is high, because half of the time it doesn’t work well, fast or at all; and from time to time you can see how your mail or your notes in Kjots have dissapeared.
    I honestly hope this hipothetical next version will finally WORK.

    For the fanatics: I don’t expect any response to this message. I’m not a coder, I’m not a power Linux user, nor I’m not going to fiddle with obscure config files; I’m just an average experienced user who knows when things work and when don’t. Thunderbird works, my Mac works, Outlook works. Kmail/Kontact/Kjots/Akonadi, don’t. Period.

    Best regards.

    1. “Will it make Kmail open folders without having to wait for a couple of minutes because Akonadi is doing who knows what in its databases?”

      How fast remains to be seen, but faster it will be. For one: no IPC-induced latency (serializing and unserializing messages, SQL queries, sending results between processes). Even at this non-optimized stage, Christian is getting some 40,000 (!) entries per second from storage into the client, and we have quite a bit more planned.

      One thing I’m really excited about (both because it presents some interesting development challenges and because it should radically improve performance in many common cases) is the ability to store and then incrementally update threading … so a folder with 100k messages with a threaded view will not need to have the threads re-calculated from scratch each time it is shown. Best of all, the currently planned implementation means no client-side caching tricks (which also implies the memoization lives between app restarts and even different clients!) *and* it can be updated incrementally by a pipeline plugin in the resource as new messages appear (which implies an incremental cost on new messages rather than a full recalc)

      “Will it stop wiping messages and notes that are in the databases but not in the app’s message and note storage folder?”

      These kinds of synchronization problems should be far easier to fix. There is no guarantee they won’t happen as bugs are always possible to create with developers being “creative” enough 😉 but at least the simpler code paths in this design will make ferreting out such problems far simpler and should help avoid many such issues in the first place (thanks in large part to the reactive design for modifications)

      “Akonadi is such a great idea, and when it workd it’s wonderful, but the pice is high”

      Yep, so we’re working hard to keep the great ideas, and lower the price.

      “Kmail/Kontact/Kjots/Akonadi, don’t. Period.”

      Heh .. yeah, they NEVER work. *hyperbole induced eye-roll* C’mon, let’s keep this in the realm of reality (on all sides): Kontact works, but not as well and consistently as we need it to. Akonadi works, but the complexities in its design cause maintenance issues and produces subtle bugs that can have unfortunate results. These are the things we need to work on .. and that’s what we’re doing.

      What would be awesome in the meantime is if we can all remain realistic about where we are and where we are going. Akonadi/Kontact work more than “don’t. Period”, and out of respect for those who have put so much effort into them we ought to recognize that. There are few things more discouraging to those working on improving things than for negative perfectionist statements to be thrown about.

      1. I didn’t expect an answer, especially as explaining as yours, at least it’s first three qurters. So I think you deserve a response for that time you have kindly spent.

        Ok, our efforts are encomiable. You, guys, are spending your free time on this, and giving it for free. That’s really generous and noble, seriously. But you are being selfindulgent. Your optimistic “half full, not half empty” point of view doesn’t make reality different: if something doesn’t serve its purpose almost always -and yes, those “perfectionists” who like things to work, have a rather narrow concept for that «almost»-, and especially if destroys the user’s “possesions” it isn’t working, sorry. Name it like you want, but more than -how much, 6, 7, 8 years?- of life and still not working, it’s not what most people would cal a success.
        No, if you couldn’t use your car as you expect from a car the 99% of the time, without issues -starting when you want it to start, not stopping or reducing speed to ridicule, keeping the controls, it’s “interface”, in the same place, the same way every day and always responsive- 4, maybe 5 days a week, and if from time to time it made you end in the hospital with some injuries I don’t think you would be indulgent at all and say that your car «works, but the complexities in its design cause maintenance issues and produces subtle bugs that can have unfortunate results».
        Yes, my illustration is a bit exaggerated: nobody has been phsically hurt due to a PIM data loss; and mecanical machines are always more solid and responsive than software “machines”, and I suppose they’ll always be; but everybody get’s the idea, I hope.

        Most average Linux users understand that FLOSS development can’t make solid software as quickly as private companies who have sufficient manpower working on a release 35-40 hours/week during months or years, yes, but release after release, year after year we have been suffereing the same errors in KDEPIM (and a bit in KDE4 itself, let’s be honest; although in KDE the progress and improvement has been quite noticeable), severe failures, like deletion of user’s data or impossibility of using the mail client because Akonadi wasn’t ready for minutes, or rather frequently and the user had to restart it -fortunately there’s Tunderbird when you need something on which trust always. It’s not integrated with KDE but, hey, works, always.-.

        Nothing farther from my will than discourage you, quite the opposite, to call a bit your attention to what really matters: to make Akonadi finally work. Stop adding cool and modern features before it really works, please. Do a poll or so, I’m sure most users will prefer a solid PIM suite even it it doesn’t support cloud integration that one that fails more than a shotgun in a fair’s attraction. Listen a bit to the users, most aren’t so stupid as meny devs seem to believe, honestly.
        We imagine that experimenting with code and change things every few years even if the former never were really ended, must be superfun, but, paraphrasing you, there are few things more discouraging to those trying to avoid “captivity” in software and to convince other people to use and support open source software -I’ll save you to read what I’ve been told several times, regarding software quality, when talking to some people about the need of donations for FLOSS projects-, than having to admit that, in the end, is completely undestandable that people use Windows and such even if they know and have used Linux some times. We may not like it, but you hardly hear of people losing their mails in Outlook or seeing their desktop crashing, their apps vanishing in thin air in the middle of the work session or devices just stop working the morning after without having done any software update nor config files edition.

        So, I REALLY ENCOURAGE you, guys, to keep up the work and make Akonadi what it should have been since one or two years after its initial release: a really reliable framework that even if didn’t support Facebook integration or XMPP/Telegram/whatever’s next in KTP, we could be sure that our mail won’t disappear; that we can make a backup of our notes folder and be sure that all our notes are in said folder, not gods know where into some database common users don’t know how to read nor should need to.
        I sincerely hope this new prototype will solve almost -a thin «almost» again- all the important bugs that make, above all, Kmail so behind Thunderbird and famous proprietary solutions when it comes to reliability and responsiveness. And -I know this is a silly bug not related to Akonadi, but…- if the link in Kmail’s main window that asks to click on it to get online would ever work, then you would crown. xD

        I do wish you the best of the luck. I love my semantic desktop, especially when the PIM part works. I haven’t stopped donating during the last 6 years and I’ll not cease doing it even if I’m not very happy with some projects, because I do understand how much effort you put, guys, but FLOSS desktop -ah, if desktop software worked half as well as console’s!…- projects need to fulfill higher quality and stability standards, but I know of people who have decided no to donate anymore when they see the software they need keeps having almost the same bugs release after release, and a one of them was relatively active reporting bugs, and I remark «was»*.
        Being selfindulgent or denying the fact that a piece of software that is unusable half of the time is not working. Have you really used you KDEPIM apps, say, 10 times in a row without any issue, without Akonadi sitting there for a couple of minutes, without Akonadi protesting again some resource it says is not valid anymore, when 5 minutes ago it was? No, my respected aseigo, I don’t think so. So good luck and best whishes from the bottom of my heart, but keep in mind that the most important is to accomplish to make Akonadi WORK.

        *http://www.linux-magazine.com/Online/Blogs/Off-the-Beat-Bruce-Byfield-s-Blog/Why-I-rarely-file-bug-reports

        P. S: Excuse the extension. But now I don’t feel like resuming it. :-s

      2. >”How fast remains to be seen, but faster it will be.”
        This kind of statements drives me nuts!
        Your requirements document should have as item #1:

        #1 – Maximum time opening a folder with 2K messages using a single core 2GHz CPU may not take longer than 1 sec.

    1. Already did, along with just about every other embedded key/value store that we could find. unqlite is one of the few systems that seems to also fit our needs, and there is indeed an implementation of the storage layer in common/storage_unqlite.cpp. I haven’t tested it recently, however, and it has bitrotted a bit .. I’ll update it later today to keep it fresh. That said, there is no particular advantage we’ve seen with unqlite over lmdb at this point.

      If you’d like to look into this, I’d be happy to collaborate with you a bit on it … aseigo at kde.org or on G+ as aseigo …

      1. ok, the unqlite storage “driver” now compiles again .. however, the issue of duplicate keys remains, which means that indexes won’t work properly … something of a show-stopper unless indexes are refactored to allow for such a restriction.

    2. Hm… seems unqlite has no support for tables with duplicate keys … that’s something we’re relying on currently for creating indexes. :/ Hopefully I’m wrong and I’m just missing something in the docu.

  5. (I’m a different Luis, BTW)

    Hey,

    Please don’t forget to think about connection problems (a momentary ADSL reconnection, for instance) and resource conflicts. Both kind of errors are still a kind of nightmare. Whenever any of them happens when using Kaddress Book, KOrganizer or Kmail, Akonadi enters in a locked up state and never leaves. All of it also happens silently and no information is ever synchronized again with the server.

    We are having a very hard time with those bugs because we use CardDav, CalDav and IMAP a lot. Every week or so, my friends, which don’t know anything about computers, but have asked me to help them setup their computer systems for home and a small shop, have some kind of Akonadi trouble. They tell me that contacts, calendars and, much less often, emails, “stopped showing up on their computers and phones”. On those occasions I always go to Akonadi server and it is stuck with a CalDav/CardDav cryptic synchronization error. So far, the only thing that I managed to get working to restore synchronization was deleting all local Akonadi resources and recreating them afterwards. Besides losing a lot of data (all work for 1-2 weeks, usually), it takes a lot of time to do.

    I have even begged them to only make changes in the same computer and user, leaving all other instances (home computers and smartphones) only for browsing (read only). It didn’t solve the problem, because either their cheap VPS or humble ISP often fails for a few seconds. When a synchronization is under way during a failure, Akonadi might get stuck again.

    I’ve tried to replace Kontact with Thunderbird + Lightning, but they really liked Kontact better. Also, I couldn’t find any replacement for Kaddressbook (a CalDav client). I’m the one who convinced them to use free software and KDE and I’m a 100% free software guy. Since it seems that we have already done so much things, I’m really worried that this “last mile” of fixing those issues could push people back to proprietary software.

    Please, either in the new Akonadi or in the old one (or in both, ideally), give “service availability”, “recovery” and “conflict resolution” more love!

    Thanks for your wonderful work.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s