ostree & Flatpak at CERN

A week and a half ago I spent a few days in Geneva and gave a presentation about ostree and Flatpak at the CERN Computing Seminar. I started by briefly introducing Endless to give some context of the problems we’re trying to solve and how we’re using ostree and Flatpak for that, then proceeded to talk more in detail about these technologies. In the end, there were several questions, and I was happy to learn afterwards that among the audience there were some of the people working at the CVMFS project: a software distribution service to help deploy data-processing infrastructure and tools. I don’t know the full details about the project’s implementation, but from the problems they’re trying to solve it seems like ostree (or more specifically libostree) could perhaps be used to replace part of the core, which would leverage all the niceties of using a complex Open Source project (more eyeballs looking into bugs, more testing, etc.). I also think more use-cases could be found in the organization, so I hope my talk was a small seed to help introduce these projects at CERN in the medium/long term. The presentation has been recorded if you’re interested.

Getting authorization to access CERN this time was also different, as for the first time I got an entrance pass as a member of the CERN Alumni. So I would like to thank Antonella Del Rosso for the Alumni initiative and also for allowing me to kindly borrow her EU-CH power adapter when I forgot mine at my friends’ home. In the end Antonella also interviewed me about my experience at CERN and after I left, and produced this summary if you want to check it out.
I would also like to thank Miguel Ángel Marquina of the CERN Computing Seminar for organizing the presentation and all the details around it.

Photo showing the author and his daughter sitting close to the lake in Geneva.

Sitting by the lake with my daughter

Having spent more than 2 years in the region, it is the friends we have there that we miss the most. So it was great to meet them and old colleagues again.
My family traveled there with me and we stayed with friends from Spain, so it was funny to see our daughter (who used to play with those friends’ kids all the time when we lived there) excusing her shyness for not speaking Spanish. But after a day or two they were all successfully playing together; it’s amazing how children can get along no matter what differences or barriers they find, while adults often resort to stupid feelings and dangerous actions.
The mountains landscape is another thing we miss in Berlin and the Spring’s clear weather allowed us to fully gaze at the Jura or the Mont Blanc which should last us for another few more months. After that, I guess I’ll try to find some graffiti of mountains around Berlin 🙂

Updates on the Endless App Center / GNOME Software

The great majority of my work at Endless is to (try to) tame GNOME Software and apply the changes that make it what we simply call “the App Center” (repo here) in the Endless OS.
This is a lot of work and usually I’d love to share more often what I am doing but end up neglecting the blog due to the lack of time. So here’s a summary of what I have done the past few months.

New App Tiles

From the times where it was called the App Store (and not based on GNOME Software like know), the Endless OS’s App Center used to have what we called internally as “app thumbnails”. These were images carefully produced for each app that Endless distributed, that worked as a way to provide some visual hint and attractiveness that is many times not achieved by the apps’ own icons. Here’s a screenshot of that version:

Old version of App Center in the Endless OS: displays colorful images in squares representing the apps available

Old version of the App Center


There was a couple of problems with “app thumbnails” like that: 1) we started shipping Flathub as a remote by default, and it’s simply not scalable to go and create an image for every app that is available in the repository; and 2) even if visually appealing, the app tiles make it a bit difficult to correctly display text on them, and depending on how apps appear next to each other, it can become visually quite bloated.

Thus the solution we came up with for the 2nd problem was to give dim a little bit the effect that the thumbnails have by placing a translucid layer on top of them, and to have a dedicated area for textual information. This means we still use app thumbnails for the apps that have it, but they will all seem a bit less intense and more alike.
That still leaves us with the first problem of not having thumbnails for most apps. For fixing that we create a background from the main colors presented in the app’s logo. The background is composed of 4 gradients, each with one of the icon’s main colors. Using the logos’ colors ensures there’s some harmony between the logos and their backgrounds, and I am very happy with the result:

Generated backgrounds for apps: shows squares where the background is a mix of gradients with colors picked from the app's icon

Automatically generated backgrounds for apps

Combination of app tiles that have a thumbnail image, and some that have automatically generated backgournds

Updates from USB and LAN

As you know, Endless’ mission is to give access to computers (and all that comes with them: knowledge, entertainment, productivity) to those who are often in remote areas, with very weak or innexistent connectivity. Maybe you’ve already heard that at Endless we’re developing an “asynchronous internet” and optimize the use of the little data connectivity some of our users have. So it’s only logical that we can give the possibility for users to share the data among themselves without an internet connection. To do that, we (more specifically Philip Withnall, kudos to him!) have implemented a way in ostree for local repositories to be found, particularly, repositories in removable drives (e.g. USB keys) and LAN. This means that e.g. a teacher’s computer in a class room can download app updates from the interwebs and the sudents’ computers will just get the updates through LAN (without the need for an external connection). In the case of the USB, as you probably guessed, a user can just set up a repository in a USB, and share the drive with friends.
For the USB case, in which the number of apps available may differ a lot from the user machine’s catalog, we need to show which of those apps are available, so when inserting the USB drive in the machine, the App Center should just pop up and show a new category “USB” with the apps that are contained in it. See the following screenshot:

App Center showing the one app in an inserted USB  key, as a category

App Center showing the one app in an inserted USB key

Performance Constraints

One of our constant concerns is that our OS and apps run smoothly even on less powerful machines, since it’s what many of our users have. GNOME Software spawns a thread for every main operation that the user does, like installing or updating an app, and we’ve noticed that when a few of these operations are running in parallel, some machines will just freeze. Moreover, downloading a bunch of data in parallel may easily occupy the whole bandwidth without actually completing any of the downloads.
One can think of a lot of smart approaches for dealing with this, but for now we just implemented a restriction on the number of operations that can be run in parallel. This was implemented together with upstream and the number of possible operations in parallel is one per GB of RAM. You may argue this is good a heuristic as any other, but it gives a low number of operations for slower machines, while still allowing powerful machines to have multiple parallel operations.
For the Endless OS we just opted to limit this number to 1, but may revisit it later. Google Play also performs just one update/install at a time, so this is not such a crazy thing.

We need also to inform the user that an app is waiting for it’s turn to install/update, so in such cases an empty progress bar with a message is shown:

An app waiting for its turn to be updated: has an empty progress bar with the message "Pending update..." on top.

An app waiting for its turn to be updated

For consistency, we’ve changed also how we showed the “queued-for-install” state (a state that happens when the user clicks install and there’s no internet connection), to be consistent with the UI shown above.

Auto Updates

Another push we’re taking together with upstream is auto-updates. GNOME Software has had something similar to auto-updates for a while, though it was more like auto-downloads. This was heavily based on the fact that it’s not very safe to just go and install new versions of packages right away (apps may be running… ). So it’s up to the user to choose when to (reboot and) install those.
Flatpak though, has no such problems. Apps can be updated even while running, as the way the new update replaces an existing version happens atomically, without touching the running app’s files.

So the ideal thing to do for Flatpak is to have real auto-updates (that is, download and deploy them right away). But since GNOME Software still has to support other sorts of app distribution, it required a bit of creativity when designing the new UI for this, which Allan Day kindly did, and very patiently with me and all my opinionated views of it 🙂

I have implemented auto-updates downstream first, without some of the niceties of the new mockups, since we needed them for our very-soo-to-come new version. But the idea is to do the real implementation upstream soon.
Surely enough, even when turned on, auto-updates only happen when on unmetered connections currently; and Philip Withnall is working on a creative solution for metered connections (soon to be announced).

Misc Fixes

This post is long enough so it’s not really sensible to enumerate all the fixes done in the last months. So I will just mention that recently we have fixed important issues (upstream as well) like installing new runtimes’ extensions when an app update needs them; cancelling auto updates/downloads when their running and the connection is switched to a metered one; setting an app as updatable if one of its runtime extensions has an update (otherwise it was not possible to install those extensions), etc.

If you’re still reading this, thank you! But especially thanks to Richard and Allan for their patience and leadership upstream!
Hope you liked this. I will try to keep the updates more frequent!

Attending FOSDEM 2018

On Friday I will attend FOSDEM 2018 after having missing last year’s (but I had a good excuse).

It’s pretty sad that there’s no longer a Desktop devroom, but there is still plenty of other interesting topics to follow.

Thanks to my company Endless for making my attendance easier, and if you’re willing to know more about the problems we’re solving just reach out, as FOSDEM is the perfect place for that!

FOSDEM

Going to GUADEC 2017

I am currently waiting for my flight to Manchester to attend another GUADEC!
This year I am also giving a presentation about the application story in Endless OS. Our infrastructure, our changes to GNOME Software, our heavy use of Flatpak, etc. Hopefully you’ll find it interesting.

A good number of Endlessers will also be around and giving other talks. We’re friendly people, so come talk to us!

Yay going to GUADEC!

Many thanks to Endless for making it possible for me to attend the event, and I am looking forward to meeting everyone again!

Hope for a good 2017

As Berlin’s fireworks roar outside (for hours now!), I want to write my typical end of year post with some thoughts of 2016 and what’s coming.

We are leaving behind a year that many (most?) of us will not miss. I hope 2017 will be a better one but the global events that took place throughout 2016 do not make me very confident about that. With unusual political events affecting the lives of millions of people, an apparently acceptance of indecency, bigotry, and hate, together with the ongoing humanitarian crisis and senseless violence, this is surely not the world I had pictured for my children.
Still, I want to be positive and hope that by seeing what’s happening in some places, people can make good choices in 2017 (I am talking in broad terms but we got some important elections coming soon in Europe).

On a more positive note — because personally 2016 was actually a very good year –, this past year I also started working for Endless and moved from SwitzerFrance to Berlin, Germany. I love my job and I have been working very hard to do my share of Endless’ mission. The current direction of the world only validates our mission more, and motivates me to work harder. It’s difficult however to make time for everything, and again my pet projects took the hit, so no big updates in that subject this year.

The other big news is that Helena and I will be parents again soon! The joy of raising a child is something so special that it is hard for me to put into words so I can just say that we are of course extremely happy and curious (and scared too) about how life will be with two kids.

Happy 2017 everyone!