The road to UDisks 2.9.0

While the world is going crazy these days we continue to march in full strength towards the next UDisks release. It’s still a couple of weeks away and there are some interesting features still pending to be merged. With all the changes we’re bound with the promise to keep the public D-Bus and C API stable and that won’t change even that there were major changes under the hood. Overall we’ve been focusing on general stability and predictability, fixing various race conditions. But we’ve also added a couple of new interesting features.

Configurable mount options

UDisks carries a set of predefined mount options for well-known filesystems. With the primary focus on desktop use the defaults were chosen to fit typical use cases. For example the vfat filesystem defaults define the flush mount option being a typical filesystem for removable flash media and it’s important to make write operations more synchronous to prevent users tearing their drives out while kernel still writing back the cached data. Such defaults may not suit everyone and users were asking for a way to change that. We’ve even been receiving contradicting merge requests to add or remove certain mount option. Some users on public forums even suggested modifying the udisksd binary how ridiculous may that sound.

There were couple of patches and suggestions in the past that served as a valuable feedback. At the end we decided to combine the best ideas and allow redefining mount options by the config file and from udev rules as each approach has specific use cases. Each way has a defined priority so e.g. udev rules would cover the config file definitions. As redefining and allowing certain mount options may have severe security implications, only a sysadmin owned global config file in /etc/udisks2 and udev rules are considered. Redefining mount options may serve either for lockdown purposes or for a more liberal environment.

Comprehensive documentation will be provided along the API docs as this is pretty complex topic. Just a glimpse of how the current config file syntax currently looks like:

[defaults]
allow=exec,noexec,nodev,nosuid,atime,noatime,nodiratime,ro,rw,sync,dirsync,noload

vfat_defaults=uid=$UID,gid=$GID,shortname=mixed,utf8=1,showexec,flush
vfat_allow=uid=$UID,gid=$GID,flush,utf8,shortname,umask,dmask,fmask,codepage,iocharset

# common options for both the native kernel driver and exfat-fuse
exfat_defaults=uid=$UID,gid=$GID,iocharset=utf8,errors=remount-ro
exfat_allow=uid=$UID,gid=$GID,dmask,errors,fmask,iocharset,namecase,umask

ntfs_defaults=uid=$UID,gid=$GID,windows_names
ntfs_allow=uid=$UID,gid=$GID,umask,dmask,fmask,locale,norecover,ignore_case,windows_names

...

D-Bus object properties updates

The reach of UDisks has evolved in recent years from being a simple automounter backend to a complete stateful storage management API. UDisks has gained internal modularity for non-standard storage technologies (more on that in upcoming blog posts!) and is more often used in scripts. Historically the design and intended use was mostly asynchronous where clients were supposed to hook up on D-Bus signals and just update the GUI afterwards. Properties on D-Bus objects were updated usually as a result of incoming uevent, often significantly later after a method call returned. That caused race conditions on client side that expected all D-Bus properties being up-to-date once a method call returns.

We embraced the work done by Peter Rajnoha on synthetic uevent tagging that’s available since Linux kernel 4.13.0. When we expect properties being updated as a result of native uevent we queue a synthetic one with a tag and we wait until it’s received. The uevent queue is more or less serialized and this makes sure the queue is fully processed before the tag. This allows us to hold on returning from a method call and let the objects be updated with fresh data from udev. When running UDisks on older Linux kernels the new measure has no effect and works just like before.

This however brings a change the clients may observe – the D-Bus property change notifications are fired earlier. I’m not sure there was any (API stability) guarantee with regards to D-Bus properties anywhere, this might however need attention in your application.

VDO (Virtual Data Optimizer)

While you might not be aware of what VDO is or does as the kvdo kernel module still hasn’t been upstreamed and adopted by major distributions, UDisks 2.8.0 brought a basic vdo module that runs on top of standalone vdo/vdo-manager commands. There were a couple of design shortcomings leading from VDO limitations itself and together with little to no adoption of this module and recent integration of dm-vdo in LVM a decision has been made to prefer the new lvmvdo integration coming in UDisks 2.9.0 and deprecate the standalone vdo module instead.

All this work is based on libblockdev support in case a stateless API fits you more.


There are even more changes queued for the 2.9.0 release, stay tuned for upcoming articles about UDisks, the API and practical use cases!

bscalc – storage size calculator

Introduction

In the spring of this year, a new major version of the libbytesize library was released. And as part of the release this small but very useful library was amended with a small but very useful tool. It is called bscalc where the first two letters BS stand for Byte Size, of course, and calc suggests it does some calculations.

In fact, it is a simple calculator that works with storage sizes. Computers are good at working with big and long numbers representing sizes in bytes, humans are not. The simplest and probably most common problem is a question like “What the hell does 9663676416 actually mean?” Or the opposite case when some input has to be in bytes (or KiB, MiB,…) and one wants to specify say 12 GiB. But there are more complex questions like “How many 512B sectors does this disk have?” or “If I backup this data on blue-rays, how many will I need and how well will the data fit?”.

Continue reading “bscalc – storage size calculator”

Release time again for libblockdev and udisks!

A new month has come and that means new releases of libblockdev and UDisks2 have come too. We are trying to stick to the golden rule of successful open-source projects – “Release early, release often.” – and even if there are no major changes and no new major features, we do regular releases every month. Usually the target date is the end of the month which then in reality means a new release is done at the beginning of the month that follows. And that is exactly what happened this time too. libblockdev-2.15 was released on December 1st and UDisks-2.7.5 on December 4th.

Continue reading “Release time again for libblockdev and udisks!”

UDisks 2.7.0 released

A new upstream version of UDisks2 was released on Friday (June 2nd) — version 2.7.0. People following the recent development of UDisks2 and our recent blog posts [1] [2] should know that this is a big version bump which can only mean one thing: the pull request changing UDisks to use libblockdev where possible was merged! Which is almost 100 commits with changes.

Continue reading “UDisks 2.7.0 released”

UDisks to build on libblockdev!?

As a recent blog post mentioned, there is a pull request for UDisks proposing the master-libblockdev branch to be merged into master. What would that mean? master-libblockdev is a parallel branch we have been working on in the last few months which has custom code in UDisks replaced by calls of the libblockdev library where possible. So for things like creating partitions, setting up MD RAID, LVM, etc. it’s not using the CLI tools, but instead calls libblockdev functions.

Continue reading “UDisks to build on libblockdev!?”

Reporting and monitoring storage actions

Two recent blog posts are focusing on reporting and monitoring of storage events related to failures, recoveries and in general device state changes. However, there are other things happening to storage. The storage configuration is from time to time changed either by administrator(s) or automatically as a reaction to some trigger. And there are components of the system, together with its users, that could/would benefit from getting the information about such changes.

Continue reading “Reporting and monitoring storage actions”

Storaged merged with UDisks, new releases 2.6.4 and 2.6.5

Quite a lot has changed since our last blogpost about the storaged project. The biggest news is that we are no longer working on Storaged. We are now “again” working on UDisks1.

Storaged originally started as a fork of udisks project in 2015 and had a lot of attention and development since then. Storaged even replaced UDisks in Fedora 25 providing backwards compatible API and new features.

Continue reading “Storaged merged with UDisks, new releases 2.6.4 and 2.6.5”

Storage event reporting and monitoring – PoC

In the previous blog post we have presented a proposal for reporting and monitoring storage-related events using journald and structured logging. To test if the proposal is viable we need some proof of concept. Such a PoC should demonstrate the complexity of the proposed solution as well as the sufficiency of the proposed set of stored (logged) items and the catalog entry.

Continue reading “Storage event reporting and monitoring – PoC”