util-linux 2.22

I released the latest util-linux version 2.22 one week ago. As usually the list of the changes in our ReleaseNotes is huge (it would be nice to release more often ;-)).

For me, the most important thing is that number of the project contributors is growing and we are able to coordinate our changes with another upstream projects like coreutils, procps-ng or systemd and maintainers from distributions contribute to the project.

The commands like mount(8), umount(8) or swapon(8) support new tags PARTLABEL=
and PARTUUID=.  It means that you can address partitions by name or UUID independently on the filesystem on the device.  You don't have to care about your fstab after mkfs or mkswap. The setting with PARTUUID= will be always valid.

Finally, we have dmesg --follow to print new kernel messages (like tail -f). This feature depends on readable /dev/kmsg (since kernel 3.5.0). I have also implemented a new dmesg output format --reltime (suggested by Linus on lkml):

$ dmesg --reltime
...
[Aug26 10:58] scsi_debug: host protection
[  +0.000004] scsi84 : scsi_debug, version 1.82 [20100324], dev_size_mb=50, opts=0x0
[  +0.000546] scsi 84:0:0:0: Direct-Access     Linux    scsi_debug       0004 PQ: 0 ANSI: 5
[  +0.000173] sd 84:0:0:0: Attached scsi generic sg1 type 0
[  +0.000356] sd 84:0:0:0: [sdb] 102400 512-byte logical blocks: (52.4 MB/50.0 MiB)
[  +0.000988] sd 84:0:0:0: [sdb] Write Protect is off
[  +0.000004] sd 84:0:0:0: [sdb] Mode Sense: 73 00 10 08

The low-level userspace tools consolidation continues:

• sulogin(1) and utmpdump(1) from sysvinit merged into util-linux (the goal is to remove all init independent utils from sysvinit package)
• eject(1) reimplemented to use /proc and /sys information and libmount and moved to util-linux
• new command lslocks(8) as replacement for dead lslk(8)

The command findmnt(8) has initial support for per-process mount tables (namespaces), for example findmnt --task will print mount table for the PID. In the next release I'd like to have a new option --unshared-tasks to print all processes with unshared mounts.

The tool lsblk(8) supports reverse trees, it means that you can see whole stack of the block devices from top to down:

$ lsblk -s /dev/mapper/luks-10d813de-fa82-4f67-a86c-23d5d0e7c30e
NAME    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
luks-10d813de-fa82-4f67-a86c-23d5d0e7c30e (dm-0)
        253:0    0  39.1G  0 crypt /home/kzak
└─sda6    8:6    0  39.1G  0 disk
  └─sda   8:0    0 149.1G  0 disk

The most invasive change is a new non-recursive build-system (just for the record: autotools are the best :-)). The result is faster build, binaries are in one top-level build directory rather than in many subdirectories, make distcheck calls our regression etc.

The another big change is fdisk refactoring. This is slow and painful work, but the result should be GPT support in release 2.23 (patches from Davidlohr Bueso are already in mailing list). I hope that one day the default fdisk will be nice, readable (colored?) low-level tool without some obsolete junk like CHS.

Note that in the next util-linux release 2.23 we're going to remove cryptoloop support. Yes, cryptoloop is bad and dead, use dm-crypt. (Note that util-linux upstream has never supported loop-AES.)

lslocks(8)

The command lslk(1) is unmaintained since year 2001 and it seems that there is no usable replacement for this command in distributions.

So, we (Davidlohr Bueso and I) decided to write a new implementation from scratch. The new implementation is based on the same concept like findmnt(8), lsblk(8) etc. It means that you can control output columns and output format.

# lslocks
COMMAND           PID  TYPE SIZE MODE  M      START                 END PATH
iscsiuio         1043 POSIX   5B WRITE 0          0                   0 /run/iscsiuio.pid
iscsid           1051 POSIX   5B WRITE 0          0                   0 /run/iscsid.pid
crond            1076 POSIX   5B WRITE 0          0                   0 /run/crond.pid
libvirtd         1264 POSIX   4B WRITE 0          0                   0 /run/libvirtd.pid
(unknown)        1304 FLOCK   0B WRITE 0          0                   0 /run
tracker-store    2088 POSIX 6.7M READ  0 1073741826          1073742335 /home/kzak/.cache/tracker/meta.db
tracker-store    2088 POSIX  32K READ  0        128                 128 /home/kzak/.cache/tracker/meta.db-shm
tracker-miner-f  2089 POSIX 6.7M READ  0 1073741826          1073742335 /home/kzak/.cache/tracker/meta.db
tracker-miner-f  2089 POSIX  32K READ  0        128                 128 /home/kzak/.cache/tracker/meta.db-shm
firefox         17151 POSIX   0B WRITE 0          0                   0 /home/kzak/.mozilla/firefox/zf4j57bz.de
firefox         17151 POSIX 416K READ  0 1073741826          1073742335 /home/kzak/.mozilla/firefox/zf4j57bz.de
firefox         17151 POSIX  20M READ  0 1073741826          1073742335 /home/kzak/.mozilla/firefox/zf4j57bz.de
firefox         17151 POSIX  32K READ  0        128                 128 /home/kzak/.mozilla/firefox/zf4j57bz.de
firefox         17151 POSIX 1.5M READ  0 1073741826          1073742335 /home/kzak/.mozilla/firefox/zf4j57bz.de
firefox         17151 POSIX  32K READ  0        128                 128 /home/kzak/.mozilla/firefox/zf4j57bz.de
firefox         17151 POSIX 416K WRITE 0 1073741824          1073742335 /home/kzak/.mozilla/firefox/zf4j57bz.de
java            25348 POSIX   0B WRITE 0          0                   0 /opt/xmind/Commons/configuration/org.ec
java            25348 POSIX   0B WRITE 0          0                   0 /opt/xmind/Commons/configuration/org.ec
java            25348 POSIX   0B WRITE 0          0                   0 /opt/xmind/Commons/configuration/org.ec
java            25348 POSIX   0B WRITE 0          0 9223372036854775806 /opt/xmind/Commons/data/workspace-cathy
atd              3348 POSIX   5B WRITE 0          0                   0 /run/atd.pid
sendmail         3400 POSIX  33B WRITE 0          0                   0 /run/sendmail.pid
sendmail         3419 POSIX  49B WRITE 0          0                   0 /run/sm-client.pid

Select process:

# lslocks --pid $(pidof crond) 
COMMAND   PID  TYPE SIZE MODE  M START END PATH
crond    1076 POSIX   5B WRITE 0     0   0 /run/crond.pid

 or use it in scripts:

 for x in $(lslocks -rn -o PID); do kill $x; done

lslocks(1) will be available in util-linux 2.22 (now -rc1).

eject(1), sulogin(1), wdctl(1)

The commands eject(1) and sulogin(1) have been merged into util-linux upstream.

eject(1)

The original eject(1) upstream is inactive for more than 4 years, but the command is actively used by many users and it's also used by system components like udisks. The command has been completely rewritten (by Michal Luscon and me). Changes:

• linked with libmount
• device may be addressed by UUID or LABEL
• partition name is always converted to whole-disk device name to avoid eject SCSI ioctls on partition devices
• uses /sys to get list of all partitions (all partitions are unmounted before eject)
• patches from distributions (mostly Fedora) have been merged
• code is about 1/3 shorter

Note that users who depend on the command volname(1) from the original eject package have to use

  blkid -p -o value -s LABEL device. 

sulogin(1)
 
util-linux and systemd upstreams ambition is to move initd independent commands to the generic util-linux package and minimize number of packages required for basic system.

We merged mountpoint(1) from sysvinit-tools into util-linux two releases ago, and sulogin(1) now. The command sulogin(1) is like login(1) but it's used in single user mode to supply the root password before a shell is started. The nice advantage is that we maintain login(1) and agetty(1) in the same source tree so we can share code and coordinate development.

wdctl(1)
 
wdctl(1) is a new small util that will be available in util-linux-2.22. The 'wd' means watchdog (unfortunately 'watchdog' is name already used for watchdog daemon, so we cannot use this name for command line util...).

wdctl(1) prints the current watchdog setting (flags), watchdog identity information and timeouts.

Default output:

  # wdctl                
  Identity:      iTCO_wdt [version 0]
  Timeout:       30 seconds
  Timeleft:       2 seconds
 
  FLAG           DESCRIPTION               STATUS BOOT-STATUS
  KEEPALIVEPING  Keep alive ping reply          0           0
  MAGICCLOSE     Supports magic close char      0           0
  SETTIMEOUT     Set timeout (in seconds)       0           0

or in script:

  KEEPALIVEPING=$(wdctl -ITnr -f KEEPALIVEPING -o STATUS)

Thanks to Lennart Poettering who provided the first prototype.

mutt fork

You know that mutt is the best e-mail client ever ;-) Unfortunately, mutt upstream is too conservative and too inactive for impatient people like me. I (we) would like to have mutt with modern features like fulltext search, tags, virtual folders, gmail-like archive, sidebar etc.

It's Open Source, let's fork it! I announced this idea at notmuch mailing list in January. The current status of this project is:

• mailing list: http://admin.fedoraproject.org/mailman/listinfo/mutt-kz (already 21 members, thanks!)
• git repository: https://github.com/karelzak/mutt-kz/
• wiki & screenshots: https://github.com/karelzak/mutt-kz/wiki


• already implemented features:
  
  
  • notmuch support (fast fulltext search and tags)
  • sidebar support (for regular and virtual folders
  • virtual folders based on notmuch queries (from .muttrc)
  • on-the-fly generate virtual folders from notmuch queries
  • add / remove / modify notmuch tags
  • support for inbox-->archive use case
  • all is implemented by regular mutt commands, so all is usable in user defined macros


• TODO:
  
  
  • integrate mutt patches from distributions (now ~20 patches from Debian in queue)
  • improve stability
  • packages (already for OpenSuse)

For more information (including .muttrc example) see: README.notmuch

Thanks to all contributors.

libblkid maintainer's brain dump

This article is about the low-level probing libblkid code, and it's really dump, nothing more ;-)

High and Low level

The library contains two APIs.

• high-level - this is the original library code from e2fsprogs. All results are cached in the file /etc/blkid.tab (or /run/blkid/blkid.tab). The advantage is that information about LABELs and UUIDs are accessible for non-root users and the cache has positive impact on performance.
  
  This advantage is no more valid on many systems where all necessary information are stored in udev db, and things like LABEL and UUID are accessible by /dev/disk/by-* udev symlinks.
  
  This is reason why for newly written programs are recommended blkid_evaluate_* functions which are able to use udev symlinks as well as the original libblkid cache. This functionality is also accessible from command line by the blkid -L|-U command.
  
  
• low-level - this part of the API completely bypass the cache and allows to work directly with library probing functions. The rest of this article is about the low-level part of the library.
  

The library contains three chains of the probing functions:

1. superblocks
2. partitions
3. topology

The superblocks probing is enabled by default. The command "blkid -p -o udev" (or built-in code in udevd) enables partitions probing chain too.

There are two basic probing methods:

• safeprobe - this is recommended method. This method cares about collisions between filesystems, raids or partition tables.
  
• fullprobe - don't check for conflicts, used for example in wipefs(8)

For the superblock is available NAME=value based API only. For topology and partitions is available binary interface too. See the docs link below.

Superblocks

• three basic "usage" groups: filesystems, raids, crypto and others
• RAIDs (MD, LVM, ...) are probed before filesystems
• don't check for filesystems when a RAID signature is detected
  
• don't check for RAIDs or others (swap-area) on CD-ROMs
  
• don't check for RAIDs on tiny devices (< 1 MiB)
  
• don't read whole FAT root directory (to lookup LABEL) on tiny devices (< 1 MiB)

exceptions / extra cases:

• MD RAID is ignored if detected within a valid partition during whole-disk probing
  
  [use case: partitioned disk, last partition used as a RAID member and the RAID has metadata at the end of the last partition (so end of the disk)]
  
  
• LVM signature is ignored if another signature is detected within first 8KiB of the device (LVM wipes this area, so there should not be any filesystem superblock)
  
  [use case: disk with LVM, user stops to use LVM and creates a new partition table by fdisk, result is MBR and obsolete LVM signature on the same device]

Partitions

• disabled by default, enabled for udev (see ID_PART_ENTRY_* in udev db)
  
• parse partition tables (aix, minix, bsd, mbr, gpt, mac, sgi, solaris, sun, ultrix and unixware)
• detect nested partition tables (e.g. BSD) within partitions
  
• if given device is a partition (e.g. sda1) then open whole disk (e.g. sda) to read details about the partition from partition table. This feature has to be enabled by BLKID_PARTS_ENTRY_DETAILS flag.
  
• partition table is ignored if a valid RAID superblock is detected at the end of the device
  
  [use case: partitioned RAID1 (mirror) -- the partition table is visible from underlaying devices]

Topology

• rarely used
• designed for mkfs-like or fdisk-like programs to get info about I/O topology
• for kernel >= 2.6.3x uses ioctl or sysfs
• as fallback for old kernels uses code originally from xfsprogs

Tips for users

• please use wipefs(8) before fdisk, mkfs or mkswap. The latest version is able to remove really all possible backup signatures, partition tables and at first glance invisible things. Don't rely on mkfs developers :-)
  
• think twice before you start to use some complex setups (for example partitioned RAIDs) to avoid misinterpretation by kernel or system tools.
  
• don't forget that blkid without -p might returns cached results
  

Tips for developers

• use blkid_evaluate_* functions to convert LABEL or UUID to device name (it's portable and able to use udev /dev/disk/by-* symlinks)
  
• libblkid uses symbols versioning and soname will NOT change, API is stable.
  
• API docs: http://ftp.kernel.org/pub/linux/utils/util-linux/v2.21/libblkid-docs/index.html
  

.... I'll try to keep these notes updated.

login(1) changes

I have completely refactored login(1). The new login(1) merges features from Suse login(1) back into util-linux version and is more compatible with login(1) from shadow-utils. I believe that now we have login(1) implementation which is usable in all mainstream Linux distributions.

The original util-linux login(1) code is derived from 4.3 BSD (so older than Linux kernel).

Changes:

• PAM only
  
  It's obvious that in all mainstream distributions is PAM de facto standard for users authentication and maintain any non-PAM methods in login(1) is waste of time.
  
  If you don't like this change (really?) than you can use login(1) from shadow-utils.
  
  
• support /etc/login.defs(5) config file
  
  Supported options: MOTD_FILE, LOGIN_TIMEOUT, LOGIN_RETRIES, FAIL_DELAY, TTYPERM, TTYGROUP HUSHLOGIN_FILE, DEFAULT_HOME, LOG_UNKFAIL_ENAB, ENV_PATH, ENV_ROOTPATH, ENV_SUPATH
  
  
• print hostname in the login prompt, the default prompt is compatible with initial prompt from agetty
  
  
• add -H for compatibility with Suse version. This option suppresses the hostname printing in the login prompt.
  
  
• global hush mode for all accounts (enabled if /etc/hushlogins exists, but is empty). The global hush mode allows to use PAM for "Last login" message.

More details: http://thread.gmane.org/gmane.linux.utilities.util-linux-ng/4866

prlimit(1)

prlimit(1) is a new util that will be available in util-linux-2.21 (now -rc1). This new util is very nice and flexible command line interface to prlimit(2) Linux syscall (supported since Linux 2.6.36).

prlimit(1) allows to get or set one or more process resource limits for given PID. When a command is given instead of PID, prlimit(1) will run this command with the given resources.

The output is flexible like output from lsblk(8) or findmnt(8). You can define the output columns, use parsable output etc.

See the default output:

  $ prlimit --pid $$
  RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
  AS         address space limit                unlimited unlimited bytes
  CORE       max core file size                         0 unlimited blocks
  CPU        CPU time                           unlimited unlimited seconds
  DATA       max data size                      unlimited unlimited bytes
  FSIZE      max file size                      unlimited unlimited blocks
  LOCKS      max number of file locks held      unlimited unlimited
  MEMLOCK    max locked-in-memory address space     65536     65536 bytes
  MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
  NICE       max nice prio allowed to raise             0         0
  NOFILE     max number of open files                1024      4096
  NPROC      max number of processes                 1024     62809
  RSS        max resident set size              unlimited unlimited pages
  RTPRIO     max real-time priority                     0         0
  RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
  SIGPENDING max number of pending signals          62809     62809
  STACK      max stack size                       8388608 unlimited bytes

or redefine the output and ask for max number of open files only:

  $ prlimit  --nofile --output RESOURCE,SOFT,HARD --pid $$
  RESOURCE SOFT HARD
  NOFILE   1024 4096

and now let's modify the soft limit of maximal core file size and maximal number of open files:

  $ prlimit --core=1000000: --nofile=100: --pid $$

the notation used for the limits is:

  soft:hard    specify both limits
  soft:        specify only the soft limit
  :hard        specify only the hard limit
  value        specify both soft and hard limits to the same value

and check the result:

  $ prlimit  --nofile --core --pid $$
  RESOURCE DESCRIPTION                 SOFT      HARD UNITS
  NOFILE   max number of open files     100      1024
  CORE     max core file size       1000000 unlimited blocks

and revert the core file soft limit:

  $ prlimit --core=unlimited: --pid $$

  $ prlimit --core --pid $$
  RESOURCE DESCRIPTION             SOFT      HARD UNITS
  CORE     max core file size unlimited unlimited blocks

Do you want to restrict CPU time for given command (sort(1) in this example):

   $ prlimit --cpu=10 sort -u hugefile

I think prlimit(1) is much better than the shell built-in command ulimit.

Thanks to Davidlohr Bueso who found time to implement prlimit(1) for util-linux 2.21.

-- In memory of Dennis M. Ritchie

frustrating gnome-control-center network

The "gnome-control-center network" is pretty incomplete application. Unfortunately, this application is called by gnome-shell if you want to modify your network settings. And all this is default in Fedora-16. Grrrr...

The most stupid thing is that you cannot configure wireless connections if your wireless is disabled. And if you enable wireless then you will be affected by your broken configuration...

Fortunately, old good nm-connection-editor works as expected. So, all you need is to kick off the idiotic network stuff from gnome-shell and start old good Network Manager Applet:

 mv /usr/share/gnome-shell/js/ui/status/network.js \
       /usr/share/gnome-shell/js/ui/status/network.disabled

and restart gnome-shell (command 'r').

gnome-shell is not so bad

It seems that I will be able to use GNOME 3. I love the minimalism provided by gnome-shell. It's gnome, so it's tricky to fully customize the desktop, but it's possible.

My requirements:

• no animations (gnome "switch workspace" is pure hell...)
• tagging -- at least for terminal (fluxbox supports this for arbitrary application, gnome has at least gnome-terminal with tabs)
• minimalistic windows decoration
• no window title for maximized windows
• tiny window title for normal and modal windows
• no max/min/close buttons for windows (I have keyboard...)
• pretty visible focused window (e.g. green border)
  
• only one tiny panel
• WM has to remember workspace for applications (e.g. firefox = 2nd workspace)

We all love screenshots, right? ;-) My old good fluxbox here, and the "same" with GNOME 3 is here.

The first step is to install some gnome-shell extensions:

• Native Window Placement Extension
• Auto move windows extension
• Disable Window Animation Extension
• Remove User Name Extension
• windowNavigator Extension

The next steps is to make windows decorations more minimalistic. This step is more tricky, because you have to modify window manager theme (for more details see gnome bug 594879). My theme is available at my home page.

The next step is to customize desktop files for some applications, for example I want to start gnome-terminal with --hide-menubar option, so

cp /usr/share/applications/gnome-terminal.desktop \
   ~/.local/share/applications/myterm.desktop

and modify Exec and Name entries in the file. Then you can add the application to your gnome-shell Favorites.

The last step is to customize keyboard shortcuts, this is simple (see "keyboard" in gnome-control center).

Note than many things like info about CPU temperature does not have to waste any place on your desktop. IMHO it's better to use keyboard shortcuts and print necessary information to the screen on demand. For example I use osd_cat:

#!/bin/bash

BATT=$(acpitool -b | awk -F ':' '/Battery/ { print $2 }' | sed 's/ //')
AC=$(acpitool -a | awk -F ':' '/AC/ { print $2 }' | sed 's/ //g')
TEMPE=$(sensors | awk '/temp/ { print $2 }' | sed 's/ //g')
FAN=$(sensors | awk '/fan/ { print $2 }'  | sed 's/ //g')

(printf "    Battery: %-20s\n" "$BATT"
printf "         AC: %-20s\n" "$AC"
printf "Temperature: %-20s \n" "$TEMPE"
printf "        Fan: %-20s\n" "$FAN") | osd_cat --delay 4 --pos bottom      \
                                                          --align right --offset 45 --indent 10 \
                                                          --color green --font "-misc-fixed-*-*-*-*-20-*-*-*-*-*-*-*"