Karel Zak's blog

libblkid maintainer's brain dump

2012-02-15T00:29:00.005+01:00

This article is about the low-level probing libblkid code, and it's really dump, nothing more ;-)

High and Low level

The library contains two APIs.

high-level - this is the original library code from e2fsprogs. All results are cached in the file /etc/blkid.tab (or /run/blkid/blkid.tab). The advantage is that information about LABELs and UUIDs are accessible for non-root users and the cache has positive impact on performance.

This advantage is no more valid on many systems where all necessary information are stored in udev db, and things like LABEL and UUID are accessible by /dev/disk/by-* udev symlinks.

This is reason why for newly written programs are recommended blkid_evaluate_* functions which are able to use udev symlinks as well as the original libblkid cache. This functionality is also accessible from command line by the blkid -L|-U command.
low-level - this part of the API completely bypass the cache and allows to work directly with library probing functions. The rest of this article is about the low-level part of the library.

The library contains three chains of the probing functions:

superblocks
partitions
topology

The superblocks probing is enabled by default. The command "blkid -p -o udev" (or built-in code in udevd) enables partitions probing chain too.

There are two basic probing methods:

safeprobe - this is recommended method. This method cares about collisions between filesystems, raids or partition tables.
fullprobe - don't check for conflicts, used for example in wipefs(8)

For the superblock is available NAME=value based API only. For topology and partitions is available binary interface too. See the docs link below.

Superblocks

three basic "usage" groups: filesystems, raids, crypto and others
RAIDs (MD, LVM, ...) are probed before filesystems
don't check for filesystems when a RAID signature is detected
don't check for RAIDs or others (swap-area) on CD-ROMs
don't check for RAIDs on tiny devices (< 1 MiB)
don't read whole FAT root directory (to lookup LABEL) on tiny devices (< 1 MiB)

exceptions / extra cases:

MD RAID is ignored if detected within a valid partition during whole-disk probing

[use case: partitioned disk, last partition used as a RAID member and the RAID has metadata at the end of the last partition (so end of the disk)]
LVM signature is ignored if another signature is detected within first 8KiB of the device (LVM wipes this area, so there should not be any filesystem superblock)

[use case: disk with LVM, user stops to use LVM and creates a new partition table by fdisk, result is MBR and obsolete LVM signature on the same device]

Partitions

disabled by default, enabled for udev (see ID_PART_ENTRY_* in udev db)
parse partition tables (aix, minix, bsd, mbr, gpt, mac, sgi, solaris, sun, ultrix and unixware)
detect nested partition tables (e.g. BSD) within partitions
if given device is a partition (e.g. sda1) then open whole disk (e.g. sda) to read details about the partition from partition table. This feature has to be enabled by BLKID_PARTS_ENTRY_DETAILS flag.
partition table is ignored if a valid RAID superblock is detected at the end of the device

[use case: partitioned RAID1 (mirror) -- the partition table is visible from underlaying devices]

Topology

rarely used
designed for mkfs-like or fdisk-like programs to get info about I/O topology
for kernel >= 2.6.3x uses ioctl or sysfs
as fallback for old kernels uses code originally from xfsprogs

Tips for users

please use wipefs(8) before fdisk, mkfs or mkswap. The latest version is able to remove really all possible backup signatures, partition tables and at first glance invisible things. Don't rely on mkfs developers :-)
think twice before you start to use some complex setups (for example partitioned RAIDs) to avoid misinterpretation by kernel or system tools.
don't forget that blkid without -p might returns cached results

Tips for developers

use blkid_evaluate_* functions to convert LABEL or UUID to device name (it's portable and able to use udev /dev/disk/by-* symlinks)
libblkid uses symbols versioning and soname will NOT change, API is stable.
API docs: http://ftp.kernel.org/pub/linux/utils/util-linux/v2.21/libblkid-docs/index.html

.... I'll try to keep these notes updated.

login(1) changes

2012-02-10T16:00:00.000+01:00

I have completely refactored login(1). The new login(1) merges features from Suse login(1) back into util-linux version and is more compatible with login(1) from shadow-utils. I believe that now we have login(1) implementation which is usable in all mainstream Linux distributions.

The original util-linux login(1) code is derived from 4.3 BSD (so older than Linux kernel).

Changes:

PAM only

It's obvious that in all mainstream distributions is PAM de facto standard for users authentication and maintain any non-PAM methods in login(1) is waste of time.

If you don't like this change (really?) than you can use login(1) from shadow-utils.
support /etc/login.defs(5) config file

Supported options: MOTD_FILE, LOGIN_TIMEOUT, LOGIN_RETRIES, FAIL_DELAY, TTYPERM, TTYGROUP HUSHLOGIN_FILE, DEFAULT_HOME, LOG_UNKFAIL_ENAB, ENV_PATH, ENV_ROOTPATH, ENV_SUPATH
print hostname in the login prompt, the default prompt is compatible with initial prompt from agetty
add -H for compatibility with Suse version. This option suppresses the hostname printing in the login prompt.
global hush mode for all accounts (enabled if /etc/hushlogins exists, but is empty). The global hush mode allows to use PAM for "Last login" message.

More details: http://thread.gmane.org/gmane.linux.utilities.util-linux-ng/4866

prlimit(1)

2012-01-25T22:20:00.002+01:00

prlimit(1) is a new util that will be available in util-linux-2.21 (now -rc1). This new util is very nice and flexible command line interface to prlimit(2) Linux syscall (supported since Linux 2.6.36).

prlimit(1) allows to get or set one or more process resource limits for given PID. When a command is given instead of PID, prlimit(1) will run this command with the given resources.

The output is flexible like output from lsblk(8) or findmnt(8). You can define the output columns, use parsable output etc.

See the default output:

  $ prlimit --pid $$
  RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
  AS         address space limit                unlimited unlimited bytes
  CORE       max core file size                         0 unlimited blocks
  CPU        CPU time                           unlimited unlimited seconds
  DATA       max data size                      unlimited unlimited bytes
  FSIZE      max file size                      unlimited unlimited blocks
  LOCKS      max number of file locks held      unlimited unlimited
  MEMLOCK    max locked-in-memory address space     65536     65536 bytes
  MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
  NICE       max nice prio allowed to raise             0         0
  NOFILE     max number of open files                1024      4096
  NPROC      max number of processes                 1024     62809
  RSS        max resident set size              unlimited unlimited pages
  RTPRIO     max real-time priority                     0         0
  RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
  SIGPENDING max number of pending signals          62809     62809
  STACK      max stack size                       8388608 unlimited bytes

or redefine the output and ask for max number of open files only:

  $ prlimit  --nofile --output RESOURCE,SOFT,HARD --pid $$
  RESOURCE SOFT HARD
  NOFILE   1024 4096

and now let's modify the soft limit of maximal core file size and maximal number of open files:

  $ prlimit --core=1000000: --nofile=100: --pid $$

the notation used for the limits is:

  soft:hard    specify both limits
  soft:        specify only the soft limit
  :hard        specify only the hard limit
  value        specify both soft and hard limits to the same value

and check the result:

  $ prlimit  --nofile --core --pid $$
  RESOURCE DESCRIPTION                 SOFT      HARD UNITS
  NOFILE   max number of open files     100      1024
  CORE     max core file size       1000000 unlimited blocks

and revert the core file soft limit:

  $ prlimit --core=unlimited: --pid $$

  $ prlimit --core --pid $$
  RESOURCE DESCRIPTION             SOFT      HARD UNITS
  CORE     max core file size unlimited unlimited blocks

Do you want to restrict CPU time for given command (sort(1) in this example):

   $ prlimit --cpu=10 sort -u hugefile

I think prlimit(1) is much better than the shell built-in command ulimit.

Thanks to Davidlohr Bueso who found time to implement prlimit(1) for util-linux 2.21.

-- In memory of Dennis M. Ritchie

frustrating gnome-control-center network

2012-01-09T14:19:00.003+01:00

The "gnome-control-center network" is pretty incomplete application. Unfortunately, this application is called by gnome-shell if you want to modify your network settings. And all this is default in Fedora-16. Grrrr...

The most stupid thing is that you cannot configure wireless connections if your wireless is disabled. And if you enable wireless then you will be affected by your broken configuration...

Fortunately, old good nm-connection-editor works as expected. So, all you need is to kick off the idiotic network stuff from gnome-shell and start old good Network Manager Applet:

 mv /usr/share/gnome-shell/js/ui/status/network.js \
       /usr/share/gnome-shell/js/ui/status/network.disabled

and restart gnome-shell (command 'r').

gnome-shell is not so bad

2012-01-05T15:52:00.001+01:00

It seems that I will be able to use GNOME 3. I love the minimalism provided by gnome-shell. It's gnome, so it's tricky to fully customize the desktop, but it's possible.

My requirements:

no animations (gnome "switch workspace" is pure hell...)
tagging -- at least for terminal (fluxbox supports this for arbitrary application, gnome has at least gnome-terminal with tabs)
minimalistic windows decoration
no window title for maximized windows
tiny window title for normal and modal windows
no max/min/close buttons for windows (I have keyboard...)
pretty visible focused window (e.g. green border)
only one tiny panel
WM has to remember workspace for applications (e.g. firefox = 2nd workspace)

We all love screenshots, right? ;-) My old good fluxbox here, and the "same" with GNOME 3 is here.

The first step is to install some gnome-shell extensions:

Native Window Placement Extension
Auto move windows extension
Disable Window Animation Extension
Remove User Name Extension
windowNavigator Extension

The next steps is to make windows decorations more minimalistic. This step is more tricky, because you have to modify window manager theme (for more details see gnome bug 594879). My theme is available at my home page.

The next step is to customize desktop files for some applications, for example I want to start gnome-terminal with --hide-menubar option, so

cp /usr/share/applications/gnome-terminal.desktop \
   ~/.local/share/applications/myterm.desktop

and modify Exec and Name entries in the file. Then you can add the application to your gnome-shell Favorites.

The last step is to customize keyboard shortcuts, this is simple (see "keyboard" in gnome-control center).

Note than many things like info about CPU temperature does not have to waste any place on your desktop. IMHO it's better to use keyboard shortcuts and print necessary information to the screen on demand. For example I use osd_cat:

#!/bin/bash

BATT=$(acpitool -b | awk -F ':' '/Battery/ { print $2 }' | sed 's/ //')
AC=$(acpitool -a | awk -F ':' '/AC/ { print $2 }' | sed 's/ //g')
TEMPE=$(sensors | awk '/temp/ { print $2 }' | sed 's/ //g')
FAN=$(sensors | awk '/fan/ { print $2 }'  | sed 's/ //g')

(printf "    Battery: %-20s\n" "$BATT"
printf "         AC: %-20s\n" "$AC"
printf "Temperature: %-20s \n" "$TEMPE"
printf "        Fan: %-20s\n" "$FAN") | osd_cat --delay 4 --pos bottom      \
                                                          --align right --offset 45 --indent 10 \
                                                          --color green --font "-misc-fixed-*-*-*-*-20-*-*-*-*-*-*-*"

Monitor a list of currently mounted filesystems

2011-12-05T12:39:00.008+01:00

You know that /proc/mounts and /proc/self/mountinfo contain list of currently mounted filesystems. These files is possible to monitor by poll(2) or select(2) functions. The util findmnt(8) exports this functionality to command line.

Examples:

session A (monitor):                                          
   findmnt --poll
session B (event):
   mount /home/fs-images/ext2.img /mnt/test
                                                 
session A (findmnt output after event):       
   ACTION     TARGET    SOURCE     FSTYPE OPTIONS
   mount      /mnt/test /dev/loop0 ext2   rw,relatime,user_xattr,acl,barrier=1

The another examples; wait until /mnt/test is unmounted:

   findmnt --poll=umount --first-only /mnt/test

Inform me about all ext2, ext3, and ext4 remounts to read-only mode:

   findmnt --poll=remount --types ext2,ext3,ext4 --options ro

You can also define output columns, for example if you want to know more about "mount --move" operations:

   # findmnt --poll=move -o OLD-TARGET,TARGET,SOURCE
   OLD-TARGET TARGET   SOURCE                      
   /mnt/test  /mnt/foo /dev/loop0

the event in this example was generated by "mount --move /mnt/test /mnt/foo" command.

And for example if you want info about old and new options after remount:

   # findmnt --poll=remount -o TARGET,OLD-OPTIONS,OPTIONS
   TARGET   OLD-OPTIONS                          OPTIONS
   /mnt/foo ro,relatime,user_xattr,acl,barrier=1 rw,user_xattr,acl,barrier=1

this event was generated by "mount -o remount /mnt/foo -o rw,strictatime" command.

All this is available in util-linux 2.20 (e.g. Fedora 16).

wipefs(8) improvements

2011-11-25T11:48:00.004+01:00

I finally found time to improve the command wipefs(8). The most visible change is support for partition tables. You can use wipefs(8) to remove MBR as well as GPT and many others partition tables.

The another important change (well.. bugfix) is that "wipefs -a" really erases everything what is possible to detect by libblkid (blkid(8)).

Now it calls libblkid detection code also after magic string erasing to ensure that nothing is possible to found on the device. This is important for stuff like GPT where is backup table on another place or for filesystems like FAT where is more ways to detect the superblock.

The last important change is a new command line option "-t". Now you can specify filesystem, raid or partition table name. For example

       wipefs -a -t ext4

will erase 'ext4' only. The option is interpreted in the same way how -t for mount(8) or findmnt(8), so you can specify more filesystems and you can prefix all or selected filesystems by 'no' prefix, for example:

       wipefs -a -t noext4,ext3,ext2

all but ext4, ext3 and ext2 filesystems will be erased.

If you're filesystem tools (e.g. mkfs.type) developer then you should know that now libblkid contains a new function blkid_do_wipe():

      
    blkid_probe pr = blkid_new_probe_from_filename("/dev/sda1");

    blkid_probe_enable_superblock(pr, true);
    blkid_probe_set_superblocks_flags(pr, BLKID_SUBLKS_MAGIC);

    while (blkid_do_probe(pr) == 0)
            blkid_do_wipe(pr, 0);

    blkid_free_probe(pr);

and all superblocks are undetectable...

By the way, it's also good idea to call wipefs -a from system installer to avoid some unexpected problems. I have seen many bug reports from people with mess on their disks (unexpected mix of partition table and raid superblocks, swap and LUKS or ReiserFS ...etc.).

The changes will be available in the next util-linux release 2.21 (beta planned next month).

util-linux stats

2011-11-07T23:05:00.005+01:00

Today Sami Kerola (util-linux contributor) sent me nice R graphs from git commit logs.

... yes, the project is growing

...but weekend is weekend

... oh, I thought that we all love summer!

dmesg(1) changes for util-linux 2.20

2011-07-20T17:08:00.004+02:00

I have re-written the dmesg(1). That's the first large change in the code in last 18 years.

New features:

--decode facility and level number to human readable prefixes

$ dmesg --decode
kern  :info  : [26443.677632] ata1.00: configured for UDMA/100
kern  :info  : [26443.830225] PM: resume of devices complete after 2452.856 msecs
kern  :debug : [26443.830606] PM: Finishing wakeup.
kern  :warn  : [26443.830608] Restarting tasks ... done.

filter out messages according to the --facility and --level options, for example

$ dmesg --level=err,warn

$ dmesg --facility=daemon,user

$ dmesg --facility=daemon --level=debug

-u, --userspace to print only userspace messages

-k, --kernel to print only kernel messages

-t, --notime to skip [...] timestamps

-T, --ctime to print human readable timestamp in ctime()-like format. Unfortunately, this is useless on laptops if you have used suspend/resume. (The kernel does not use the standard system time as a source for printk() and it's not updated after resume.)

--show-delta to print time delta between printed messages

$ dmesg --show-delta
[35523.876281 <    4.016887>] usb 1-4.1: new low speed USB device using  hci_hcd and address 12
[35523.968398 <    0.092117>] usb 1-4.1: New USB device found, idVendor=413c, idProduct=2003
[35523.968408 <    0.000010>] usb 1-4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[35523.968416 <    0.000008>] usb 1-4.1: Product: Dell USB Keyboard

bind mounts, mtab and read-only

2011-04-20T16:51:00.002+02:00

The bind mount feature is supported since Linux 2.4. It's pretty long time, but many users still think that bind mounts are something completely different to the normal mounts.

Example 1:

 # mount /dev/sdb1 /mnt/A
 # mount /dev/sdb1 /mnt/B

This is not a bug. It's possible to mount the same filesystem on two places.

Example 2:

 # mount /dev/sdb1 /mnt/A
 # mount --bind /mnt/A /mnt/B

The result from both examples is the same, see /proc/self/mountinfo:

 # grep mnt /proc/self/mountinfo
 48 20 8:17 / /mnt/A rw,relatime - ext4 /dev/sdb1 rw,barrier=1,stripe=64,data=ordered
 49 20 8:17 / /mnt/B rw,relatime - ext4 /dev/sdb1 rw,barrier=1,stripe=64,data=ordered

This is very important, from kernel point of view is it the same thing. The same filesystem is mounted on two places.

The kernel does not maintain anywhere information that /mnt/B was created by bind mount (MS_BIND mount(2) syscall flags). There is not dependence between /mnt/A and /mnt/B (for example you can umount /mnt/A).

Unfortunately, the situation in the /etc/mtab file is completely different:

 # grep mnt /etc/mtab
 /dev/sdb1 /mnt/A ext4 rw 0 0
 /mnt/A    /mnt/B none rw,bind 0 0

This is confusing for many users. Try:

 # umount /mnt/A
 # rm -rf /mnt/A

 # grep mnt /etc/mtab
 /mnt/A /mnt/B none rw,bind 0 0

Does the information in mtab make any sense? I don't think so... Keep this kind of information in userspace is mistake. Yeah, mtab is evil.

Everyone who uses bind mounts on system without mtab (where mtab is symlink to /proc/mounts) has to undestand that "bind" flag is no more stored anywhere. For example you have to explicitly add the flag to the mount options if you want to use read-only bind mount.

 # rm -f /etc/mtab
 # ln -s /proc/mounts /etc/mtab

(or install Fedora 15:-)

Let's use findmnt(8) rather than grep /proc/self/mountinfo:

 # findmnt -o TARGET,VFS-OPTIONS,FS-OPTIONS /dev/sda1
 TARGET VFS-OPTIONS FS-OPTIONS
 /mnt/A rw,relatime rw,errors=continue,user_xattr,acl,barrier=0,data=ordered
 /mnt/B rw,relatime rw,errors=continue,user_xattr,acl,barrier=0,data=ordered

What will happen if we try to remount with bind flag? See:

  # mount -o remount,ro,bind /mnt/B

 # findmnt -o TARGET,VFS-OPTIONS,FS-OPTIONS /dev/sda1
 TARGET VFS-OPTIONS FS-OPTIONS
 /mnt/A rw,relatime rw,errors=continue,user_xattr,acl,barrier=0,data=ordered
 /mnt/B ro,relatime rw,errors=continue,user_xattr,acl,barrier=0,data=ordered

The filesystem (superblock) is still read-write, but the /mnt/B mountpoint is in VFS marked as read-only.

And now the same thing without the bind flag:

 # mount -o remount,ro /mnt/B

 # findmnt -o TARGET,VFS-OPTIONS,FS-OPTIONS /dev/sda1
 TARGET VFS-OPTIONS FS-OPTIONS
 /mnt/A rw,relatime ro,errors=continue,user_xattr,acl,barrier=0,data=ordered
 /mnt/B ro,relatime ro,errors=continue,user_xattr,acl,barrier=0,data=ordered

the superblock has been remounted read-only, so the filesystem is read-only everywhere in the system.

Again, all this is possible independently on the way how /mnt/B has been mounted to the system (examples 1 and 2).

BTW, you can also set the block device as read-only by blockdev --setro. So we have three layers (device -> FS -> VFS) where is possible to set read-only attribute :-)

findmnt(8) and submounts

2011-01-04T14:08:00.002+01:00

I just applied (to the util-linux upstream) a patch that allows to list all submounts for defined filesystem(s). For example:

$ findmnt --submounts /sys
TARGET                     SOURCE    FSTYPE  OPTIONS
/sys                       /sys      sysfs   rw,relatime
├─/sys/fs/cgroup           tmpfs     tmpfs   rw,nosuid,nodev,noexec,relat
│ ├─/sys/fs/cgroup/systemd cgroup    cgroup  rw,nosuid,nodev,noexec,relat
│ ├─/sys/fs/cgroup/cpuset  cgroup    cgroup  rw,nosuid,nodev,noexec,relat
│ ├─/sys/fs/cgroup/ns      cgroup    cgroup  rw,nosuid,nodev,noexec,relat
│ ├─/sys/fs/cgroup/cpu     cgroup    cgroup  rw,nosuid,nodev,noexec,relat
│ ├─/sys/fs/cgroup/cpuacct cgroup    cgroup  rw,nosuid,nodev,noexec,relat
│ ├─/sys/fs/cgroup/memory  cgroup    cgroup  rw,nosuid,nodev,noexec,relat
│ ├─/sys/fs/cgroup/devices cgroup    cgroup  rw,nosuid,nodev,noexec,relat
│ ├─/sys/fs/cgroup/freezer cgroup    cgroup  rw,nosuid,nodev,noexec,relat
│ ├─/sys/fs/cgroup/net_cls cgroup    cgroup  rw,nosuid,nodev,noexec,relat
│ └─/sys/fs/cgroup/blkio   cgroup    cgroup  rw,nosuid,nodev,noexec,relat
├─/sys/kernel/security     systemd-1 autofs  rw,relatime,fd=22,pgrp=1,tim
├─/sys/kernel/debug        systemd-1 autofs  rw,relatime,fd=24,pgrp=1,tim
└─/sys/fs/fuse/connections fusectl   fusectl rw,relatime

returns info about /sys and all /sys submounts.

Now you can implement recursive umount in shell, something like:

for d in $(findmnt --list --submounts $MOUNTPOINT -o TARGET -n | tac); do
  umount $d
done

I hope that umount(8) will support something like this ASAP.

lsblk(8)

2010-12-16T12:53:00.004+01:00

lsblk(8) is a new util that will be available in util-linux-2.19 (coming soon;-). The util lists all block devices as a tree. This output is very useful on machines with complicated storage setup (e.g. RAID, dm-crypt, ...). It's so useful that we'll probably backport lsblk(8) to RHEL-6 to make life easier for people who need to debug their systems.

The original idea comes from "dmsetup ls --tree", but lsblk(8) is better :-) It uses "holders" and "slaves" from /sys filesystem. This means that the util is usable without root permissions and it works for all types of block devices (dmsetup uses DM ioctls).

For example my laptop with dm-crypted $HOME and partitioned RAID0 (md8) on scsi_debug device (sdc):

$ lsblk
NAME                 MAJ:MIN RA   SIZE RO MOUNTPOINT
sda                    8:0    0  93.2G  0
├─sda1                 8:1    0   102M  0 /mnt/test
├─sda2                 8:2    0     1K  0
├─sda3                 8:3    0   2.3G  0 [SWAP]
├─sda4                 8:4    0  76.2G  0 /
├─sda5                 8:5    0    10G  0
│ └─kzak-home (dm-0) 253:0    0    10G  0 /home/kzak
└─sda6                 8:6    0   4.7G  0 /boot
sdc                    8:32   0   500M  0
├─sdc1                 8:33   0   250M  0
│ └─md8                9:8    0 498.9M  0
│   ├─md8p1          259:0    0   100M  0
│   ├─md8p2          259:1    0   100M  0
│   └─md8p3          259:2    0 297.9M  0
└─sdc2                 8:34   0   249M  0
  └─md8                9:8    0 498.9M  0
    ├─md8p1          259:0    0   100M  0
    ├─md8p2          259:1    0   100M  0
    └─md8p3          259:2    0 297.9M  0

You can also list more details about devices. The next example is from scsi_debug device with 4KiB sectors and enabled alignment offset:

$ lsblk --topology /dev/sdb
NAME   ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED
sdb         3584   4096  32768    4096     512    0 cfq
├─sdb1         0   4096  32768    4096     512    0 cfq
└─sdb2         0   4096  32768    4096     512    0 cfq

lsblk(8) is also usable in script, for example:

$ lsblk --nodeps --noheading --raw -o ALIGNMENT /dev/sdb
3584

returns only alignment-offset for device sdb.

Thanks to Milan Broz (from Red Hat) who is author of the original lsblk(8) prototype.

util-linux is without -ng

2010-12-02T11:41:00.003+01:00

The util-linux-ng project was officially merged into util-linux in last days. It means that there is no more -ng fork. Fortunately, the change was pretty simple, because the original util-linux project was inactive in last four years. It was enough to rename mailing list, git repository and directories at kernel.org. Note that the old addresses still work and all is redirected to the new stuff.

The last remaining problem is to rename the mailing list at gmane.org (http://news.gmane.org/gmane.linux.utilities.util-linux-ng/). I don't want to lost the list history, so the renaming seems as a better way than remove old list and add a new list (without -ng). Let's hope that gmane admins will be able to do the change.

See
http://article.gmane.org/gmane.linux.file-systems/49137 for more details about new URLs and addresses.

Thanks to Kay, Adrian, John and David.

findmnt(8)

2010-07-15T22:26:00.002+02:00

I released util-linux-ng 2.18 two weeks ago. There is many changes, for example completely new libmount (not stable API yet), new fsfreeze(8) and findmnt(8) utils and some important changes in fdisk(8).

From my point of view the most attractive for end users is findmnt(8). This new util is a command line interface to the libmount library, the util is able to search in /etc/fstab, /etc/mtab or /proc/self/mountinfo.

Default output (mounted filesystems):

$ findmnt
TARGET                       SOURCE                FSTYPE   OPTIONS
/                            /dev/sda4             ext3     rw,noatime,errors=co
├─/proc                      /proc                 proc     rw,relatime
│ ├─/proc/bus/usb            /proc/bus/usb         usbfs    rw,relatime
│ ├─/proc/sys/fs/binfmt_misc none                  binfmt_m rw,relatime
│ └─/proc/fs/nfsd            nfsd                  nfsd     rw,relatime
├─/sys                       /sys                  sysfs    rw,relatime
├─/dev                       udev                  devtmpfs rw,relatime,size=197
│ ├─/dev/pts                 devpts                devpts   rw,relatime,gid=5,mo
│ └─/dev/shm                 tmpfs                 tmpfs    rw,relatime
├─/boot                      /dev/sda1             ext3     rw,noatime,errors=co
├─/home/kzak                 /dev/mapper/kzak-home ext4     rw,noatime,barrier=1
│ └─/home/kzak/.gvfs         gvfs-fuse-daemon      fuse.gvf rw,nosuid,nodev,rela
├─/var/lib/nfs/rpc_pipefs    sunrpc                rpc_pipe rw,relatime
├─/mnt/foo                   //sr.net.home/foo     cifs     rw,relatime,mand,unc
└─/mnt/test                  /dev/sda6             btrfs    rw,relatime

Get info about a mountpoint:

$ findmnt /home/kzak
TARGET     SOURCE                FSTYPE OPTIONS
/home/kzak /dev/mapper/kzak-home ext4   rw,noatime,barrier=1,data=ordered

Get all mounted extN filesystems:

$ findmnt -t ext4,ext3
TARGET     SOURCE                FSTYPE OPTIONS
/          /dev/sda4             ext3   rw,noatime,errors=continue,user_xattr
/boot      /dev/sda1             ext3   rw,noatime,errors=continue,user_xattr
/home/kzak /dev/mapper/kzak-home ext4   rw,noatime,barrier=1,data=ordered

The same thing, but from fstab:

$ findmnt --fstab -t ext4,ext3
TARGET     SOURCE                                    FSTYPE OPTIONS
/          UUID=d3a8f783-df75-4dc8-9163-975a891052c0 ext3   noatime,defaults
/boot      UUID=f1cd38fa-c887-4ab8-834b-c8ee659b97fe ext3   noatime,defaults
/home/kzak /dev/mapper/kzak-home                     ext4   noatime,defaults

Don't like LABLEs/UUIDs?

$ findmnt --fstab --evaluate -t ext4,ext3
TARGET     SOURCE                FSTYPE OPTIONS
/          /dev/sda4             ext3   noatime,defaults
/boot      /dev/sda1             ext3   noatime,defaults
/home/kzak /dev/mapper/kzak-home ext4   noatime,defaults

or convert UUID to mountpoint:

$ findmnt -o TARGET --noheadings UUID=f1cd38fa-c887-4ab8-834b-c8ee659b97fe
/boot

4096-byte sector hard drives

2010-05-12T11:49:00.003+02:00

Maybe you're already read some blogs/articles about new 4KiB disks and Linux. These articles usually share one important thing -- WDxxEARS hard drives. Unfortunately, it seems that WDC made a brown-paper-bag bug here. The disks report 512-bytes physical sector size instead of 4096...

hdparm -I /dev/sdb

ATA device, with non-removable media
  Model Number:       WDC WD15EARS-00Z5B1
  Serial Number:      WD-WMAVUxxxxxxx
  Firmware Revision:  80.00A80
[...]
  Logical/Physical Sector size:           512 bytes

Fortunately, it seems that newer models correctly report 4K sectors:

ATA device, with non-removable media
  Model Number:       WDC WD15EARS-00Z5B1               
  Serial Number:      WD-WMAVUxxxxxxx
  Firmware Revision:  80.00A80
[...]
  Logical  Sector size:                   512 bytes
  Physical Sector size:                  4096 bytes
  Logical Sector-0 offset:                  0 bytes

Now the good news. Fedora-13 and all related upstream projects are ready for 4096-byte sector disks. The libparted, fdisk, mkfs.{ext[234],xfs,gfs2} and cryptsetup (upstream, Fedora-14 and RHEL6) have been enhanced to use the new I/O topology to properly align things on the devices.

The I/O topology (aka "I/O limits") is supported since kernel 2.6.31. The topology is exported to userspace by sysfs, for example:

  $ cat /sys/block/sdb/queue/physical_block_size
  4096
  $ cat /sys/block/sdb/queue/logical_block_size
  512
  $ cat /sys/block/sdb/queue/optimal_io_size
  32768

The kernel also supports topology ioctls since 2.6.32. The parted, fdisk and mkfs.{ext,xfs} use libblkid to get the topology, but some other tools directly use ioctls. So it's better to have kernel 2.6.32 or .33.

The fdisk(8) command uses 1MiB offset and grain to align partitions by default. So the final partition table is usable on hard drives with 4096-byte sectors by default. It means independently on the disk topology. This is the good news for WDxxEARS users.

If you want to use fdisk(8) then think twice and don't forget that fdisk is a low-level tool. Some fdiks(8) hints:

use fdisk from util-linux-ng >= 2.17.2
read warnings
don't use DOS-compatible mode (for backward compatibility this mode is enable by default, you have to use command 'c' or '-c' command line option to disable DOS mode. Note that for the next major release the DOS mode will be disable by default.)
use sectors as display units (command 'u' or '-u' command line option)
all default sizes/offsets are aligned to the physical block boundary (e.g. "First sector" dialog always provides aligned default)
use +size{M,G} convention to specify "Last sector" (e.g. +5G to create 5GiB partition) then fdiskl aligns the size to physical block boundary
don't forget that fdisk(8) always follows your wishes -- it means that if you explicitly define first/last sector number then the partition could be misaligned
the 'p' (print) command checks for partitions alignment

For more information about 4KiB sectors read:
https://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
http://people.redhat.com/msnitzer/docs/io-limits.txt

git-pull from translationproject.org

2010-03-27T01:26:00.007+01:00

I don't like localization services (portals) that require a write access to the upstream VCS, because distributed version control systems (e.g. git) are designed for a different type of work flow.

Unfortunately, the access to VCS is a preferred method used by transifex.net (note that now transifex allows to pull .po files from .tar.gz and then the final translations are send by e-mail (!) to the package maintainer).

So... I started to use translationproject.org 3 years ago. And it works. It works very successfully without any automatic interaction with upstream VCS. The final translated .po files are available by HTTP or RSYNC. For example:

  rsync  -Lrtvz  rsync://translationproject.org/tp/latest/$PROJECT/$LANG.po  po

downloads selected .po file to the po/ directory. I usually sync my private GIT repository with the latest PO stuff from translationproject.org few hours before package release. The next step after .po files download is to call git-add (for new .po files) and git-commit. The ideal solution is to commit with the --author= option where the author name is a "Last-Translator" from PO file.

Because I'm lazy, I have a script for this task:

  git-tp-sync <project-name> [<lang> ...]

And here is an example from git-log:

  commit d609d0fb8ba1d2302d8bfd0c85903316af83c402
  Author: Clytie Siddall 
  Date:   Mon Mar 22 09:36:51 2010 +0100

    po: update vi.po (from translationproject.org)

http://people.redhat.com/kzak/git-scripts/git-tp-sync

Highlighted patches inside mutt

2010-02-03T10:58:00.005+01:00

Tons of patches are delivered by mailing lists. Unfortunately, there is not a native way how properly highlight patches inside mutt. This problem could be resolved by two different methods:

1) call an external command for diffs. You need something like

  text/x-diff;  cat %s | colordiff;copiousoutput
  text/x-patch; cat %s | colordiff;copiousoutput

in your ~/.mailcap and

  auto_view text/x-diff
  auto_view text/x-patch
  set allow_ansi

in your ~/.muttrc. Unfortunately this method sucks, because a lot of patches are distributed in the body of the mail. Note that the mail body is usually better for patches that attachments (see The Perfect Patch).

2) use mutt highlighting. This method is also not perfect, but I use it for years and I am very happy with it. You need to add

  color body green default "^diff \-.*"
  color body green default "^index [a-f0-9].*"
  color body green default "^\-\-\- .*"
  color body green default "^[\+]{3} .*"
  color body cyan default "^[\+][^\+]+.*"
  color body red  default "^\-[^\-]+.*"
  color body brightblue default "^@@ .*"

to you ~/.muttrc. That's all :-)

util-linux-ng 2.17

2010-01-08T15:09:00.002+01:00

I released v2.17 this morning (CET). A few numbers about the release:

404 patches
488 files changed, 66816 insertions, 48504 deletions
41 developers
22 developers contributed first time
5 months

Thanks to all contributors!

Release highlights

fallocate

this NEW COMMAND is a command line interface to fallocate Linux syscall and allows to preallocate blocks to a file.

unshare

this NEW COMMAND is a command line interface to unshare Linux syscall and allows to run program with some namespaces unshared from parent.

wipefs

this NEW COMMAND is based on libblkid and allows to remove filesystem or RAID signatures from a device.

libblkid

libblkid allows to gather information about block device topology, currently supported methods are:

ioctl - supported since kernel 2.6.32
sysfs - supported since kernel 2.6.31
fallback for DM, MD, LVM and EVMS on old kernels (base on code from xfsprogs/libdisk)

libblkid supports partition tables parsing (currently supported are aix, bsd, dos, mac, gpt, minix, sgi, solaris, sun and unixware). This functionality is designed for mkfs programs, DeviceKits, [k]partx or so.

libblkid API documentation is available at http://ftp.kernel.org/pub/linux/utils/util-linux-ng/libblkid-docs/

blockdev

supports all new topology ioctls

fdisk

aligns newly created partitions to minimum_io_size boundary ("minimum_io_size" is physical sector size or stripe chunk size on RAIDs).
supports disks with alignment_offset now.

Complete ReleaseNotes:
ftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/v2.17/v2.17-ReleaseNotes

Tarball:
ftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/v2.17/util-linux-ng-2.17.tar.bz2

unshare(1)

2009-12-09T15:36:00.003+01:00

The unshare(1) is a new command line interface to unshare Linux syscall and allows a program to run with some parts of the process execution context unshared from parent. This new command is available in util-linux-ng 2.17 (thanks to Mikhail Gusarov who is author of the command).

Currently, the unshare(1) command allows to disassociates:

mount namespace
UTS (since kernel 2.6.19, independent hostname and domainname)
IPC namespace (System V message queues, semaphores, and shared memory)
network namespace (since kernel 2.6.24, independent IP stacks, IP routing tables, firewall, ...)

Probably the most attractive is unshared mount namespace (see Mike Hommey's "newns" or pam_namespace).

For example I have two xterms, let's start a new bash with unshared mount namespace in the first session:

Session1:


  # unshare --mount  /bin/bash
  # mount /dev/sda1 /mnt/test
  # grep test /proc/mounts
  /dev/sda1 /mnt/test ext3 rw,relatime,errors=continue,user_xattr,acl,data=ordered 0 0

Session2:


  # grep test /proc/mounts
  # ll /mnt/test
  total 0

... the /mnt/test filesystem is visible in the Session1 only. That's all, pretty simple and pretty useful :-)

wipefs(8)

2009-11-25T17:06:00.003+01:00

This new command from util-linux-ng-2.17 is user friendly solution for people who have block devices with more valid filesystem signatures.

It's usually mkfs bug when old signatures are not removed from the device. The situation when you have more signatures on the device could be pretty dangerous. For example you can call swapon(8) for a wrong device and lost your data (see fedora bug #473514).

The libblkid was fixed few months ago and it does not return ambivalent information now. All devices with more signatures are ignored (well, there is exception for RAIDs, volume managers and some special filesystems like iso9660 and UDF). The blkid(8) command returns:


# blkid -p /dev/loop0
/dev/loop0: ambivalent result (probably more filesystems on the device)

Now imagine that you have data on the device, what next?

Use dd(1) and hexdump(1)... well that's joke :-) Use wipefs(8) and remove old (unwanted) filesystem signature.

The command without any option lists details about all detected filesystems:


# wipefs /dev/loop0
offset               type
----------------------------------------------------------------
0xff6                swap   [other]
                     UUID:  da6c54ea-77d9-470a-82f1-e793fd5d9131

0x36                 vfat   [filesystem]
                     UUID:  497F-3013

Now remove unwanted swap signature:


# wipefs --offset 0xff6  /dev/loop0
10 bytes [53 57 41 50 53 50 41 43 45 32] erased at offset 0xff6 (swap)

The wipefs(8) command does not erase whole filesystem superblock or so. It removes magic strings only. Now the device is useful for blkid(8):


# blkid -p /dev/loop0
/dev/loop0: SEC_TYPE="msdos" UUID="497F-3013" VERSION="FAT12" TYPE="vfat" USAGE="filesystem"

The other change in util-linux-ng-2.17 is that detailed information about ambivalent probing results are reported to udevd ($ID_FS_AMBIVALEN variable) so it will be possible to notify users by UI (e.g. Palimpsest).

libblkid

2009-11-12T22:04:00.004+01:00

I think I can say that filesystem probing (identification) code in Linux distributions has been successfully consolidated to the one library. The libblkid.

The library is Ted Tso's baby and was maintained and distributed within e2fsprogs package for many years. Unfortunately, the library didn't match with udev requirements. The udev project requires lightweight probing library without any high-level logic. This was reason why udev developers created a new probing library -- libvolume_id.

And finally and unfortunately there was very old filesystem probing code in the mount(8) utility too. The useless code in mount(8) was ignored by mainstream Linux distributions (for example Suse linked their utils against libvolume_id and RHEL/Fedora against libblkid) .... and this is not end of nightmare.

The other filesystem probing code was in xfsprogs project (libdisk), and the other code is still in the parted (libparted).

And now about consolidation. The libblkid was merged into util-linux-ng (one year ago). The old version in e2fsprogs is still maintained for people who are very conservative. All new features go to libblkid in util-linux-ng project.

Some important functionality from libvolume_id was merged into libblkid including a new low-level API. The blkid(8) command supports new "-p" (low-level probe) and "-o udev" (udev output format) options now.

The libvolume_id was completely removed from udev project a few months ago. The mount(8) command uses only libblkid.

The libblkid has been heavily extended in the latest (incoming) util-linux-ng-2.17. It supports partition tables probing and allows to gather information about blkdev topology. This latest change allows to use libblkid instead of libdisk in mkfs programs (already in mke2fs and mkfs.xfs).

And finally libparted could be linked against libblkid. Currently, it uses libblkid for topology information only, but I hope one day it will use it for filesystems probing too.

The last remaining place where is any filesystem probing code is cfdisk. It will be fixed in some next util-linux-ng release.

So... now is excellent opportunity to say THANKS to Ted (e2fsprogs), Kay (udev), Christoph (xfsprogs) and to others who help me with this task.

[...yeah, this is my first blog post ... and yeah, my English is horrible:-)]