lsblk(8)

lsblk(8) is a new util that will be available in util-linux-2.19 (coming soon;-). The util lists all block devices as a tree. This output is very useful on machines with complicated storage setup (e.g. RAID, dm-crypt, ...). It's so useful that we'll probably backport lsblk(8) to RHEL-6 to make life easier for people who need to debug their systems.

The original idea comes from "dmsetup ls --tree", but lsblk(8) is better :-) It uses "holders" and "slaves" from /sys filesystem. This means that the util is usable without root permissions and it works for all types of block devices (dmsetup uses DM ioctls).

For example my laptop with dm-crypted $HOME and partitioned RAID0 (md8) on scsi_debug device (sdc):

$ lsblk
NAME                 MAJ:MIN RA   SIZE RO MOUNTPOINT
sda                    8:0    0  93.2G  0
├─sda1                 8:1    0   102M  0 /mnt/test
├─sda2                 8:2    0     1K  0
├─sda3                 8:3    0   2.3G  0 [SWAP]
├─sda4                 8:4    0  76.2G  0 /
├─sda5                 8:5    0    10G  0
│ └─kzak-home (dm-0) 253:0    0    10G  0 /home/kzak
└─sda6                 8:6    0   4.7G  0 /boot
sdc                    8:32   0   500M  0
├─sdc1                 8:33   0   250M  0
│ └─md8                9:8    0 498.9M  0
│   ├─md8p1          259:0    0   100M  0
│   ├─md8p2          259:1    0   100M  0
│   └─md8p3          259:2    0 297.9M  0
└─sdc2                 8:34   0   249M  0
  └─md8                9:8    0 498.9M  0
    ├─md8p1          259:0    0   100M  0
    ├─md8p2          259:1    0   100M  0
    └─md8p3          259:2    0 297.9M  0

You can also list more details about devices. The next example is from scsi_debug device with 4KiB sectors and enabled alignment offset:

$ lsblk --topology /dev/sdb
NAME   ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED
sdb         3584   4096  32768    4096     512    0 cfq
├─sdb1         0   4096  32768    4096     512    0 cfq
└─sdb2         0   4096  32768    4096     512    0 cfq

lsblk(8) is also usable in script, for example:

$ lsblk --nodeps --noheading --raw -o ALIGNMENT /dev/sdb
3584

returns only alignment-offset for device sdb.

Thanks to Milan Broz (from Red Hat) who is author of the original lsblk(8) prototype.

util-linux is without -ng

The util-linux-ng project was officially merged into util-linux in last days. It means that there is no more -ng fork. Fortunately, the change was pretty simple, because the original util-linux project was inactive in last four years. It was enough to rename mailing list, git repository and directories at kernel.org. Note that the old addresses still work and all is redirected to the new stuff.

The last remaining problem is to rename the mailing list at gmane.org (http://news.gmane.org/gmane.linux.utilities.util-linux-ng/). I don't want to lost the list history, so the renaming seems as a better way than remove old list and add a new list (without -ng). Let's hope that gmane admins will be able to do the change.

See
http://article.gmane.org/gmane.linux.file-systems/49137 for more details about new URLs and addresses.

Thanks to Kay, Adrian, John and David.

findmnt(8)

I released util-linux-ng 2.18 two weeks ago. There is many changes, for example completely new libmount (not stable API yet), new fsfreeze(8) and findmnt(8) utils and some important changes in fdisk(8).

From my point of view the most attractive for end users is findmnt(8). This new util is a command line interface to the libmount library, the util is able to search in /etc/fstab, /etc/mtab or /proc/self/mountinfo.

Default output (mounted filesystems):

$ findmnt
TARGET                       SOURCE                FSTYPE   OPTIONS
/                            /dev/sda4             ext3     rw,noatime,errors=co
├─/proc                      /proc                 proc     rw,relatime
│ ├─/proc/bus/usb            /proc/bus/usb         usbfs    rw,relatime
│ ├─/proc/sys/fs/binfmt_misc none                  binfmt_m rw,relatime
│ └─/proc/fs/nfsd            nfsd                  nfsd     rw,relatime
├─/sys                       /sys                  sysfs    rw,relatime
├─/dev                       udev                  devtmpfs rw,relatime,size=197
│ ├─/dev/pts                 devpts                devpts   rw,relatime,gid=5,mo
│ └─/dev/shm                 tmpfs                 tmpfs    rw,relatime
├─/boot                      /dev/sda1             ext3     rw,noatime,errors=co
├─/home/kzak                 /dev/mapper/kzak-home ext4     rw,noatime,barrier=1
│ └─/home/kzak/.gvfs         gvfs-fuse-daemon      fuse.gvf rw,nosuid,nodev,rela
├─/var/lib/nfs/rpc_pipefs    sunrpc                rpc_pipe rw,relatime
├─/mnt/foo                   //sr.net.home/foo     cifs     rw,relatime,mand,unc
└─/mnt/test                  /dev/sda6             btrfs    rw,relatime

Get info about a mountpoint:

$ findmnt /home/kzak
TARGET     SOURCE                FSTYPE OPTIONS
/home/kzak /dev/mapper/kzak-home ext4   rw,noatime,barrier=1,data=ordered

Get all mounted extN filesystems:

$ findmnt -t ext4,ext3
TARGET     SOURCE                FSTYPE OPTIONS
/          /dev/sda4             ext3   rw,noatime,errors=continue,user_xattr
/boot      /dev/sda1             ext3   rw,noatime,errors=continue,user_xattr
/home/kzak /dev/mapper/kzak-home ext4   rw,noatime,barrier=1,data=ordered

The same thing, but from fstab:

$ findmnt --fstab -t ext4,ext3
TARGET     SOURCE                                    FSTYPE OPTIONS
/          UUID=d3a8f783-df75-4dc8-9163-975a891052c0 ext3   noatime,defaults
/boot      UUID=f1cd38fa-c887-4ab8-834b-c8ee659b97fe ext3   noatime,defaults
/home/kzak /dev/mapper/kzak-home                     ext4   noatime,defaults

Don't like LABLEs/UUIDs?

$ findmnt --fstab --evaluate -t ext4,ext3
TARGET     SOURCE                FSTYPE OPTIONS
/          /dev/sda4             ext3   noatime,defaults
/boot      /dev/sda1             ext3   noatime,defaults
/home/kzak /dev/mapper/kzak-home ext4   noatime,defaults

or convert UUID to mountpoint:

$ findmnt -o TARGET --noheadings UUID=f1cd38fa-c887-4ab8-834b-c8ee659b97fe
/boot

4096-byte sector hard drives

Maybe you're already read some blogs/articles about new 4KiB disks and Linux. These articles usually share one important thing -- WDxxEARS hard drives. Unfortunately, it seems that WDC made a brown-paper-bag bug here. The disks report 512-bytes physical sector size instead of 4096...

hdparm -I /dev/sdb

ATA device, with non-removable media
  Model Number:       WDC WD15EARS-00Z5B1
  Serial Number:      WD-WMAVUxxxxxxx
  Firmware Revision:  80.00A80
[...]
  Logical/Physical Sector size:           512 bytes

Fortunately, it seems that newer models correctly report 4K sectors:

ATA device, with non-removable media
  Model Number:       WDC WD15EARS-00Z5B1               
  Serial Number:      WD-WMAVUxxxxxxx
  Firmware Revision:  80.00A80
[...]
  Logical  Sector size:                   512 bytes
  Physical Sector size:                  4096 bytes
  Logical Sector-0 offset:                  0 bytes

Now the good news. Fedora-13 and all related upstream projects are ready for 4096-byte sector disks. The libparted, fdisk, mkfs.{ext[234],xfs,gfs2} and cryptsetup (upstream, Fedora-14 and RHEL6) have been enhanced to use the new I/O topology to properly align things on the devices.

The I/O topology (aka "I/O limits") is supported since kernel 2.6.31. The topology is exported to userspace by sysfs, for example:

  $ cat /sys/block/sdb/queue/physical_block_size
  4096
  $ cat /sys/block/sdb/queue/logical_block_size
  512
  $ cat /sys/block/sdb/queue/optimal_io_size
  32768

The kernel also supports topology ioctls since 2.6.32. The parted, fdisk and mkfs.{ext,xfs} use libblkid to get the topology, but some other tools directly use ioctls. So it's better to have kernel 2.6.32 or .33.

The fdisk(8) command uses 1MiB offset and grain to align partitions by default. So the final partition table is usable on hard drives with 4096-byte sectors by default. It means independently on the disk topology. This is the good news for WDxxEARS users.

If you want to use fdisk(8) then think twice and don't forget that fdisk is a low-level tool. Some fdiks(8) hints:

• use fdisk from util-linux-ng >= 2.17.2
• read warnings
• don't use DOS-compatible mode (for backward compatibility this mode is enable by default, you have to use command 'c' or '-c' command line option to disable DOS mode. Note that for the next major release the DOS mode will be disable by default.)
• use sectors as display units (command 'u' or '-u' command line option)
• all default sizes/offsets are aligned to the physical block boundary (e.g. "First sector" dialog always provides aligned default)
• use +size{M,G} convention to specify "Last sector" (e.g. +5G to create 5GiB partition) then fdiskl aligns the size to physical block boundary
• don't forget that fdisk(8) always follows your wishes -- it means that if you explicitly define first/last sector number then the partition could be misaligned
• the 'p' (print) command checks for partitions alignment

For more information about 4KiB sectors read:
https://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
http://people.redhat.com/msnitzer/docs/io-limits.txt

git-pull from translationproject.org

I don't like localization services (portals) that require a write access to the upstream VCS, because distributed version control systems (e.g. git) are designed for a different type of work flow.

Unfortunately, the access to VCS is a preferred method used by transifex.net (note that now transifex allows to pull .po files from .tar.gz and then the final translations are send by e-mail (!) to the package maintainer).

So... I started to use translationproject.org 3 years ago. And it works. It works very successfully without any automatic interaction with upstream VCS. The final translated .po files are available by HTTP or RSYNC. For example:

  rsync  -Lrtvz  rsync://translationproject.org/tp/latest/$PROJECT/$LANG.po  po

downloads selected .po file to the po/ directory. I usually sync my private GIT repository with the latest PO stuff from translationproject.org few hours before package release. The next step after .po files download is to call git-add (for new .po files) and git-commit. The ideal solution is to commit with the --author= option where the author name is a "Last-Translator" from PO file.

Because I'm lazy, I have a script for this task:

  git-tp-sync <project-name> [<lang> ...]

And here is an example from git-log:

  commit d609d0fb8ba1d2302d8bfd0c85903316af83c402
  Author: Clytie Siddall 
  Date:   Mon Mar 22 09:36:51 2010 +0100

    po: update vi.po (from translationproject.org)

http://people.redhat.com/kzak/git-scripts/git-tp-sync

Highlighted patches inside mutt

Tons of patches are delivered by mailing lists. Unfortunately, there is not a native way how properly highlight patches inside mutt. This problem could be resolved by two different methods:

1) call an external command for diffs. You need something like

  text/x-diff;  cat %s | colordiff;copiousoutput
  text/x-patch; cat %s | colordiff;copiousoutput

in your ~/.mailcap and

  auto_view text/x-diff
  auto_view text/x-patch
  set allow_ansi

in your ~/.muttrc. Unfortunately this method sucks, because a lot of patches are distributed in the body of the mail. Note that the mail body is usually better for patches that attachments (see The Perfect Patch).

2) use mutt highlighting. This method is also not perfect, but I use it for years and I am very happy with it. You need to add

  color body green default "^diff \-.*"
  color body green default "^index [a-f0-9].*"
  color body green default "^\-\-\- .*"
  color body green default "^[\+]{3} .*"
  color body cyan default "^[\+][^\+]+.*"
  color body red  default "^\-[^\-]+.*"
  color body brightblue default "^@@ .*"

to you ~/.muttrc. That's all :-)

util-linux-ng 2.17

I released v2.17 this morning (CET). A few numbers about the release:

• 404 patches
• 488 files changed, 66816 insertions, 48504 deletions
• 41 developers
• 22 developers contributed first time
• 5 months

Thanks to all contributors!

Release highlights

fallocate

• this NEW COMMAND is a command line interface to fallocate Linux syscall and allows to preallocate blocks to a file.

unshare

• this NEW COMMAND is a command line interface to unshare Linux syscall and allows to run program with some namespaces unshared from parent.

wipefs

• this NEW COMMAND is based on libblkid and allows to remove filesystem or RAID signatures from a device.

libblkid

• libblkid allows to gather information about block device topology, currently supported methods are:
• ioctl - supported since kernel 2.6.32
• sysfs - supported since kernel 2.6.31
• fallback for DM, MD, LVM and EVMS on old kernels (base on code from xfsprogs/libdisk)

The topology support is mostly designed for mkfs programs or partitioning tools (already used in mkfs.xfs, mkex2fs, libparted and fdisk)

• libblkid supports partition tables parsing (currently supported are aix, bsd, dos, mac, gpt, minix, sgi, solaris, sun and unixware). This functionality is designed for mkfs programs, DeviceKits, [k]partx or so.

• libblkid API documentation is available at http://ftp.kernel.org/pub/linux/utils/util-linux-ng/libblkid-docs/

blockdev

• supports all new topology ioctls

fdisk

• aligns newly created partitions to minimum_io_size boundary ("minimum_io_size" is physical sector size or stripe chunk size on RAIDs).
• supports disks with alignment_offset now.

Complete ReleaseNotes:
ftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/v2.17/v2.17-ReleaseNotes

Tarball:
ftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/v2.17/util-linux-ng-2.17.tar.bz2