4096-byte sector hard drives

Maybe you're already read some blogs/articles about new 4KiB disks and Linux. These articles usually share one important thing -- WDxxEARS hard drives. Unfortunately, it seems that WDC made a brown-paper-bag bug here. The disks report 512-bytes physical sector size instead of 4096...

hdparm -I /dev/sdb

ATA device, with non-removable media
  Model Number:       WDC WD15EARS-00Z5B1
  Serial Number:      WD-WMAVUxxxxxxx
  Firmware Revision:  80.00A80
[...]
  Logical/Physical Sector size:           512 bytes

Fortunately, it seems that newer models correctly report 4K sectors:

ATA device, with non-removable media
  Model Number:       WDC WD15EARS-00Z5B1               
  Serial Number:      WD-WMAVUxxxxxxx
  Firmware Revision:  80.00A80
[...]
  Logical  Sector size:                   512 bytes
  Physical Sector size:                  4096 bytes
  Logical Sector-0 offset:                  0 bytes

Now the good news. Fedora-13 and all related upstream projects are ready for 4096-byte sector disks. The libparted, fdisk, mkfs.{ext[234],xfs,gfs2} and cryptsetup (upstream, Fedora-14 and RHEL6) have been enhanced to use the new I/O topology to properly align things on the devices.

The I/O topology (aka "I/O limits") is supported since kernel 2.6.31. The topology is exported to userspace by sysfs, for example:

  $ cat /sys/block/sdb/queue/physical_block_size
  4096
  $ cat /sys/block/sdb/queue/logical_block_size
  512
  $ cat /sys/block/sdb/queue/optimal_io_size
  32768

The kernel also supports topology ioctls since 2.6.32. The parted, fdisk and mkfs.{ext,xfs} use libblkid to get the topology, but some other tools directly use ioctls. So it's better to have kernel 2.6.32 or .33.

The fdisk(8) command uses 1MiB offset and grain to align partitions by default. So the final partition table is usable on hard drives with 4096-byte sectors by default. It means independently on the disk topology. This is the good news for WDxxEARS users.

If you want to use fdisk(8) then think twice and don't forget that fdisk is a low-level tool. Some fdiks(8) hints:

• use fdisk from util-linux-ng >= 2.17.2
• read warnings
• don't use DOS-compatible mode (for backward compatibility this mode is enable by default, you have to use command 'c' or '-c' command line option to disable DOS mode. Note that for the next major release the DOS mode will be disable by default.)
• use sectors as display units (command 'u' or '-u' command line option)
• all default sizes/offsets are aligned to the physical block boundary (e.g. "First sector" dialog always provides aligned default)
• use +size{M,G} convention to specify "Last sector" (e.g. +5G to create 5GiB partition) then fdiskl aligns the size to physical block boundary
• don't forget that fdisk(8) always follows your wishes -- it means that if you explicitly define first/last sector number then the partition could be misaligned
• the 'p' (print) command checks for partitions alignment

For more information about 4KiB sectors read:
https://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
http://people.redhat.com/msnitzer/docs/io-limits.txt

git-pull from translationproject.org

I don't like localization services (portals) that require a write access to the upstream VCS, because distributed version control systems (e.g. git) are designed for a different type of work flow.

Unfortunately, the access to VCS is a preferred method used by transifex.net (note that now transifex allows to pull .po files from .tar.gz and then the final translations are send by e-mail (!) to the package maintainer).

So... I started to use translationproject.org 3 years ago. And it works. It works very successfully without any automatic interaction with upstream VCS. The final translated .po files are available by HTTP or RSYNC. For example:

  rsync  -Lrtvz  rsync://translationproject.org/tp/latest/$PROJECT/$LANG.po  po

downloads selected .po file to the po/ directory. I usually sync my private GIT repository with the latest PO stuff from translationproject.org few hours before package release. The next step after .po files download is to call git-add (for new .po files) and git-commit. The ideal solution is to commit with the --author= option where the author name is a "Last-Translator" from PO file.

Because I'm lazy, I have a script for this task:

  git-tp-sync <project-name> [<lang> ...]

And here is an example from git-log:

  commit d609d0fb8ba1d2302d8bfd0c85903316af83c402
  Author: Clytie Siddall 
  Date:   Mon Mar 22 09:36:51 2010 +0100

    po: update vi.po (from translationproject.org)

http://people.redhat.com/kzak/git-scripts/git-tp-sync

Highlighted patches inside mutt

Tons of patches are delivered by mailing lists. Unfortunately, there is not a native way how properly highlight patches inside mutt. This problem could be resolved by two different methods:

1) call an external command for diffs. You need something like

  text/x-diff;  cat %s | colordiff;copiousoutput
  text/x-patch; cat %s | colordiff;copiousoutput

in your ~/.mailcap and

  auto_view text/x-diff
  auto_view text/x-patch
  set allow_ansi

in your ~/.muttrc. Unfortunately this method sucks, because a lot of patches are distributed in the body of the mail. Note that the mail body is usually better for patches that attachments (see The Perfect Patch).

2) use mutt highlighting. This method is also not perfect, but I use it for years and I am very happy with it. You need to add

  color body green default "^diff \-.*"
  color body green default "^index [a-f0-9].*"
  color body green default "^\-\-\- .*"
  color body green default "^[\+]{3} .*"
  color body cyan default "^[\+][^\+]+.*"
  color body red  default "^\-[^\-]+.*"
  color body brightblue default "^@@ .*"

to you ~/.muttrc. That's all :-)

util-linux-ng 2.17

I released v2.17 this morning (CET). A few numbers about the release:

• 404 patches
• 488 files changed, 66816 insertions, 48504 deletions
• 41 developers
• 22 developers contributed first time
• 5 months

Thanks to all contributors!

Release highlights

fallocate

• this NEW COMMAND is a command line interface to fallocate Linux syscall and allows to preallocate blocks to a file.

unshare

• this NEW COMMAND is a command line interface to unshare Linux syscall and allows to run program with some namespaces unshared from parent.

wipefs

• this NEW COMMAND is based on libblkid and allows to remove filesystem or RAID signatures from a device.

libblkid

• libblkid allows to gather information about block device topology, currently supported methods are:
• ioctl - supported since kernel 2.6.32
• sysfs - supported since kernel 2.6.31
• fallback for DM, MD, LVM and EVMS on old kernels (base on code from xfsprogs/libdisk)

The topology support is mostly designed for mkfs programs or partitioning tools (already used in mkfs.xfs, mkex2fs, libparted and fdisk)

• libblkid supports partition tables parsing (currently supported are aix, bsd, dos, mac, gpt, minix, sgi, solaris, sun and unixware). This functionality is designed for mkfs programs, DeviceKits, [k]partx or so.

• libblkid API documentation is available at http://ftp.kernel.org/pub/linux/utils/util-linux-ng/libblkid-docs/

blockdev

• supports all new topology ioctls

fdisk

• aligns newly created partitions to minimum_io_size boundary ("minimum_io_size" is physical sector size or stripe chunk size on RAIDs).
• supports disks with alignment_offset now.

Complete ReleaseNotes:
ftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/v2.17/v2.17-ReleaseNotes

Tarball:
ftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/v2.17/util-linux-ng-2.17.tar.bz2

unshare(1)

The unshare(1) is a new command line interface to unshare Linux syscall and allows a program to run with some parts of the process execution context unshared from parent. This new command is available in util-linux-ng 2.17 (thanks to Mikhail Gusarov who is author of the command).

Currently, the unshare(1) command allows to disassociates:

• mount namespace
• UTS (since kernel 2.6.19, independent hostname and domainname)
• IPC namespace (System V message queues, semaphores, and shared memory)
• network namespace (since kernel 2.6.24, independent IP stacks, IP routing tables, firewall, ...)

Probably the most attractive is unshared mount namespace (see Mike Hommey's "newns" or pam_namespace).

For example I have two xterms, let's start a new bash with unshared mount namespace in the first session:

Session1:

  # unshare --mount  /bin/bash
  # mount /dev/sda1 /mnt/test
  # grep test /proc/mounts
  /dev/sda1 /mnt/test ext3 rw,relatime,errors=continue,user_xattr,acl,data=ordered 0 0

Session2:

  # grep test /proc/mounts
  # ll /mnt/test
  total 0

... the /mnt/test filesystem is visible in the Session1 only. That's all, pretty simple and pretty useful :-)

wipefs(8)

This new command from util-linux-ng-2.17 is user friendly solution for people who have block devices with more valid filesystem signatures.

It's usually mkfs bug when old signatures are not removed from the device. The situation when you have more signatures on the device could be pretty dangerous. For example you can call swapon(8) for a wrong device and lost your data (see fedora bug #473514).

The libblkid was fixed few months ago and it does not return ambivalent information now. All devices with more signatures are ignored (well, there is exception for RAIDs, volume managers and some special filesystems like iso9660 and UDF). The blkid(8) command returns:

# blkid -p /dev/loop0
/dev/loop0: ambivalent result (probably more filesystems on the device)

Now imagine that you have data on the device, what next?

Use dd(1) and hexdump(1)... well that's joke :-) Use wipefs(8) and remove old (unwanted) filesystem signature.

The command without any option lists details about all detected filesystems:

# wipefs /dev/loop0
offset               type
----------------------------------------------------------------
0xff6                swap   [other]
                     UUID:  da6c54ea-77d9-470a-82f1-e793fd5d9131

0x36                 vfat   [filesystem]
                     UUID:  497F-3013

Now remove unwanted swap signature:

# wipefs --offset 0xff6  /dev/loop0
10 bytes [53 57 41 50 53 50 41 43 45 32] erased at offset 0xff6 (swap)

The wipefs(8) command does not erase whole filesystem superblock or so. It removes magic strings only. Now the device is useful for blkid(8):

# blkid -p /dev/loop0
/dev/loop0: SEC_TYPE="msdos" UUID="497F-3013" VERSION="FAT12" TYPE="vfat" USAGE="filesystem"

The other change in util-linux-ng-2.17 is that detailed information about ambivalent probing results are reported to udevd ($ID_FS_AMBIVALEN variable) so it will be possible to notify users by UI (e.g. Palimpsest).

libblkid

I think I can say that filesystem probing (identification) code in Linux distributions has been successfully consolidated to the one library. The libblkid.

The library is Ted Tso's baby and was maintained and distributed within e2fsprogs package for many years. Unfortunately, the library didn't match with udev requirements. The udev project requires lightweight probing library without any high-level logic. This was reason why udev developers created a new probing library -- libvolume_id.

And finally and unfortunately there was very old filesystem probing code in the mount(8) utility too. The useless code in mount(8) was ignored by mainstream Linux distributions (for example Suse linked their utils against libvolume_id and RHEL/Fedora against libblkid) .... and this is not end of nightmare.

The other filesystem probing code was in xfsprogs project (libdisk), and the other code is still in the parted (libparted).

And now about consolidation. The libblkid was merged into util-linux-ng (one year ago). The old version in e2fsprogs is still maintained for people who are very conservative. All new features go to libblkid in util-linux-ng project.

Some important functionality from libvolume_id was merged into libblkid including a new low-level API. The blkid(8) command supports new "-p" (low-level probe) and "-o udev" (udev output format) options now.

The libvolume_id was completely removed from udev project a few months ago. The mount(8) command uses only libblkid.

The libblkid has been heavily extended in the latest (incoming) util-linux-ng-2.17. It supports partition tables probing and allows to gather information about blkdev topology. This latest change allows to use libblkid instead of libdisk in mkfs programs (already in mke2fs and mkfs.xfs).

And finally libparted could be linked against libblkid. Currently, it uses libblkid for topology information only, but I hope one day it will use it for filesystems probing too.

The last remaining place where is any filesystem probing code is cfdisk. It will be fixed in some next util-linux-ng release.

So... now is excellent opportunity to say THANKS to Ted (e2fsprogs), Kay (udev), Christoph (xfsprogs) and to others who help me with this task.

[...yeah, this is my first blog post ... and yeah, my English is horrible:-)]