Friday, June 9, 2017

util-linux v2.30 -- what's new?

The command tailf is dead thing. (RIP ... years ago I had nice time to improve it with inotify:) You have to use "tail -f" from coreutils project.

blkzone -- this new command is excellent example of the open source collaboration. The command has been developed by people from WD, Seagate and SanDisk (thanks to Shaun Tancheff, Damien Le Moal and others). The goal is to have command line interface to run zone commands on block devices that support Zoned Block Commands (ZBC) or Zoned-device ATA Commands (ZAC). For now the supported zone commands are "reset" and "report". See for more details about zones.

fincore (file in core)-- this is nice useful command to get information about number of memory pages used by file content. For example my fulltext email DB:

# fincore ~/Mail/Maildir/.notmuch/xapian/*.DB
 60.1M  15392  4.6G /home/kzak/Mail/Maildir/.notmuch/xapian/position.DB
687.4M 175982  3.5G /home/kzak/Mail/Maildir/.notmuch/xapian/postlist.DB
  328K     82 18.6M /home/kzak/Mail/Maildir/.notmuch/xapian/record.DB
190.5M  48758  2.1G /home/kzak/Mail/Maildir/.notmuch/xapian/termlist.DB

Fortunately RAM is cheap :) Thanks to Masatake Yamato from Red Hat.

lsmem (list memory) and chmem (change memory) -- another new commands. The commands have been originally implemented in Perl for s390-tools, now re-implemented in C in more generic way and to be usable on another architectures too. (thanks to Clemens von Mann and Heiko Carstens from IBM.)

The command fallocate supports an "insert range" operation now.

We continue on hwclock cleanup, some things in the code have been simplified, dead and useless things removed. (thanks to J William Piggott)

The code behind "column -t|--table" uses libsmartcols now. This change dramatically increased number of available features for table formatting. Now it's possible to define header for columns, truncate text in cells, align text to the right, change order of columns, JSON output or create tree-like output. Now almost all libsmartcols features are available on command line, example:
pstree-like output:

 $ ps -h -o pid,ppid,comm | column --table --tree 3 --tree-id 1 --tree-parent 2 --table-hide 2 --table-right 1
 1799  bash
 2254  bash
28427  └─mutt
 4263    └─vim
 7409  bash
10641  └─man
10657    └─less
16775  bash
11486  ├─ps
11487  └─column

 $ column /proc/diskstats --table --table-columns MAJ,MIN,NAME,READ-COMP,\
      --table-hide MAJ,MIN \
      --table-right 4,5,6,7,8,9,10,11,12,13,14 \
sda     13486466     149085  1288469300    9715620    45556082     7788088  1600182109   150180178        0  12935701  159902109
sda1         463        170       19002        131          91           0         161         331        0       334        462
sda2         778         16       63140        276         434         261      507574       12616        0      2382      12889
sda3    10710224     109592  1052352266    8018950    43983768     7022717  1153182094   126210185        0  11002854  134299501
sda4     1630396      32476    67166050    1039837     1197142      665798   343331344    23264993        0   2148041   24306932
sda5     1140435        241   168747746     655625      225373       73891   102920032      637906        0    627834    1293105
sda6        3703       6590       99512        691        4691       25421      240904        8418        0      6402       9108
sdb          448          0       22506       3088        1887           4         128         275        0      1449       3363
sdb1         404          0       19370       3035          12           4         128          60        0      1187       3095
loop0      22086          0      347311       2025       10738           0      844888        2226        0      1129       4265
loop1        947          0       26940        325        1100           0      133316         734        0       411       1058
md8            0          0           0          0           0           0           0           0        0         0          0

passwd in JSON:
 $ grep -v nologin /etc/passwd | \
     column --separator : --table --table-name passwd --json \
            --table-columns USERNAME,PWD,UID,GID,GECOS,HOME,SHELL \
            --table-hide PWD
   "passwd": [
      {"username": "root", "uid": "0", "gid": "0", "gecos": "root", "home": "/root", "shell": "/bin/bash"},
      {"username": "sync", "uid": "5", "gid": "0", "gecos": "sync", "home": "/sbin", "shell": "/bin/sync"},
      {"username": "shutdown", "uid": "6", "gid": "0", "gecos": "shutdown", "home": "/sbin", "shell": "/sbin/shutdown"},
      {"username": "halt", "uid": "7", "gid": "0", "gecos": "halt", "home": "/sbin", "shell": "/sbin/halt"},
      {"username": "kzak", "uid": "1000", "gid": "1000", "gecos": "Karel Zak,Home,,,", "home": "/home/kzak", "shell": "/bin/bash"},
      {"username": "gamer", "uid": "1001", "gid": "1001", "gecos": null, "home": "/home/gamer", "shell": "/bin/bash"},
      {"username": "test", "uid": "1002", "gid": "1002", "gecos": null, "home": "/home/test", "shell": "/bin/bash"}

findmnt-like output:
 $ column /proc/self/mountinfo \
     --table-order TARGET,SOURCE,TYPE,VFS-OPTS \
     --tree TARGET \
     --tree-id ID \
     --tree-parent PARENT
TARGET                             SOURCE       TYPE         VFS-OPTS
/                                  /dev/sda4    ext4         rw,relatime
├─/sys                             sysfs        sysfs        rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security           securityfs   securityfs   rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                 tmpfs        tmpfs        ro,nosuid,nodev,noexec
│ │ ├─/sys/fs/cgroup/systemd       cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/blkio         cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/cpu,cpuacct   cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/devices       cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/hugetlb       cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/pids          cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/memory        cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/cpuset        cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
... and so on ... 
Thanks to all contributors. The next version v2.31 is planned for September 2017.

Friday, January 6, 2017

10 years with util-linux project!

I can't believe that it's already 10 years we have active community around basic Linux utils.

Yes, we had util-linux before (and many thanks to Adrian Bunk and Andries E. Brouwer), but I believe that with git and close collaboration between Linux distributions and Linux kernel community it better now :-)
  • ~11000 commits
  • ~460 unique contributors
  • ~630 regression tests
  • ~100 utils, 5 shared libs with public API
  • ~16 major releases (from v2.13 to v2.29)
  • ~26 translated languages
  • ~10000 e-mails on mailing list

  • ported to GNU Hurd, FreeBSD and XOS
  • used by all mainstream Linux distros (as well as by Systemd haters ;-)
  • modern autotools based build system

  • merges from another projects
    • libblkid, libuuid and fsck from e2fsprogs
    • sulogin, last, utmpdump  and mounpoint from sysvinit
    • su from coreutils

  • many new utils (lsblk, findmnt, wipefs, rtcwake, unshare, nsenter, prlimit, blkdiscard, flock, fstrim, ipcmk, ldattach, lscpu, lsipc, lslocks, lslogins, resizepart, setarch, setpriv, switch_root, swaplabel)

  • many rewrites (libblkid probing code, mount, fdisks, etc.)

  • new shared libraries
    • libmount
    • libfdisk
    • libsmartcols

...the first original announce:

... gource video with all the ten years :-)

Thanks to all contributors!

Thursday, October 6, 2016

util-linux v2.29 -- what's new?

The release v2.29 (now rc1) is without dramatical changes, the small exception is libsmartcols where we have many improvements. 

The old good cal(1) is more user-friendly now. It's possible to specify month by name (e.g. "cal January 2017") and use relative placeholders, for example:

        cal now
        cal '1 year ago'
        cal '+2 months'

fdisk(8) allows to wipe newly created partitions -- the feature is possible to control by new command line option --wipe-partitions[==auto|never|default]. 
 The default in the interactive mode is to ask user when a filesystem or RAID signature is detected. The goal is to be sure that new block devices are usable without any collisions and extra wipefs(8) step (because users are lazy and mkfs-like programs are often no smart enough to wipe the device). 

findmnt --verify is probably the most attractive new feature for admins. The command scans /etc/fstab and tries to verify the configuration. The traditional way is to use "mount -a" for this purpose, but it's overkill. The new --verify does not call mount(2), but it checks parsability, LABEL/UUID/etc. translation to paths, mountpoints order, support for specified FS types. The option --verify together with --verbose provides many details. 

For example my ext4 filesystems:

# findmnt --verify --verbose -t ext4
   [ ] target exists
   [ ] LABEL=ROOT translated to /dev/sda4
   [ ] source /dev/sda4 exists
   [ ] FS type is ext4
   [W] recommended root FS passno is 1 (current is 2)
   [ ] target exists 
   [ ] UUID=c5490147-2a6c-4c8a-aa1b-33492034f927 translated to /dev/sda2
   [ ] source /dev/sda2 exists
   [ ] FS type is ext4
   [ ] target exists
   [ ] UUID=196972ad-3b13-4bba-ac54-4cb3f7b409a4 translated to /dev/sda3
   [ ] source /dev/sda3 exists
   [ ] FS type is ext4
   [E] unreachable on boot required target: No such file or directory 
   [ ] UUID=e8ce5375-29d4-4e2f-a688-d3bae4b8d162 translated to /dev/sda5
   [ ] source /dev/sda5 exists
   [ ] FS type is ext4
0 parse errors, 1 error, 1 warning

When you create multiple loop block devices from one backing file then Linux kernel does not care about possible collisions and the same on-disk filesystem is maintained by multiple independent in-memory filesystem instances. The result is obvious -- data lost and filesystem damage.

Now mount(8) rejects requests to create another device and mount filesystem for the same backing file. The command losetup --nooverlap reuse loop device if already exists for the same backing file. All the functionality calculate with offset and sizelimit options of course, so it's fine to have multiple regions (partitions) in the same image file and mount all of them in the same time. The restriction is that the regions should not overlap. Thanks to Stanislav Brabec from Suse! 

Heiko Carstens from IBM (thanks!) has improved lscpu(1) for s390. Now it supports "drawer" topology level, static and dynamic MHz, machine type and a new option --physical. 

The most important libsmartcols change is probably better support for multi-line cells. Now the library supports custom cell wrap functions -- this allows to wrap your text in cells after words, line breaks, etc. See multi-line cells (WRAPNL column) output: 

aaaa            1      0 aaa
├─bbb           2      1 bbbbb
│ ├─ee          5      2 hello
│ │                      baby
│ └─ffff        6      2 aaa
│                        bbb
│                        ccc
│                        ddd
├─ccccc         3      1 cccc
│ │                      CCCC
│ └─gggggg      7      3 eee
│   ├─hhh       8      7 fffff
│   │ └─iiiiii  9      8 g
│   │                    hhhhh
│   └─jj       10      7 ppppppppp
└─dddddd        4      1 dddddddd

The another change is support for user defined padding chars; we use this feature for LIBSMARTCOLS_DEBUG_PADDING=on|off, for example: 


For me really important is that we have regression tests for all libsmartcols table and tree formatting code now :-) 

Igor Gnatenko from Red Hat (thanks!) continues to work on Python binding for libsmartcols, see and see example below.

The idea is to use libsmartcols as output formatter for Fedora/RHEL dnf (package manager for RPM-based Linux distributions, yum replacement). This is also reason why libsmartcols has been massively extended and improved in the last releases. 

That's all. Thanks also to Werner Fink, Sami Kerola, Ruediger Meier and many others contributors! 

import smartcols
tb = smartcols.Table()
name = tb.new_column("NAME")
name.tree = True
age = tb.new_column("AGE")
age.right = True
ggf = tb.new_line()
ggf[name] = "John"
ggf[age] = "70"
gfa = tb.new_line(ggf)
gfa[name] = "Donald"
gfa[age] = "50"
fa = tb.new_line(gfa)
fa[name] = "Benny"
fa[age] = "30"
ln = tb.new_line(fa)
ln[name] = "Arlen"
ln[age] = "5"
ln = tb.new_line(fa)
ln[name] = "Gerge"
ln[age] = "7"
fa = tb.new_line(gfa)
fa[name] = "Berry"
fa[age] = "32"
ln = tb.new_line(ggf)
ln[name] = "Alex"
ln[age] = "44"
NAME        AGE
John         70
├─Donald     50
│ ├─Benny    30
│ │ ├─Arlen   5
│ │ └─Gerge   7
│ └─Berry    32
└─Alex       44

Tuesday, March 22, 2016

util-linux v2.28 -- what's new?

The release v2.28 does not contain any dramatical changes and huge improvements. It's another release to keep users happy and absorb new features provided by kernel. We all love this kind of release, right? :-)

The fdisk programs (sfdisk, cfdisk and fdisk) have been improved to wipe old filesystem, RAID and partition tables from the device before libfdisk writes a new partition table.

The fdisk-like programs traditionally care about begin of the device, but it's insufficient. This new feature has been introduced to avoid collisions between new partition table and old unwanted signatures and it's possible to control it by --wipe[=auto|never|always]. For backward compatibility on non-terminals (non interactive fdisk execution) the feature is disabled by default.

We have a new command lsns, see and sfdisk provides new functionality, see:
Linux kernel 3.14 is really not a hot news, but standard Linux userspace still does not support DEADLINE scheduler. chrt since v2.28 supports the DEADLINE scheduling class and the new options --sched-runtime --sched-period and --sched-deadline.

The command logger supports RFC 5424 structured data through the new options --sd-id and --sd-param. For example:
   logger --rfc5424 --sd-id zoo@123     \  
          --sd-param tiger=\"hungry\"   \            
          --sd-param zebra=\"running\"  \            
          --sd-id manager@123           \            
          --sd-param onMeeting=\"yes\"  \            
          "this is message"
 <13>1 2015-10-01T14:07:59.168662+02:00 ws kzak - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="218616"][zoo@123 tiger="hungry" zebra="running"][manager@123 onMeeting="yes"] this is message

The library libsmartcols has been massively improved (thanks to Igor Gnatenko for testing and reviews). Now it's possible to specify title for table, table supports multi-line cells, it's possible to print subset of table and the library supports continuous printing.

Igor works on Python binding, it's available at

The portability of the util-linux package is not our primary goal, but in many cases port code to the another libc or another operation system (if possible) is a way how to detect code disadvantages, obsolete functions etc. v2.28 is possible to compile on OSX and improved has been also support for kFreeBSD and GNU Hurd (of course you cannot compile Linux specific stuff, but build-system is smart enough to automatically disable utils irrelevant for your OS).

This is in connection with our regression tests suite, where many things have been improved to make the tests more stable in all random environments. (thanks to Ruediger Meier).

The complete list of all changes (~480 patches) is at

and in details at:

Thanks to all (~40) contributors!

Tuesday, December 1, 2015

lsns(8) new command to list Linux namespaces

The namespaces are commonly used way how to isolate global (ipc, mount, net, ...) resource instances. Unfortunately, we have no command line tool to list namespaces. The new command lsns(8) tries to fill this gap.


# lsns
4026531836 pid      276     1 root   /usr/lib/systemd/systemd --system --deserialize 15
4026531837 user     276     1 root   /usr/lib/systemd/systemd --system --deserialize 15
4026531838 uts      276     1 root   /usr/lib/systemd/systemd --system --deserialize 15
4026531839 ipc      276     1 root   /usr/lib/systemd/systemd --system --deserialize 15
4026531840 mnt      269     1 root   /usr/lib/systemd/systemd --system --deserialize 15
4026531857 mnt        1    63 root   kdevtmpfs
4026531963 net      275     1 root   /usr/lib/systemd/systemd --system --deserialize 15
4026532189 mnt        1   545 root   /usr/lib/systemd/systemd-udevd
4026532390 net        1   776 rtkit  /usr/libexec/rtkit-daemon
4026532478 mnt        1   776 rtkit  /usr/libexec/rtkit-daemon
4026532486 mnt        1   847 colord /usr/libexec/colord
4026532518 mnt        3  6500 root   -bash
and list namespace content:
# lsns 4026532518
 6500  6372 root -bash
19572  6500 root └─/usr/bin/mc -P /tmp/mc-root/mc.pwd.6500
19575 19572 root   └─bash -rcfile .bashrc
help output with columns description:
# lsns -h
 lsns [options] [namespace]
List system namespaces.
 -J, --json             use JSON output format
 -l, --list             use list format output
 -n, --noheadings       don't print headings
 -o, --output list      define which output columns to use
 -p, --task pid         print process namespaces
 -r, --raw              use the raw output format
 -u, --notruncate       don't truncate text in columns
 -t, --type name        namespace type (mnt, net, ipc, user, pid, uts)
 -h, --help     display this help and exit
 -V, --version  output version information and exit
Available columns (for --output):
          NS  namespace identifier (inode number)
        TYPE  kind of namespace
        PATH  path to the namespace
      NPROCS  number of processes in the namespace
         PID  lowers PID in the namespace
        PPID  PPID of the PID
     COMMAND  command line of the PID
         UID  UID of the PID
        USER  username of the PID
For more details see lsns(8).
The important detail is that you can see only namespaces accessible from currently mounted /proc filesystem. The lsns(8) is not able to list persistent namespaces without processes where the namespace instance is hold by bind mounts of the /proc/[pid]/ns/[type] files and the output may be affected by unshared PID namespace and unshared /proc (see unshare(8) for more details).

... it will be probably available in util-linux v2.28 (~ January 2016).

Tuesday, October 6, 2015

logger v2.28

logger is small util to send log messages from command line. It supports (relatively) a new systemd journal as well as classic syslog. The syslog is still de-facto standard for enterprise admins and the latest logger version add support for RFC5424. This RFC introduces "structured-data" and since v2.28 logger is going to support this feature too.

The structured data is parse-able part of the message in format:

 [SD-ID[@digits] SD-PARAM="value" SD-PARAM="value" ...]
for exmaple:
 <13>1 2015-10-01T14:07:59.168662+02:00 ws kzak - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="218616"] message
is complete message. The message structured data contains one element with ID "timeQuality". This is standardized element, the custom user defined elements have to use "@digits" suffix in the ID name. It's possible to have arbitrary number of the structured data elements.

And now this functionality is completely exported to logger command line to provide control over the elements, v2.28 is going to introduce two new options, --sd-id to specify structured data element ID and --sd-param to specify one SD-PARAM=value pair, for example:

 logger --rfc5424 --sd-id zoo@123                \
                  --sd-param tiger=\"hungry\"    \
                  --sd-param zebra=\"running\"   \
                  --sd-id manager@123            \
                  --sd-param onMeeting=\"yes\"   \
                  "this is message"   
  <13>1 2015-10-01T14:07:59.168662+02:00 ws kzak - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="218616"][zoo@123 tiger="hungry" zebra="running"][manager@123 onMeeting="yes"] this is message
message with three SD elements: timeQuality build-in element, zoo@123 and manager@123 user defined elements.

Now all you need is smart server side or log indexing tool that understand RFC5424 (e.g. rsyslog).

Thursday, September 17, 2015

what's cooking in sfdisk for v2.28

I have worked on new features for sfdisk during v2.27 stabilisation. The most user visible change is partitions resize/move improvements. Now (in git tree) sfdisk is able to move partition together with data on the partition. Let's imagine you have disk with three partitions:
Device       Start     End Sectors  Size Type
/dev/sdc1     2048  206847  204800  100M Linux filesystem
/dev/sdc2   206848 1230847 1024000  500M Linux filesystem
/dev/sdc3  1230848 2047966  817119  399M Linux filesystem
where the last partition is your ex-girlfriend /home and the first partition is almost full of pictures. Now you want to remove the last unnecessary partition and enlarge the first partition. And you don't want to lost data on the first and second partitions.

(You have backup of all data, because you understand that all operations related to the partition table are risky. Right? :-)

Let's delete the last partition:

  # sfdisk /dev/sdc --delete 3
Now move the second partition to the end of the device. The size of the deleted sdc3 was 817119 sectors. The sfdisk expects input in format
where "+num" means offset or size relative to the original partition setting. If any number is unspecified than default is to use the current setting -- it means that "+817119" is enough. (Note that -N 2 means second partition.)
# echo '+817119' | sfdisk /dev/sdc -N 2 --move-data
Old situation:
Device      Start     End Sectors  Size Type
/dev/sdc1    2048  206847  204800  100M Linux filesystem
/dev/sdc2  206848 1230847 1024000  500M Linux filesystem
New situation:                        

Device       Start     End Sectors  Size Type            
/dev/sdc1     2048  206847  204800  100M Linux filesystem
/dev/sdc2  1023967 2047966 1024000  500M Linux filesystem

Data move:                            
 typescript file: /root/sfdisk-sdc2.move
 old start: 206848, new start: 1023967 (move 1024000 sectors)
The important detail is --move-data option, it forces sfdisk after partition table modification copy data from old area to the new offsets. The source and target may overlap (sfdisk copy sectors in backward order if necessary). The file /root/sfdisk-sdc2.move is log with details about the change.

The last step is to enlarge the first partition to use all available free space (space originally used by the second partition). The string ",+" keeps start offset unchanged and the size is specified as "+" (all available space).

# echo ',+' | sfdisk /dev/sdc -N 1
Old situation:
Device       Start     End Sectors  Size Type
/dev/sdc1     2048  206847  204800  100M Linux filesystem
/dev/sdc2  1023967 2047966 1024000  500M Linux filesystem
New situation:
Device       Start     End Sectors  Size Type
/dev/sdc1     2048 1023966 1021919  499M Linux filesystem
/dev/sdc2  1023967 2047966 1024000  500M Linux filesystem
And now inform filesystem about a new space:
# resize2fs /dev/sdc1
and test all by mount:
# mount /dev/sdc1 /mnt/A
# mount /dev/sdc2 /mnt/B
# lsblk /dev/sdc
sdc      8:32   0 1000M  0 disk
├─sdc1   8:33   0  499M  0 part /mnt/A
└─sdc2   8:34   0  500M  0 part /mnt/B