Thursday, October 19, 2017

util-linux v2.31 -- what's new?

uuidparse -- this is a new small command to get more information about UUIDs "hash". The command provides info about UUID type, variant and time. For example:

$ (uuidgen; uuidgen -t) | uuidparse
UUID                                  VARIANT TYPE       TIME
8f251893-d33a-40f7-9bb3-36988ec77527  DCE     random
66509634-b404-11e7-aa8e-7824af891670  DCE     time-based 2017-10-18 15:01:04,751570+0200

The command su has been refactored and extended to create pseudo terminal for the session (new option --pty). The reason is CVE-2016-2779, but the issue addressed by this CVE is pretty old and all the problem is silently ignored for for years on many places (on only su(1)). The core of the problem is that unprivileged user (within su(1) session) shares terminal file descriptor with original root's session. The new option --pty forces su(1) to create independent pseudo terminal for the session and than su(1) works as proxy between the terminals. The feature is experimental and not enabled by default (you have to use su --pty).

standard su session (all on pts/0):
               
24909 pts/0    S      0:02          \_ -bash                  
13607 pts/0    S      0:00              \_ su - kzak          
13608 pts/0    S      0:00                  \_ -bash          
13679 pts/0    R+     0:00                      \_ ps af      

su --pty session (root pts/0; user pts/5):
               
24909 pts/0    S      0:02          \_ -bash                  
13857 pts/0    S+     0:00              \_ su --pty - kzak    
13858 pts/5    Ss     0:00                  \_ -bash          
13921 pts/5    R+     0:00                      \_ ps af      

rfkill -- this is a new command in util-linux. The command was originally written by Johannes Berg and Marcel Holtmann and maintained for years as standalone package. We believe that it's better to maintain and distribute it with another commands on one place. The util-linux version is backwardly compatible with the original implementations. The command has been also improved (libsmartcols ouotput, etc.), the new default output:
# rfkill       
ID TYPE      DEVICE                   SOFT      HARD          
 0 bluetooth tpacpi_bluetooth_sw unblocked unblocked          
 1 wlan      phy0                unblocked unblocked          
 4 bluetooth hci0                  blocked unblocked

The library libuuid and command uuidgen support hash-based UUIDs v3 (md5) and v5 (sha1) as specified by RFC-4122 now. The library also provides UUID templates for dns, url, oid, or x500. For example:
 
$ uuidgen --sha1  --namespace @dns --name foobar.com
e361e3ab-32c6-58c4-8f00-01bee1ad27ec

and it's expected to use v3 and v5 UUIDs as hierarchy, so you can use this UUID (or arbitrary other UUID) as a namespace:
 
$ uuidgen --sha1  --namespace e361e3ab-32c6-58c4-8f00-01bee1ad27ec --name mystuff
513f905c-7df2-5afa-9470-4e82382dbf00

I can imagine system where for example per-user or per-architecture partition UUIDs are based on this system. For example use UUID specific for the system root as --namespace and username as --name, or so. 

wipefs and libblkid have been improved to provide all possible string permutations for a device. It means that wipefs does not return the first detected signature, but it continues and tries another offsets for the signature. This is important for filesystems and partitions tables where the superblock is backuped on multiple places (e.g. GPT) or detectable by multiple independent ways (FATs). This all is possible without a device modification (the old version provides the same, but only in "wipe" mode). 

The libfdisk has been extended to use BLKPG ioctls to inform the kernel about changes. This means that cfdisk and fdisk will not force your kernel to reread all of the partition table, but untouched partitions may remain mounted and used by the system. The typical use-case is resizing the last partition on the system disk. 

You can use cfdisk to resize a partition. Yep, cool.

The hwclock command now significantly reduces system shutdown times by not reading the RTC before setting it (except when the --update-drift option is used). This also mitigates other potential shutdown and RTC setting problems caused by requiring an RTC read.



Friday, June 9, 2017

util-linux v2.30 -- what's new?

The command tailf is dead thing. (RIP ... years ago I had nice time to improve it with inotify:) You have to use "tail -f" from coreutils project.

blkzone -- this new command is excellent example of the open source collaboration. The command has been developed by people from WD, Seagate and SanDisk (thanks to Shaun Tancheff, Damien Le Moal and others). The goal is to have command line interface to run zone commands on block devices that support Zoned Block Commands (ZBC) or Zoned-device ATA Commands (ZAC). For now the supported zone commands are "reset" and "report". See http://www.storagereview.com/methods_of_smr_data_management for more details about zones.

fincore (file in core)-- this is nice useful command to get information about number of memory pages used by file content. For example my fulltext email DB:

# fincore ~/Mail/Maildir/.notmuch/xapian/*.DB
   RES  PAGES  SIZE FILE
 60.1M  15392  4.6G /home/kzak/Mail/Maildir/.notmuch/xapian/position.DB
687.4M 175982  3.5G /home/kzak/Mail/Maildir/.notmuch/xapian/postlist.DB
  328K     82 18.6M /home/kzak/Mail/Maildir/.notmuch/xapian/record.DB
190.5M  48758  2.1G /home/kzak/Mail/Maildir/.notmuch/xapian/termlist.DB

Fortunately RAM is cheap :) Thanks to Masatake Yamato from Red Hat.

lsmem (list memory) and chmem (change memory) -- another new commands. The commands have been originally implemented in Perl for s390-tools, now re-implemented in C in more generic way and to be usable on another architectures too. (thanks to Clemens von Mann and Heiko Carstens from IBM.)

The command fallocate supports an "insert range" operation now.

We continue on hwclock cleanup, some things in the code have been simplified, dead and useless things removed. (thanks to J William Piggott)

The code behind "column -t|--table" uses libsmartcols now. This change dramatically increased number of available features for table formatting. Now it's possible to define header for columns, truncate text in cells, align text to the right, change order of columns, JSON output or create tree-like output. Now almost all libsmartcols features are available on command line, example:
pstree-like output:

 $ ps -h -o pid,ppid,comm | column --table --tree 3 --tree-id 1 --tree-parent 2 --table-hide 2 --table-right 1
 
 1799  bash
 2254  bash
28427  └─mutt
 4263    └─vim
 7409  bash
10641  └─man
10657    └─less
16775  bash
11486  ├─ps
11487  └─column

diskstat:
 
 $ column /proc/diskstats --table --table-columns MAJ,MIN,NAME,READ-COMP,\
      READ-MERG,READ-SECS,READ-TIME,WRITE-COMP,WRITE-MERG,WRITE-SECS,\
      WRITE-TIME,IO-CURR,IO-TIME,WTIME \
      --table-hide MAJ,MIN \
      --table-right 4,5,6,7,8,9,10,11,12,13,14 \
 
NAME   READ-COMP  READ-MERG   READ-SECS  READ-TIME  WRITE-COMP  WRITE-MERG  WRITE-SECS  WRITE-TIME  IO-CURR   IO-TIME      WTIME
sda     13486466     149085  1288469300    9715620    45556082     7788088  1600182109   150180178        0  12935701  159902109
sda1         463        170       19002        131          91           0         161         331        0       334        462
sda2         778         16       63140        276         434         261      507574       12616        0      2382      12889
sda3    10710224     109592  1052352266    8018950    43983768     7022717  1153182094   126210185        0  11002854  134299501
sda4     1630396      32476    67166050    1039837     1197142      665798   343331344    23264993        0   2148041   24306932
sda5     1140435        241   168747746     655625      225373       73891   102920032      637906        0    627834    1293105
sda6        3703       6590       99512        691        4691       25421      240904        8418        0      6402       9108
sdb          448          0       22506       3088        1887           4         128         275        0      1449       3363
sdb1         404          0       19370       3035          12           4         128          60        0      1187       3095
loop0      22086          0      347311       2025       10738           0      844888        2226        0      1129       4265
loop1        947          0       26940        325        1100           0      133316         734        0       411       1058
md8            0          0           0          0           0           0           0           0        0         0          0


passwd in JSON:
 
 $ grep -v nologin /etc/passwd | \
     column --separator : --table --table-name passwd --json \
            --table-columns USERNAME,PWD,UID,GID,GECOS,HOME,SHELL \
            --table-hide PWD
{
   "passwd": [
      {"username": "root", "uid": "0", "gid": "0", "gecos": "root", "home": "/root", "shell": "/bin/bash"},
      {"username": "sync", "uid": "5", "gid": "0", "gecos": "sync", "home": "/sbin", "shell": "/bin/sync"},
      {"username": "shutdown", "uid": "6", "gid": "0", "gecos": "shutdown", "home": "/sbin", "shell": "/sbin/shutdown"},
      {"username": "halt", "uid": "7", "gid": "0", "gecos": "halt", "home": "/sbin", "shell": "/sbin/halt"},
      {"username": "kzak", "uid": "1000", "gid": "1000", "gecos": "Karel Zak,Home,,,", "home": "/home/kzak", "shell": "/bin/bash"},
      {"username": "gamer", "uid": "1001", "gid": "1001", "gecos": null, "home": "/home/gamer", "shell": "/bin/bash"},
      {"username": "test", "uid": "1002", "gid": "1002", "gecos": null, "home": "/home/test", "shell": "/bin/bash"}
   ]
} 
 

findmnt-like output:
 
 $ column /proc/self/mountinfo \
     --table-columns ID,PARENT,MAJMIN,ROOT,TARGET,VFS-OPTS,PROP,SEP,TYPE,SOURCE,FS-OPTS \
     --table-hide=SEP,ID,PARENT,ROOT,PROP,FS-OPTS,MAJMIN \
     --table-order TARGET,SOURCE,TYPE,VFS-OPTS \
     --tree TARGET \
     --tree-id ID \
     --tree-parent PARENT
 
TARGET                             SOURCE       TYPE         VFS-OPTS
/                                  /dev/sda4    ext4         rw,relatime
├─/sys                             sysfs        sysfs        rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security           securityfs   securityfs   rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                 tmpfs        tmpfs        ro,nosuid,nodev,noexec
│ │ ├─/sys/fs/cgroup/systemd       cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/blkio         cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/cpu,cpuacct   cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/devices       cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/hugetlb       cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/pids          cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/memory        cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/cpuset        cgroup       cgroup       rw,nosuid,nodev,noexec,relatime
... and so on ... 
 
 
Thanks to all contributors. The next version v2.31 is planned for September 2017.

Friday, January 6, 2017

10 years with util-linux project!

I can't believe that it's already 10 years we have active community around basic Linux utils.

Yes, we had util-linux before (and many thanks to Adrian Bunk and Andries E. Brouwer), but I believe that with git and close collaboration between Linux distributions and Linux kernel community it better now :-)
  • ~11000 commits
  • ~460 unique contributors
  • ~630 regression tests
  • ~100 utils, 5 shared libs with public API
  • ~16 major releases (from v2.13 to v2.29)
  • ~26 translated languages
  • ~10000 e-mails on mailing list

  • ported to GNU Hurd, FreeBSD and XOS
  • used by all mainstream Linux distros (as well as by Systemd haters ;-)
  • modern autotools based build system

  • merges from another projects
    • libblkid, libuuid and fsck from e2fsprogs
    • sulogin, last, utmpdump  and mounpoint from sysvinit
    • su from coreutils

  • many new utils (lsblk, findmnt, wipefs, rtcwake, unshare, nsenter, prlimit, blkdiscard, flock, fstrim, ipcmk, ldattach, lscpu, lsipc, lslocks, lslogins, resizepart, setarch, setpriv, switch_root, swaplabel)

  • many rewrites (libblkid probing code, mount, fdisks, etc.)

  • new shared libraries
    • libmount
    • libfdisk
    • libsmartcols

...the first original announce: https://lkml.org/lkml/2006/12/18/12

... gource video with all the ten years https://youtu.be/YcG6hHGD-78 :-)

Thanks to all contributors!