[Verifying a file on the system] [Verifying a Package] [Managing keys in RPM] [Verifying a Distribution] [Verifying a burned CD] [Checking GPG Signatures] [References]

How to verify Linux software from RPMs

(as used by RedHat, Fedora, CentOS, Scientific Linux and other RedHat derivitives)

Linux software is typically distributed via networks of non-commercial and possibly vulnerable servers, by sharing within the community, and by Peer-to-Peer transmission such as BitTorrent. Nevertheless, it is possible to trust software packages because they use cryptographically strong hashes and public-key signatures to authenticate the contents.

Verifying a file on the system

If you want to check the integrity of a command like ps (which shows running processes and is a favourite target of hackers, who like to modify it to hide their activity), you can first identify the system file with which, and then use rpm -qf to see which package it belongs to. Then you can use rpm -V to verify the package contents against the RPM database. E.g.
$ which ps
$ rpm -qf /bin/ps
$ rpm -V procps
S.5....T  /bin/ps
This shows that ps has been tampered with - the MD5 checksum (5) does not match, nor does the file modification time (T) or size (S).

The commands can be combined into one line, e.g.

$ rpm -Vf `which named`
S.5....T  c /etc/sysconfig/named
here, a "c" indicates that the file is a configuration file, that might be expected to be changed as normal practice.

It would be good practice to also verify loadable libraries:

$ ldd /bin/ps
        linux-vdso.so.1 => (0x00007fff0eba4000)
        libproc-3.2.7.so => /lib64/libproc-3.2.7.so (0x00000037cfc00000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00000037d0000000)
        libc.so.6 => /lib64/libc.so.6 (0x00000037cf800000)
        /lib64/ld-linux-x86-64.so.2 (0x00000037cf400000)
$ rpm -qf /lib64/libproc-3.2.7.so /lib64/libdl.so.2 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2
$ rpm -V glibc-2.5-58.el5_6.3.x86_64
and also of course ldd itself, and rpm
$ rpm -V rpm 
$ rpm -Vf `which ldd`

Verifying a Package

The process above verifies a file against the RPM database, but how do you know that the database itself can be trusted, and that trusted packages were installed in the first place ?

The packages and database contain a SHA1 digest, and usually also a GPG signature. If a bad package is forcibly installed (using --nosignature --nodigest) and then checked, error messages will be generated:

$ rpm -qf /usr/share/doc/anthy-7900/README 
error: rpmdbNextIterator: skipping h#    3369 Header V3 DSA signature: BAD, key ID 82fd17b2
$ rpm -qf /usr/share/doc/anthy-7900/README --nosignature 
error: rpmdbNextIterator: skipping h#    3369 Header SHA1 digest: BAD Expected(c0b6ac13a922be1b2d4690940f860a2cd44c4bf9) != (8fd4bce6fc2bd76a0184c9c7f49677edd81febea)
$ rpm -qf /usr/share/doc/anthy-7900/README --nosignature --nodigest
It is quite unlikely that someone would tamper with a digest, but it's possible that they would repackage a modified version of a standard program which could then be installed by a hacker or unwary administrator. RPM will normally refuse to install a package with a bad signature, but it issues only warnings for packages with an unknown or missing signature:
# rpm -hi abrt-1.1.0-1.fc13.x86_64.rpm 
warning: abrt-1.1.0-1.fc13.x86_64.rpm: Header V3 RSA/SHA256 signature: NOKEY, key ID e8e40fde
In most cases, RPM GPG keys are packaged with distribution media, e.g. in a file on a CD such as RPM-GPG-KEY-fedora. They are also installed in /etc/pki/rpm-gpg, from where they can be imported into GPG or RPM. E.g.
# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
$ gpg --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
gpg: key 217521F6: public key "Fedora EPEL <epel@fedoraproject.org>" imported

Managing keys in RPM

The list of keys installed in the RPM database may be generated by
$ rpm -qa gpg-pubkey*
Each individual key may be examined using rpm, e.g.
$ rpm -qi gpg-pubkey-217521f6-45e8a532 | egrep 'Version|Release|Summary'
Version     : 217521f6                          Vendor: (none)n|Release|Summary' 
Release     : 45e8a532                      Build Date: Wed Jul 22 18:11:11 2009
Summary     : gpg(Fedora EPEL <epel@fedoraproject.org>)
Note that the version (217521f6) of the RPM key matches the GPG public key (217521F6).

Note that RPM allows keys to be added to the database even if they are already present, causing duplicates to build up. To remove duplicate keys, use e.g.

rpm -e --allmatches gpg-pubkey-0b86274e-48b5dd6f
If desired, a separate GPG key database may be used for RPM keys than for personal keys, by using the homedir option. Likewise, the "root" option maybe used with rpm:
$ gpg --homedir /opt/rpmgpg --list-keys
$ rpm --root /opt/rpmgpg -qa

Verifying a Distribution

GPG keys included in a distrubution can be used to authenticate software updates and additional packages. But how do you know the distribution media is good ?

If you download an ISO image, there are usually GPG-signed hash files (md5sum.txt.asc, sha1sum.txt.asc) in the same directory. These may be verified using GPG, e.g.

$ gpg --verify-files md5sum.txt.asc
gpg: Signature made Sat 17 Dec 2011 06:39:47 AM PST using RSA key ID C105B9DE
gpg: Good signature from "CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org>"
gpg: WARNING: This key is not certified with a trusted signature!
while a corrupted file generates errors
$ gpg --verify-files md5sum.bad.txt.asc 
gpg: CRC error; F95A7F - 530B62
gpg: no signature found
$ gpg --verify-files md5sum.bad.txt.asc 
gpg: Signature made Sat 17 Dec 2011 06:39:47 AM PST using RSA key ID C105B9DE
gpg: BAD signature from "CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org>"
The downloaded image may then be checked against the hash, e.g.
$ sha1sum CentOS-6.2-x86_64-LiveDVD.iso
086f102c4c3ee228c5663cbaa2f1d6bf2edf5c4b  CentOS-6.2-x86_64-LiveDVD.iso
$ grep 086f102c4c3ee228c5663cbaa2f1d6bf2edf5c4b sha1sum.txt.asc 
086f102c4c3ee228c5663cbaa2f1d6bf2edf5c4b  CentOS-6.2-x86_64-LiveDVD.iso

Verifying a burned CD or DVD

You can check the volume and block sizes of a downloaded image using isoinfo, e.g.
$ isoinfo -d -i CentOS-6.2-x86_64-LiveDVD.iso | grep size
Volume set size is: 1
Logical block size is: 2048
Volume size is: 827854
Once you have burned a CD, you can then check the sizes for the burned image:
$ isoinfo -d -i /dev/cdrom | grep size
Volume set size is: 1
Logical block size is: 2048
Volume size is: 827854
A mismatch means there was a serious error writing the disk.

You may then generate a hash of the image optical medium. Hashing the entire device may generate spurious errors since padding may have been added while burning, so the block count may be restricted using dd E.g.

$ dd if=/dev/cdrom bs=2048 count=827854 conv=notrunc status=noxfer | sha1sum
086f102c4c3ee228c5663cbaa2f1d6bf2edf5c4b  -
This hash should match that of the ISO image (and the signed published hashfile)

Checking GPG Signatures

So you have a good GPG signature on a package, but how do you know that the signature is genuine and not the work of an impostor ?

GPG (like PGP before it) uses the concept of key signing. At a grassroots level, it can use the six degrees of separation idea that a package author is connected to you via someone you know personally. So some package keys are signed personally by the authors, whose keys are available on public keyservers and are in turn signed by other software developers, colleagues and friends who are able to assert their identity. It is also possible to sign a distribution key with a master key, or with a key from an earlier distribution that you may have already installed. E.g.

$ gpg --homedir .gnupg --list-sigs 4F2A6FD2
pub   1024D/4F2A6FD2 2003-10-27
uid                  Fedora Project <fedora@redhat.com>
sig 3        4F2A6FD2 2003-10-27  Fedora Project <fedora@redhat.com>
sig 3        DB42A60E 2003-10-27  Red Hat, Inc <security@redhat.com>
sig          8DF56D05 2003-10-28  Fedora Linux (RPMS) <security@fedora.us>
sig          2048C528 2004-03-30  [User ID not found]
sig          220A3F8F 2004-05-17  [User ID not found]
sig          11E60E88 2004-08-07  Ghigheanu Tudor (BadZen) <badzen@as.ro>

$ gpg --homedir .gnupg --list-sigs 8DF56D05
pub   1024D/8DF56D05 2003-03-27
uid                  Fedora Linux (RPMS) <security@fedora.us>
sig          F1A2DED5 2003-03-30  Eric Harrison <eharrison@mesd.k12.or.us>
sig          3ED6F034 2003-03-31  Adrian Reber <adrian@lisas.de>
sig          AD2CEC66 2003-04-16  [User ID not found]
sig          BCD241CB 2003-06-03  Ville Skytt� <ville.skytta@iki.fi>

Sometimes you can follow this process for dozens of steps to thousands of signatures. Even if you don't find someone you know, the statistics look good - the odds of someone tampering with keys from hundreds of individuals at different locations using different operating systems and working for different companies are very low.

In other cases, packages keys are just self-signed and published on a company website, such as the Fedora Project. Standard Web techniques for verifying site identity should be used (look for SSL, check the certificate). SSL uses a hierarchical trust model rather than a peer-based one; trust is ultimately guaranteed by the browser vendor (Mozilla, Microsoft, Google etc.) who pre-load public certificates of trusted certificate authorities. Note that the fact that a site identity is verified is independant of whether the content can be trusted (it might have been hacked).

In some cases, key fingerprints are published. These can be manually checked against fingerprints generated in GPG, e.g.

$ gpg --homedir .gnupg/ --fingerprint 22B3B81A
pub   4096R/22B3B81A 2012-01-10
      Key fingerprint = 37B8 48AB 4B93 16CC 2006  A9AD 0983 1293 22B3 B81A
uid                  Fedora (18) <fedora@fedoraproject.org>
Fingerprints for an individual may also be checked over the phone.

If GPG is properly configured, missing keys may be downloaded from a keyserver, e.g.

$ gpg --homedir .gnupg --recv-keys 2048C528
gpg: requesting key 2048C528 from hkp server ice.mudshark.org
gpg: key 2048C528: public key "Justin Blumer <justinblumer@gmail.com>" imported
In this case, the file .gnupg/options contains an entry "keyserver ice.mudshark.org"


March 2012 A.Daviel