System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.

html

System Administration Toolkit: Monitoring disk space and usage

Martin Brown (mc@mcslp.com), Freelance Writer, Freelance Developer

Summary: Look at methods for determining disk usage across multiple UNIX® systems and how to create a simple warning system to alert you of potential
problems. Keeping an eye on your file systems and ensuring they don't fill up is a trivial, but vital process in the day-to-day management of your UNIX systems.
In this article, you'll look at methods for keeping an eye on disk space, discovering which files, users, or applications are using up the most space, and how to
make use of quotas and other solutions to find the information you need.

Date: 13 Jun 2006

Level: Intermediate
Activity: 4665 views
Comments: 0 (Add comments)

Average rating (based on 71 votes)

About this series

The typical UNIX® administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key
utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the
tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools
across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment.

Getting started: using df

Monitoring disk space is a vital part of your job as a UNIX administrator. This article gives you the tools you need to be successful, including the use of df, du,
find, and even the use of quotas. Let's get started by taking a look at how useful df can be.

Use of the df tool can be a bit like a nervous twitch for many administrators, as it gives you a snapshot view of the storage space used and available across all of
your file systems through a single command. Depending on your UNIX environment, the default output from df might contain a range of different information.

Most modern df variants show disk space, usage and availability, and usually the mountpoint (and sometimes the device). For example, Mac OS X, based on
BSD, shows the information in Listing 1.

Listing 1. Default disk space information for Mac OS X

Filesystem 512-blocks Used Avail Capacity Mounted on

/dev/disk0s2 268435456 119741344 148182112 45% /
devfs 195 195 0 100% /dev
fdesc 2 2 0 100% /dev
<volfs> 1024 1024 0 100% /.vol
/dev/disk0s3 43474520 7251448 36223072 17% /Volumes/Untitled

Note that the size is displayed in 512 byte blocks, not kilobytes. Nearly all traditional UNIX operating systems report the size in 512 byte blocks rather than
kilobytes by default. To get the information in 1K blocks, use the -k command-line option, as shown in Listing 2.

Listing 2. Using the -k option with the df command

$ df -k
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/disk0s2 134217728 59870704 74091024 45% /
devfs 97 97 0 100% /dev
fdesc 1 1 0 100% /dev
<volfs> 512 512 0 100% /.vol
/dev/disk0s3 21737260 3625724 18111536 17% /Volumes/Untitled

Some variants might also support alternative block sizes, such as megabytes (use -m) and gigabytes (use -g), as demonstrated in Listing 3.

Listing 3. Using the -g option with the df command

$ df -g
Filesystem 1G-blocks Used Avail Capacity Mounted on
/dev/disk0s2 128 57 70 45% /
devfs 0 0 0 100% /dev
fdesc 0 0 0 100% /dev
<volfs> 0 0 0 100% /.vol
/dev/disk0s3 20 3 17 17% /Volumes/Untitled

Obviously, as you increase the block size shown, the level of detail in the information starts to reduce, but using alternative sizes can be a useful way to quickly
monitor disks, particularly very large ones. Other versions of the df tool (notably Solaris) report the bare block and file availability information by default. Most
file systems have a limited number of files that can be stored (the number is so large that you are unlikely to reach the limit), so it is possible to fill up a system
with files, but still have disk capacity and be unable to store any more files.

1 de 7 28/08/09 18:59
System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

You can see a sample of the default output of Solaris in Listing 4.

Listing 4. Default output of Solaris

$ df
/ (/dev/dsk/c0t0d0s0 ):14877208 blocks 914042 files
/devices (/devices ): 0 blocks 0 files
/system/contract (ctfs ): 0 blocks 2147483618 files
/proc (proc ): 0 blocks 16109 files
/etc/mnttab (mnttab ): 0 blocks 0 files
/etc/svc/volatile (swap ): 5737216 blocks 147177 files
/system/object (objfs ): 0 blocks 2147483532 files
/usr (/dev/dsk/c0t0d0s3 ): 9076010 blocks 863695 files
/dev/fd (fd ): 0 blocks 0 files
/var (/dev/dsk/c0t0d0s4 ): 8110796 blocks 483714 files
/tmp (swap ): 5737216 blocks 147177 files
/var/run (swap ): 5737216 blocks 147177 files
/export/home (/dev/dsk/c0t0d0s7 ):69362510 blocks 4272812 files

Using the -k command-line option switches the display to one similar to that shown with earlier examples, where the data is summarized into a more easily
readable format (see Listing 5).

Listing 5. Using the -k option

$ df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 7644629 206026 7362157 3% /
/devices 0 0 0 0% /devices
ctfs 0 0 0 0% /system/contract
proc 0 0 0 0% /proc
mnttab 0 0 0 0% /etc/mnttab
swap 2868600 1016 2867584 1% /etc/svc/volatile
objfs 0 0 0 0% /system/object
/dev/dsk/c0t0d0s3 8261237 3723232 4455393 46% /usr
fd 0 0 0 0% /dev/fd
/dev/dsk/c0t0d0s4 4130238 74849 4014087 2% /var
swap 2867584 0 2867584 0% /tmp
swap 2867624 40 2867584 1% /var/run
/dev/dsk/c0t0d0s7 35611388 930133 34325142 3% /export/home

All variants of df accept a directory or path, and then the information about that disk space for the file system that contains that path is displayed. For example, the
following shows you how to get the space on the root file system:

$ df -k /
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 7644629 206026 7362157 3% /

Or, the following uses df to show the space on a user's home directory:

$ df -k ~mc
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s7 35611388 930133 34325142 3% /export/home

Using the df output

The df tool is actually short for disk free, and that is precisely what the tool shows. If you look at some sample output again, you can extract the important
individual elements:

Filesystem kbytes used avail capacity Mounted on

/dev/dsk/c0t0d0s7 35611388 930133 34325142 3% /export/home

The first column, usually with a header according to the block size data, shows the total size of the disk. The used column shows the number of blocks that have
been used on the file system or device. The avail column shows the number of unused (available) blocks on the file system.

The capacity column gives a percentage calculation of how much disk space out of the total size has actually been used. On its own, this information can give a
quick indication of the space available.

By default, UNIX creates file systems that can be filled up to 90 percent capacity with data by users. The remaining 10 percent is reserved by the system for use
by root to perform any emergency maintenance. If a user tries to add or create a file that goes beyond that level, an error is returned (out of file system space).

That emergency maintenance space is there so it can be used to recover from an overfull disk. For example, as an administrator, you could use the space to create
a quick backup, or to compress existing data without having to use a secondary file system for the purpose. You can adjust the amount of free space that is
reserved (using the tunefs tool), or when creating the file system. This is vital for large disks, where a 10 percent buffer could equal many gigabytes of potential
space. It is generally a good idea to reserve at least one percent to give you a minimal buffer before you completely run out of space. Before you get that far,
especially if a lot of your disk space is being used, you should determine who is using all of that space.

2 de 7 28/08/09 18:59
System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

Using du

The du command, instead of showing disk free space, shows disk usage information. The du tool is used to determine the disk usage for files and directories. To
use it, change to a directory and run the tool (see Listing 6).

Listing 6. du command

$ cd /var
$ du
16 ./lost+found
4 ./sadm/install/admin
22 ./sadm/install/logs
28448 ./sadm/install
4 ./sadm/pkg/SUNWocfd/install
4 ./sadm/pkg/SUNWocfd/save/pspool/SUNWocfd/install
16 ./sadm/pkg/SUNWocfd/save/pspool/SUNWocfd
18 ./sadm/pkg/SUNWocfd/save/pspool
20 ./sadm/pkg/SUNWocfd/save
28 ./sadm/pkg/SUNWocfd
4 ./sadm/pkg/SUNWcsu/inst
...

The output, shown in Listing 6 above, has been trimmed. By default, du shows the file usage for every file and directory beneath the current, or specified,
directory. The value given is the size of the file in the default block size for that system, just as that used with df. This might, or might not, be 1K; you can force
the display to 1K blocks using the -k command-line option.

You will probably want to summarize the information by the top-level file, or directory, that you are examining. Use the -s option to switch on the summary view.
Here's a summarized version, shown in the /var directory, of a Solaris installation:

$ du -sk
70818 .

Note that it shows the summary information for the current directory (.). To get summary information for all of the files and directories, use the * wildcard (see
Listing 7).

Listing 7. Using the * wildcard to get summary information

$ du -sk *
382 adm
950 apache
683 apache2
6837 appserver
1 audit
162 cache
3 cc-ccr
2 crash
4 cron
31 dmi
22 dt
6 fm
2 imq
1 inet
3 krb5
4 ld
1 ldap
937 lib
6 log
8 lost+found
2 lp
2 mail
1 mysql
1 news
3 nfs
38 nis
2 ntp
10034 opt
1 preserve
96 run
49687 sadm
15 saf
3 samba
2 sma_snmp
131 snmp
39 spool
4 statmon
663 svc
14 tmp
10 uucp
24 yp

Be careful when using that tool, particularly in user directories, as the information shown will not include hiddenfiles -- in other words, those files and directories
with a single period prefix. You might want to use the following line in a user directory to get all of the summary information:

$ du -sk * .[a-zA-Z0-9]*

3 de 7 28/08/09 18:59
System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

One final very useful option for du is -d, which prevents du from following file system boundaries. For example, to determine the disk usage on the root file
system but not any other file systems, you would use the -d option:

$ du -dsk /

Some systems do not have this option but do have the -x command-line option, which only includes the file usage for files on the same device or file system as the
path you specify.

Finding disk usage for a specific user

To find the disk space being used by a single user, you need to combine du with the find command to only report disk usage for a specific user:

$ find . -user mc -type f -exec du -k {} \;

The -user option allows you to specify that find will only report files that are owned by the specified user. The -type option forces find to only return the path of
items of a specific type (in this case, files); this prevents du from including directories, which might be owned by one user, but contain files for many users. Then,
for each found path, the du command is executed to report the disk usage.

To get summary information, in other words, the total space used by a specific user, you can use awk to total the information and print out the final value:

$ find . -user mc -type f -exec du -k {} \;|awk '{ s =

s+$1 } END { print "Total used: ",s }'
Total used: 123721

You use the same principle with groups using the -group option to find:

$ find . -group mcslp -type f -exec du -k {} \;|awk '{ s = s+$1 } END { print
"Total used: ",s }'
Total used: 542485

There is, however, an easier way, if you enable disk quotas.

Using quotas

The quota system automatically monitors the disk usage for users on a file system by a file system basis. The quota environment not only enables you to monitor
disk usage, but you can also set limits for usage that provide a warning and, more explicitly, prevent users from using disk space over their assigned quota value.
The lower limit (warning) is called the soft limit and the higher limit (which prevents creation of files over that level) is called the hard limit. Some systems might
also allow you to control the number of files owned by each user.

The precise method required to enable quotas on your machine is dependent on the operating system you are using. Most UNIX systems include quota support by
default. Linux® systems might need to build a new kernel that includes quota support. Most, however, use a single file on each file system, usually called quotas,
into which you place the limits for each user.

To enable quotas, you should first create this file and ensure that the quotas file is only editable by root:

$ touch /export/home/quotas
$ chmod 600 /export/home/quotas

Then switch on quotas using the quotaon command:

$ quotaon /export/home

Finally, you must edit the appropriate quota for each user. You can do this by using the edquota command, specifying the user:

$ edquota mc

Your default editor (or vi, if you haven't set an alternative) will be opened with a simple form for setting quota values. You can see below that quotas have been
enabled on the users home directory file system with a soft limit of 200,000KB and a hard limit of 400,000KB. The settings of zero for file limits indicate that
there is no set limit and the user can set and create as many files as they like.

fs /export/home blocks (soft = 200000, hard = 400000) inodes (soft = 0,

hard = 0)

4 de 7 28/08/09 18:59
System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

If you have more file systems with quotas, then you will need to configure more rows for the file systems.

If you want to configure the quota for multiple users, first set the quota for a single user, then use the -p command-line option to edquota. This uses the specified
user as the basis for the new users. For example, to set the quota using the settings for mc, for usernames slp, tw, and sh, do the following:

$ edquota -p mc slp tw sh

File limit warnings with quotas

When the user creates a file that goes over their soft limit, they will get the following warning:

quota_ufs: Warning: over disk limit (pid 1738, uid 101, inum 94, fs /export/home)

Note that the user is given seven days to correct the problem; you can alter the duration by using edquota -t.

If the user tries to create a file that pushes them over the hard limit, the system terminates the write process and truncates the file to the limit:

$ mkfile 210000k overlimit

quota_ufs: over hard disk limit (pid 1843, uid 101, inum 130, fs
/export/home)
overlimit: initialized 191873024 of 215040000 bytes: Disc quota exceeded

Any user can check their own quota limits and disk usage by running the quota command:

$ quota
Over disk quota on /export/home, remove 199993K within 7.0 days

Quota administration

The systems administrator can check the quota of any user using the quota command; you should use the -v command-line option to provide a full report on the
file system, usage, and limit information (see Listing 8).

Listing 8. Using the -v option

$ quota -v mc
Disk quotas for mc (uid 101):
Filesystem usage quota limit timeleft files quota limit
timeleft
/export/home 399993 200000 400000 6.9 days 151 0 0

To get a report about disk and quota usage use on a file system basis detailing all users, use the repquota command, specifying the file system to report (see
Listing 9).

Listing 9. Using the repquote command

$ repquota -v /export/home
/dev/dsk/c0t0d0s7 (/export/home):
Block limits File limits
User used soft hard timeleft used soft hard timeleft
mc +- 399993 200000 400000 6.9 days 151 0 0

To ensure that the quota information is kept up to date, you should use the quotacheck command. This validates the file storage data with the quota information
used to report quota information. You should run this automatically using cron -- once a day should be fine (the process is comparatively time consuming).

Quota tricks and traps

The quota system provides the best combination of monitoring and automatic administration, but you should be careful about the file systems on which you
enable quotas and the limits that you set, as they might actually end up hindering the work of your users rather than controlling their disk usage.

For example, programmers might need significantly more space to actually build an application than you might have configured for their use. You can get around
this issue by providing an unrestricted area for compilation (by setting an alternative temporary directory) while retaining quotas on their home directories, or by
setting appropriate soft limits while setting the hard limits very high -- maybe even the maximum size of the file system.

The result should be a warning when the soft limit is reached, but no enforcement of the hard limit. Users should still be able to build and create temporary files
but, as they are naturally deleted during the build process, they should never be prevented from creating the files they need.

Quotas can also be used to help monitor and alert you to usage of special user accounts. I have, in the past, used quotas to monitor Web user accounts, so called
nobody accounts and others, to ensure that they do not have the ability to write files to file systems they should not otherwise have access to. To do this, set the
hard limit to 1KB for the file system and user you want to protect.

5 de 7 28/08/09 18:59
System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

Automatic monitoring

Monitoring disk space usage by hand is fine, but you don't want to be constantly running df (or even du) to determine the disk space being used, or that is
available. You can automate the process and automatically send the administrator (or an administration group) an e-mail when the available space gets below a
certain level. The script in Listing 10 monitors disk space, and you can set the warning (warninglimit) and emergency (lowlimit) limits, as well as the list of file
systems that are checked.

Listing 10. Monitoring disk space

#!/bin/sh

warninglimit=500000
lowlimit=250000

filesystems="/export/data /export/home /"

for fs in $filesystems
do
size=`df -k $fs|grep $fs|awk '{ print $4; }'`
if [ $size -le $lowlimit ]
then
mailx -s "URGENT: Low disk space for $fs ($size)"
break
fi
if [ $size -le $warninglimit ]
then
mailx -s "WARNING: Low disk space for $fs ($size)"
fi
done

The key is the line that extracts the amount of free disk space for each file system:

size=`df -k $fs|grep $fs|awk '{ print $4; }'`

The script uses df, extracts only the line you want with grep, and then uses awk to extract the fourth column of data, which is the amount of free space.

You can then check the free space against the warning/low limits and generate a suitable error. To prevent the system from raising both a low and a warning error,
the script checks for a low warning first and uses break to skip to the next file system in the loop before the warning test is tried.

Summary

Monitoring disk space is a vital part of your job as an administrator; run out of disk space and you could seriously impair the work of your users and, in severe
cases, lose data, or crash systems as they run out of the vital disk space required to operate.

You can determine the free space and usage for entire file systems using df, but this only gives you part of the picture. To determine where that disk space is being
used, you need to use the du tool to study individual directories. Through the use of find, you can even use du to find out how much space specific users are using.
For more automated user-level disk usage and control, quotas are a much better option.

Resources

Learn

System Administration Toolkit: Check out other parts in this series.

developerWorks Podcasts: Listen to interesting interviews and discussions for software developers.

IBM Redbooks: IBM has a Redbook on AIX for Sun Solaris Administrators that contains useful information on the differences between AIX from the
perspective of Solaris system administrators.

HP-UX 11i System Administration Handbook and Toolkit: Marty Poniatowski's (Prentice Hall PTR, ISBN: 0-13-060081-3) book contains information not
only about HP-UX, but also compares the HP-UX commands and generated output to other UNIX operating systems.

AIX® and UNIX articles: Check out other articles written by Martin Brown.

Search the AIX and UNIX library by topic:

System administration
Application development
Performance
Porting
Security
Tips
Tools and utilities
Java technology
Linux
Open source

AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and

6 de 7 28/08/09 18:59
System Administration Toolkit: Monitoring disk space and usage http://www.ibm.com/developerworks/aix/library/au-satdiskmon.html

expanding your UNIX skills.

New to AIX and UNIX: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.

AIX 5L™ Wiki: A collaborative environment for technical information related to AIX.

Safari bookstore: Visit this e-reference library to find specific technical resources.

developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.

Podcasts: Tune in and catch up with IBM technical experts.

Get products and technologies

IBM trial software: Build your next development project with software for download directly from developerWorks.

Discuss

Participate in the discussion forum.

Participate in the developerWorks blogs and get involved in the developerWorks community.

Participate in the AIX and UNIX forums:

AIX 5L -- technical forum
AIX for Developers Forum
Cluster Systems Management
IBM Support Assistant
Performance Tools -- technical
Virtualization -- technical
More AIX and UNIX forums

About the author

Martin Brown has been a professional writer for more than seven years. He is the author of numerous books and articles across a range of topics. His expertise
spans myriad development languages and platforms -- Perl, Python, Java™, JavaScript, Basic, Pascal, Modula-2, C, C++, Rebol, Gawk, Shellscript, Windows®,
Solaris, Linux, BeOS, Mac OS X and more -- as well as Web programming, systems management, and integration. He is a Subject Matter Expert (SME) for
Microsoft and regular contributor to ServerWatch.com, LinuxToday.com, and IBM developerWorks. He is also a regular blogger at Computerworld, The Apple
Blog, and other sites. You can contact him through his Web site.

Trademarks | My developerWorks terms and conditions

7 de 7 28/08/09 18:59