Howto rollover a file when size exceeds using unix find

Here is a one liner to rollover a file to file.old when it exceeds the size using find command. Lets say we have a script in cron that runs and prints messages in a log file. Overtime the log file will grow and we would want to rollover the log file to log.old. Many solutions exist by finding the size and comparing it. Here is one elegant solution in one liner. Thanks to my colleague Vlad who gave the idea for using find and exec, and I added the automatic substitution or brace expansion from my knowledge-base. Happy sharing of knowledge.

find /var/log/ -name myapp.log -size +1M -exec mv {}{,.old} ;

Continue reading

Unix : find affected by current working directory

On may Unix variants, find first looks for current working directory before proceeding with what it was asked to find. Ubuntu 10.10 and Debian Squezze not affected and I did not check older versions, but debian 5.0.6 or Lenny is affected and list includes Solaris 10 and Solaris 11 Express. It is very easy to fall in this pitfall if you have some automated package installation which may invoke some scripts for starting applications at the end of installation while cleaning up the temporary directory the package was running from. I wasted couple of hours in going over all my scripts to understand what was going on. The ls command was working but find was not able to get me the list of files to process from unrelated directories. So I ended up redirecting find’s error out to standard out and viola, solution presents itself. That redirection should have been on top of my list. It tells you that “find : cannot get the current working directory”. Why it needs that? I don’t know. Linux has this fixed for some time now, but for some reason SunOS is still using the old find variant including Solaris 11 Express which is the latest version out. Maybe some historical reasons. If anyone know, please share.

So the solution to the problem was that before invoking the command that will continue to run and may need to call find, start it in a directory that will persist after package installation is complete, e.g. / or /tmp.
Continue reading

SunOS : List dynamic libraries loaded by a running process or core

Both SunOS and Linux provides ldd which provides the list of shared objects or dynamic libraries that are required for the executable to run. It will tell which libraries will be required and from where they will be loaded when that executable is started. This is controlled by LD_LIBRARY_PATH and/or compilation flags. Here I will address how to find which objects were really picked when executable was started because what you see and what may be happening are two different things.

Take for example a running process that starts to misbehave or even crash. For troubleshooting it is a must that real dynamic libraries that were loaded by the application are found. ldd on the executable lists what it is supposed to use (based on our current environment) but not what that running process is currently using or it was using when it was running (if crashed). Maybe environment was setup differently when that executable was started and process loaded shared libraries from a different location. So ldd on the same binary when it started may be different than what it is currently, baffling the troubleshooter.

SunOS provides pldd (from the suite of /proc utilities) which lists the dynamic libraries that were loaded in reality when the process in question was started (or during its lifetime). pldd can also be used on a corefile to list the dynamic libraries that were used by that process. You simply run pldd with process Id or the the core to get the list of dynamic libraries that the process is using or was using before crash.

Continue reading

Most often used Network Sniffer flags for tcpdump and snoop

Most often used flags for tcpdump and snoop are below. Snoop is for SunOS while tcpdump is available for mostly all Unix/Linux kinds. We will discuss filters in another post. And to read back and process the captured files, I use wireshark which provides lots of options to deep dive into the packet stream.

Purpose snoop tcpdump Description
Select Interface -d <iface> -i <iface> Not needed if system has only one interface (ignoring localhost)
Capture full -s 0 -s 0 Snap length controls how much to capture. Zero means capture all.
Write capture to a file -o <filename> -w <filename>
Avoid DNS lookups for IPs in capture -r -n So no DNS lookups are performed when displaying real-time capture, good for efficiency

Continue reading

How to find number of CPUs on unix system

Newer processors are multi-core and could have hyper-threading enabled. So there are time when user may need to know how many cores (virtual) processors are available and how many Physical (real) processors are installed. CPU information can be retrieved via /usr/sbin/psrinfo on SunOS while it is available in /proc/cpuinfo on Linux. /proc/cpuinfo on Linux contains information about each available core. Following command combinations can be used to retrieve the number of CPUs.

OS Physical Processors Cores available
SunOS /usr/sbin/psrinfo -p /usr/sbin/psrinfo | wc -l
Linux grep “^physical id” /proc/cpuinfo | awk ‘{print $NF}’ | sort -u | wc -l grep processor /proc/cpuinfo | wc -l
IRIX hinv | grep -i processor | head -n1 | cut -d’ ‘ -f1

Update: Thanks Matias for tip on IRIX.

find broken symlinks

If you have a big project with multiple shared libraries, there is a chance to run into soft links that are no longer pointing anywhere. Here are couple of ways to find those dangling soft links that no longer point to any real file using the Unix find command.

These are in preferred order.

find /path/to/search -type l | (while read FN ; do /usr/bin/test -e "$FN" || echo "$FN"; done)
find /path/to/search -type l ! -exec /usr/bin/test -r {} ; -print
find -L /path/to/search -type l

So here is how these can be used to clean out those dangling soft links (again in preferred order). We use rm to ensure that any aliases don’t kick in. First one is the best way which is portable to most Unix platforms.

find /path/to/search -type l | (while read FN ; do /usr/bin/test -e "$FN" || rm -f "$FN"; done)
find -L /path/to/search -type l -exec rm -f {} ;
find -L /path/to/search -type l -delete

Note: When using in script, make sure to escape the properly.

Interoperability Issues between SunOS and Linux

-delete is not supported on SunOS find. It is available on GNU findutils. So using -exec to invoke rm command would be portable. As well as the test for existence /usr/bin/test -e is portable. Make sure to use /usr/bin/test instead of shell built in test because on sh the flag -e is not available.

Update (2011-08-12) : /usr/bin/test should be used for interoperability.

bash : Automate add/modify/delete of cron jobs from a script

If you have auto installing packages there could be times when a cron job needs to be added. So the script has to be able to create new cron entries or delete old ones. One solution is to create temporary files in between to hold the other unaffected cron entries that are currently installed, add the new entry and then install this file using crontab. Creating of temporary files should be avoided in between as there are risks. So here is an elegant solution which uses piping in the output of multiple commands.

To remove already existing cron job (rdate for user unixite in example below) use a syntax like

crontab -l -u unixite | grep -v rdate | crontab -u unixite -

This pipe chain lists the existing crontab entries, removes any containing the string rdate, then reloads the resulting data by piping it back to crontab of user unixite. “-” is for reading from the stream or terminal (see Note below). No useless temporary file, no security risk.

To add new crontab entries Continue reading