Syntax for grep to search for a pattern in a file is very well-known. But there are times when one has to grep for the pattern from a certain location or after a certain offset in the file. For example if we are searching for a pattern in a log file which could appear multiple times. Each time we grep, it will provide us all the matching lines from top to bottom of the file and then we have to find which lines were new since our last run. Using dd, the file can be sliced and then grep can be applied for the pattern on that slice. Lets see an example. Continue reading “bash : grep for pattern from certain location in the file”
bash : search multiple file patterns using single find command
One reader asked how find can be used to find various file patterns. For example in a directory which could be littered with various logs and other files, how do I use a single find command to find all shell scripts, perl scripts and say php scripts. Simple answer is to use multiple -name arguments combined with -o (or ORed) and if needed with -a (or ANDed). Other find conditions (like -mtime, -type etc) can be combined as well. Continue reading “bash : search multiple file patterns using single find command”
Howto rollover a file when size exceeds using unix find
Here is a one liner to rollover a file to file.old when it exceeds the size using find command. Lets say we have a script in cron that runs and prints messages in a log file. Overtime the log file will grow and we would want to rollover the log file to log.old. Many solutions exist by finding the size and comparing it. Here is one elegant solution in one liner. Thanks to my colleague Vlad who gave the idea for using find and exec, and I added the automatic substitution or brace expansion from my knowledge-base. Happy sharing of knowledge.
find /var/log/ -name myapp.log -size +1M -exec mv {}{,.old} ;
Continue reading “Howto rollover a file when size exceeds using unix find”
bash : Count number of recurrence of lines
Say, we have a file or data that has many duplicate rows or entries and we want to find how many time each one has repeated and maybe want to know which is repeated most of the time. Here is an elegant script that can do that in single line.
sort input.file | uniq -c | sort -n -r
Explanation:
First sort will sort the records in the file. Then uniq -c will count how many times each record is duplicated. And finally sort -n -r will sort the output of uniq -c in reverse order giving us the records that repeated most often to the least often.
Continue reading “bash : Count number of recurrence of lines”
bash : Self redirect of Script’s output to a file
Everyone knows how to use redirection operators to send output of a script to a file. Simply use “>” or “>>” on command prompt after the script or application name and it does the magic of storing the output of the script to the file of choice. But what if one wants that their script shall create its own file and store the output there. In other words, self redirect of output.
Lets take a use case. Imagine you are developing a script that will bring up some applications when a system reboots. When it is run from command line it works fine but when it is run during system startup, it misbehaves. How do you troubleshoot? First response is to use “-x” to print how the script is triggering. But what we want is that when our startup script runs, all of its output (both stdout and stderr) stored in a file that we could use for troubleshooting later. For redirecting output of a startup script we have to use a wrapper script to trigger it and use redirection operator to store its output which is simply a workaround. Or, use the magic word “exec“.
Continue reading “bash : Self redirect of Script’s output to a file”
How to find number of CPUs on unix system
Newer processors are multi-core and could have hyper-threading enabled. So there are time when user may need to know how many cores (virtual) processors are available and how many Physical (real) processors are installed. CPU information can be retrieved via /usr/sbin/psrinfo on SunOS while it is available in /proc/cpuinfo on Linux. /proc/cpuinfo on Linux contains information about each available core. Following command combinations can be used to retrieve the number of CPUs.
| OS | Physical Processors | Cores available |
|---|---|---|
| SunOS | /usr/sbin/psrinfo -p | /usr/sbin/psrinfo | wc -l |
| Linux | grep “^physical id” /proc/cpuinfo | awk ‘{print $NF}’ | sort -u | wc -l | grep processor /proc/cpuinfo | wc -l |
| IRIX | hinv | grep -i processor | head -n1 | cut -d’ ‘ -f1 |
Update: Thanks Matias for tip on IRIX.
bash : Signal Handling
A script could have lock file or is producing temporary files which need to be renamed before exiting. What if the script receives a signal and quits in the middle leaving temporary files scattered around? Bash provides “trap” function to trap a signal and right a signal handler.
if [ ! -e $lockfile ]; then
# Arm the handler
trap "rm -f $lockfile; exit" INT TERM EXIT
touch $lockfile
critical-section
rm $lockfile
# Disarm the handler
trap - INT TERM EXIT
fi
Or you can write a function like below.
Handle_Exit_Safely()
{
rm -f $lockfile
exit
}
if [ ! -e $lockfile ]; then
# Arm the handler
trap Handle_Exit_Safely INT TERM EXIT
touch $lockfile
critical-section
rm $lockfile
# Disarm the handler
trap - INT TERM EXIT
fi
find broken symlinks
If you have a big project with multiple shared libraries, there is a chance to run into soft links that are no longer pointing anywhere. Here are couple of ways to find those dangling soft links that no longer point to any real file using the Unix find command.
These are in preferred order.
find /path/to/search -type l | (while read FN ; do /usr/bin/test -e "$FN" || echo "$FN"; done)
find /path/to/search -type l ! -exec /usr/bin/test -r {} ; -print
find -L /path/to/search -type l
So here is how these can be used to clean out those dangling soft links (again in preferred order). We use rm to ensure that any aliases don’t kick in. First one is the best way which is portable to most Unix platforms.
find /path/to/search -type l | (while read FN ; do /usr/bin/test -e "$FN" || rm -f "$FN"; done)
find -L /path/to/search -type l -exec rm -f {} ;
find -L /path/to/search -type l -delete
Note: When using in script, make sure to escape the properly.
Interoperability Issues between SunOS and Linux
-delete is not supported on SunOS find. It is available on GNU findutils. So using -exec to invoke rm command would be portable. As well as the test for existence /usr/bin/test -e is portable. Make sure to use /usr/bin/test instead of shell built in test because on sh the flag -e is not available.
Update (2011-08-12) : /usr/bin/test should be used for interoperability.
bash : Automate add/modify/delete of cron jobs from a script
If you have auto installing packages there could be times when a cron job needs to be added. So the script has to be able to create new cron entries or delete old ones. One solution is to create temporary files in between to hold the other unaffected cron entries that are currently installed, add the new entry and then install this file using crontab. Creating of temporary files should be avoided in between as there are risks. So here is an elegant solution which uses piping in the output of multiple commands.
To remove already existing cron job (rdate for user unixite in example below) use a syntax like
crontab -l -u unixite | grep -v rdate | crontab -u unixite -
This pipe chain lists the existing crontab entries, removes any containing the string rdate, then reloads the resulting data by piping it back to crontab of user unixite. “-” is for reading from the stream or terminal (see Note below). No useless temporary file, no security risk.
To add new crontab entries Continue reading “bash : Automate add/modify/delete of cron jobs from a script”
bash : brace expansion for automatic substitution
Bash expands a list of strings separated by commas with in braces to a list of strings separated by spaces. For example
[unixite@theunixtips:~/> echo 111-{aa,bb,cc}+{xx,yy,zz}-222
111-aa+xx-222 111-aa+yy-222 111-aa+zz-222 111-bb+xx-222 111-bb+yy-222 111-bb+zz-222 111-cc+xx-222 111-cc+yy-222 111-cc+zz-222
unixite@theunixtips:~/>
There should not be any space between suffix/prefix and the start/end bracket. Otherwise it is considered as a separate list. Now coming to a real world example where you can make use of this.
mv process.log{,.old} #This will move process.log to process.log.old
mv process.log.{old,oldest} #This will move process.log.old to process.log.oldest