Compare files ignoring a field or column using Process Substitution

Lets say the data contains multiple fields/columns separated by space or comma or some other delimiter. And we want to compare two files ignoring a specific column. Lets divide work in two small issues. First is to ignore the provided field/column.

If we simply want to ignore the first column, we can use one of the following cut constructs.

cut -d',' -f 1 --complement datafile
cut -d',' -f 2- fileName.csv

If we want to ignore a specific one we can use awk in following manner which is much more generalized because you can specify which column to ignore, be it first, third or last.

This can be used as

awk -F',' -v FieldToIgnore=3 -f ignoreField.awk datafile

Next part is to diff the output after ignoring (read removing) the column. That is where process substitution comes handy. Here are two examples.

# ignore 1st column from two csv datafiles while comparing
diff -u <(cut -d, -f 2- datafile1) <(cut -d, -f 2- datafile2)
# ignore column 3 from two csv datafiles while comparing
diff -u <(awk -F',' -v FieldToIgnore=3 -f ignoreField.awk datafile1) <(awk -F',' -v FieldToIgnore=3 -f ignoreField.awk datafile2)

So instead of giving it two real files, we give it two redirected streams. Same solution can be used to pre-process files differently (e.g. ignore any comments or empty lines or compare two unsorted files).

See below for more information on Process Substitution.
http://www.tldp.org/LDP/abs/html/process-sub.html
http://wiki.bash-hackers.org/syntax/expansion/proc_subst

Application Logging Improvement – Part 3 Making it Readable

This is part three of my Application Logging improvement plan. So far I have discussed that log should be machine readable for application performance, management and monitoring. In this post I give an example of how to make the log readable to human (or make the log just like everyone has been used to seeing them). I am going to use vim to view the log files and have it configured so it knows how to handle the file with syntax etc.

First thing is configure vim to recognize the format. Continue reading

Application Logging Improvement – Part 2 Multithreading

Multi-threading is now becoming a norm. Obvious issue with logging is how to synchronize between threads. As discussed in last post Application Logging Improvement Plan – Part 1, we want to log as much as possible in machine readable format. So there comes a problem with multiple threads trying to log at the same time. Two possible implementations come to mind but both are flawed.

  1. Synchronize between threads for logging – Disk writes are slow and now locking contention would only make it worse. This slows down the business logic and is a big no-no.
  2. Log without synchronizing – Business logic works but logs get jumbled up because multiple threads are trying to log at the same time. This leaves logs in worst shape and unusable.

We can do better by combining both of above to get a solution. We will create a per thread logging buffer (lets call it LogBuffer) where each thread would log without any conflicts. And at a certain threshold, threads synchronize and log their LogBuffer to the disk (lets call this Flush).  Continue reading

Application Logging Improvement Plan – Part 1

People are divided on how to log, what to log, how much to log. A never ending discussion this is. In addition many open source libraries are available for logging. Not to mention many standards. I am not going to go in details of what is available out there. Use Google to pick your poison. What I am going to discuss here is what I think makes most sense with available technology.
Continue reading

mimikatz : Export non-exporteable Private certificate from Symantec PKI

Recently our organization started to provision Private certificates using Symantec Managed PKI Service. It has lot more appeal for IT admins because it takes out all user intervention which always creates support nightmares.

Previously I had direct access to the private key so it was easy to export it to all my devices and use for VPN and other secure stuff that needed to verify that I am indeed the real user. Because Symantec PKI is not available for Linux, it broke the VPN access from my Ubuntu system. Naturally I started to look for ways to export the key out of windows system. So here is what I did to get me out of the bind.

How to export certificates

First I installed Symantec PKI client on a windows 7 system. That was a no brainer because there was no other choice. I did not try with Windows 8 so YMMV. The main issue was that Windows certificate manager showed that the private key was not exportable. If it was then my quest would have been over right there. But I had to take another step. Mimikatz was the answer which marks them exportable and also allows to export them. Note: The patching that it does only lasts for that session. Once you reboot windows system you have to patch again using mimikatz. I used latest version which is 2.0 at the writing of this post. Continue reading

Setup WebEx on 64 bit Ubuntu 12.04 using 32 bit Oracle Java

WebEx would not work on Ubuntu 12.04 64 bit with default configuration. It requires 32 bit java. WebEx control window would launch but desktop sharing, application sharing, white-board etc. do not show up. Neither I could see other people’s shared content nor I could share mine even if I am the host of the meeting.

Starting Firefox from command line on a terminal shows ELFCLASS32 error from WebEx shared objects. So it was clear that WebEx would not work on 64 bit system as is and would need 32 bit java to work. Because I use 64 bit system I do not want to downgrade to a 32 bit version just for the sake of WebEx.

In brief, these three steps cover the fix.

  1. Install 32 bit Oracle Java locally. Oracle Java is must and OpenJDK would not cut. Warning: because it is local installation, user would need to manually keep on updating as new java becomes available. Recently there have been many releases from Oracle which came with very little time in between addressing major security issues so this would be concerning.
  2. Install Firefox locally so it can be configured to use this 32 bit java. Add a different profile and use a different theme so it does not conflict with the native Firefox and clearly stands out if both are running.
  3. (Optional) Add shortcut in Unity HUD for quick access.
    Continue reading

Add a custom script in Unity HUD

Here is how to add a custom script in Unity HUD/dash for quick access. /usr/share/applications directory has all shortcuts for Unity desktop. So create a file named “mycustomscript.desktop” (or any_name_you_like.desktop) there which has information about the custom script. Additionally an icon could be added by pointing to an image. Files in /usr/share/applications directory have to be created as root.

 [Desktop Entry]
 Version=1.0
 Type=Application
 Terminal=false
 StartupNotify=true
 Icon=/home/vishalj/Pictures/mycustomscript.jpeg
 Name=MyCustomScript
 Comment=My Custom Script for X, Y and Z
 Exec=/home/vishalj/scripts/mycustomscript.sh
 Categories=Application;Productivity
 

Then run sudo update-desktop-database after which you will be able to use Unity HUD for invoking the custom script. Also note that each time you update a .desktop file you have to run update-desktop-database.

vpn : Split Tunnel Concept

Once a user starts a vpn client to connect to company extranet, all network traffic is diverted to the vpn tunnel. Routing gets setup by VPN client such that everything would go down the tunnel. Split tunnel can fix that by keeping traffic for internet from tunnel and only direct extranet traffic to the tunnel. But it comes with few risks on its own. Lets review the concept for a minute.

The VPN tunnel can be configured to work in two modes.

  1. Mandatory (default)
    While a client tunnel is established in mandatory mode, all client traffic is tunneled through it by default. This is the default vpn mode. So accessing yahoo.com will go through vpn tunnel to company extranet which will then route it via its own internet connection after applying access policy etc.
  2. Split Tunneled mode
    Split Tunneling allows configuring specific network routes that are then tunneled and sent to the client’s Extranet adapter; any other traffic goes to the local PC Ethernet or Dialup adapter interface. So Split tunneling allows the user to get access to the Internet or print locally even while the system is tunneled into the company Extranet. But this comes with a security issue because it opens a backdoor into the secure office network from internet via the home system. A hacker can exploit the home system and can use that as a jump box to get into the company network. Or if the system at home is infected it will further that infection into office network. That is why organizations want vpn users to ensure they are up to date and have anti-virus installed and most will provide vpn clients that are tightly controlled to enable the Default mode. Continue reading

VPN : Connecting to Nortel VPN from Linux

I moved to Ubuntu full-time a while back. See the first part Migrating from Windows to Linux. But VPN into my office from Linux remained an open issue because my organization uses Nortel Contivity. Ubuntu comes with vpn client but that does not work with Nortel. Then I found vpnc-nortel branch of vpnc open source project that can connect to Nortel vpn servers. In this series I will explain how to connect to Nortel VPN from Ubuntu. You may need to tweak few of these instructions for your favourite Linux version. I am on Ubuntu 11.10 x86_64, using vpnc version 0.5.3-481.

Continue reading