VirtualBox : Use Raw Disk to load Windows under Linux

Here I explain how to use a Physical disk partition for a guest OS under VirtualBox. This is also called Raw Disk partition use for VirtualBox. My use case was to run WindowsXP as guest OS from a physical installation under Linux and still be able to boot up the system in same Windows installation when needed. My system is running Ubuntu 11.04 on core2 duo, 3GB memory, two hard disks (one with Ubuntu and other with WindowsXP installation), with VirtualBox 4.0.4. Process is simple but took quite a while to get all information/steps collected and tested. In nutshell, first we need to know the partition that we will use, then the user who is going to use it needs to have access to it. After that an mbr has to be created and finally a vmdk file is created to use the Raw Disk. Keep reading for the full process. Continue reading

Capture local network traffic for multi-homed host

If both end-points of a socket are on local system, network traffic will be seen on loopback interface even if applications are using non-loopback interface (e.g. eth0, wlan0…). Capturing data over loopback is quite obvious. But here I am discussing that applications are using one of the external interfaces (e.g. eth0, eth1, wlan0 …..).

Since both end-points are on  local system, kernel will shunt the traffic and not send it to the wire. The data will be delivered internally by queuing it to the read queue of other end-point. So we cannot capture the traffic on that particular interface, but this traffic is visible on loopback interface. Lets see an example.

I use netcat for setting up our test client and server program. nc -kl 9090 will run server on all interfaces on port 9090. And nc 10.1.1.100 9090 will setup a client. Here 10.1.1.100 is the external IP of my system(wlan0). Now instead of using the interface name associated with that IP (in my case wlan0), we have to use loopback interface lo to capture the traffic as below.

tcpdump -i lo tcp port 9090

Now anything that is typed on the client terminal when sent will be seen by tcpdump. Problem solved.

Compare files ignoring a field or column using Process Substitution

Lets say the data contains multiple fields/columns separated by space or comma or some other delimiter. And we want to compare two files ignoring a specific column. Lets divide work in two small issues. First is to ignore the provided field/column.

If we simply want to ignore the first column, we can use one of the following cut constructs.

cut -d',' -f 1 --complement datafile
cut -d',' -f 2- fileName.csv

If we want to ignore a specific one we can use awk in following manner which is much more generalized because you can specify which column to ignore, be it first, third or last.

This can be used as

awk -F',' -v FieldToIgnore=3 -f ignoreField.awk datafile

Next part is to diff the output after ignoring (read removing) the column. That is where process substitution comes handy. Here are two examples.

# ignore 1st column from two csv datafiles while comparing
diff -u <(cut -d, -f 2- datafile1) <(cut -d, -f 2- datafile2)
# ignore column 3 from two csv datafiles while comparing
diff -u <(awk -F',' -v FieldToIgnore=3 -f ignoreField.awk datafile1) <(awk -F',' -v FieldToIgnore=3 -f ignoreField.awk datafile2)

So instead of giving it two real files, we give it two redirected streams. Same solution can be used to pre-process files differently (e.g. ignore any comments or empty lines or compare two unsorted files).

See below for more information on Process Substitution.
http://www.tldp.org/LDP/abs/html/process-sub.html
http://wiki.bash-hackers.org/syntax/expansion/proc_subst

Application Logging Improvement – Part 3 Making it Readable

This is part three of my Application Logging improvement plan. So far I have discussed that log should be machine readable for application performance, management and monitoring. In this post I give an example of how to make the log readable to human (or make the log just like everyone has been used to seeing them). I am going to use vim to view the log files and have it configured so it knows how to handle the file with syntax etc.

First thing is configure vim to recognize the format. Continue reading

Application Logging Improvement – Part 2 Multithreading

Multi-threading is now becoming a norm. Obvious issue with logging is how to synchronize between threads. As discussed in last post Application Logging Improvement Plan – Part 1, we want to log as much as possible in machine readable format. So there comes a problem with multiple threads trying to log at the same time. Two possible implementations come to mind but both are flawed.

  1. Synchronize between threads for logging – Disk writes are slow and now locking contention would only make it worse. This slows down the business logic and is a big no-no.
  2. Log without synchronizing – Business logic works but logs get jumbled up because multiple threads are trying to log at the same time. This leaves logs in worst shape and unusable.

We can do better by combining both of above to get a solution. We will create a per thread logging buffer (lets call it LogBuffer) where each thread would log without any conflicts. And at a certain threshold, threads synchronize and log their LogBuffer to the disk (lets call this Flush).  Continue reading

Application Logging Improvement Plan – Part 1

People are divided on how to log, what to log, how much to log. A never ending discussion this is. In addition many open source libraries are available for logging. Not to mention many standards. I am not going to go in details of what is available out there. Use Google to pick your poison. What I am going to discuss here is what I think makes most sense with available technology.
Continue reading

Create Post Excerpt Intelligently in Jekyll

meta-descriptions or OpenGraph Descriptions are the short descriptions that Search Engines/Facebook display on result pages. They provide a brief introduction to the content and grab user attention. So when possible, try to use keywords in meta description that describes in short what that page is about. Ensure to limit meta description length to 160 characters which seems to be a norm for SEO now-a-days.

Jekyll by default supports post.excerpt which would automatically take the value of excerpt from Front Matter (if present). Otherwise it will fall-back to post content from beginning till excerpt_separator or end of post. Whatever separator you choose, have it defined as excerpt_separator in _config.yaml. With this done you can use {% raw %}{{ post.excerpt | strip_html }}{% endraw %} wherever you need to show it (e.g. meta-description/og-description). I use strip_html to remove any html which will cause issues with description tags.

I personally like to show a bigger teaser on my main page to entice user attention. And on other pages (like search results, Category/Tag index) show a brief description. Here is how I do it.

  1. Define meta-description tag in Front Matter of the post and describe in few words what the post is about. Then use {% raw %}{{ post.meta-description }}{% endraw %} during publishing HTTP headers for the post and on Category/Tag index pages. And on main page OR index.html use {% raw %}{{ post.excerpt }}{% endraw %} which is auto-generated by Jekyll based on Front Matter or excerpt_separator. I personally stopped defining excerpt in Front Matter to avoid namespace collision with Jekyll.
    Continue reading

mimikatz : Export non-exporteable Private certificate from Symantec PKI

Recently our organization started to provision Private certificates using Symantec Managed PKI Service. It has lot more appeal for IT admins because it takes out all user intervention which always creates support nightmares.

Previously I had direct access to the private key so it was easy to export it to all my devices and use for VPN and other secure stuff that needed to verify that I am indeed the real user. Because Symantec PKI is not available for Linux, it broke the VPN access from my Ubuntu system. Naturally I started to look for ways to export the key out of windows system. So here is what I did to get me out of the bind.

How to export certificates

First I installed Symantec PKI client on a windows 7 system. That was a no brainer because there was no other choice. I did not try with Windows 8 so YMMV. The main issue was that Windows certificate manager showed that the private key was not exportable. If it was then my quest would have been over right there. But I had to take another step. Mimikatz was the answer which marks them exportable and also allows to export them. Note: The patching that it does only lasts for that session. Once you reboot windows system you have to patch again using mimikatz. I used latest version which is 2.0 at the writing of this post. Continue reading

postfix : Configure outgoing relay server

Update /etc/postfix/main.cf and add the name of your outgoing/relaying mailhost as “relayhost”. Ensure that the relay server is accepting your email first.

e.g. if the outgoing relay is mailhost.xyzserver.com sendmail configuration should look like following.

# INTERNET OR INTRANET

# The relayhost parameter specifies the default host to send mail to
# when no entry is matched in the optional transport(5) table. When
# no relayhost is given, mail is routed directly to the destination.
#
# On an intranet, specify the organizational domain name. If your
# internal DNS uses no MX records, specify the name of the intranet
# gateway host instead.
#
# In the case of SMTP, specify a domain, host, host:port, [host]:port,
# [address] or [address]:port; the form [host] turns off MX lookups.
#
# If you're connected via UUCP, see also the default_transport parameter.
#
#relayhost = $mydomain
#relayhost = [gateway.my.domain]
#relayhost = [mailserver.isp.tld]
#relayhost = uucphost
#relayhost = [an.ip.add.ress]
relayhost = mailhost.xyzserver.com

After that restart postscript.

service postscript restart