VirtualBox : Use Raw Disk to load Windows under Linux

Here I explain how to use a Physical disk partition for a guest OS under VirtualBox. This is also called Raw Disk partition use for VirtualBox. My use case was to run WindowsXP as guest OS from a physical installation under Linux and still be able to boot up the system in same Windows installation when needed. My system is running Ubuntu 11.04 on core2 duo, 3GB memory, two hard disks (one with Ubuntu and other with WindowsXP installation), with VirtualBox 4.0.4. Process is simple but took quite a while to get all information/steps collected and tested. In nutshell, first we need to know the partition that we will use, then the user who is going to use it needs to have access to it. After that an mbr has to be created and finally a vmdk file is created to use the Raw Disk. Keep reading for the full process. Continue reading

AgentX Tutorial using Net-SNMP – Trap implementation

Third one in this series is implementation of SNMP Traps, the method which is the main strength of SNMP. These Traps provide an event based system where NMS relies heavily on the monitored stations to tell them if something of interest has happened. mib2c again provides an easy way to implement traps. This is actually one of the easiest ones of the three in this series. Again as I said before clone my repository and checkout the code to find what was done and how. Following are three commands of interest.


git clone https://github.com/jainvishal/agentx-tutorial.git
git checkout step3_traps_autogenerated
git checkout step3_traps_implement

mib2c provides mib2c.notify.conf to compile the MIB and generate C code. By default it uses SNMPv2 for traps and I leave it as is. Implementation is as simple as copying the data into right variables when time comes and they will be delivered. Continue reading

AgentX Tutorial using Net-SNMP – Simple Table MIB for Dummies

Second in the series of implementing an AgentX agent I describe a simple table implementation. Net-SNMP provides multiple implementations from easy one to more elaborate ones. An approach should be accepted based on the need. For most of the needs MIB for Dummies work (mib2c -c mib2c.mfd.conf). Points to consider for using MIB for dummies are :

  1. Data could be cached and smaller time windows are acceptable where data may be stale compared to real world values (or application can implement a logic to mark cache expired on real value change)
  2. Data reloads are not resource intensive
  3. Data set for table is small so memory foot print of copying to cache will be small

Continue reading

AgentX Tutorial using Net-SNMP – Simple Scalars

Since no good tutorials are available I put together what I learnt. My approach was step by step learning and that is what I describe here. So on day 1 or step one we will see how to create a simple MIB that has only two simple read-only scalars and how to implement them in the agent. I have configured net-snmpd agentx mode to run on TCP. Check this post on how to configure that.

You should clone my git repository and follow the README which describes the changes made. I have created tags in the git repository so it is easy to reference what is done when and for what.

git clone https://github.com/jainvishal/agentx-tutorial.git
git checkout step1_scalars_autogenerated
git checkout step1_scalars_implement

Notes

  1. I used net-snmp 5.7.3 for this tutorial.
  2. My test MIB AGENTX-TUTORIAL-MIB is under experimental branch (.1.3.6.1.3) so my MIB branch is .1.3.6.1.3.9999.
  3. smilint was used to verify that MIB was syntactically correct.
  4. mib2c was used with mib2c.scalar.conf to compile the MIB and generate the code, makefile and agent code.

Now wait for second tutorial where I will implement a simple table using MIB-For-Dummies configuration.

Net-SNMP configuration for non-root AgentX application

It is imperative to run the application when developing it. You may be testing it, debugging it or troubleshooting something. By default Net-SNMP uses named socket for AgentX communication which does not allow a non-root user to connect making troubleshooting difficult. There are security reasons for not allowing this kind of widely open access so do not set this up in your production environment. There are other ways to control the access which I will narrate in future posts.

To enable AgentX and allow non-root applications/Agents to connect to snmpd you can setup TCP socket as follows. TCP socket provides a cleaner access and allows easier troubleshooting e.g you could capture network traffic between snmpd and the AgentX application. Update /etc/snmp/snmpd.conf and ensure that following directives are set for TCP based AgentX communication.

rocommunity public default # or whatever community string/access control you want
master agentx # Run as an AgentX master agent
agentXSocket tcp:localhost:705 # Listen for localhost network connections instead of /var/agentx/master

Restart snmpd (/etc/init.d/snmpd restart)

Alternate is to set correct permissions for /var/agentx/master named socket or whatever you have configured.

Capture local network traffic for multi-homed host

If both end-points of a socket are on local system, network traffic will be seen on loopback interface even if applications are using non-loopback interface (e.g. eth0, wlan0…). Capturing data over loopback is quite obvious. But here I am discussing that applications are using one of the external interfaces (e.g. eth0, eth1, wlan0 …..).

Since both end-points are on  local system, kernel will shunt the traffic and not send it to the wire. The data will be delivered internally by queuing it to the read queue of other end-point. So we cannot capture the traffic on that particular interface, but this traffic is visible on loopback interface. Lets see an example.

I use netcat for setting up our test client and server program. nc -kl 9090 will run server on all interfaces on port 9090. And nc 10.1.1.100 9090 will setup a client. Here 10.1.1.100 is the external IP of my system(wlan0). Now instead of using the interface name associated with that IP (in my case wlan0), we have to use loopback interface lo to capture the traffic as below.

tcpdump -i lo tcp port 9090

Now anything that is typed on the client terminal when sent will be seen by tcpdump. Problem solved.

Compare files ignoring a field or column using Process Substitution

Lets say the data contains multiple fields/columns separated by space or comma or some other delimiter. And we want to compare two files ignoring a specific column. Lets divide work in two small issues. First is to ignore the provided field/column.

If we simply want to ignore the first column, we can use one of the following cut constructs.

cut -d',' -f 1 --complement datafile
cut -d',' -f 2- fileName.csv

If we want to ignore a specific one we can use awk in following manner which is much more generalized because you can specify which column to ignore, be it first, third or last.

This can be used as

awk -F',' -v FieldToIgnore=3 -f ignoreField.awk datafile

Next part is to diff the output after ignoring (read removing) the column. That is where process substitution comes handy. Here are two examples.

# ignore 1st column from two csv datafiles while comparing
diff -u <(cut -d, -f 2- datafile1) <(cut -d, -f 2- datafile2)
# ignore column 3 from two csv datafiles while comparing
diff -u <(awk -F',' -v FieldToIgnore=3 -f ignoreField.awk datafile1) <(awk -F',' -v FieldToIgnore=3 -f ignoreField.awk datafile2)

So instead of giving it two real files, we give it two redirected streams. Same solution can be used to pre-process files differently (e.g. ignore any comments or empty lines or compare two unsorted files).

See below for more information on Process Substitution.
http://www.tldp.org/LDP/abs/html/process-sub.html
http://wiki.bash-hackers.org/syntax/expansion/proc_subst

Application Logging Improvement – Part 3 Making it Readable

This is part three of my Application Logging improvement plan. So far I have discussed that log should be machine readable for application performance, management and monitoring. In this post I give an example of how to make the log readable to human (or make the log just like everyone has been used to seeing them). I am going to use vim to view the log files and have it configured so it knows how to handle the file with syntax etc.

First thing is configure vim to recognize the format. Continue reading

Application Logging Improvement – Part 2 Multithreading

Multi-threading is now becoming a norm. Obvious issue with logging is how to synchronize between threads. As discussed in last post Application Logging Improvement Plan – Part 1, we want to log as much as possible in machine readable format. So there comes a problem with multiple threads trying to log at the same time. Two possible implementations come to mind but both are flawed.

  1. Synchronize between threads for logging – Disk writes are slow and now locking contention would only make it worse. This slows down the business logic and is a big no-no.
  2. Log without synchronizing – Business logic works but logs get jumbled up because multiple threads are trying to log at the same time. This leaves logs in worst shape and unusable.

We can do better by combining both of above to get a solution. We will create a per thread logging buffer (lets call it LogBuffer) where each thread would log without any conflicts. And at a certain threshold, threads synchronize and log their LogBuffer to the disk (lets call this Flush).  Continue reading