Named pipes

If you are familiar with UNIX, you are familiar with pipes. For example, I can do:

ps -ef | sort | more

The ps command will output a list of all processes to stdout. Normally, this would be to the console window. The pipe (|) will tell UNIX to take the output of ps and make it the input to sort. Then the output from sort will become the input to more.

Without using pipes I could do:

ps -ef > temp_file1
sort < temp_file1 > temp_file2
rm temp_file1
more temp_file2
rm temp_file2

This is like using the pipe but instead we put the output of ps into temp_file1. Then we use temp_file1 as the input to sort and send the output to temp_file2. Finally, we use temp_file2 as the input to more. You should be able to see how this is a lot like the first example using pipes.

Now here is a third way using Named Pipes. To create a named pipe use:

mkfifo temp_file1

If you list this entry using ls -l you will see something like:

prw-r--r--   1 dgrainge staff          0 Mar 23 08:13 stdout

Notice the first letter is not - for a file or even d for a directory. It is p for a named pipe. Also the size of the 'file' is 0. We will need two shells to do this.

# shell 1
mkfifo temp_pipe1
mkfifo temp_pipe2
ps -ef > temp_pipe1            # this will block so switch to shell 2

# shell 2
sort < temp_pipe1 > temp_pipe2 # this will block so switch back to shell 1

# shell 1
more temp_pipe2
rm temp_pipe1
rm temp_pipe2

The interesting thing about this example is that we needed two shells to do this. At first this might seem like a downside but the truth is, this is a positive. I can do something like:

mkfifo stdout
mkfifo stderr

# shell 2
more stdout

# shell 3
more stderr

# shell 1
sh -x some_script.sh 1> stdout 2> stderr

The -x will turn on trace. Debug information will be output to stderr. By using the named pipes, I can redirect the regular output to shell 2 and the debug information to shell 3.

Windows Manage

Originally, there was Windows 3.1, Windows 9x and Windows NT. The 3.1 version was pretty much dead when Windows 95 came out. Windows 95/98/ME were okay but did not have the robustness of Windows NT.

If you wanted to use a home computer you'd go for Windows 9x but for a work environment they liked Windows NT. Windows NT 4.0 came before Windows 2000. Windows 2000 is REALLY Windows NT 5.0. If you look in the registry (regedit.exe) you will see evidence of NT 5.0. Windows XP Professional is Windows NT 6.0. Windows 2003 Server and Windows XP Professional are on par. One is server class and the other is workstation class.

Since Windows 2000, all of these (including Windows Vista) have an extra menu option when you right click on "My Computer". It is the Manage option. When you select Manage it opens up the Microsoft Management Console (mmc). You can even find this on Windows XP Home.

MMC is a framework that you can open other programs inside. If you go to the Control Panel, you will find things like Services, Disk Management, Event Viewer, etc. When you open up Manage, it opens the MMC with *ALL* these administrative tools in it. You can edit users/groups, manage disk drives, view events, create performance logs, etc.

Any place you can see "My Computer" you can right click on it and select Manage. This is a great shortcut to these administrative tools. You should give this area a peek and see what is available in there. Be careful though; this is an area where you can lock yourself out of the system or disable the bootable hard drive if you are not careful.

Extracting part of a log using Bourne shell

Someone recently asked me how to select a range of text from a log file. Because it was a log file, each line started with the date and time for each log entry.

She wanted to extract all the log entries from a start time to an end time. For example, all log entries from 08:07 to 08:16 on March 8th, 2007. The format for the timestamp would be:

2007-03-08 08:07:ss.sss [log message]

where ss.sss was the seconds and [log message] was the actual text message written to the log.

My solution, using Bourne shell, was to determine the first occurance of "2007-03-08 08:07" using grep. The GNU grep command would be:

START=`grep -n -m1 "2007-03-08 08:07" logfile.log | cut -d: -f1`

The -n will prefix the results with the line number. The -m1 tells it to quit after the first match. The output is going to be something like:

237:2007-03-08 08:07:ss.sss [log message]

where 237 is the line number. So the cut -d: will break the line at the semicolons and the -f1 will take the first field, i.e. 237.

Next you want to find the last occurance of 08:16. I would suggest looking for 08:17 using the same grep command, e.g.

END=`grep -n -m1 "2007-03-08 08:17" logfile.log | cut -d: -f1`

The reason you want to look for the value after the real END time is because a log might have many entries for 08:16. By looking for 08:17 we know we have captured all the entries for 08:16 rather than just the first entry.

This will give us the line AFTER the line we want, so we do the following to decrement it by one:

END=`expr $END - 1`

Now we want to extract everything from START to END in the log. We start by extracting everything from 1 to the END using the head command:

head -n $END logfile.log

Now we want to trim off the first START lines from this. For that we can use the tail command. But the tail command wants to know how many lines are to be kept. The value of START is the number of lines we want to get rid of. So we really want $END - $START + 1. So:

LINES=`expr $END - $START + 1`

Finally we would have:

head -n $END logfile.log | tail -n $LINES

and this will display only the lines from 08:07 to 08:16 on March 8, 2007.