Named pipes

If you are familiar with UNIX, you are familiar with pipes. For example, I can do:

ps -ef | sort | more

The ps command will output a list of all processes to stdout. Normally, this would be to the console window. The pipe (|) will tell UNIX to take the output of ps and make it the input to sort. Then the output from sort will become the input to more.

Without using pipes I could do:

ps -ef > temp_file1
sort < temp_file1 > temp_file2
rm temp_file1
more temp_file2
rm temp_file2

This is like using the pipe but instead we put the output of ps into temp_file1. Then we use temp_file1 as the input to sort and send the output to temp_file2. Finally, we use temp_file2 as the input to more. You should be able to see how this is a lot like the first example using pipes.

Now here is a third way using Named Pipes. To create a named pipe use:

mkfifo temp_file1

If you list this entry using ls -l you will see something like:

prw-r--r--   1 dgrainge staff          0 Mar 23 08:13 stdout

Notice the first letter is not - for a file or even d for a directory. It is p for a named pipe. Also the size of the 'file' is 0. We will need two shells to do this.

# shell 1
mkfifo temp_pipe1
mkfifo temp_pipe2
ps -ef > temp_pipe1            # this will block so switch to shell 2

# shell 2
sort < temp_pipe1 > temp_pipe2 # this will block so switch back to shell 1

# shell 1
more temp_pipe2
rm temp_pipe1
rm temp_pipe2

The interesting thing about this example is that we needed two shells to do this. At first this might seem like a downside but the truth is, this is a positive. I can do something like:

mkfifo stdout
mkfifo stderr

# shell 2
more stdout

# shell 3
more stderr

# shell 1
sh -x some_script.sh 1> stdout 2> stderr

The -x will turn on trace. Debug information will be output to stderr. By using the named pipes, I can redirect the regular output to shell 2 and the debug information to shell 3.

Windows Manage

Originally, there was Windows 3.1, Windows 9x and Windows NT. The 3.1 version was pretty much dead when Windows 95 came out. Windows 95/98/ME were okay but did not have the robustness of Windows NT.

If you wanted to use a home computer you'd go for Windows 9x but for a work environment they liked Windows NT. Windows NT 4.0 came before Windows 2000. Windows 2000 is REALLY Windows NT 5.0. If you look in the registry (regedit.exe) you will see evidence of NT 5.0. Windows XP Professional is Windows NT 6.0. Windows 2003 Server and Windows XP Professional are on par. One is server class and the other is workstation class.

Since Windows 2000, all of these (including Windows Vista) have an extra menu option when you right click on "My Computer". It is the Manage option. When you select Manage it opens up the Microsoft Management Console (mmc). You can even find this on Windows XP Home.

MMC is a framework that you can open other programs inside. If you go to the Control Panel, you will find things like Services, Disk Management, Event Viewer, etc. When you open up Manage, it opens the MMC with *ALL* these administrative tools in it. You can edit users/groups, manage disk drives, view events, create performance logs, etc.

Any place you can see "My Computer" you can right click on it and select Manage. This is a great shortcut to these administrative tools. You should give this area a peek and see what is available in there. Be careful though; this is an area where you can lock yourself out of the system or disable the bootable hard drive if you are not careful.

Extracting part of a log using Bourne shell

Someone recently asked me how to select a range of text from a log file. Because it was a log file, each line started with the date and time for each log entry.

She wanted to extract all the log entries from a start time to an end time. For example, all log entries from 08:07 to 08:16 on March 8th, 2007. The format for the timestamp would be:

2007-03-08 08:07:ss.sss [log message]

where ss.sss was the seconds and [log message] was the actual text message written to the log.

My solution, using Bourne shell, was to determine the first occurance of "2007-03-08 08:07" using grep. The GNU grep command would be:

START=`grep -n -m1 "2007-03-08 08:07" logfile.log | cut -d: -f1`

The -n will prefix the results with the line number. The -m1 tells it to quit after the first match. The output is going to be something like:

237:2007-03-08 08:07:ss.sss [log message]

where 237 is the line number. So the cut -d: will break the line at the semicolons and the -f1 will take the first field, i.e. 237.

Next you want to find the last occurance of 08:16. I would suggest looking for 08:17 using the same grep command, e.g.

END=`grep -n -m1 "2007-03-08 08:17" logfile.log | cut -d: -f1`

The reason you want to look for the value after the real END time is because a log might have many entries for 08:16. By looking for 08:17 we know we have captured all the entries for 08:16 rather than just the first entry.

This will give us the line AFTER the line we want, so we do the following to decrement it by one:

END=`expr $END - 1`

Now we want to extract everything from START to END in the log. We start by extracting everything from 1 to the END using the head command:

head -n $END logfile.log

Now we want to trim off the first START lines from this. For that we can use the tail command. But the tail command wants to know how many lines are to be kept. The value of START is the number of lines we want to get rid of. So we really want $END - $START + 1. So:

LINES=`expr $END - $START + 1`

Finally we would have:

head -n $END logfile.log | tail -n $LINES

and this will display only the lines from 08:07 to 08:16 on March 8, 2007.

Explaining URLs

I've been a little busy with work so we are looking at almost a month since my last post.

Here is an example URL:

http://128.23.64.233:8080/contest/reward.html?key=value&key2=value2

Here is the break down to the parts of the URL:

- http:// is the transmission protocol (humans talk french, engish, etc.; computers talk http, ftp, telnet, etc.)
- 128.23.64.233 is the IP address of the website. If you use www.google.ca, your computer will turn this into 64.233.167.104.
- On a website computer it typically listens on port 80 for http talking. The :8080 tells our web browser to send the message to port 8080 on the website computer. You can access Google using http://www.google.ca:80/ but the :80 is not necessary.
- The /contest is call the context root. The web site sees the /contest and knows that all the files for /context root are stored in a specific location on the hard drive. Typically there will be a root to the website, e.g. C:\www\http\. For this context root the file could be in C:\www\http\contest\.
- The reward.html is the file that gets processed. The website will process different files different ways. It knows how to send you an HTML file or a JSP file.
- Everything after the ? are parameters for the web page. If I have a form on a web page it it has the following fields: name, password. Then I can enter the text in the textbox or I can do: http://somesite/login.html?name=myname&password=secret

So that is a simple break down for an HTTP URL. There is more to it but this is a start. I'll post more later and maybe a tutorial on how to peek at websites without using a web browser. Good way to avoid website viruses.

J2EE, what is it?

It has been a while since I posted. I've been busy training new co-op students and trying to get two projects out the door.

One of the projects runs on JBoss application server. The interface for configuring the application is deployed as a WAR file. So we can update the configuration tools on future releases by just shipping a new WAR file to the customers. The student didn't understand how this works. Here was my 5 minute explanation...

In Java you have JAR files. A JAR file is a ZIP file with a bunch of Java classes in it. You can create a JAR file such that it can be run using 'java -jar'. It just requires the JAR file to have specific files in specific locations. A WAR file is the same idea.

For example, if I want to create the WAR file console.war I might do the following:

- Create the directory C:\console
- Create the directory C:\console\WEB-INF
- Create the directory C:\console\META-INF
- Create the file C:\console\WEB-INF\web.xml
- Create the file C:\console\META-INF\Manifest.mf

This is the basic structure. Now I need the code that will actually get run. Let's say I create the class AdminConsole in the package com.mycompany.myproject.console. Or, in other words, I created the file C:\console\classes\com\mycompany\myproject\console\AdminConsole.class.

The web.xml is defined by J2EE WAR file standards to have a specific format. It might contain something like:

<web-app>
<servlet>
<servlet-name>
com.mycompany.myproject.console.AdminConsole
</servlet-name>
<servlet-class>
com.mycompany.myproject.console.AdminConsole
</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>
com.mycompany.myproject.console.AdminConsole
</servlet-name>
<url-pattern>
/console/Admin
</url-pattern>
</servlet-mapping>
</web-app>

The <servlet> tag tells the application server about the class file and the <servlet-mapping> tells the application server when to run the code. So if the server in on the machine darrell.blogger.com the user could enter the following in a web browser:

http://darrell.blogger.com/console/Admin

The <url-pattern> takes the /console/Admin and knows which Java code to run. It will run the AdminConsole.class code. The AdminConsole class has to have specific methods implemented. Just like a desktop application knows to start at:

public static void main(Strings[] args)

The AdminConsole class will extend javax.servlet.http.HttpServlet and will have doGet() and possibly doPost() methods.

With a desktop application, if I wanted to print to the screen I'd use System.out, e.g.

System.out.println("Hello");

but with a servlet I'd have to get the output stream and write to it, e.g.

ServletOutputStream sos = response.getOutputStream();
// uses the methods of ServletOutputStream to write to the web browser
// e.g. sos.write().

Additionally, when you write to the web browser you have to indicate it the data is an image, text, HTML, etc. Before you write you can do things like:

response.setContentType("text/html");

So if you can write a Java application, it is fairly easy to convert to a web application.

Making shell scripts atomic

What do you do if you have a shell script that cannot be run twice at the same time, i.e. you have to wait until the script finishes before you can run it a second time.

The solution is to make the script check to see if it is running. If it is not running then let it start. The problem with this is it is possible to have the following scenario:

- start the script
- it checks to see if it is running
- start the script running a second time
- the second script checks to see if it is running
- flag that the first script is running and enter the critical section
- flag that the second script is running and enter the critical section

In other words, the check and the setting of the flag have to be atomic, i.e. you cannot have them be two steps in the script.

The solution is to use ln to create a link. The link will be the setting of the flag and the check. Then you can check the status of the ln command (the flag).

So here is the code to do it:

# Bourne Shell
#!/bin/sh

# create a filename for the ln
LOCK=`echo $0 | awk -F/ '{print $NF}' | awk -F. '{print $1}'`.LOCK

# create the link
ln $0 ${LOCK} 2> /dev/null

# see if we are already running
if [ $? -eq "0" ]; then
echo "running atomic script here"
echo "it just sleeps for 5 seconds"
sleep 5
/bin/rm ${LOCK}
else
echo "script is already running"
fi

Or if you perfer the C shell:

#C shell
#!/bin/csh

# create a filename for the ln
set LOCK=${0:r}.LOCK

# create the link
ln $0 ${LOCK} >& /dev/null

# see if we are already running
if ( ! $status ) then
echo "running atomic script here"
echo "it just sleeps for 5 seconds"
sleep 5
/bin/rm ${LOCK}
else
echo "script is already running"
endif

System V versus BSD UNIX

Now a days you will most likely be exposed to AT&T System V UNIX. Years ago there were dozens of different 'versions' of UNIX. At some point things started to converge to either AT&T System V (pronounced A T and T System Five) or BSD UNIX.

If you go for a job interview they might ask you things like, "How do you list all processes?" If you are using System V then the answer is "ps -ef" but it is "ps -aux" if you are using BSD UNIX.

Additionally, a lot of the System V installations have the BSD implementations of commands as well. The idea is that I might have a script that assumes BSD format commands. If I was to parse the output of ps to do a mass kill of all processes and its children then the format of the ps command matters.

If the script was written assuming the BSD version of ps and my computer was configured for System V, the script will fail. The solution, the default path might be /usr/bin but the BSD version of the commands are stored in /usr/ucb. So I could just edit the script so it finds the /usr/ucb/ps command rather than the default /usr/bin/ps.

Finally, if you are using MacOS X you are using a GUI front end to an implementation of UNIX. The UNIX is BSD UNIX. So things like "ps -ef" will not work.

So if you want to impress an interviewer who asks how to list all the processes on a UNIX machine, the answer is "If your computer is configured for System V UNIX then ps -ef will do the trick, but if /usr/ucb exists in the path before /usr/bin then ps -aux will do the trick. Additionally, if you are using MacOS X, which is BSD UNIX, then the default path will use ps -aux."

There are some obvious wrinkles to this. I'm assuming the default set up and configuration for the system. You'd impress them more if you asked which verison of UNIX and then noted, "if they are using the common configuration..."

If you want to read more of the details behind the history of UNIX, give this web site a try. Additionally, this site talks about creation of different versions of UNIX. Click on the picture at the top to see how nuts it got.