“You see,” he explained, “I consider that a man’s brain originally is like a little empty attic, and you have to stock it with such furniture as you choose. A fool takes in all the lumber of every sort that he comes across, so that the knowledge which might be useful to him gets crowded out, or at best is jumbled up with a lot of other things so that he has a difficulty in laying his hands upon it. Now the skilful workman is very careful indeed as to what he takes into his brain-attic. He will have nothing but the tools which may help him in doing his work, but of these he has a large assortment, and all in the most perfect order. It is a mistake to think that that little room has elastic walls and can distend to any extent. Depend upon it there comes a time when for every addition of knowledge you forget something that you knew before. It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones.” (Sherlock Holmes, in “A Study in Scarlet”)

Most good Unix sysadmins are very smart guys. They are smart not just about computers. Good sysadmins rely on automation. They write scripts and set up cron jobs to perform most everyday tasks. This reduces manual work and minimizes errors. And this also allows sysadmins plenty of free time to browse the Web and read. And that’s why they are smart. When you see your sysadmin doing nothing, you should be happy: all the servers and running and all the users are working. On the other hand, if you see your sysadmin constantly running around like a chicken without head, it means either that your department is seriously understaffed or that the guy doesn’t know what he is doing.

An experienced sysadmin spent a lifetime working with computers. He knew more about computers when he was in junior high than most people ever want to learn. He spent four or five years in college and years on the job learning what he knows today. System administration is in part about knowing the theory, but it is primarily about having practical experience. And experience takes time, long years of time.

When a hospital interviews a surgeon, they don’t ask him to explain the function of the lungs. When a university hires a math professor, they don’t ask him to recite the multiplication table. So why in the hell would you ask a system administrator to explain how the “df” command works? You want to know how it works? You want to know every option for this command? You really, really do? Just type “man df” at the command prompt and it will tell you everything there is to know about “df”. Would this make you happy?

There are hundreds, thousands Unix commands and most have dozens of options. These commands and options can be combined in a virtually infinite number of sequences. A human brain is not for storing this type of nonsense. There are commands and options that you use on a daily basis. These you have memorized to a point when typing them becomes a reflex. As for the rest of them: there are “man” pages, documentation, Google and books.

Here is a characteristic example of all the wrong questions to ask a sysadmin during an interview. This questionnaire comes from a very large IT services company (and is probably copyright of whatever nitwit that wrote it). We use it here just as an example of what not to ask when interviewing a sysadmin. Consider some of the questions: “What is an inode?” Inode describes files, directories, and filesystems. You want to know more – go to Wikipedia’s “inode” article. There, problem solved. Here’s another one: “What kind of permission would “chmod 755″ yield on a file?” This is like asking a race car driver to explain the purpose of the clutch.

Questioners like this were slapped together in about twenty minutes right before an interview, just because the interviewing manager knew nothing about Unix and needed something to go by. The kind of a job candidate who would correctly answer these questions is a fresh-out-of-college rookie who spent the past week browsing Wikipedia and reading “man” pages to get ready for this interview. So congratulations on your new sysadmin – after a few years of on-the-job training and countless screwups he will be able to work on his own.

Another pet peeve of mine is certificates. Sun Certified Professional, Certified Novel Administrator, IBM Certified Specialist, etc., etc., etc. These pieces of paper help you land a job not because they confirm your qualifications, but because they help the hiring manager cover his ass. In case the new employee turns out to be incompetent, as often is the case, the hiring manager can always say: “Hey, this guy was Sun-certified!” Everybody – even the managers – knows that the actual value of these certificates is marginal at best. If you are good at memorizing things and keeping them in memory for a week, you can take a five-day course and get your certificate.

So what questions should you ask a candidate for a Unix sysadmin position at your company? Ask general, systemic, procedural, even philosophical questions. Don’t forget to ask a few questions that have nothing to do with system administration. Remember: a good sysadmin uses system automation to have more time to browse the Web and to learn. So ask him about neutron stars or the String Theory. Always ask a few questions on computer networking theory. Ask general questions about the fundamentals of Unix security. Ask him about major filesystem types and data storage hardware. Your goal is to test the candidate’s breadth of knowledge and analytical skills, not to check if he has photographic memory and if he uses his brain for storing technical manuals.

Below is a list of question you may want to ask a candidate for a senior Unix/Linux sysadmin position or for the position of a Unix system architect. This questionnaire is designed to test general knowledge of Unix, networking, hardware and to test analytical skills. If you absolutely require knowledge of specific software, hardware, or programming languages, you should ask about it at the beginning of the interview to get it out of the way and not to waste anyone’s time. But don’t demand too much diverse knowledge from such a candidate: you are looking for a specialist – not a renaissance man.

Unix Fundamentals

A Unix sysadmin should have strong knowledge of Unix history and fundamentals. This kind of knowledge separates Unix hacks with a couple years of ad-hoc Unix training from the true experts.

Q: What defines a Unix OS? How is it fundamentally different from other types of operating systems, like VMS or Windows?

A: The candidate should tell you that Unix was created to be a portable, multitasking, multiuser operating environment designed for time-sharing, distributed configurations and supporting hierarchical filesystems, networking, and inter-process communications.

Q: Show an example of how one Unix process can communicate with another process.

A: The candidate may provide at least a simple example of using the “pipe” (something like ” ps -ef | awk ‘{print $2}’ “) or any other method of passing information from one process to another.

Q: What are the main functions of kernel?

A: The primary purpose of kernel is to schedule access to hardware to avoid system resource conflicts. Kernel also provides services to start and stop programs, handles filesystems and performs other so-called “low-level” tasks.

Q: What is the purpose of IPC sockets?

A: Generally speaking, IPC sockets are virtual byte stream channels allowing separate processes to communicate with each other.

Q: What is Unix time?

A: It is the number of seconds elapsed since January 1, 1970.

Unix Security

Understanding of Unix security fundamentals is not just for keeping the hackers out. Most simple but time-consuming problems on Unix systems arise from incorrect ownership and permissions settings. A good sysadmin must see the big picture of Unix security – not just know how to use the chmod command.

Q: Are Unix operating systems susceptible to computer viruses and worms?

A: Yes, they are. Inexperienced Unix sysadmins are under the impression that Unix is impervious to computer viruses. This is a potentially dangerous misconception. In fact, the first ever Internet worm – the Morris worm – was created back in 1988 to specifically target Unix systems.

Q: What is the fundamental difference between TELNET and SSH?

A: Anything you send or receive via a TELNET session travels across the network as open text. Intercepted data streams can be easily read. SSH encrypts data before sending.

Q: Automated OS patching: is it a good idea?

A: Many Unix and Unix-like operating systems provide tools for automating the process of downloading and installing OS patches. An experienced sysadmin will rarely use such tools. While all patches are tested by the OS vendor, bugs slip through on a regular basis and unexpected interactions with software and hardware are common. Patches in the Unix world are installed to address specific issues – not just to bring the OS to the current “service pack” level.

Networking Basics

Many Unix administration and troubleshooting activities will require dealing with networks. A sysadmin who doesn’t know networking basics should not be allowed anywhere near your servers.

Q: What are the four abstraction layers of the TCP/IP model?

A: Data link, network, transport, and application layers.

Q: For the OSI model, provide a few examples of the network layer and the transport layer.

A: For the network layer any of the following will be a good answer: IP (IPv4, IPv6), OSPF, IS-IS, IPsec, ARP, RARP, RIP, ICMP, ICMPv6, IGMP. For the transport layer: TCP, UDP, DCCP, SCTP, RSVP, ECN.

Q: What are the basic differences between TCP and UDP protocols?

A: TCP delivers data packets in a specific order. TCP verifies delivery and integrity of data packets and, if necessary, retransmits them. UDP sends data packets in no particular order and does not verify their delivery or integrity.

Q: What is the most fundamental difference between a network hub and a network switch?

A: A network hub connects multiple Ethernet devices to the same network segment. A hub is a layer-one physical device. A network switch connects different network segments. A switch can operate on multiple layers.

Q: What would be the fastest way to transfer 100Tb of data from a Unix server in Philadelphia to a server in Los Angeles?

A: Copy data to a locally-attached disk array and FedEx the disk array to LA. An experienced sysadmin is familiar with the difficulties transferring lots of data over the network. With the current networking infrastructure, physically moving storage hardware is the fastest, most reliable and often the cheapest way of transferring massive amounts of data.

Data Storage Basics

Large enterprises have dedicated storage administrators who support nothing but SANs and NASes. Still, Unix sysadmins must have a good understanding of storage subsystems to effectively support their servers.

Q: What is the basic difference between SAN (Storage Area Network) and NAS (Network Attached Storage)?

A: The client operating system sees a SAN as locally-attached storage and requests disk blocks. A NAS uses file-based protocols, such as NFS or SMB/CIFS, to serve abstract files to the client operating system.

Q: RAID 1+0 is commonly used for high-load databases. How do you calculate usable space in this type of RAID configuration?

A: S x N / 2, where S is the size of the smallest drive and N is the number of drives in the set.

Q: What is the minimum number of drives required to implement a RAID 5 set?

A: Three.

Shell Scripting

Any Unix sysadmin should have strong shell scripting skills for automating repetitive tasks. There can be no system administration without scripting. Lazy sysadmins have excellent scripting skills. That’s how they can be lazy and still keep their jobs. In the world of system administrators “lazy”=”competent”.

Q: Review the following simple Korn shell script and describe its purpose:

#!/bin/ksh
for i in opt usr
do
   find /${i} -type f -name "core" | while read line
   do
      gzip "${line}"
      mv "${line}.gz" /var/tmp/
   done
done

A: The script searches /opt and /usr for files named “core”, compresses them, and moves them to /var/tmp

Q: Take a look at the following short script and describe what it does:

#!/bin/ksh
i=1
while [ $i -le 50 ]
do
   ssh node$i "echo '135.122.12.9   host.mydomain.com' >> /etc/hosts"
   (( i = i + 1 ))
done

A: This script will connect via SSH to remote systems called node1 through node50 and it will append a new entry to the /etc/hosts file on each node. Short while loops like this can be useful for managing clusters.

Q: Review the following script and describe what it does:

#!/bin/ksh
ps -ef | grep [h]ttpd | awk '{print $2}' | while read LINE
do
   kill -9 $LINE
done

A: The script looks for active “httpd” processes, determines their PIDs and kills them.

Q: Take a look at the following Korn shell script and describe what it does:

#!/bin/ksh
logfile=/var/log/messages
if [ `tail -100 $logfile | grep -c "ERROR"` -gt 0 ]
then
   echo "Errors found on `hostname` in $logfile" | mailx -s "Error report from `hostname`" admin@mydomain.com
fi

A: The script will check the last 100 line of the /var/log/messages for error messages. If it finds any, it will send an email to the admin.

Afterword

When interviewing a sysadmin, keep this in mind: the person sitting across the desk from you – should you choose to hire him – will have the keys to your kingdom. You are looking for experience. Don’t bother asking about unimportant details: command options, configuration file syntax, dictionary definitions of technical terms and other such nonsense.

Experienced sysadmins tend to come off as bumbling idiots when confronted by trivial questions. You may mistake their apparent confusion for incompetence. Beware of job applicants who can recite “man” pages from memory, but are unable to explain how Unix is fundamentally different from Windows.

A good friend of mine – a highly skilled Unix sysadmin with decades of experience – used to bring the 1000-page “Unix Power Tools” book to the interviews when he was looking for his next consulting job. Whenever the interviewing manager would ask him an inconsequential question about some command or option, he would take out the book and find the relevant page. Usually the hiring “gurus” would get the hint.