Skip to main content

Adventures In Server-land: Power Up (Post-Boot) Server Status Email

Lately my projects have taken a back seat to some other higher priority stuff (including setting up the new web page for CCCKC (my local hackerspace) and various KC Maker Faire duties.

On my own time I've been working to build a low power server. Something I can leave on all the time without burning holes in my wallet. Really that's just an excuse...any money I would have saved was dumped into good hardware and setup time, but whatever. I think the main point is that (when I've finished) I should have a server that I can leave on and have reliable file storage/access anywhere. And I will have learned something (supposedly).

The Server:
At full load this server uses less than 70 watts! (My main PC uses 160W at idle, 300-400W at full load...)

I don't think I'll go into full detail with every software thing I've done so far (I can't remember most of the steps, and I'm not really qualified to explain them). I will however give the short list:
  • Ubuntu Server 11.4
  • LAMP (Apache, mySQL, PHP) Mostly so I can have a quick easy status check from any device.
  • smartmontools - For S.M.A.R.T hard drive monitoring
  • mdadm - For raid (still playing with raid drives, burning in disks and practicing for failures.)
  • samba - still tinkering... local file share access (tests show ~100MB up/down speeds...awesome!)
  • sftp - still tinkering... remote file share access (dunno yet...)
  • lm-sensors - CPU temperature
  • hddtemp - HD temperatures
Anyways I started this post to talk about a script I made while tinkering today. Firstly I followed this guide to get command line email via gmail working...and then went about making a robust post-boot status report script. This serves three purposes:
  • After a power failure/reboot I get an email immediately.
  • The email has important/useful status information about the server before any users get there hands in the mix.
  • The script serves as a library of useful sys-admin commands. Things I wouldn't normally use and, therefore, forget easily.
Here's what I did:

Create a file called onboot-report.sh with the following:
Note: You should change some@email.com to your own email...and you may need to change the FILE and MAIL_FILE locations...
  
#!/bin/bash
#The report file (will remain after script exits...until next run)
FILE=/latest-report.txt

#Clear file by using ">" instead of ">>" and at the same time
#add current date/time
date > $FILE
echo "***********************************************************" >> $FILE
echo " Begin Report " >> $FILE
echo "***********************************************************" >> $FILE

#report active users
echo "***********************************************************" >> $FILE
echo " Users " >> $FILE
echo "***********************************************************" >> $FILE
w >> $FILE

#report disk statistics
echo "***********************************************************" >> $FILE
echo " I/O Stats " >> $FILE
echo "***********************************************************" >> $FILE
iostat >> $FILE
echo "***********************************************************" >> $FILE
echo " Disk Space " >> $FILE
echo "***********************************************************" >> $FILE
df -aT --total >> $FILE
echo "***********************************************************" >> $FILE
echo " SMART Stats " >> $FILE
echo "***********************************************************" >> $FILE
echo "--------------------------/dev/sda-------------------------" >> $FILE
hddtemp /dev/sda >> $FILE
smartctl -a /dev/sda >> $FILE
echo "--------------------------/dev/sdb-------------------------" >> $FILE
hddtemp /dev/sdb >> $FILE
smartctl -a /dev/sdb >> $FILE
echo "--------------------------/dev/sdc-------------------------" >> $FILE
hddtemp /dev/sdc >> $FILE
smartctl -a /dev/sdc >> $FILE
echo "--------------------------/dev/sdd-------------------------" >> $FILE
hddtemp /dev/sdd >> $FILE
smartctl -a /dev/sdd >> $FILE
echo "--------------------------/dev/sde-------------------------" >> $FILE
hddtemp /dev/sde >> $FILE
smartctl -a /dev/sde >> $FILE
echo "--------------------------/dev/sdf-------------------------" >> $FILE
hddtemp /dev/sdf >> $FILE
smartctl -a /dev/sdf >> $FILE
echo "***********************************************************" >> $FILE
echo " Mounts " >> $FILE
echo "***********************************************************" >> $FILE
cat /proc/mounts >> $FILE

#memory usage.
echo "***********************************************************" >> $FILE
echo " Memory " >> $FILE
echo "***********************************************************" >> $FILE
vmstat >> $FILE

#processor usage
echo "***********************************************************" >> $FILE
echo " CPU Usage " >> $FILE
echo "***********************************************************" >> $FILE
mpstat -P ALL >> $FILE

echo "***********************************************************" >> $FILE
echo " Processes " >> $FILE
echo "***********************************************************" >> $FILE
ps axjf >> $FILE

echo "***********************************************************" >> $FILE
echo " Sensors " >> $FILE
echo "***********************************************************" >> $FILE
sensors >> $FILE

echo "***********************************************************" >> $FILE
echo " Network " >> $FILE
echo "***********************************************************" >> $FILE
ss >> $FILE
netstat -l >> $FILE

echo "***********************************************************" >> $FILE
echo " End Report " >> $FILE
echo "***********************************************************" >> $FILE

#build an email and include the report data...html formatted so
#we can display it in a monospaced font.

MAIL_FILE=/mail_html.html
#Clear file by using ">" instead of ">>" and at the same time
#add the first part of the email content.
echo "To: some@email.com" > $MAIL_FILE
echo "Subject: Server Report" >> $MAIL_FILE
echo "Content-Type: text/html; charset=\"us-ascii\"" >> $MAIL_FILE
#Use the <pre> tag so html parsers ignore all the crap in the
#report that's already formatted. style = courier = monospaced...
echo "<html><pre style=\"font-family:courier\">" >> $MAIL_FILE
#replace any less than greater than signs in the file.
sed -i 's/</\&amp;lt;/g' $FILE
sed -i 's/</\&amp;gt;/g' $FILE
cat $FILE >> $MAIL_FILE
echo "</pre></html>" >> $MAIL_FILE
#email the report.
sendmail some@email.com < $MAIL_FILE
rm $MAIL_FILE

Most of those commands are explained very well here.
Now the only thing left is to make onboot-report.sh run after power-up:

In Ubuntu you just have to run the script in the file /etc/rc.local by adding the following:
bash /root/onboot-report.sh &
I also have the following in /etc/rc.local I find it extremely useful because it lets me know that I can reestablish ssh sessions after a reboot without plugging in a monitor...

beep
sleep .25s
beep
sleep .25s
beep

It's not very complicated, it just makes the PC speaker beep three times after it's completely powered up.

Oh, maybe it's obvious, but after you install "beep" (sudo apt-get install beep) you can only test it from the server itself (tty1-ttyX) not via ssh (which will just beep your local PCs speaker).

Comments

New comments are not allowed.