Many WordPress attacks will create a backdoor WordPress account that has administrative privileges. One way to tell if your WordPress instance has been compromised is to check for the presence of an administrator account that you did not add.

The latest WordPress worm, as described in Matt Mullenweg’s recent post titled How to Keep WordPress Secure, uses javascript on the user page to hide the backdoor administrator account. It’s not enough to log in and check the list of users. You need to check the database itself.

Dougal Campbell had a good post, Checking Your WordPress Security, that talks about how to find the current list of accounts having administrative privileges in your WordPress instance. He provided an SQL query that you could run to get the list.

I’d rather have an email sent to me if the list of administrators changes, rather than having to check manually. I created a simple setup that checks my database every 10 minutes and sends me an email if the list changes. It checks to see if new administrator accounts are added and also checks to see if one was removed and replaced.

Security

At first I thought I could use the WordPress environment itself to make database access easier, but I realized that by doing so I could potentially run compromised code. I chose to directly connect to the database using the credentials in wp-config.php and run my queries using the mysql_* PHP functions.

Implementation

Directory Structure

First I got the directory structure in place. As root:

mkdir /root/etc
mkdir /root/bin

Files

• admin_list_common.php – Common Code – This code is shared by the other two utilities.
• check_admin_list.php – Admin List Checker – This checks to see if the current list of admins matches the known good list.
• create_admin_list_file.php – Admin List Creator – This script will create the initial list of admin users.

Common Include File

IMPORTANT NOTE: Make sure to update the database variables to match your setup. At a minimum, you will need to modify $wordpress_docroot and the database credentials.

In /root/bin/admin_list_common.php I put this:

<?php
 
$wordpress_docroot = "/www/llamalabs.com/content";
$admin_list_file = "/root/etc/myblog.wp_admin_list";
 
// get these variables from wp-config.php, which is in the document root for your blog.
$db_host = "localhost";
$db_user = "user";
$db_pass = "password";
$db_name = "db_name";
 
$db = mysql_connect($db_host, $db_user, $db_pass);
if ( ! $db ) {
    echo "Unable to connect to DB: " . mysql_error() . "\n";
    exit(1);
}
 
if ( ! mysql_select_db($db_name, $db) ) {
    echo "Unable to select db: " . mysql_error() . "\n";
    exit;
}
 
// based off of get_users_of_blog in wp-includes/user.php
function get_admin_users() {
    global $db;
    $sql = "SELECT um.user_id AS ID, u.user_login "
         . "FROM wp_users u, wp_usermeta um "
         . "WHERE u.ID = um.user_id "
         . "    AND um.meta_key = 'wp_capabilities' "
         . "    AND um.meta_value LIKE '%administrator%' "
         . "ORDER BY um.user_id";
 
    $res = mysql_query( $sql, $db );
    if ( ! $res ) {
        echo "Could not execute query: " . mysql_error() . "\n";
        exit(1);
    }
 
    if ( mysql_num_rows( $res ) == 0 ) {
        return array();
    }
 
    $users = array();
    while ( $row = mysql_fetch_assoc( $res ) ) {
        $users[] = $row;
    }
    return $users;
}
 
function get_prev_admin_users() {
    global $admin_list_file;
    $fp = fopen( $admin_list_file, "r" );
    if ( ! $fp ) {
        echo  "Error opening file.\n";
        exit( 1 );
    }
 
    $data = "";
    while ( ! feof( $fp ) ) {
        $data .= fgets( $fp, 4096 );
    }
    return unserialize( $data );
}

Check Admin List

In /root/bin/check_admin_list.php I put this:

<?php
 
require_once 'admin_list_common.php';
 
$prev_admin_users = get_prev_admin_users();
$admin_users = get_admin_users();
 
if ( count( $prev_admin_users ) != count( $admin_users ) ) {
    show_warning_and_exit( $prev_admin_users, $admin_users );
}
 
$admin_count = count( $admins );
 
for ( $i = 0 ; $i < $admin_count ; $i++ ) {
    if ( $admin_users[$i]["ID"] != $prev_admin_users[$i]["ID"] ) {
        show_warning_and_exit( $prev_admin_users, $admin_users );
    }
}
 
function show_warning_and_exit( $prev_admin_users, $admin_users ) {
    echo "Warning! The list of admin users differs.\n\n";
 
    echo "Current Admin Users\n";
    print_admin_users( $admin_users );
 
    echo "Previous Admin Users\n";
    print_admin_users( $prev_admin_users );
 
    exit(1);
}
 
function print_admin_users( $admin_users ) {
    $admin_count = count( $admin_users );
    echo "------------------------------------\n";
    for ( $i = 0 ; $i < $admin_count ; $i++ ) {
        echo "ID({$admin_users[$i]["ID"]}) user_login({$admin_users[$i]["user_login"]})\n";
    }
    echo "\n";
}

Create Admin List File

In create_admin_list_file.php I put this:

<?php
 
require_once 'admin_list_common.php';
 
$fp = fopen( $admin_list_file, 'w' );
if ( ! $fp ) {
    echo "Error opening file .\n";
    exit( 1 );
}
 
$admin_users = get_admin_users();
if ( fwrite( $fp, serialize( $admin_users ) ) === FALSE ) {
    echo "Cannot write to file ($admin_list_file)";
    exit( 1 );
}
 
fclose($fp);

Setup and Testing

Run the command to populate the known list of admin accounts. Then verify that the file was created:

[root@host bin]# php create_admin_list_file.php 
[root@host bin]# cat /root/etc/myblog.wp_admin_list 
a:3:{i:0;a:2:{s:2:"ID";s:1:"1";s:10:"user_login";s:5:"admin";}i...<snipped>
[root@host bin]#

Run the check script and make sure you get an exit status of 0.

[root@host bin]# php check_admin_list.php 
[root@host bin]# echo $?
0
[root@host bin]#

Now, check to make sure that the check works. To do this you can erase the contents of the known good list, test, and then restore. Make sure you get an exit status of 1 from your check scrpt. This will cause cron to mail you the report.

[root@host bin]# php check_admin_list.php 
Warning! The list of admin users differs.
 
Current Admin Users
------------------------------------
ID(1) user_login(admin)
ID(2) user_login(bob)
ID(3) user_login(harry)
 
Previous Admin Users
------------------------------------
ID() user_login()
 
[root@host bin]# echo $?
1
[root@host bin]# php create_admin_list_file.php 
[root@host bin]# php check_admin_list.php 
[root@host bin]# echo $?
0
[root@host bin]#

Set up the Recurring Check

Set up the crontab entry using the crontab command:

crontab -u root -e

The contents of my entry are:

*/10 * * * * php /root/bin/check_admin_list.php

Final Notes

It’s important to test your setup. Make sure that when you add a new admin account you are alerted about the change.

Some people may not want to run the check as root. That’s fine, simply set it up under a different account.

Make sure that cron emails for the user running the cron job are making it to your inbox. Alternatively you could use the PHP mail functions to send yourself an email with the warning.

If you have nonstandard table prefixes you will need to update the SQL queries to reflect your custom table prefixes.

If I didn’t have full root access (ie: shared hosting, etc), I’d look into whether or not my hosting provider gave me the ability to create cron jobs. If so, I’d use whatever facilities were available, keeping the files out of the document root for the site. Yes, you could run these php scripts under your document root, but you should obfuscate them and understand that it’s not as safe to do so.

If you have PHP installed in a non-standard location, make sure to specify the path as well as the binary name when editing your crontab.

If you’re not getting cron emails, check out your system’s maillog and other log files to see what’s happening.

If you do need to add a new administrator account and you get emails about the difference, simply run the create_admin_list_file.php file again in order to update your list of known admin accounts.

Related Links

• http://wordpress.org/development/2009/09/keep-wordpress-secure/
• http://dougal.gunters.org/blog/2009/09/05/checking-your-wordpress-security
• http://www.news-hub.eu/2009/09/wordpress-blogs-under-hack-attack/
• http://lorelle.wordpress.com/2009/09/04/old-wordpress-versions-under-attack/
• http://secunia.com/advisories/product/6745/?task=advisories_2008
• http://scobleizer.com/2009/09/05/i-dont-feel-safe-with-wordpress-hackers-broke-in-and-took-things/

When there’s more than one person making changes to system configuration, I like to keep that system configuration under version control. It makes it easier to track who is making configuration changes, it provides a history of the changes, and you can set up a post-commit hook to notify team members about the changes that are being committed.

One nice feature in Subversion is Keyword Substitution. Keyword substitution allows you to embed keyword anchors into your file that get expanded to show useful information about the file such as the revision, revision date, URL.

I like to embed the Id and the Url near the top of configuration files in a comment block. The Id contains filename, revision, revision time, and user and the Url describes the full URL to the latest version of the file in the repository.

Whenever I encounter a configuration file in a system that has these substitutions in place, I can immediately tell where I need to go in order to check out and commit changes to the file.

Below is an example of how to use Subversion’s keyword substitution. I’ll set up a local subversion repository, add a file and do the keyword substitution. I’m going to do this on my macbook.

First, create an empty repository called ‘system’:

macbook ~ $ sudo mkdir /usr/local/svn
macbook ~ $ sudo chown $USER /usr/local/svn
macbook ~ $ sudo chgrp $USER /usr/local/svn
macbook ~ $ svnadmin create /usr/local/svn/system
macbook ~ $

Next, check out the repository and change directory into the checked out copy. Note, I’m skipping the normal repository structure (trunk, tags, branches) to keep things simple.

macbook ~ $ svn co file:///usr/local/svn/system system
Checked out revision 0.
macbook ~ $ cd system/
macbook ~ $

Make a sample configuration file full of comments:

macbook system $ cat >sample.cfg<<EOD
> # this is a sample configuration file
> #
> # \$Id\$
> # \$URL\$
> #
> EOD
macbook system $

Add the file to the repository:

macbook system $ svn add sample.cfg 
A         sample.cfg
macbook system $ svn ci -m 'adding sample config file to demonstrate keyword substitution'
Adding         sample.cfg
Transmitting file data .
Committed revision 1.
macbook system $

Notice how the keyword anchors are in the file, not the substitutions.

macbook system $ cat sample.cfg 
# this is a sample configuration file
#
# $Id$
# $URL$
#
macbook system $

We’ll need to enable the keyword substitution for sample.cfg using ‘svn propset svn:keywords’.

macbook system $ svn propset svn:keywords "Id URL" sample.cfg
property 'svn:keywords' set on 'sample.cfg'
macbook system $ svn st
 M     sample.cfg
macbook system $ svn ci -m 'enabling keyword substitution for Id and URL on sample.cfg'
Sending        sample.cfg
 
Committed revision 2.
macbook system $

Now let’s look at our configuration file:

macbook system $ cat sample.cfg 
# this is a sample configuration file
#
# $Id: sample.cfg 2 2009-07-22 06:46:35Z dustin $
# $URL: file:///usr/local/svn/system/sample.cfg $
#
macbook system $

There’s our Id and URL. Note, if you checked out an existing repository over HTTP(S) or SSH, the URL would reflect that URL. So for a team accessing the SVN repository through Apache+WebDAV the example may look like this:

macbook system $ cat sample.cfg 
# this is a sample configuration file
#
# $Id: sample.cfg 2 2009-07-22 06:46:35Z dustin $
# $URL: https://svn.example.com/svn/system/sample.cfg $
#
macbook system $

Make sure to read through the docs at Keyword Substitution for more examples and additional details.

This seems to be my life living with EC2 instances…

Last night the server hosting the Dice Game failed. At first there were some filesystem errors. Then the root filesystem was remounted in read-only mode. All this happened while I was pushing out some changes to the game.

I issued a reboot of the underlying server and waited for it to come back up. I heard nothing from it. Remote console access wasn’t working; nothing was showing up on the console. How long should I wait before finding a new home for the Dice Game? The server was powercycled and there was still no output on the console. I think it’s dead.

It just so happens that I have another OpenVZ container at a hosting provider in Seattle, but it’s running a rather old version of CentOS. It would have to do.

I removed the incumbent version of Ruby and downloaded, built and installed the latest stable release from scratch. I installed rubygems and all of the required gems to make the Dice Game work. I had to rebuild Apache from scratch to support mod_proxy and mod_proxy_reverse (so it could send requests to the mongrel processes). I configured Apache, Mongrel Cluster and Mysql. I restored the Dice Game database backup from my last hourly backup. I installed Git so I could push out my local repository to the new server. I created a new hostname in DNS and updated the Facebook application configuration to use the new hostname.

It took about two hours to get everything built and restored on the new system, which is less time than it would have taken to order and have a brand new dedicated server delivered. The Dice Game seemed a bit more snappy on the new system too.

I found out the next morning that the original server was not completely dead. I’m not going to move the game back to it though, unless the new system dies. If I have to do so, I know that the old system is ready to be used as a replacement.

Links:

• OpenVZ
• Ruby
• Ruby GEMS
• Ruby on Rails (ROR)
• Mislav will_paginate gem
• Mongrel Cluster
• Ruby MySQL gem
• Git
• MySQL
• Apache HTTPD
• Capistrano

The Filesystem Errors:


EXT3-fs warning (device sda2): ext3_rmdir: empty directory has nlink!=2 (-1)
EXT3-fs warning (device sda2): ext3_rmdir: empty directory has nlink!=2 (-2)
EXT3-fs warning (device sda2): empty_dir: bad directory (dir #17991246) - no `.' or `..'
EXT3-fs warning (device sda2): ext3_rmdir: empty directory has nlink!=2 (1)
EXT3-fs warning (device sda2): empty_dir: bad directory (dir #17991247) - no `.' or `..'
EXT3-fs warning (device sda2): ext3_rmdir: empty directory has nlink!=2 (8)
EXT3-fs unexpected failure: !buffer_revoked(bh);
inconsistent data on disk
ext3_forget: aborting transaction: IO failure in __ext3_journal_revoke
ext3_abort called.
EXT3-fs error (device sda2): ext3_forget: error -5 when attempting revoke
Remounting filesystem read-only
Aborting journal on device sda2.
EXT3-fs error (device sda2) in ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device sda2) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sda2) in ext3_truncate: IO failure
EXT3-fs error (device sda2) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sda2) in ext3_orphan_del: Journal has aborted
EXT3-fs error (device sda2) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sda2) in ext3_delete_inode: IO failure


From http://www.dban.org/:

Darik’s Boot and Nuke (“DBAN”) is a self-contained boot disk that securely wipes the hard disks of most computers. DBAN will automatically and completely delete the contents of any hard disk that it can detect, which makes it an appropriate utility for bulk or emergency data destruction.

I’d have to say, it has definitely lived up to its name. I downloaded the ISO image, burned it to a CD and several unattended hours later I had four clean machines. You boot one, remove the CD, and move on to the next machine.

Now I just need to find a new home for these machines.

• Amazon Public Images – Windows Server 2003 R2 (32bit)
• Feature Guide: Amazon EC2 Running Windows
• Elasticfox Firefox Extension for Amazon EC2
• Elasticfox Getting Started Guide
• CoRD: Remote Desktop for Mac OS X

Once you read the directions and get your instance started, you can use your favorite Remote Desktop client to access your instance:

The new API allows you to get the administrator password. Log in as Administrator with the password you get back and you’re greeted with the desktop.

Big deal? I want to play solitaire on it. Oh, it doesn’t come with Solitaire, you’ll have to supply that yourself (Thanks Gmail for not letting me send Windows executables). It also seems that they don’t provide an easy way to access the installation media for Windows either (Not that I could install it from the install media if I wanted to, and I’ll probably be proven wrong on this anyhow).

Wasn’t that exciting? ec2kill time.

It has been demonstrated that Quercus outperforms a straight-out-of-the box installation of mod_php.

1. “Performance: Quercus outperforms a straight mod_php implementation by about 4x (for Mediawiki and Drupal).” [1]
2. “Resin backed PHP drives 4x performance improvements for Drupal” [2]

I had always wondered how Quercus compared to APC though, specifically for running a Drupal instance. I had planned on performing a bunch of tests, but they state clearly enough in the docs that it roughly matches PHP performance with accelerators like APC. [1]

That’s exactly what I needed to know. I’ll stick with APC until I need to write some new PHP functions in Java. So sorry to disappoint you with a lack of no new conclusions, but mod_php + APC keeps me pretty happy.

Links:
[1] http://www.caucho.com/resin-3.0/quercus/
[2] http://www.workhabit.com/labs/resin-backed-php-drives-4x-performance-improvements-drupal

Yesterday’s outage reminded me of my list and I wanted to get around to posting it.

Hopefully these notes will be useful to someone who is evaluating Google Apps for their email/calendar/docs solution.

Without any specific order, here are the notes I have collected.

There’s no way to enforce communication over HTTPS through the admin control panel.

Before the recent announcement of the option to require https, by default you would be redirected to http://mail.google.com after signing in. With employees working remotely, not being able to force communication using HTTPS was a real bummer. There is now the option to always use HTTPS in Google Apps Premier. It’s still possible to switch back to plain old HTTP once logged in.

Here are a few pages discussing the issue. It’s no really such an issue anymore, but they are in my notes, so I’m including them here.

• Using SSL/HTTPS for Gmail and Google Apps.
• Google Apps for Enterprise missing full-time SSL?
• SSL on Google means Semi Secure Links
• Fix A Problem > Need to be able to enforce the use of SSL

You are forced to go through the Google Checkout process to add new seats

There’s no easy way to add a new user account without going through the Google Checkout process. If you’re purchasing single seats each time you have a new hire, that means that you need to go through the Google Checkout process in addition to adding the account. You could avoid this hassle by purchasing seats in advance.

Ideally there would be an option to charge the credit card on file. It’s already stored in the account used for Google Checkout, so why not automate the process?

Can’t forward one user’s email to another user in the same domain

So, when someone leaves the company, and you need to forward new messages destined for her account to her supervisor, you’d usually put a forward in place.

When I try to do so in Google Apps I get the message “Forwarding to Google Apps hosted email addresses is not supported”. I’m not the only person having this problem.

I’m able to do so by logging in as that user and putting the forward in place under the user’s Gmail settings.

I’ve been going back and forth with Google Apps Premier support for a couple weeks now attempting to troubleshoot this issue. I’m assured it’s a problem specific to my account.

Disabling account access.

When you change the password for an account, it does not invalidate existing logged-in sessions. If the user is logged in, she will be able to to still use her account.

Why not suspend the account? That disables logging in, and will log a user out after several minutes, but it also blocks incoming mail to that user.

A third option is to log in as the user and click the “Details” link that is part of the account activity messaging. That opens a pop-up that shows you existing logged in sessions and gives you the opportunity to log those sessions out. It would be nice if that feature was in the Admin control panel.

Unreliable IMAP

I don’t personally use IMAP for my email, but there have been confirmed cases of messages being delayed for 20+ minutes when using IMAP. The messages will show up instantly through the Gmail interface,
but in an external mail reader the new messages won’t show up. YMMV.

Some messages silently ignored during migration

If you use their IMAP migration tool (Which is very handy), you may notice that some messages (containing zip files) are not migrated.

Support

It’s been hit and miss when dealing with Google Apps Technical Support.

Customer: We're having problems checking mail using IMAP.
Support: Which web browser are you using?


The online documents are nice.

Feature Suggestions

Make sure to take a look at the Feature Suggestions to make sure some critical feature you require isn’t in the pipeline.

Google Groups not included

Before signing up I looked into whether or not Google Groups was one of the included applications.

Pine Star said:

Hello everyone interested in this integration:

I had a chat with the Google Rep of Commercial (paid) version of
Google Apps yesterday.
He said they are going to release a “Premier” edition of Google Apps
by end of 2007 that wil have Google Groups integrated.

So, they are also thinking on these lines….

If you’re looking for Google Groups for your projects in Google Apps Premier, you will be disappointed to find out that it’s not included.

We were really hoping to have it so we could have archived discussions for projects.

Email Lists

You can create email lists that deliver to multiple recipients, both members of the domain and external email addresses.

A couple notes on the lists.

• No subject prefixes.
• No List-id field.
• Not as cool as having Google Groups included as an app.

Calender Importing

Aside from using the API, there appears to be no way to import an entire calendar into a new instance.

Branding with a Custom Logo

You can define a custom logo to be displayed instead of the default Google logo.

Conclusion?

Like any service, Google Apps Premier has its pros and cons. Research thoroughly before making the leap.

In addition to his configuration and aliases, I have a couple of my own that I have added. Here’s an excerpt from my .profile:

git config --global user.name "Your Name"
git config --global user.email "your email"
git config --global color.diff auto
git config --global color.status auto
git config --global color.branch auto
git config --global color.interactive auto
git config --global alias.st status
git config --global alias.ci commit
git config --global alias.co checkout
git config --global alias.br branch
git config --global merge.tool opendiff
git config --global merge.summary true
git config --global core.excludesfile ~/.gitignore

alias g='git'
alias gst='g st'
alias gci='g ci'
alias gdiff='g diff'
alias gadd='g add'

I don’t have anything against the space bar, but it does seem to be easier with those aliases. See Tim’s article for some additional comments on some of the git configuration.

The color configuration commands will also add quite a bit of richness to your git experience.

When you run ssh-agent it creates a long-running process that holds decrypted keys and spits out some environment variables that the ssh client can use:

SSH_AUTH_SOCK=/tmp/ssh-Ksfjq28070/agent.28070; export SSH_AUTH_SOCK;
SSH_AGENT_PID=28071; export SSH_AGENT_PID;
echo Agent pid 28071;

The problem is that you want to have those environment variables sourced in when you open new terminal windows. I added the following to my .profile to write those lines out to a file so they could be sourced in easily.

test -e ~/.ssh_agent && . ~/.ssh_agent
need_to_start_ssh_agent="0"
if test "" != "$SSH_AGENT_PID" ; then
  /bin/ps -p $SSH_AGENT_PID | /usr/bin/grep ssh-agent > /dev/null
  res=$?
  if test "1" == "$res" ; then
    echo "ssh-agent is not running. We need to start the agent"
    need_to_start_ssh_agent="1"
  else
    echo "ssh-agent running - pid: $SSH_AGENT_PID"
  fi
else
  echo "ssh-agent is not running. We need to start the agent"
  need_to_start_ssh_agent="1"
fi

if test "1" == "$need_to_start_ssh_agent" ; then
  `which ssh-agent` | grep -v echo > ~/.ssh_agent
  . ~/.ssh_agent
  `which ssh-add`
fi

After using this for a while I found Daniel Robbin’s article about his keychain utility http://www.ibm.com/developerworks/library/l-keyc2/. It’s basically a more-refined version of my addition to .profile. I’m still going to stick with my solution for now because it’s so lightweight.

$ dscacheutil -cachedump
DirectoryService Cache Overview:
AAAA Queries  - Disabled (link-local IPv6 addresses)
Buckets Used  - 31
Cache Size    - 16

Entry count by category:
Group  - 2
Host   - 11
User   - 3

Flush:

$ dscacheutil -flushcache
$

All gone:

$ dscacheutil -cachedump
DirectoryService Cache Overview:
AAAA Queries  - Disabled (link-local IPv6 addresses)
Buckets Used  - 0
Cache Size    - 0

Entry count by category:

Please see the previous post to see how to clear the DNS cache on OS X Tiger http://llamalabs.com/2007/04/18/how-to-flush-your-dns-cache-on-os-x/

rpm -qa --queryformat '%{SIZE} %{NAME} %{VENDOR}\n'  | sort -n

The -qa flag specifies that we wish to query all packages. The –queryformat flag specifies how we want each result to be formatted. See below for additional items you can put in your query format string. The -n flag for sort tells us to perform a numerical sort (rather than alphabetical). Try it without the -n to see how the output looks.

Here’s an example to see the 10 installed packages that take up the most space:

[user@host ~]$ rpm -qa --queryformat '%{SIZE} %{NAME} %{VENDOR}\n'  | sort -n -r | head -10
195644952 VMware-server VMware, Inc.
113550744 jdk Sun Microsystems, Inc.
66639973 glibc-common CentOS
52098364 compat-gcc-34-c++ CentOS
51851071 festival CentOS
45929176 libstdc++-devel CentOS
41768729 libgcj CentOS
41719064 MySQL-server-community MySQL AB
41653082 frysk CentOS
38859091 firefox CentOS
[user@host ~]$

There are many options that you can specify in your query format. Try this command to get a list:

[user@host ~]$ rpm --querytags
HEADERIMAGE
HEADERSIGNATURES
HEADERIMMUTABLE
HEADERREGIONS
HEADERI18NTABLE
SIGSIZE
SIGPGP
--snip--

He talked about the Using csh & tcsh book and gave some examples of some shortcuts. I remember using those shortcuts for a while but eventually forgot them all.

What I really found interesting were the CLI tools for interfacing with the clipboard in Mac OS X. I was surprised to see that those utilities existed. See the man pages for pbcopy and pbpaste for details on those tools.

I figured i’d mention a couple of useful commands that I use on a regular basis.

The first is Ctrl+L to clear the screen. Old habits can die hard, but there’s no reason to type ‘clear’ and hit enter to clear the screen if your shell supports Ctrl+L. I’m not sure when this feature came about, but it seems to work on tcsh 6.12.00 and bash 2.05 and 3.

The second thing I use frequently is history along with the !n shell shortcut (where n is the history id). History shows an integer next to each command. If you wish to run the command with the number 42 at the beginning of the line, you simply type !42 at the prompt.

The other benefit of using history is in a shared server environment. You’d be surprised at what you can learn by looking at the commands that other people run.

-p, --program
  Show the PID and name of the program to which each socket belongs.
-a, --all
  Show both listening and non-listening sockets.  With the --interfaces option,
  show interfaces that are not marked
--numeric , -n
  Show numerical addresses instead of trying to determine symbolic host,
  port or user names.

This gives a nice output like this:

[root@host ~]# netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address  Foreign Address  State  PID/Program name
tcp        0      0 0.0.0.0:22       0.0.0.0:*     LISTEN      10587/sshd
tcp        0      0 0.0.0.0:80       0.0.0.0:*     LISTEN      16872/httpd
tcp        0      0 0.0.0.0:21       0.0.0.0:*     LISTEN      26698/proftpd
tcp        0      0 0.0.0.0:25       0.0.0.0:*     LISTEN      2209/sendmail

It can be a lot faster using the netstat -anp method on heavily loaded systems.


[root@host ~]# dd if=/dev/zero of=/swapfile bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
[root@host ~]# sync
[root@host ~]# mkswap /swapfile
Setting up swapspace version 1, size = 1073737 kB
[root@host ~]# swapon /swapfile
[root@host ~]# echo "/swapfile swap swap defaults 0 0" >> /etc/fstab


The problem with just writing a ’.' is that most terminals/operating systems use a feature called “buffering”.  When you write to a file (or terminal/stdout) the operating system doesn’t immediately flush that output to where you are writing.  It will wait until a sufficient amount of data has been written, and then flush it all out at the same time to save on resources/system calls.  A single ‘.’ is nowhere near enough data to signal the OS to flush that output, so you’ll end up seeing a bunch of them all written when the loop completes, or worst yet, nothing until the script is finished.  For example:

#!/usr/bin/env ruby
20.times do
putc('.')
sleep(5)
end

On first glance, you’d think this script would print 20 dots on a line, one every 5 seconds, but it doesn’t! When run, it sits there for a full minute without outputting anything, and then spits out 20 dots right before quitting. We need to turn off write buffering on $stdout by turning on sync mode:
$stdout.sync=(true) if not $stdout.sync

If you add that line to the top of the script, each dot is written to the terminal right away.

#!/bin/sh
# ... other stuff ...
sudo cat > $CHROOTMNT/tmp/chroot.sh <<'CHROOT'
#!/bin/bash
cd /usr/local/
unzip /tmp/ec2-api-tools.zip
ln -s ec2-api-tools* ec2-api-tools
chmod -R a-w /usr/local/ec2-api-tools/
rm /tmp/ec2-api-tools.zip
cat >> /etc/profile <<'EC2APIPROFILE'

export EC2_HOME=/usr/local/ec2-api-tools
export PATH=$PATH:$EC2_HOME/bin
EC2APIPROFILE
# ... other stuff ...
CHROOT
sudo chmod a+x $CHROOTMNT/tmp/chroot.sh
sudo chroot $CHROOTMNT /tmp/chroot.sh

I was having trouble before finding this.

Disabling parameter substitution permits outputting literal text. Generating scripts or even program code is one use for this.

Without disabling parameter substitution, it was expanding "$PATH" to my current $PATH, instead of writing "$PATH" like I wanted. Sure, I could have unset it before running the script, but this is much more useful.

From the Subversion book: Commit Log Message Correction

Example:

[admin@svn]$ echo "Here is the new, correct log message" > newlog.txt
[admin@svn]$ svnadmin setlog myrepos newlog.txt -r 388 --bypass-hooks

From the command line:

sudo lookupd -flushcache

And just for fun, you can examine your resolver configuration using:

lookupd -configuration

> /path/to/file

That results in the open system call being called with the O_TRUNC flag.

open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666)

You can use dpkg to figure out what package installed a file on your system. For instance, I don’t particularly like emacs so was suprised to find /etc/emacs is installed with a default Debian Etch install.
debian:~# dpkg -S /etc/emacs
dictionaries-common: /etc/emacs
To see more information about that package (to decide what to do with it). Use apt-cache show
debian:~# apt-cache show dictionaries-common
Package: dictionaries-common
Priority: standard
Section: text
Installed-Size: 745
Maintainer: Agustin Martin Domingo
Architecture: all
Version: 0.70.10
Replaces: openoffice.org-updatedicts
Provides: openoffice.org-updatedicts
Depends: perl-base (>= 5.6.0-16), debconf (>= 0.5) | debconf-2.0
Suggests: ispell, emacsen-common, jed-extra
Conflicts: ispell (<= 3.1.20.0-1), miscfiles (<< 1.3-2.1), iamerican (<= 3.1.20.0-1), ibrazilian (<< 2.4-5.1), ibritish (<= 3.1.20.0-1), ibulgarian (<= 2.0-2), icatalan (<= 0.1-4), iczech (<= 20020628-1), idanish (<< 1.4.22-2.1), idutch (<= 1:0.1e-20), iesperanto (<< 2.1.2000.02.25-6), ifaroese (<= 0.1.16-2), ifinnish (<< 0.7-3.4), ifinnish-large (<< 0.7-3.4), ifinnish-small (<< 0.7-3.4), ifrench (<= 1.4-13), ifrench-gut (<= 1:1.0-9), igerman, ihungarian (<= 0.84-1), iitalian (<< 2.20-1.2), ingerman (<< 20010414-2), inorwegian (<< 2.0-6.1), ipolish (<< 20011004-2.1), iportuguese (<< 19980611-8), irussian (<= 0.99f0-1), ispanish (<< 1.7-5), ispell-ga, iswedish (<= 1.4.2), wbritish (<= 3.1.20.0-1), wbulgarian (<= 2.0-2), wcatalan (<= 0.1-4), wdanish (<< 1.4.22-2.1), wdutch (<= 1:0.1e-20), wenglish (<= 2.0-2), wfaroese (<= 0.1.16-2), wfinnish (<< 0.7-3.4), wfrench (<= 1.0-11), wgerman, witalian (<= 1.6), wnorwegian (<< 2.0-6.1), wngerman (<< 20010414-2), wpolish (<< 20011004-2.1), wspanish (<= 1.0.11.1), wswedish (<= 1.4.2), openoffice.org-updatedicts
Filename: pool/main/d/dictionaries-common/dictionaries-common_0.70.10_all.deb
Size: 250004
MD5sum: e9a26a3367d2373a20368d1fe6b22a73
SHA1: 16730ac0cac691e5bd9a1edc123be83902214738
SHA256: 39ce6f1684a2b568e78467656bfdcd605dae20f588dc7544b08f03aaadd1e789
Description: Common utilities for spelling dictionary tools
These are utilities shared by all ispell, myspell and wordlist
dictionaries, including support for some tools that use ispell
(like emacsen, jed and mutt). More info about naming conventions and
availability of those dictionaries in the README file.
.
Maintainers should install dictionaries-common-dev as well, and read its
documentation.
Tag: implemented-in::lisp, implemented-in::perl, role::plugin, role::program, scope::utility, special::auto-inst-parts, works-with::dictionary
http://www.debian.org/doc/manuals/reference/ch-package.en.html


[root@host ~]# rpm -qf /bin/bash
bash-3.0-19.3


So /bin/bash belongs to bash-3.0-19.3

The -q option lets rpm know that you want to do a query of the database. The f lets you query the package containing the file you specify.

Here’s another query:


[root@host ~]# rpm -qf /usr/bin/xargs
findutils-4.1.20-7.el4.1
[root@host ~]#


yum lets you get info on the findutils package:


[root@host ~]# yum info findutils
Setting up repositories
update 100% |=========================| 951 B 00:00
rpmforge 100% |=========================| 1.1 kB 00:00
base 100% |=========================| 1.1 kB 00:00
addons 100% |=========================| 951 B 00:00
extras 100% |=========================| 1.1 kB 00:00
Reading repository metadata in from local files
Installed Packages
Name : findutils
Arch : i386
Epoch : 1
Version: 4.1.20
Release: 7.el4.1
Size : 231 k
Repo : installed
Summary: The GNU versions of find utilities (find and xargs).


Description:
The findutils package contains programs which will help you locate
files on your system. The find utility searches through a hierarchy
of directories looking for files which match a certain set of criteria
(such as a filename pattern). The xargs utility builds and executes
command lines from standard input arguments (usually lists of file
names generated by the find command).


You should install findutils because it includes tools that are very
useful for finding things on your system.


So, if you are using a system that uses RPM/YUM, you can easily find out the packages that files on your filesystem belong to.

Once you have identified the package that you are looking for you may wish to find out which files are part of that package. To do so you use the query list option (-ql). Here is an example for the bash package:


[dustin@host ~]$ rpm -ql bash
/bin/bash
/bin/sh
/etc/skel/.bash_logout
/etc/skel/.bash_profile
/etc/skel/.bashrc
/usr/bin/bashbug-32
/usr/share/doc/bash-3.0
/usr/share/doc/bash-3.0/CHANGES
--snip--
/usr/share/man/man1/unset.1.gz
/usr/share/man/man1/wait.1.gz
[dustin@host ~]$