This seems to be my life living with EC2 instances…

Last night the server hosting the Dice Game failed. At first there were some filesystem errors. Then the root filesystem was remounted in read-only mode. All this happened while I was pushing out some changes to the game.

I issued a reboot of the underlying server and waited for it to come back up. I heard nothing from it. Remote console access wasn’t working; nothing was showing up on the console. How long should I wait before finding a new home for the Dice Game? The server was powercycled and there was still no output on the console. I think it’s dead.

It just so happens that I have another OpenVZ container at a hosting provider in Seattle, but it’s running a rather old version of CentOS. It would have to do.

I removed the incumbent version of Ruby and downloaded, built and installed the latest stable release from scratch. I installed rubygems and all of the required gems to make the Dice Game work. I had to rebuild Apache from scratch to support mod_proxy and mod_proxy_reverse (so it could send requests to the mongrel processes). I configured Apache, Mongrel Cluster and Mysql. I restored the Dice Game database backup from my last hourly backup. I installed Git so I could push out my local repository to the new server. I created a new hostname in DNS and updated the Facebook application configuration to use the new hostname.

It took about two hours to get everything built and restored on the new system, which is less time than it would have taken to order and have a brand new dedicated server delivered. The Dice Game seemed a bit more snappy on the new system too.

I found out the next morning that the original server was not completely dead. I’m not going to move the game back to it though, unless the new system dies. If I have to do so, I know that the old system is ready to be used as a replacement.

Links:

• OpenVZ
• Ruby
• Ruby GEMS
• Ruby on Rails (ROR)
• Mislav will_paginate gem
• Mongrel Cluster
• Ruby MySQL gem
• Git
• MySQL
• Apache HTTPD
• Capistrano

The Filesystem Errors:


EXT3-fs warning (device sda2): ext3_rmdir: empty directory has nlink!=2 (-1)
EXT3-fs warning (device sda2): ext3_rmdir: empty directory has nlink!=2 (-2)
EXT3-fs warning (device sda2): empty_dir: bad directory (dir #17991246) - no `.' or `..'
EXT3-fs warning (device sda2): ext3_rmdir: empty directory has nlink!=2 (1)
EXT3-fs warning (device sda2): empty_dir: bad directory (dir #17991247) - no `.' or `..'
EXT3-fs warning (device sda2): ext3_rmdir: empty directory has nlink!=2 (8)
EXT3-fs unexpected failure: !buffer_revoked(bh);
inconsistent data on disk
ext3_forget: aborting transaction: IO failure in __ext3_journal_revoke
ext3_abort called.
EXT3-fs error (device sda2): ext3_forget: error -5 when attempting revoke
Remounting filesystem read-only
Aborting journal on device sda2.
EXT3-fs error (device sda2) in ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device sda2) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sda2) in ext3_truncate: IO failure
EXT3-fs error (device sda2) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sda2) in ext3_orphan_del: Journal has aborted
EXT3-fs error (device sda2) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sda2) in ext3_delete_inode: IO failure


It has been demonstrated that Quercus outperforms a straight-out-of-the box installation of mod_php.

1. “Performance: Quercus outperforms a straight mod_php implementation by about 4x (for Mediawiki and Drupal).” [1]
2. “Resin backed PHP drives 4x performance improvements for Drupal” [2]

I had always wondered how Quercus compared to APC though, specifically for running a Drupal instance. I had planned on performing a bunch of tests, but they state clearly enough in the docs that it roughly matches PHP performance with accelerators like APC. [1]

That’s exactly what I needed to know. I’ll stick with APC until I need to write some new PHP functions in Java. So sorry to disappoint you with a lack of no new conclusions, but mod_php + APC keeps me pretty happy.

Links:
[1] http://www.caucho.com/resin-3.0/quercus/
[2] http://www.workhabit.com/labs/resin-backed-php-drives-4x-performance-improvements-drupal

Yesterday’s outage reminded me of my list and I wanted to get around to posting it.

Hopefully these notes will be useful to someone who is evaluating Google Apps for their email/calendar/docs solution.

Without any specific order, here are the notes I have collected.

There’s no way to enforce communication over HTTPS through the admin control panel.

Before the recent announcement of the option to require https, by default you would be redirected to http://mail.google.com after signing in. With employees working remotely, not being able to force communication using HTTPS was a real bummer. There is now the option to always use HTTPS in Google Apps Premier. It’s still possible to switch back to plain old HTTP once logged in.

Here are a few pages discussing the issue. It’s no really such an issue anymore, but they are in my notes, so I’m including them here.

• Using SSL/HTTPS for Gmail and Google Apps.
• Google Apps for Enterprise missing full-time SSL?
• SSL on Google means Semi Secure Links
• Fix A Problem > Need to be able to enforce the use of SSL

You are forced to go through the Google Checkout process to add new seats

There’s no easy way to add a new user account without going through the Google Checkout process. If you’re purchasing single seats each time you have a new hire, that means that you need to go through the Google Checkout process in addition to adding the account. You could avoid this hassle by purchasing seats in advance.

Ideally there would be an option to charge the credit card on file. It’s already stored in the account used for Google Checkout, so why not automate the process?

Can’t forward one user’s email to another user in the same domain

So, when someone leaves the company, and you need to forward new messages destined for her account to her supervisor, you’d usually put a forward in place.

When I try to do so in Google Apps I get the message “Forwarding to Google Apps hosted email addresses is not supported”. I’m not the only person having this problem.

I’m able to do so by logging in as that user and putting the forward in place under the user’s Gmail settings.

I’ve been going back and forth with Google Apps Premier support for a couple weeks now attempting to troubleshoot this issue. I’m assured it’s a problem specific to my account.

Disabling account access.

When you change the password for an account, it does not invalidate existing logged-in sessions. If the user is logged in, she will be able to to still use her account.

Why not suspend the account? That disables logging in, and will log a user out after several minutes, but it also blocks incoming mail to that user.

A third option is to log in as the user and click the “Details” link that is part of the account activity messaging. That opens a pop-up that shows you existing logged in sessions and gives you the opportunity to log those sessions out. It would be nice if that feature was in the Admin control panel.

Unreliable IMAP

I don’t personally use IMAP for my email, but there have been confirmed cases of messages being delayed for 20+ minutes when using IMAP. The messages will show up instantly through the Gmail interface,
but in an external mail reader the new messages won’t show up. YMMV.

Some messages silently ignored during migration

If you use their IMAP migration tool (Which is very handy), you may notice that some messages (containing zip files) are not migrated.

Support

It’s been hit and miss when dealing with Google Apps Technical Support.

Customer: We're having problems checking mail using IMAP.
Support: Which web browser are you using?


The online documents are nice.

Feature Suggestions

Make sure to take a look at the Feature Suggestions to make sure some critical feature you require isn’t in the pipeline.

Google Groups not included

Before signing up I looked into whether or not Google Groups was one of the included applications.

Pine Star said:

Hello everyone interested in this integration:

I had a chat with the Google Rep of Commercial (paid) version of
Google Apps yesterday.
He said they are going to release a “Premier” edition of Google Apps
by end of 2007 that wil have Google Groups integrated.

So, they are also thinking on these lines….

If you’re looking for Google Groups for your projects in Google Apps Premier, you will be disappointed to find out that it’s not included.

We were really hoping to have it so we could have archived discussions for projects.

Email Lists

You can create email lists that deliver to multiple recipients, both members of the domain and external email addresses.

A couple notes on the lists.

• No subject prefixes.
• No List-id field.
• Not as cool as having Google Groups included as an app.

Calender Importing

Aside from using the API, there appears to be no way to import an entire calendar into a new instance.

Branding with a Custom Logo

You can define a custom logo to be displayed instead of the default Google logo.

Conclusion?

Like any service, Google Apps Premier has its pros and cons. Research thoroughly before making the leap.

In addition to his configuration and aliases, I have a couple of my own that I have added. Here’s an excerpt from my .profile:

git config --global user.name "Your Name"
git config --global user.email "your email"
git config --global color.diff auto
git config --global color.status auto
git config --global color.branch auto
git config --global color.interactive auto
git config --global alias.st status
git config --global alias.ci commit
git config --global alias.co checkout
git config --global alias.br branch
git config --global merge.tool opendiff
git config --global merge.summary true
git config --global core.excludesfile ~/.gitignore

alias g='git'
alias gst='g st'
alias gci='g ci'
alias gdiff='g diff'
alias gadd='g add'

I don’t have anything against the space bar, but it does seem to be easier with those aliases. See Tim’s article for some additional comments on some of the git configuration.

The color configuration commands will also add quite a bit of richness to your git experience.

rpm -qa --queryformat '%{SIZE} %{NAME} %{VENDOR}\n'  | sort -n

The -qa flag specifies that we wish to query all packages. The –queryformat flag specifies how we want each result to be formatted. See below for additional items you can put in your query format string. The -n flag for sort tells us to perform a numerical sort (rather than alphabetical). Try it without the -n to see how the output looks.

Here’s an example to see the 10 installed packages that take up the most space:

[user@host ~]$ rpm -qa --queryformat '%{SIZE} %{NAME} %{VENDOR}\n'  | sort -n -r | head -10
195644952 VMware-server VMware, Inc.
113550744 jdk Sun Microsystems, Inc.
66639973 glibc-common CentOS
52098364 compat-gcc-34-c++ CentOS
51851071 festival CentOS
45929176 libstdc++-devel CentOS
41768729 libgcj CentOS
41719064 MySQL-server-community MySQL AB
41653082 frysk CentOS
38859091 firefox CentOS
[user@host ~]$

There are many options that you can specify in your query format. Try this command to get a list:

[user@host ~]$ rpm --querytags
HEADERIMAGE
HEADERSIGNATURES
HEADERIMMUTABLE
HEADERREGIONS
HEADERI18NTABLE
SIGSIZE
SIGPGP
--snip--

He talked about the Using csh & tcsh book and gave some examples of some shortcuts. I remember using those shortcuts for a while but eventually forgot them all.

What I really found interesting were the CLI tools for interfacing with the clipboard in Mac OS X. I was surprised to see that those utilities existed. See the man pages for pbcopy and pbpaste for details on those tools.

I figured i’d mention a couple of useful commands that I use on a regular basis.

The first is Ctrl+L to clear the screen. Old habits can die hard, but there’s no reason to type ‘clear’ and hit enter to clear the screen if your shell supports Ctrl+L. I’m not sure when this feature came about, but it seems to work on tcsh 6.12.00 and bash 2.05 and 3.

The second thing I use frequently is history along with the !n shell shortcut (where n is the history id). History shows an integer next to each command. If you wish to run the command with the number 42 at the beginning of the line, you simply type !42 at the prompt.

The other benefit of using history is in a shared server environment. You’d be surprised at what you can learn by looking at the commands that other people run.

-p, --program
  Show the PID and name of the program to which each socket belongs.
-a, --all
  Show both listening and non-listening sockets.  With the --interfaces option,
  show interfaces that are not marked
--numeric , -n
  Show numerical addresses instead of trying to determine symbolic host,
  port or user names.

This gives a nice output like this:

[root@host ~]# netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address  Foreign Address  State  PID/Program name
tcp        0      0 0.0.0.0:22       0.0.0.0:*     LISTEN      10587/sshd
tcp        0      0 0.0.0.0:80       0.0.0.0:*     LISTEN      16872/httpd
tcp        0      0 0.0.0.0:21       0.0.0.0:*     LISTEN      26698/proftpd
tcp        0      0 0.0.0.0:25       0.0.0.0:*     LISTEN      2209/sendmail

It can be a lot faster using the netstat -anp method on heavily loaded systems.


[root@host ~]# dd if=/dev/zero of=/swapfile bs=1024 count=1048576
1048576+0 records in
1048576+0 records out
[root@host ~]# sync
[root@host ~]# mkswap /swapfile
Setting up swapspace version 1, size = 1073737 kB
[root@host ~]# swapon /swapfile
[root@host ~]# echo "/swapfile swap swap defaults 0 0" >> /etc/fstab


The problem with just writing a ’.' is that most terminals/operating systems use a feature called “buffering”.  When you write to a file (or terminal/stdout) the operating system doesn’t immediately flush that output to where you are writing.  It will wait until a sufficient amount of data has been written, and then flush it all out at the same time to save on resources/system calls.  A single ‘.’ is nowhere near enough data to signal the OS to flush that output, so you’ll end up seeing a bunch of them all written when the loop completes, or worst yet, nothing until the script is finished.  For example:

#!/usr/bin/env ruby
20.times do
putc('.')
sleep(5)
end

On first glance, you’d think this script would print 20 dots on a line, one every 5 seconds, but it doesn’t! When run, it sits there for a full minute without outputting anything, and then spits out 20 dots right before quitting. We need to turn off write buffering on $stdout by turning on sync mode:
$stdout.sync=(true) if not $stdout.sync

If you add that line to the top of the script, each dot is written to the terminal right away.

#!/bin/sh
# ... other stuff ...
sudo cat > $CHROOTMNT/tmp/chroot.sh <<'CHROOT'
#!/bin/bash
cd /usr/local/
unzip /tmp/ec2-api-tools.zip
ln -s ec2-api-tools* ec2-api-tools
chmod -R a-w /usr/local/ec2-api-tools/
rm /tmp/ec2-api-tools.zip
cat >> /etc/profile <<'EC2APIPROFILE'

export EC2_HOME=/usr/local/ec2-api-tools
export PATH=$PATH:$EC2_HOME/bin
EC2APIPROFILE
# ... other stuff ...
CHROOT
sudo chmod a+x $CHROOTMNT/tmp/chroot.sh
sudo chroot $CHROOTMNT /tmp/chroot.sh

I was having trouble before finding this.

Disabling parameter substitution permits outputting literal text. Generating scripts or even program code is one use for this.

Without disabling parameter substitution, it was expanding "$PATH" to my current $PATH, instead of writing "$PATH" like I wanted. Sure, I could have unset it before running the script, but this is much more useful.

From the Subversion book: Commit Log Message Correction

Example:

[admin@svn]$ echo "Here is the new, correct log message" > newlog.txt
[admin@svn]$ svnadmin setlog myrepos newlog.txt -r 388 --bypass-hooks

From the command line:

sudo lookupd -flushcache

And just for fun, you can examine your resolver configuration using:

lookupd -configuration

> /path/to/file

That results in the open system call being called with the O_TRUNC flag.

open("/path/to/file", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666)

You can use dpkg to figure out what package installed a file on your system. For instance, I don’t particularly like emacs so was suprised to find /etc/emacs is installed with a default Debian Etch install.
debian:~# dpkg -S /etc/emacs
dictionaries-common: /etc/emacs
To see more information about that package (to decide what to do with it). Use apt-cache show
debian:~# apt-cache show dictionaries-common
Package: dictionaries-common
Priority: standard
Section: text
Installed-Size: 745
Maintainer: Agustin Martin Domingo
Architecture: all
Version: 0.70.10
Replaces: openoffice.org-updatedicts
Provides: openoffice.org-updatedicts
Depends: perl-base (>= 5.6.0-16), debconf (>= 0.5) | debconf-2.0
Suggests: ispell, emacsen-common, jed-extra
Conflicts: ispell (<= 3.1.20.0-1), miscfiles (<< 1.3-2.1), iamerican (<= 3.1.20.0-1), ibrazilian (<< 2.4-5.1), ibritish (<= 3.1.20.0-1), ibulgarian (<= 2.0-2), icatalan (<= 0.1-4), iczech (<= 20020628-1), idanish (<< 1.4.22-2.1), idutch (<= 1:0.1e-20), iesperanto (<< 2.1.2000.02.25-6), ifaroese (<= 0.1.16-2), ifinnish (<< 0.7-3.4), ifinnish-large (<< 0.7-3.4), ifinnish-small (<< 0.7-3.4), ifrench (<= 1.4-13), ifrench-gut (<= 1:1.0-9), igerman, ihungarian (<= 0.84-1), iitalian (<< 2.20-1.2), ingerman (<< 20010414-2), inorwegian (<< 2.0-6.1), ipolish (<< 20011004-2.1), iportuguese (<< 19980611-8), irussian (<= 0.99f0-1), ispanish (<< 1.7-5), ispell-ga, iswedish (<= 1.4.2), wbritish (<= 3.1.20.0-1), wbulgarian (<= 2.0-2), wcatalan (<= 0.1-4), wdanish (<< 1.4.22-2.1), wdutch (<= 1:0.1e-20), wenglish (<= 2.0-2), wfaroese (<= 0.1.16-2), wfinnish (<< 0.7-3.4), wfrench (<= 1.0-11), wgerman, witalian (<= 1.6), wnorwegian (<< 2.0-6.1), wngerman (<< 20010414-2), wpolish (<< 20011004-2.1), wspanish (<= 1.0.11.1), wswedish (<= 1.4.2), openoffice.org-updatedicts
Filename: pool/main/d/dictionaries-common/dictionaries-common_0.70.10_all.deb
Size: 250004
MD5sum: e9a26a3367d2373a20368d1fe6b22a73
SHA1: 16730ac0cac691e5bd9a1edc123be83902214738
SHA256: 39ce6f1684a2b568e78467656bfdcd605dae20f588dc7544b08f03aaadd1e789
Description: Common utilities for spelling dictionary tools
These are utilities shared by all ispell, myspell and wordlist
dictionaries, including support for some tools that use ispell
(like emacsen, jed and mutt). More info about naming conventions and
availability of those dictionaries in the README file.
.
Maintainers should install dictionaries-common-dev as well, and read its
documentation.
Tag: implemented-in::lisp, implemented-in::perl, role::plugin, role::program, scope::utility, special::auto-inst-parts, works-with::dictionary
http://www.debian.org/doc/manuals/reference/ch-package.en.html


[root@host ~]# rpm -qf /bin/bash
bash-3.0-19.3


So /bin/bash belongs to bash-3.0-19.3

The -q option lets rpm know that you want to do a query of the database. The f lets you query the package containing the file you specify.

Here’s another query:


[root@host ~]# rpm -qf /usr/bin/xargs
findutils-4.1.20-7.el4.1
[root@host ~]#


yum lets you get info on the findutils package:


[root@host ~]# yum info findutils
Setting up repositories
update 100% |=========================| 951 B 00:00
rpmforge 100% |=========================| 1.1 kB 00:00
base 100% |=========================| 1.1 kB 00:00
addons 100% |=========================| 951 B 00:00
extras 100% |=========================| 1.1 kB 00:00
Reading repository metadata in from local files
Installed Packages
Name : findutils
Arch : i386
Epoch : 1
Version: 4.1.20
Release: 7.el4.1
Size : 231 k
Repo : installed
Summary: The GNU versions of find utilities (find and xargs).


Description:
The findutils package contains programs which will help you locate
files on your system. The find utility searches through a hierarchy
of directories looking for files which match a certain set of criteria
(such as a filename pattern). The xargs utility builds and executes
command lines from standard input arguments (usually lists of file
names generated by the find command).


You should install findutils because it includes tools that are very
useful for finding things on your system.


So, if you are using a system that uses RPM/YUM, you can easily find out the packages that files on your filesystem belong to.