Posts

• Jun 20, 2012

  Setting up and using SSD drives in Fedora Linux

  Stevea has written a comprehensive post on the Fedora Forums about using SSD drives on Fedora Linux. Good thing his post is general enough that it can also be applied to other distributions.

  If you are planning on have SSDs on your GNU/Linux servers, you might find it a good read.

  · · ·
• Jun 19, 2012

  Upgrading to MySQL 5.5 on CentOS 6

  MySQL server shipped with CentOS 6 / RHEL 6 is still stuck at 5.1, which will soon be EOL. It will be prudent and sane to start preparing the migration to MySQL 5.5.

  · · ·
• Jun 18, 2012

  Amazon AWS EC2 Outage (us-east-1)

  AWS Status Page:

  12:15 AM PDT We would like to share some detail about the Amazon Elastic Compute Cloud (EC2) service event last night when power was lost to some EC2 instances and Amazon Elastic Block Store (EBS) volumes in a single Availability Zone in the US East Region.

  At approximately 8:44PM PDT, there was a cable fault in the high voltage Utility power distribution system. Two Utility substations that feed the impacted Availability Zone went offline, causing the entire Availability Zone to fail over to generator power. All EC2 instances and EBS volumes successfully transferred to back-up generator power. At 8:53PM PDT, one of the generators overheated and powered off because of a defective cooling fan. At this point, the EC2 instances and EBS volumes supported by this generator failed over to their secondary back-up power (which is provided by a completely separate power distribution circuit complete with additional generator capacity). Unfortunately, one of the breakers on this particular back-up power distribution circuit was incorrectly configured to open at too low a power threshold and opened when the load transferred to this circuit. After this circuit breaker opened at 8:57PM PDT, the affected instances and volumes were left without primary, back-up, or secondary back-up power. Those customers with affected instances or volumes that were running in multi-Availability Zone configurations avoided meaningful disruption to their applications; however, those affected who were only running in this Availability Zone, had to wait until the power was restored to be fully functional.

  The generator fan was fixed and the generator was restarted at 10:19PM PDT. Once power was restored, affected instances and volumes began to recover, with the majority of instances recovering by 10:50PM PDT. For EBS volumes (including boot volumes) that had inflight writes at the time of the power loss, those volumes had the potential to be in an inconsistent state. Rather than return those volumes in a potentially inconsistent state, EBS brings them back online in an impaired state where all I/O on the volume is paused. Customers can then verify the volume is consistent and resume using it. By 1:05AM PDT, over 99% of affected volumes had been returned to customers with a state ‘impaired’ and paused I/O to the instance.

  Separate from the impact to the instances and volumes, the EBS-related EC2 API calls were impaired from 8:57PM PDT until 10:40PM PDT. Specifically, during this time period, mutable EBS calls (e.g. create, delete) were failing. This also affected the ability for customers to launch new EBS-backed EC2 instances. The EC2 and EBS APIs are implemented on multi-Availability Zone replicated datastores. The EBS datastore is used to store metadata for resources such as volumes and snapshots. One of the primary EBS datastores lost power because of the event. The datastore that lost power did not fail cleanly, leaving the system unable to flip the datastore to its replicas in another Availability Zone. To protect against datastore corruption, the system automatically flipped to read-only mode until power was restored to the affected Availability Zone. Once power was restored, we were able to get back into a consistent state and returned the datastore to read-write mode, which enabled the mutable EBS calls to succeed. We will be implementing changes to our replication to ensure that our datastores are not able to get into the state that prevented rapid failover.

  Utility power has since been restored and all instances and volumes are now running with full power redundancy. We have also completed an audit of all our back-up power distribution circuits. We found one additional breaker that needed corrective action. We’ve now validated that all breakers worldwide are properly configured, and are incorporating these configuration checks into our regular testing and audit processes.

  We sincerely apologize for the inconvenience to those who were impacted by the event.

  Moral of the story: use multi-AZ deployment or multiple cloud providers.

  · · ·
• Jun 14, 2012

  pwqgen—My Favourite Password Generator

  Security best practices encourage different passwords to be used for each site requiring a login. This is because should a particular site suffer a security breach, where all the user passwords are leaked publicly, your password that was leaked is unique to that site. Any malicious person who got hold of your account credentials will not be able to login to your other accounts, like your email.

  So, you may ask, how do you track the various passwords used for each site?

  There is a two prong solution to this. First, use easy-to-remember, but hard to guess passwords, and second, use a cross-platform password manager.

  To generate good but easy-to-use passwords is a challenge. Most password managers include built-in generators but they are awful. These generators have an implied assumption that you’ll never have to type them out by copying visually. Hence, these passwords are hard to pronounce and impossible to remember.

  I have found one that finally meets my requirements and is now my favourite password generator—pwqgen, from the passwdqc package.

  Here’s an example: Strict!Spin6cairo.

  It is about 17 characters long, pronounceable, easy to type visually, but hard to brute-force. With a different one of these on each of your logins, security breaches will just be a matter of resetting that particular password. Of course, you are to regenerate your passwords every 6–12 months.

  There are often times when I needed to generate a password on-the-go, such as at a remote computer or even on my iPad, where I don’t have access to pwqgen. As such, I’ve created a web-interface for it, where it generates a random password on every visit. You might find it useful.

  As for a password manager, I use 1Password which syncs and works across all my Macs, Windows, and iOS devices. iOS support is exceptionally crucial as it allows you to access your passwords on-the-go at a public terminal. While it is not free, it is worth every cent, and its utility is perpetual.

  If you want something free, KeePass is another alternative, but with a much awful interface.

  · · ·