Another bad day at the office
A software error during routine maintenance caused an ISP, Charter Communications, to delete the contents of 14,000 customer email accounts.
The news article doesn't mention whether the "software error" was an unfortunate and evidently untested change to the maintenance scripts (indicating a hole in their change management processes), a genuine bug in the code (possible I guess), or a simple human error by an operator/systems manager (seems entirely possible). Since the lost email accounts disappeared forever in a puff of logic, it seems the ISP had no backups of customer data - not just 'no recent backups' but 'no backups whatsoever' (a gaping hole as far as their customers are concerned but no doubt a legitimate money-saving measure from the ISPs perspective).
This incident cost the ISP $50 credits to the affected customers, presumably rather less than 14,000x$50 ($700k) as some will defect before using up all their credit. The reputational damage could be even costlier, although the truth is that such unfortunate incidents can and indeed occasionally do strike most organizations.
The Silicon Valley piece ends rather lamely with "Computer experts advise backing up all important e-mail.", implying in effect that customers are to blame for losing their emails. In some ways that is true (presumably any small businesses or power users will have been using local emaiil clients such as Outlook to download and read their emails and so should have local backup copies) but I would advise Charter Comms to look long and hard at its information security arrangements.
"Charter gives each new Internet user a free e-mail account, but some customers opt to use other accounts instead. So every three months the company deletes inactive accounts, Lamont said. "During this maintenance we erroneously deleted active accounts along with the others," Lamont said. "It's never happened before. They are taking steps to make sure it never happens again."
The news article doesn't mention whether the "software error" was an unfortunate and evidently untested change to the maintenance scripts (indicating a hole in their change management processes), a genuine bug in the code (possible I guess), or a simple human error by an operator/systems manager (seems entirely possible). Since the lost email accounts disappeared forever in a puff of logic, it seems the ISP had no backups of customer data - not just 'no recent backups' but 'no backups whatsoever' (a gaping hole as far as their customers are concerned but no doubt a legitimate money-saving measure from the ISPs perspective).
This incident cost the ISP $50 credits to the affected customers, presumably rather less than 14,000x$50 ($700k) as some will defect before using up all their credit. The reputational damage could be even costlier, although the truth is that such unfortunate incidents can and indeed occasionally do strike most organizations.
The Silicon Valley piece ends rather lamely with "Computer experts advise backing up all important e-mail.", implying in effect that customers are to blame for losing their emails. In some ways that is true (presumably any small businesses or power users will have been using local emaiil clients such as Outlook to download and read their emails and so should have local backup copies) but I would advise Charter Comms to look long and hard at its information security arrangements.
Labels: Availability, Bugs, Email, Incidents, Office
Links to this post:
