Dealing With Hacks

At Canvas Host, we pride ourselves in operating as secure a network as we can. We work hard every day to provide a safe hosting environment, to help you run a successful website. In recent months, a few hacks have circulated around the Internet, and we thought this would be a good time to update you on the steps we have taken, and changes to our procedures we continually employ, to protect you and your data.

Patching Is Crucial

Every software vendor on the planet has at some point or another issued a patch for their code. Some providers and frameworks, like WordPress, are constantly releasing updates, not only for the core code base, but also themes and plugins. While some of the updates are to roll out functionality improvements, many of the incremental releases are to patch vulnerabilities — weakness in the code that can be exploited — and which have come to light through testing, or in some cases, pure chance.

Code updates are a good thing. They are necessary to ensuring your website and hosting environment are kept running quickly and securely. Update notices can be annoying, but they’re there for a good reason.

Alerts Are There For A Reason

If you subscribe to our WordPress hosting service line, you should receive regular alerts about vulnerabilities our own scanning systems have detected in your installations. It could be due to an outdated code base or old plugin, or we may have detected a suspicious file that is lying dormant in your website.

In our notifications, we try to address the precise file or set of files that are of concern. As a customer, it is your responsibility to clean up your website, have your webmaster/webmistress do it for you, or hire us to do it. If you don’t act on our warnings, and we later determine that your site has been compromised, we will most likely suspend the account until you have a chance to address and resolve the issue(s).

We understand how this may impact you, and even informing you of potential concerns may be alarming, but that’s the point. It’s our responsibility to protect our systems and network, so in turn we can protect you and all customers of our services.

That said, no system is foolproof. Inevitably, any system is going to have weakness. Sometimes, those weaknesses will be exploited.

Understanding The Impact Of Vulnerabilities

In January, we issued a statement about a previously unknown set of vulnerabilities that potentially impacted every CPU on the planet, including those in our web servers, that we had been made aware of, known as Spectre and Meltdown. Hardware vendors worldwide scrambled to release patches for operating systems, including some that we use, to prevent the vulnerabilities from becoming a major issue. We applied those patches, and all was well.

In mid-April, we became aware of an update to an operating system technology we utilize called CloudLinux. CloudLinux provides a virtualized environment that functions in ways very similar to virtual private servers, in that we can allocate precise amounts of RAM and CPU processes to a given website. It’s a fantastic technology that can prevent server spikes and website outages.

It was not indicated that the release was an urgent patch. At that same time, we were diagnosing a mystery hack for a handful of websites hosted on one of our servers that use CloudLinux. In the release notes, we learned the patch corrected an issue that was related to the site hacks. We applied the patch as soon as we became aware of it, but unfortunately, the hack had already occurred. Though not considered a zero day exploit, it is our belief hackers immediately seized upon the vulnerability, before we and other providers had an opportunity to apply the patch. In the end, fewer than 40 websites were defaced, and we communicated directly with those customers throughout the process.

It could have been much, much worse. Take, for example, this release from Drupal about an extremely critical vulnerability that could lead to an entire server becoming compromised. We have been in communication with several Drupal developers whose own websites were hit by that vulnerability, and unfortunately, it impacted their entire server.

Responding To A Hack

When a hack happens, how a service provider responds is crucial. And yet, disclosure of a vulnerability is one of the greatest challenges in dealing with a hack. As a B Corporation, we champion transparency throughout our operations, including admitting errors or faults in our systems. At the same time, when we’re dealing with a potential security risk, we don’t want to broadcast it to the world. It’s not because we’re afraid of admitting fault; Rather, we don’t want to draw additional attention and risk to the situation than is necessary. We also don’t want to unnecessarily raise alarms to customers that were not affected.

Every situation is a learning opportunity, and a chance to improve a process. In the case of the CloudLinux hack, we identified a weakness in one our Managed WordPress service line, and have implemented a change to how we manage backups for those websites. The change has dramatically improved its utility, not to mention added security for those customers subscribed to it.

As part of the service, we make weekly backups of WordPress websites prior to applying patches and other updates. Historically, those backups were being stored locally, within the customer’s hosting account. We have amended this process, and are now storing those backups at our secondary datacenter in Bend. Beyond protecting those backups from a potentially compromised hosting account, the data is also stored in an Earthquake-proof hosting environment. This is one silver lining that has come out of a situation of concern.

How You Can Protect Your Website

Here are a few tips you can employ to protect your website against hacks:

  1. Patch, patch, and patch again! Keep your website updated.
  2. When we notify you of a vulnerability, act on it.
  3. If you’re managing your own website, look for announcements from the application’s project team.
  4. If you’re not sure how to patch or manage your website, ask us for help.
  5. Change your website and hosting account passwords frequently.
  6. If it’s been a while since you last patched your website, revisit step 1, or ask us to perform a free vulnerability scan of your website.

If ever you have questions or concerns about our hosting services, please ask us. We’re always open to your inquiries and suggestions. We recognize that no system is perfect, and it is our goal to learn from a situation, and from it create an even better service to you.

Thank you,

David Anderson

David Anderson

This blog is published by David Anderson, Principal and owner of Canvas Host.