PEAK Premium Support is here

You can stop searching for where to get additional computer support. PEAK Premium Support is now available for everyone with a computer! You don’t have to drag your computer down to a computer repair shop anymore, or call those geek people for help who will charge you hundreds of dollars. With PEAK’s new Premium Support we will help you fix your computer problems for a low one-time fee of $49.99 per problem repair.

Continue Reading »

PEAK’s State of Spam and Phishing

Let’s face it; spam and phishing email are here and not going away. In fact, for many, it’s getting worse. While e-mail filters have become more effective at detecting junk and scam email, spammers are equally ingenious in finding new ways of annoying us with false offers. Fortunately at PEAK we have managed to stay ahead of the spam ‘curve’ by putting filtering appliances in place and providing other verification mechanisms to control these threats. Despite this success however, our work is not complete in this ever-evolving world of email messaging.

PEAK’s ongoing efforts and spam summaries will be detailed in the ‘State of Spam and Phishing’ reports periodically delivered throughout the year.

PEAK’s State of Spam and Phishing

Upwards of 90 percent of worldwide email is considered spam or other unwanted email messages. At PEAK this figure holds true where on average 1.5 million messages are processed each day and only 10% are actually delivered. Of this figure, the majority of messages are delivered from spam botnets, which are networks of hijacked computers or servers that send email messages around the globe with intent to lure users into fraudulent account activity.

Top Spam Message Subject Lines

Spam Subject Lines

Most common spam email message subject lines (click to expand)

Spam Rates

Spam Rates

Rates of spam and other e-mail threats (click to expand)

Botnets

Botnet threats

Detected Botnets (click to expand)

Updates of phishing scams

Phishing is another form email messaging scam that threatens users by deceptively attempting to garner account access credentials or financial information via fraudulent notification. Examples of this may be message with urgent topics such as, expiring domain names or bank account messages. These notes then will ask you to access via a link contained in the email, which leads to a fraudulent site resembling your destination where your access information is harvested.

These campaigns are not limited to nationwide brands. PEAK occasionally has been the target of phishing scams that attempt to lure customer with fraudulent messages related to PEAK’s email or billing account access. The most recent example was in March with a message that had the subject “Account Alert” and that detailed “peak.org” security information update. This message was fake, but it did lure some users into the scam. PEAK’s staff quickly took action and notified the user of the error.

To the best of our efforts we will notify customers when they have succumbed to a phishing scam, but we cannot be totally accurate with our monitoring. Be sure to check the validity of all messages asking for information and never click through email links to enter access information. If you concerned at any time, please don’t hesitate to contact PEAK.

PEAK’s hardware to eliminate spam

PEAK has been diligent in making investments to filter spam from the end users accounts. In 2010, PEAK made the investment in Red Condor spam filtering appliances that operate separate, but in line with your email application (or desktop spam solution).

The Red Condor service has been extremely successful in blocking the amount of delivered spam and catching different botnet or phishing attempts. Each day the Red Condor devices process over a million messages, filtering the vast majority before they even reach the destination mail box.  Each user should receive a daily digest from the Red Condor, detailing which messages were not delivered to your mail box because of suspicion or because of virus detection.

Recent Microsoft seizure and botnets causing problems

Big news in the reduction of spam came earlier in March when Microsoft announced it had collaborated with the federal government to take down the Rustock botnet, which was estimated to consist of roughly one million slave computers that were malware infected.

Rustock was among the world’s largest spam botnets and was capable of sending as many as 30 billion emails per day. Much of the email sent by Rustock advertised counterfeit or unapproved knock-off versions of drugs like Viagra, while other spam tried to dupe people with bogus Microsoft lottery notices.

March Checklist: Protecting your business, your family members and/or employees

Do

  • Unsubscribe from legitimate mailings that you no longer want to receive. When signing up to receive mail, verify what additional items you are opting into at the same time. Deselect items you do not want to receive.
  • Be selective about the Web sites where you register your email address.
  • Avoid publishing your email address on the Internet. Consider alternate options – for example, use a separate address when signing up for mailing lists, get multiple addresses for multiple purposes, or look into disposable address services.
  • Using directions provided by your mail administrators report missed spam if you have an option to do so.
  • Delete all spam.
  • Avoid clicking on suspicious links in email or IM messages as these may be links to spoofed websites. We suggest typing web addresses directly in to the browser rather than relying upon links within your messages.
  • Always be sure that your operating system is up-to-date with the latest updates, and employ a comprehensive security suite.

Do Not

  • Open unknown email attachments. These attachments could infect your computer.
  • Reply to spam. Typically the sender’s email address is forged, and replying may only result in more spam.
  • Fill out forms in messages that ask for personal or financial information or passwords. A reputable company is unlikely to ask for your personal details via email. When in doubt, contact the company in question via an independent, trusted mechanism, such as a verified telephone number, or a known Internet address that you type into a new browser window (do not click or cut and paste from a link in the message).
  • Buy products or services from spam messages.
  • Open spam messages.
  • Forward any virus warnings that you receive through email. These are often hoaxes.

Data Center UPS Upgrade

Dear Valued PEAK Customer,

I would like to take a moment to describe the sequence of events that lead up to the complete data center outage, which occurred on Saturday, October 9th at 3:15am. This outage affected all PEAK customer services.

During planned maintenance to the Uninterruptible Power Supply (UPS) we experienced an unplanned electrical interruption to the PEAK data center (DC). For more information regarding the project, see the PEAK blog at:
http://blog.peakinternet.com/monthly-newsletter/the-anatomy-of-a-data-center-ups-upgrade/

Part of our plan was to ensure continuous operation during the upgrade. To accomplish this, we started our back-up generator to supply continuous power to the DC in the event of interruption of city power. Second, we installed a temporary back-feed electrical circuit between the electrical panel feed by the generator and the electrical panel normally fed by the UPS, which feeds all equipment in the DC.

The electrical system of the DC is a three-phase system, which runs at 208 volts. During the weeks leading up to the upgrade, engineers moved workloads to get the power load at the time of the upgrade to less than 80 amps per phase to be in safe tolerance with the circuit breakers. At the time of the upgrade loads per phase were: A:78 amps B:80 amps C:79 amps.

OLD UPS

OLD UPS

Electricians installed a circuit capable of supplying 100 amps of load between the generator panel and UPS panel. We used 100 amp three-phase breakers on both sides of the temporary back-feed circuit.

At 2:30 AM, the stand-by generator was started and electrical load was transferred from CITY to GENERATOR power.

At 3:00 AM, we initiated procedures to shutdown the old Liebert UPS, and put the UPS in maintenance-bypass mode.  This allowed electrical power to flow from the generator panel through the UPS to the UPS sub-panel. At this time, we closed (turned on) the temporary back-feed breakers, which put the UPS in parallel with the temporary back-feed circuit. Next we completely shut down the UPS. This transferred the load in the DC to the temporary back-feed circuit. We amp probed the back-feed circuit and confirmed loads at A:78 amps B:80 amps C:79 amps. The electricians felt comfortable that this circuit would hold the load properly.

PEAK engineers, equipment movers and electricians began the process of safely disconnecting and removing the existing UPS from the DC, while the generator supplied power.

OLD UPS

OLD UPS

Around 3:15 AM, for an un-known reason, the 100-amp breaker on the generator side of the back-feed circuit tripped, causing complete electrical loss of power to the DC. At this time, the decision was made to shut down the electrical sub-panel supplying power to the DC co-location area, which would remove around 50 amps from our workload.

Around 3:20 AM, we re-closed the tripped breaker and power was restored to the DC.

There were cascading failures caused by the sudden unplanned loss of electrical power to the DC.  Our infrastructure relies on hundreds of devices which all work together to provide Internet and information technology services to our customers.  The procedures to re-start these systems are tedious, time-consuming, and must be done in a specific order. Engineers immediately and swiftly initiated this re-start procedure.

The most significant failure during the re-start was our network switching/routing core, which runs on Juniper EX4200 switches. Two of the four switches did not re-start properly and required a re-load of the operating system, which runs on the device.

In addition, a Network Appliance Filer for data storage did not re-start. The controller for this device completely failed and a replacement has been procured, which will arrive on Tuesday. Most of the data that runs on the filer was moved to alternate servers, but there were a few services, which rely on this server. The most significant service is customer personal web space.

At 5:00 AM, the existing UPS was completely removed from the DC and the process of bringing in the new UPS begun.

APC TECHNICIAN

ELECTRICIAN

Around 6:00 AM, the factory service technician from APC arrived on-site. The new UPS is a modular 3 cabinet system consisting of an inverter cabinet, battery cabinet and distribution cabinet. Electricians and APC started the process of cabling the cabinets and connecting incoming and outgoing power to the distribution cabinet.

At 9:00 AM, after APC completed start-up checklist procedures, power was supplied to the UPS to confirm proper operation and cabling.

At 9:15 AM, the new UPS was put in maintenance bypass mode, which put the new UPS in parallel with the temporary back-feed circuit. At this time, the breakers to the temporary back-feed circuit were opened (turned off) and the new UPS carried the full DC load. After a few minutes, the UPS taken out of maintenance bypass mode and it began normal operation protecting electrical loads in the DC.

At 9:20 AM, the stand-by generator was shut down and incoming electrical load was transferred from the GENERATOR to CITY power.

At 9:35 AM, power to the co-location area was restored.

NEW UPS

NEW UPS

This marked the completion of the UPS upgrade project. Although we experienced a significant service interruption to our customers, which was not planned, the new UPS will provide increased reliability and capacity.

The new UPS is an APC PX80 Symmetra Modular UPS. Some of the features and benefits of this system are:

  • Modular design: Provides fast serviceability and reduced maintenance requirements via self-diagnosing, field-replaceable modules
  • Configurable for N+1 internal redundancy: Provides high availability through redundancy by allowing configuration with one more Power Module than is necessary to support the connected load.
  • Redundant Intelligence Modules: Provides higher availability to the UPS connected loads by giving redundant communication paths to critical UPS functions.
  • Hot-swappable intelligence modules: Ensures clean, uninterrupted power to protected equipment during Intelligence Module replacement.
  • Hot-swappable power modules: Ensures clean, uninterrupted power to protected equipment during Power Module replacement.
  • Hot-swappable batteries: Ensures clean, uninterrupted power to protected equipment while batteries are being replaced
  • Power Modules connected in parallel: Enhances availability by allowing immediate, seamless recovery from isolated module failures.
  • Battery modules connected in parallel: Delivers higher availability through redundant batteries.
  • Automatic internal bypass: Supplies utility power to the connected loads in the event of a UPS overload condition or fault.
  • Automatic restart of loads after UPS shutdown: Automatically starts up the connected equipment upon the return of utility power.
  • Power conditioning: Protects connected loads from surges, spikes, lightning, and other power disturbances.

In our configuration, the UPS is operating at 45% capacity, provides 15 minutes of operation (while the stand-by generator starts) and provides N+3 power module redundancy.

We sincerely apologize for the interruption of service this outage caused you. We take great pride in the reliability of our infrastructure. You can be confident that we will continue to work on improving our infrastructure and procedures to ensure highly available service delivery to our valued customers.

If you have specific questions or concerns, please do not hesitate to contact me directly.

Sincerely,

David Placko

Chief Technology Officer
PEAK Internet
david.placko@peakinternet.com

The Anatomy of a data center UPS upgrade

*Update – 10/14* : Read the post-mortem log, recapping the upgrade and explaining the events that transpired, as recorded by PEAK CTO David Placko. Data Center UPS Upgrade Blog


Providing a centralized source for surge filtered and redundant electricity, the Uninterruptable Power Supply (UPS) is easily the most critical infrastructure component of a data center. This critical disposition makes an upgrade to a data center a complex and tedious task, especially when maintaining uptime is crucial.

PEAK’s engineers have been planning and evaluating an upgrade to a new UPS to replace the aging unit currently in place. After careful consideration, a unit was selected that will occupy less space in the data center and provide more capacity for equipment. The process to upgrade to this new unit will start early in the morning and be finalized several hours later, as the plan dicatates.

General steps involved with this upgrade include:
• Starting back-up generator and transferring all services from data center to the generator.
• Moving UPS equipment into data center in a staging position.
• Ensuring all phases on current UPS are ready for back feed. Bypass current UPS and shut it down.
• Initialize new equipment and switch power cables from old unit to the new unit.
• Remove the old UPS and position the new UPS into new space.
• Complete start-up procedures on new UPS unit and transfer load from temporary bypass circuit.
• Shut down generator and transfer load from generator to city grid power.

Originally scheduled for late August, an equipment inventory issue forced a delay in the UPS upgrade project.  The next available timeframe is coming up this month, October 8th and 9th.  This project will resume along with the originally laid plans and no outage is expected.

New features and functionality announced for your Clear Access modem

clearaccessAbout a year ago PEAK made the decision to make modems from Clear Access its primary customer premised equipment (CPE) for DSL customers.  This decision was made because Clear Access provided a higher quality device that offered the promise of added value through expanded services for customers.  Over the past year some of these features were released, providing simplification to the increasingly complex home network.  Over the next month, we would like to showcase some of the impressive functionality Clear Access is building into its devices that will soon be available to you.

Face it – you’re no network administrator.  Even if you were, the last thing you want to do when you get home is troubleshoot your daughter’s Internet connection or figure out why your latency is affecting your gaming on Xbox Live.  The complexity of today’s home networking is rapidly increasing; and with the introduction of so many Internet-enabled devices around the home it’s only getting worse.

The Clear Access modems, offered by PEAK Internet, offer a complete home networking solution so you can relax and enjoy the technology coming into your household.  Through a “connected home” concept, the new modem has the ability to provide Managed WiFi, Parental Internet Controls, Home Monitoring, and Home Network Optimizing.  In addition to these offerings, Clear Access will soon provide integrated services with other organizations to bring home Power Monitoring through Smart Grid technologies, Online Backup from Carbonite, Online Device Management (printers, etc.), and control over other bandwidth enabled services.

This is a lot of information and too many promises to deliver on in just one article, so the one takeaway is to stay tuned for more information coming out about the next generation of releases.  If you haven’t already tried them yet, several of the new services are already available.

clear-manage-wifiManaged WiFi:

New modems from Clear Access offer a Managed WiFi service that can be enabled at your request.  This lets a trusted PEAK support agent create a secure wireless broadcast with a password so unknown users aren’t accessing your network.

clear-time-blockTime Blocking:

The Time Blocking feature allows you to set limits on your devices that access an Internet connection.  If you are interested in implementing limits on computers, gaming consoles, or other devices, this feature will do the trick.

clear-time-blockParental Controls:

Content filtering allows you to set which websites and general content are allowed to be viewed on your connection.  If you need a layer of control on your connection, this is a good place to start.

Of all the features mentioned, I am most excited about the Home Control Panel for devices and integration with Application Partners.  I don’t have descriptive details yet, but from what I have seen it will provide an enriched Internet experience.

In brief, the Home Control Panel gives you an easy listing of all the devices accessing your network and all the services you are using (time blocking, parental controls, WiFi).  You are then able to take actions based on your services.

The Application Partners feature allows you to have access to a network of different services all in a portal format.  This interface will give you one screen to manage all the different home computing services you utilize.  Services like Online Backup, Home Security Monitoring, and others are planned as part of this offering.

Thanks for tuning in.  We’ll be using and reviewing these services in-depth and reporting back via PEAK Chat, so check back for additional information. If you would like to see a sample of these services, come into the PEAK office and try it out.

   Next Page »