Limited access to services
Incident Report for GetResponse
Resolved
All GetResponse services are fully operational, and GetResponse is now protected with the strongest always-on DDoS mitigation available on the market today, with 1.8 Tbps of mitigation bandwidth. The team has pulled off an amazing stunt by making a tremendous infrastructural change in less than 3 days (under normal circumstances, it can take anywhere from 2 to 4 weeks). The email & subscription queue is shrinking and we expect to be fully caught up very soon. Thank you again for your patience, understanding and loyalty.
Posted May 01, 2014 - 19:40 CEST
Update
We're currently delivering emails held in queue as well as add queued contacts to your target campaigns.
Posted May 01, 2014 - 10:25 CEST
Update
Great news: We have managed to resolve outbound mailing capability. All emails that had been scheduled in the past days will now be sent, however it may take several hours in order to provide the best possible deliverability.

New scheduled emails may have a bit longer delivery time as we are dealing with a quite large volume of emails that were scheduled in the past few days.

If you have any questions about your newsletters or account, please contact our Customer Service Department at support.getresponse.com (GetResponse 360 customers, please contact your Account Managers).
Posted May 01, 2014 - 03:01 CEST
Update
We are very close to resolve outbound mailing capability. Should you schedule your newsletters to be sent in the past few days, this is the last moment to remove those emails from waiting list in case you would like to make some content updates.
Posted May 01, 2014 - 01:29 CEST
Update
Akamai and Windstream have finished provisioning our IP space through a fully routed always-on scrubbing service (clean pipe). We are now testing our IP configuration and working as fast as possible to resolve outbound mailing capability. Shouldn't take long now...
Posted Apr 30, 2014 - 21:18 CEST
Update
We are making good progress on restoring full mailing capability. The current plan is to restore outbound email by tomorrow.
Posted Apr 30, 2014 - 18:25 CEST
Update
GetResponse CEO Statement Regarding the DDoS Attack.

Dear GetResponse Customer,

As you may know, GetResponse has had a pretty rough weekend… We suffered a major outage
caused by a DDoS (distributed denial of service) attack on an unprecedented scale, which has flooded
our network and our data center (Windstream) with malicious traffic.

What happened?

On Saturday, after hours of working together with our data center Windstream (including their
directors and VP-level executives), we received information that they have failed to mitigate
the attack and of their decision to block our network connectivity until a powerful DDoS
mitigation service can be fully implemented.

When GetResponse went down, the morale of the GetResponse community didn’t. We saw an
amazing display of support and we would like to thank our customers for rooting for us.

What’s the current status?

We brought many of the services back online on Sunday (websites, web subscriptions, API, web
forms, surveys, landing pages).

We had another attack on Monday, but we fully mitigated it, with no downtime, thanks to our new
mitigation solution.

We are still working on resolving issues with email deliveries and anticipate a complete resolution
within the next couple of days. We are literally working around the clock with VP-level executives
from our data center and DDoS mitigation partners.

What is GetResponse doing to mitigate the attacks?

Over the past 15 years, we have repelled many DDoS attacks using our infrastructure. More
recently, we employed CloudFlare Enterprise solution to help us to deal with attacks in March and
April. Unfortunately, the scale of last weekend’s attack was so huge that it overran our current
mitigation solution, and we needed to come up with a new plan.

After a conference call with our data center, within minutes we got in touch with Akamai, the
world’s leading DDoS mitigation provider, with 1.8 Tbps mitigation bandwidth. We asked them to
deploy an “always on” clean-pipe DDoS scrubbing service for GetResponse. This is the most
advanced type of protection available on the market (also most expensive). It scrubs all inbound
traffic for malicious packets of data and returns only “clean” traffic to our data center.

In total, we are spending close to half a million dollars in mitigation solutions, hardware,
connectivity and other upgrades.

Why was GetResponse attacked?

This wasn’t the first attack GetResponse has encountered recently. We endured several other
attacks in March and April. And we weren’t alone…

Over the last two months, a criminal has targeted many other large Internet companies, putting
them offline. These include Meetup, Shutterstock, MailChimp, Fotolia, Basecamp, oDesk,
SurveyGizmo, MadMimi, OkCupid, HootSite, Typepad, Elance, Aweber and others. It is
unfortunate, but these types of attacks are becoming much more frequent in today’s environment.

Why would anyone attack well-meaning companies, disrupting their business?

This email explains it all:

Subject: DDoS attack, warning

I don't have to explain myself anymore. I will stop the attack for 1.2 Bitcoin (≈ $750).
Your network will be safe from further attacks coming from several botnets, think
twice before making your final decision, as even the best global DDoS mitigation
won't be able to handle easily the incoming new Amp. methods.
Let me know if you are interested in my offer.

We weren’t interested, and won’t ever be, even if they continue to attack us. The low “offer” ($750)
was just bait; we know they’d come back for more. And paying would only encourage them to
attack other companies. Besides, we are confident that we can protect ourselves in the future with
the mitigation plan we’ve put in place. But above all, paying criminals is simply the wrong thing to
do.

I sincerely apologize for this outage. We care about you, our customers, and realize you rely on us
to get your emails out to the world. Since the attacks, we’ve been working around the clock to get
mitigation in place.

I thank you for your trust, your support and your loyalty during these difficult moments.

Regards,
Simon Grabowski
CEO
GetResponse

PS: If you have any questions about this issue, please contact our Customer Service Department
at support.getresponse.com (GetResponse 360 customers, please contact your Account Managers).
Posted Apr 29, 2014 - 22:59 CEST
Update
We are making progress with our DDoS mitigation which, once complete, will give GetResponse the strongest always-on anti-DDoS protection in the email marketing space (provided by two largest providers, Akamai and CloudFlare). GetResponse 360 IP and DNS changes are currently propagating. Many GetResponse 360 customers can already access their control panel. We are working on bringing back full access to the control panel, API, web subscriptions, landing pages, surveys for all of our 360 customers. For GetResponse SMB (shared) customers, we are taking steps to resolve issues with limited email delivery. We are working around the clock to execute the plan. We thank you for your continued support, loyalty and understanding.
Posted Apr 29, 2014 - 02:43 CEST
Update
Our engineers are currently focusing on restoring web services, customer logins, web form sign-ups, API functionality and surveys for GetResponse 360. Thank you for your continuous patience and understanding.
Posted Apr 28, 2014 - 23:57 CEST
Update
Landing pages hosted on following domains are available and running:
gr8.com
subscribemenow.com
getresponsepages.com
Posted Apr 28, 2014 - 18:30 CEST
Monitoring
Our engineers are working day and night on bringing the service back to a fully functioning state, however you might still encounter problems when using some of our features.

Your new subscribers are currently being added to a queue upon the sign-up. If they are double opt-ins - all confirmation emails will be delivered once the service is completely restored.
You can also view, edit, and create new emails and landing pages.

Unfortunately we’re still working to resolve problems with sending out the mailings as well as to bring the landing page subdomains back to life. The landing pages should be active soon.

All the emails and autoresponders that were scheduled for broadcast during the outage period couldn’t be delivered. Please reschedule in order to get them to your contacs inboxes.

All emails that you sent out starting today, are added to a queue and will be delivered as soon as it is possible.

We'd also like to assure you that none of the data stored in your GetResponse account have been compromised in any way.
Posted Apr 28, 2014 - 13:54 CEST