#MarketoDown: What Went Wrong and What Can We Learn?

At 11:03 am EST, Marketo users received an email that read:

Users that visited their website were greeted with:

Marketo didn’t renew its domain name, which is what caused the view above.

Marketo’s website was down from 7:41 am EST (4:41 am PST) through at least 9 pm EST (6 pm PST). The following day, July 26, some users could still not use the website. Those who normally use Marketo for email campaigns struggled to get any work done. Those who normally use Marketo for email campaigns, lead management, nurture, reporting, or website forms/landing pages hosted through Marketo subdomains struggled to get any work done.

While Marketo’s team scrambled to fix the problem, bored email marketers took to Twitter to watch for updates, alert Marketo of the issue, use #marketopocolypse or #marketodown, and create some hilarious content of their own.

But quickly, one Good Samaritan, Travis Prebble stepped in.

At first, users thought Prebble might be trying to sell the $2B company back its website. But he assured them otherwise.

Eventually, more than 12 hours after it began, Marketo resolved the issue, but users reported it taking as many as 48 hours to get their services back.

In the aftermath, marketers can now analyze and learn from how Marketo handled this digital disaster. There were some things they did well, and some things they did not:
 

Losses

Letting the Domain Expire (Obviously)

Auto renew.

That’s it. Someone needed to make sure that this happened. And that person should have someone to report to that would ensure that it actually happened.

Creating chains of responsibility will make it hard for this to happen again. Schedule meeting to confirm that the website is all set each time the domain renews. Redundancies are key to ensuring your digital presence (and even product) stay online.

Response and Recovery Time

Marketo is used by more than 9 thousand domains and makes up 6.26% market share for the marketing automation market.

When it went down, that means thousands of marketing teams were struggling to do their job.

The website went down just before 8 am EST, and you’ll notice there was no official word from Marketo until a tweet at 9:18 am EST. The email came out almost an hour later (which is understandable, as they likely had to send out a broadcast without using Marketo).

Understanding that 8 am EST time is 5 am PST, which is why it took a while for the team to respond. It may have been early, but companies in the UK were midway through their days and east coast companies were just starting them.

That hour lapse gave users quite a bit of time to gather some traction. The timing was unfortunate, but it’s lucky that Prebble stepped in as folks at Marketo were just starting to brush their teeth.

Then there’s the issue that it took more than 24 hours for the problem to mostly get resolved.

But this was not the case and users were still having spotty access even at 11 am EST on July 27. The company said that they were just waiting for local DNS servers to update. Here at HiP, the page was readily accessible but many users worldwide still struggled, causing Marketo to release an FAQ for those users.

It can apparently take as long as 24 to 48 hours for all local DNS servers to update to reflect the change in the domain’s status, which is likely what is cause the delay.

This is a case a small mistake that could have big consequences for Marketo. That amount of downtime equals large amounts of cash for thousands of businesses.

It will be interesting to see how Marketo makes this up to its customers. It should provide discounts on its services, at the very minimum, to make it up to its customers.

 

Wins

Social Response

Market executed a social media emergency strategy during this event. They must have had all hands on deck, replying to as many tweets as possible.

Brand Loyalty

The fact that a Marketo user renewed the domain out of his own pocket is an impressive example of brand loyalty.

He didn’t have to, he could have tried to buy the domain right out from under them and ransom it back. But he didn’t.

The fact that he likes the brand enough to do that shows a surprising amount of brand loyalty. This is creating by having a normally clean reputation, in both its marketing and customer service. It comes from creating a good product that marketers rely on.

This sort of brand loyalty that all brands strive for. When your fans have you back even when you make a mistake, that’s when you’ve reached an exceptional level of consumer success.

 


 
How did this incident effect you? Did it cause any issues that you’re still dealing with? Let us know in the comments section.
 


 

Related Posts

Leave a Reply

%d bloggers like this: