How Amazon Web Services Surged Out of Nowhere
Few people–if any–saw it coming. Even engineers who helped build the omnipresent Cloud that is Amazon Web Services (AWS) are surprised by its goliath success today.
“You never know how big something will be while you’re working on it, Christopher Brown,” an early AWS engineer told Business Insider.
When Amazon CEO Jeff Bezos first proposed selling infrastructure-as-a-service (IaaS), his board of directors raised an eyebrow. It’s understandable. For the general population, “Cloud-based technology platform” isn’t exactly the first thing that comes to mind when you think of Amazon. AWS is far removed from the core function as an e-commerce site going 21-years-strong. But today AWS zooms so far ahead in the market that its competitors are dwarfed into tiny specs in its rear view mirror. Even its own execs behind its growth strategy say they’re surprised at the speed of AWS’s claim to fame. Here’s a matrix depiction of the market share by Gartner:
For several years, AWS has been a developer’s paradise, a platform where you can build and run software without having to set up your own servers. It’s grown to be so huge that even its IaaS competitors, like DigitalOcean, depend on it. Nearly 40% of traffic runs through its infrastructure, according to an estimation by Jarrod Levitan, Chief Cloud Officer at TriNimbus. Another more concrete statistic by Deepfield Networks found that 1/3 of Internet users visit an AWS-supported website daily.
If AWS stopped working tomorrow, much of the Internet would dim. You wouldn’t be able to browse billion dollar social networking sites, like Pinterest. Your DropBox photos and files would be missing in action. You couldn’t stream Netflix. Forget scoring a great deal on Airbnb. Many of these billion dollar startup may have never disrupted their respective industries without AWS to propel them quickly on the scene. Plus, enterprises and even the ever-paranoid governmental agencies, like the CIA, NASA, Department of Defense and the Pentagon are now relying on AWS.
Not only is it massive, but its capabilities are astounding. Of the dozen services, arguably among the most innovative is Amazon Kinesis, a real-time processing service for live events and streaming. It can continuously capture and store terabytes of data each hour from hundreds of thousands of sources at once. It’s great, for instance, for helping advertising agencies make sense of live social chatter.
The team knew the initial concept was–at most–an interesting idea, but there are so many existing big players in the realm of data storage. How could Amazon, an online shopping site, compete? Having put his neck on the line in front of investors, Bezos must have known he was on to something…but no one could have predicted the scale to which AWS has escalated.
The Canny Rise of ‘the Infrastructure of the World’
There’s an old folklore about how AWS got its legs. The story goes that because Amazon’s technology capacity requirement naturally shoots off the charts during holiday season, it has excess capacity for the rest of the year. So, why not rent the excess storage to other companies that need it?
Amazon CTO Werner Vogels wonders: Why won’t this myth die?
It was never a matter of selling excess capacity, actually within 2 months after launch AWS would have already burned through the excess Amazon.com capacity.
One thing that the fable gets right is that AWS was created out of Amazon’s own need to support high volume of data. But each interface was created with design and intent that outsiders will eventually use it. Former Amazonian Steve Yegge recalls the infamous mandate put forth by Bezos around 2002:
“He issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.”
Essentially, the mandate not only required all developers to expose data through surface interfaces over network but also built to be “externalizable,” or good enough for outsiders to use. So, you see, it was always about deliberate business and innovation from its inception, stemmed by Bezos’ singular focus on obsessively serving their customers. Amazon had been perfecting its infrastructure to meet massive needs for several years. Given its internal success. it only made sense for Bezos to take his team’s invaluable skill and profit from it.
But AWS wasn’t Bezos’ brainchild.
The initial proposal paper (because Bezos doesn’t like slides) came from head of global infrastructure Chris Pinkham, who first ideated a potential “Infrastructure of the World.”
He worked closely with website engineering manager Benjamin Black:
Chris was always pushing me to change the infrastructure, especially driving better abstraction and uniformity, essential for efficiently scaling. He wanted an all IP network instead of the mess of VLANs Amazon had at the time, so we designed it, built it, and worked with developers so their applications would work with it.
From their own experience, they knew that maintaining servers eats up the majority of time and money. This push for universal infrastructure no doubt came from Bezos’ relentless prioritization of customers. Bezos famously asks every team to leave an empty seat in meetings for their customer, as a demonstration of just how customer-centric the company should be. In this case, the better, more comprehensive infrastructure Amazon offers, the better support its customers would enjoy. And so AWS was born into a world starved of affordable, reliable and all-encompassing IaaS.
But Why is Amazon the Chosen One?
Amazon has a historical lead in an industry so new that few people understand truly understand it. Some call it a Coke without a Pepsi competitor. AWS’s biggest IaaS challengers are Microsoft Azure, Google Compute Engine and IBM Softlayer. To supplement the Gartner matrix above, take a look at the difference in revenue last year–AWS is raking in more than its top competitors, including hybrid servicers, combined:
Its legendary lead is a result of three core reasons. First, much of this growth is in parallel with the rise of Cloud computing itself. You can see the steady increase of Cloud popularity and interest via Google Search Trends. It starts to rise around 2007, just one year after the release of Amazon’s two most widely acclaimed services: Elastic Compute Cloud (EC2) that maximizes compute capacity and Simple Storage Service (S3), which stores massive data.
Second, Amazon’s EC2 and S3 were the very first widely accessible virtual infrastructure services by a number of years. It’s part of cloud computing history. AWS had been working on solving a pain point that’s common to all developers, and perfected the solution long before anyone else in the industry.
Third, when the mindset started shift from apprehension about the Cloud to a necessary gravitation to the Cloud, existing players could only scramble to copy AWS services. Microsoft’s Windows Azure, for instance, started off as a Platform-as-a-Service (PaaS), but kept losing to AWS. Microsoft had to add IaaS capabilities to stay in the Cloud game, becoming a hybrid service and rebranded as “Microsoft Azure.” Then, Google followed the lead and launched its own virtual server service, marking the start of copycats.
As a result of these copycats, there’s been a significant price war to developers’ benefit. Specifically, AWS has dropped its price 49 times in 8 years. But what’s even more impressive is some scrambling competitors rely on AWS to provide IaaS to their own customer base. We mentioned DigitalOcean’s dependence on AWS earlier in this piece.
But Target.com’s story is even more telling. It had been a customer of AWS since 2001, until it decided to go its separate ways and build its own server. A little disheartened, but still amicable, AWS helped Target.com transfer its data. But after one popular promotion, the site crashed. It’s unsettling moments like this that light a fire under any infrastructure to capitalize on AWS’s offerings for additional support.
Moving forward, AWS isn’t settling to dominate the IaaS market. At the end of 2014, it announced a number of new PaaS tools to bolster developer’s paradise. This includes: Code Commit, Code Pipeline and Code Deploy. David Bernstein, CEO of Cloud Strategy Partners, sees AWS becoming any developer’s comprehensive environment.
The Cost of Supreme Excellence
After Pinkham was appointed to execute AWS, it was time to get to work. The AWS might have started with Pinkham and a handful of engineers sent to Africa, where he was based. But it’s grown to a massive organization under Amazon’s umbrella. A former EC2 engineer says there are several teams that are dedicated to each offering (e.g. S3 team).
A recent New York Times Amazon in-depth expose reveals some awful things about Amazon’s culture in general. Referring to it a “bruising workplace,” reporters cite unforgiving examples of lack of empathy by Amazonian leaders who prioritize work above all. Some Amazonians have responded with claims of inaccuracies and false depictions of isolated incidents strung together to depict a hellish culture.
Whether or not these anecdotes are truly representative of Amazon’s culture, one thing is certain: It takes herculean dedication to achieve excellence, especially to that of AWS’s caliber. This kind of market leadership doesn’t come without cost of sweat and tears. It’s always subjective to judge a company’s culture–what may be a joyous challenge to one might be relentlessly poor balance of work and life to another. Such high-performing teams can be self-selecting. After all, every single Amazonian chose to be there and most likely has InMail invitations from top tech companies waiting to be read.
Granted this is not a scientific measure, most reviews of the AWS team are glowing. Several AWS engineers can attest to the grueling workplace, but still express gratification in their role in trailblazing their industry:
I worked for EC2 in Cape Town, South Africa. It was the best!!!! I can’t imagine ever finding a working environment as cool as what I had in my team. That said, there were on-call weeks where I was cursing the company and my job at 4am, being awake for the 3rd consecutive night. But the org is aware of problems like heavy on-call load. They started a program to improve unnecessary tickets. I found AWS (at least in Cape Town) to be a really well-run place to work.
Bezos himself is very clear that this type of revolutionary environment may not be for everyone. In one meeting, a female engineer asked what Amazon will do to better the work-life balance? His response is clear as daylight and indicative of severely high expectations:
“The reason we are here is to get stuff done, that is the top priority,” he answered bluntly. “That is the DNA of Amazon. If you can’t excel and put everything into it, this might not be the place for you,” he says, according to the book on the history of Amazon.
For better or worse, this is the culture that lead to the success of AWS today. Bezos was almost prophetic when he green-lit Pinkham and Black’s proposal to sell Amazon’s internal virtual service. By focusing his talented engineers’ energy on the infrastructure, and laying out a rigid culture of excellence, AWS is “winning” the market share for now.
Developers, what has been your experience with AWS…is AWS too big to fail or can competitors knock it off the #1 spot?
If you enjoyed reading this, please subscribe to our newsletter for an occasional update when we have something new to share!
Also, if you’d like 100 free AWS credits, you can enter the upcoming World Cup CodeSprint (or online hackathon) and complete one challenge.