The geek from Egypt who’s taking on the world of big data

Techie Tuesdays

Tuesday July 25, 2017,

16 min Read

Amr Awadallah would be a professor today had he not stumbled upon the world of startups. The focus of this week's Techie Tuesdays, Amr Co-founded Cloudera, the platform for data management and analytics he built on Apache Hadoop and other open source technologies. The company went for an IPO earlier this year and is now valued at almost $2.5 billion.

During a recent conversation with Amr Awadallah, Founder and CTO of Cloudera, I not only learnt of his fascinating journey, but also about an aspect of the Egyptian life that I until then thought was peculiar to India. Listening to Amr talk about his early years, I realised that Indian parents aren't the only ones to decide or force a career path on their children. Egyptian parents too do that. I also learnt that Yahoo! has a product division called Yahoo Shopping, and that Mendel Rosenblum, one of the co-founders of VMware, is a professor of Computer Science at Stanford University.

Amr would’ve remained a computer enthusiast irrespective of his career path. If not a techie and CTO, he would have become a professor, a career path his father had chosen for him.

Things were on track until the startup bug bit Amr. He started up and exited while still a student. After getting his doctoral degree, he went back to the startup world again, this time with more confidence and knowledge. Amr founded Cloudera with three other people in 2008. Earlier this year, the company went for an IPO and is currently valued at nearly $2.5 billion. Cloudera is one of the largest platforms in the world for data management and analytics.

This week’s Techie Tuesdays delves into Amr’s journey, from Cairo to California.

Related read - From UP to the US: The journey of Abhinav Asthana and his affair with APIs

Amr was born in Cairo, Egypt in 1970. Soon after his birth, his father moved to Southampton in the UK to do his PhD in Accounting. Amr and his mother joined him a little later and they spent five years there. After finishing his PhD, his father returned home and joined Cairo University as a professor. Amr recalls his father pushing him to become a university professor from an early age. It was quickly becoming Amr’s goal too.

Amr did very well in school and graduated at the top of his class in Cairo University's Faculty of Engineering. A teaching job was seen as a logical progression for students excelling at the undergraduate level. Amr enrolled for a Master's in computer engineering.

He studied animation and worked on creating characters that walked like humans—as opposed to animators drawing them frame by frame. He explains,

Amr loved video games ever since he was a child and even saw a future in animation. He wanted to make the process of creating characters in the games easier, as well as endow them with movements that were as natural and real as possible.

After his postgraduation, he applied to many universities in the US and was accepted at Stanford besides several others.

The first few months at Stanford were tough for Amr. He had moved to the US with his wife and their little baby and did not know many people there. Also, Amr who was used to being a topper in his class in Cairo found himself to be average in comparison to other students in his class at Stanford.

Initially, Amr planned to do his PhD in computer networking and he even published a couple of papers on the subject. While at Stanford, he got exposed to entrepreneurship very early. He recalls,

All this made starting up relatively easier in the eyes of students. In 1999, Amr joined one of his friends, Thai Tran, who was building a shopping comparison service for books called BookSmart (later renamed VivaSmart). Since it was the height of the internet bubble back then, Amr could comfortably raise about $500,000 for his venture.

VivaSmart was a comparison shopping engine that could crawl all over the web and aggregate all kinds of information about a product (specifications, images, etc) along with prices from different websites. Amr adds, “We started with books because students are price sensitive about books.”

VivaSmart used Oracle for the database to aggregate a lot of information and Perl scripts and programs to build the agents to crawl the web to ferret out information. The company had an on-premise data centre.

Also read - Meet Ajay Shrivastava – the tech general behind culture and technology at Oyo and Knowlarity

In 2000, it became obvious that the internet bubble was going to burst, and a lot of companies started shutting down. Since VivaSmart had a good product but fewer customers Amr and his co-founder decided to sell the company. They approached some of their clients and one of them gave a good offer. But on the day the deal was to be inked, it was called off. Amr says, “We were devastated. But we went back and looked at other companies we had approached.”

Eventually, Yahoo! agreed to acquire VivaSmart in June 2000 for $8.9 million, and Amr and his co-founder joined Yahoo!

While focusing on VivaSmart, Amr stopped working on his PhD thesis and, as a result, by 2001, his two years of leave of absence came to an end. He was keen to go back and finish his PhD and told Yahoo! the same. But the company, wanting him to continue to work for them, agreed to pay Amr’s tuition fee and gave him enough time to work on his PhD. Professor Mendel Rosenblum, who was one of the founders of VMware, agreed to guide Amr. Amr’s thesis was vMatrix, a backward-compatible solution for improving the interactivity, scalability, and reliability of Internet applications. He adds,

In his first four years at Yahoo!, between 2000 and 2004, Amr focused on the comparison shopping service, which was formed after VivaSmart’s acquisition. VivaSmart became the backend for Yahoo Shopping and Amr helped in integrating the technology and adding new features.

In 2004, Amr shifted to a new role: measuring the performance for business intelligence and data analytics for different Yahoo products and optimising it. He holds this experience fundamental to the idea that later metamorphosed into Cloudera.

You may also like: The untold story of Orkut Büyükkökten—the man who built world’s largest social network before Mark Zuckerberg

At Yahoo!, Amr had built a business intelligence platform using Oracle technology for data warehousing and data systems where all the data was aggregated. He used IBM DataStage for data collection, and MicroStrategy for business reporting and dashboards. While doing all this, he faced the following problems in his data pipeline:

At that time, a group at Yahoo! was working on Hadoop for Yahoo! Search. Amr recalls,

Amr met the team working on this and described some of his BI problems and data science challenges. He tried Hadoop and it solved all the four problems. He adds, “Eight hours of data processing was shortened to five minutes. Cost also came down ten times on a per gigabyte basis. It was much more flexible to change the schemas and I could use many other ways to analyse data other than SQL.”

Amr finished his PhD in 2007 and left Yahoo! in 2008. By then, he had figured out that Hadoop would become big. He joined Accel Partners as entrepreneur-in-residence (EIR) and got a chance to work on idea stages of startups that were being pitched to the firm. Soon, he decided to work on Hadoop startups as he understood the problem and the solution really well.

He met his first co-founder, Jeff Hammerbacher, who was also an EIR then at Accel. Jeff started the data science team at Facebook. In fact, he, along with DJ Patil, had actually coined the term 'data science'. Jeff too had experienced the power of Hadoop for scale, agility and cost efficiency. Just a few weeks down the line, Amr and Jeff heard about two guys doing the same thing as them and decided to meet them on the advice of the investor to whom they were pitching. The duo met Mike Olson, who went on to become Cloudera's CEO for the first five years, and Christophe Bisciglia. The four, bringing complementary skills to the table, formally created Cloudera in October 2008.

Related read: Meet Kiran Bhat—the man who engineered Hulk and Tarkin to win 2017 sci-tech Oscar

Amr believes that every startup goes through three phases:

Cloudera has recently launched Altus, the platform-as-a-service version of their product. Cloudera counts Airtel as one of their largest customers globally. The company also has a couple of other customers in the Indian government.

Cloudera started with Hadoop focused on batch processing and storage, using MapReduce Service and HDFS (Hadoop Distributed File System). Currently, the company has almost 30 different projects. Amr says,

He adds, “To achieve that result, we need to have a platform that is flexible in terms of consuming any type of data but can also extract the value of the data in different ways.”

Some of the offerings of Cloudera include:

Amr notes that the challenges for a startup, though many, vary according to the stage it is in. During the product-market fit phase, Cloudera’s main challenge was to figure out the features and functions of open source projects that they should include in the Hadoop distribution to meet the needs and expectations of the widest spectrum of customers possible.

When Cloudera started signing on a lot of customers, they had a challenge in terms of providing support to these customers. They now have bots in customer support that handle almost 20 percent of the workload of the support organisation. Cloudera finds it much more scalable than having humans handle the workload, and it plans to continue investing in bots.

Amr and the Cloudera team believe that by collecting enough data of different types—not just structural data, but semi-structural data from sensors, mobile devices, as well as completely unstructured data from social media, email and documents—on how humans make different types of decisions, one can learn so much more on the subject. This has been a part of Cloudera’s DNA from early days.

Amr believes that human civilisation is now in the sixth wave of automation:

He claims that Cloudera's stack is superior to that of Amazon’s and others. He says,

In addition, Amr claims that Cloudera has the bot/AI that supports their customer clusters and analyses their data and lets them know what changes need to be made to keep the cluster performing with high security and capacity.

Also read - The untold story of Alan Cooper, the father of Visual Basic

As for Cloudera's employees, Amr looks for productive, smartpeople with good work ethics and who fit the company’s culture. He says,

Cloudera follows consensus-driven decision-making. Amr believes that debates, even if they cause the occasional delay, lead to smarter decisions.

Much of Amr's individual values shine through at Cloudera as well:

One of the prominent regrets in Amr’s life is missing out on an opportunity to prevent one of Cloudera’s key competitors from being born. He says, “HortonWorks came from Yahoo! and I knew about that team. In some ways we could have intervened to acquire the team or at least hire some of the people from the team and prevent that spin off from taking place.” HortonWorks is valued at more than $800 million now.

According to International Data Corporation, automation of decisions will be a $200-billion market by 2020. Amr believes that over the next 10-15 years, Cloudera has enough opportunity to become a $30-100 billion one. With Cloudera, Amr wants to create the Oracle or the IBM of the future. He wants to enable this wave of automation of decisions that will change finance, insurance, governance, smart cities and how we all live.

You can connect with Amr on Twitter | LinkedIn

cloudera

Jeff Hammerbacher

Cairo University

Amr Awadallah

Christophe Bisciglia

VivaSmart

Altus

1

AI Gen

Technical Guruji: Gaurav's Journey to Becoming India's Richest Tech Influencer

2

AI Gen

Indian Woman Sampriti Bhattacharyya: Creator of America's First Flying Electric Boat

3

AI Gen

Man Turns AI Passion into Rs 1.4 Crore Success in Weeks

4

AI Gen

How Aristotle's Wisdom Fuels Modern Productivity and Emotions

5

Interview

Life after CaratLane: Mithun Sacheti wants to help build businesses