Apache Hadoop and it's Distribution

To know about Hadoop deployment in the industries we have to be aware of it's distributors. Disctibutors of Hadoop are those companyes that provides Apache Hadoop-based software, support and services, and training to business customers. There are multiple vendor which exist in the market among whom two are the most widely used Apache Hadoop distribution
i.  Cloudera
ii. Hortonworks

Cloudera v. Hortonworks: Tale of the Tape 
Cloudera has plenty to boast about. It has in fact contributed significantly to the open source Apache Hadoop project and its Hadoop distribution is in production at high-profile Web companies like Groupon and Klout. It launched an innovative partner and certification program in September and Cloudera engineers continue to develop new features to help Hadoop meet enterprise-level uptime and security requirements.

In addition, Cloudera has a two-year head start over Hortonworks servicing a small but growing customer base. No question the Hortonworks team learned many valuable lessons working at Yahoo, but supporting an internal Hadoop deployment at one large technology company is a lot different than supporting a large and varied customer base of both technology and non-technology companies. In order for Hortonworks to become a self-sufficient Hadoop support juggernaut, Baldeschwieler’s stated goal, the company needs to prove it can deliver.

Finally, consider the competing Hadoop distributions themselves. Their cores are both based on the open source Apache Hadoop distribution and related sub-projects, with the real differentiation being the installation and administration management add-on tools. Cloudera Management Suite, while proprietary, includes important enterprise-level features such as automated, wizard-based Hadoop deployment capabilities, dashboards for configuration management and a resource management module for capacity and expansion planning. Ambari, Hortonworks' answer to Cloudera Management Suite, is open but is less mature and currently lacks advanced cluster management capabilities.

The reality is that Cloudera’s Hadoop distribution is largely open source and the risk of vendor lock-in due to its relatively few proprietary components is, in Wikibon’s opinion, lower than what Hortonworks marketing implies. Organizations that come to rely on Cloudera Enterprise for crucial parts of the business but later decide to move to a different Hadoop distribution or competing Big Data approach should be able to do so with little difficulty.

That said, Hortonworks’ open 100% approach means that updates and improvements to its distribution are likely to come quicker than those of Cloudera’s distribution and that partners may find it easier to integrate with HDP than Cloudera Enterprise. These are not insignificant factors that potential customers must consider.

Industry Requirement for Hadoop

Well before you dig more dipper into Hadoop you must need to know why it is creating that much of buzz in today's industries. I try to explain in the simplest term the "Industry Requirement for Hadoop" as this blog is meant for the beginners and novice hence not drilling into complex data scientific and customize data mining algorithm of it which gives the developer ample power to tweak their data warehouse on the fly.

Hadoop is the poster child for Big Data, so much so that the open source data platform has become practically synonymous with the wildly popular term for storing and analyzing huge sets of information.

While Hadoop is not the only Big Data game in town, the software has had a remarkable impact. But exactly why has Hadoop been such a major force in Big Data? What makes this software so damn special - and so important?

Sometimes the reasons behind something success can be staring you right in the face. For Hadoop, the biggest motivator in the market is simple: Before Hadoop, data storage was expensive.

Hadoop, however, lets you store as much data as you want in whatever form you need, simply by adding more servers to a Hadoop cluster. Each new server (which can be commodity x86 machines with relatively small price tags) adds more storage and more processing power to the overall cluster. This makes data storage with Hadoop far less costly than prior methods of data storage.


Spendy Storage Created The Need For Hadoop

We're not talking about data storage in terms of archiving… that's just putting data onto tape. Companies need to store increasingly large amounts of data and be able to easily get to it for a wide variety of purposes. That kind of data storage was, in the days before Hadoop, pricey.

And, oh what data there is to store. Enterprises and smaller businesses are trying to track a slew of data sets: emails, search results, sales data, inventory data, customer data, click-throughs on websites… all of this and more is coming in faster than ever before, and trying to manage it all in a relational database management system (RDBMS) is a very expensive proposition.

Historically, organizations trying to manage costs would sample that data down to a smaller subset. This down-sampled data would automatically carry certain assumptions, number one being that some data is more important than other data. For example, a company depending on e-commerce data might prioritize its data on the (reasonable) assumption that credit card data is more important than product data, which in turn would be more important than click-through data.

Assumptions Can Change

That's fine if your business is based on a single set of assumptions. But what what happens if the assumptions change? Any new business scenarios would have to use the down-sampled data still in storage, the data retained based on the original assumptions. The raw data would be long gone, because it was too expensive to keep around. That's why it was down-sampled in the first place.

Expensive RDBMS-based storage also led to data being siloed within an organization. Sales had its data, marketing had its data, accounting had its own data and so on. Worse, each department may have down-sampled its data based on its own assumptions. That can make it very difficult (and misleading) to use the data for company-wide decisions.

Hadoop: Breaking Down The Silos

Hadoop's storage method uses a distributed filesystem that maps data wherever it sits in a cluster on Hadoop servers. The tools to process that data are also distributed, often located on the same servers where the data is housed, which makes for faster data processing.

Hadoop, then, allows companies to store data much more cheaply. How much more cheaply? In 2012, Rainstor estimated that running a 75-node, 300TB Hadoop cluster would cost $1.05 million over three years. In 2008, Oracle sold a database with a little over half the storage (168TB) for $2.33 million - and that's not including operating costs. Throw in the salary of an Oracle admin at around $95,000 per year, and you're talking an operational cost of $2.62 million over three years - 2.5 times the cost, for just over half of the storage capacity.

This kind of price savings mean Hadoop lets companies afford to hold all of their data, not just the down-sampled portions. Fixed assumptions don't need to be made in advance. All data becomes equal and equally available, so business scenarios can be run with raw data at any time as needed, without limitation or assumption. This is a very big deal, because if no data needs to be thrown away, any data model a company might want to try becomes fair game.

That scenario is the next step in Hadoop use, explained Doug Cutting, Chief Architect of Cloudera and an early Hadoop pioneer. "Now businesses can add more data sets to their collection," Cutting said. "They can break down the silos in their organization."

More Hadoop Benefits

Hadoop also lets companies store data as it comes in - structured or unstructured - so you don't have to spend money and time configuring data for relational databases and their rigid tables. Since Hadoop can scale so easily, it can also be the perfect platform to catch all the data coming from multiple sources at once.

Hadoop's most touted benefit is its ability to store data much more cheaply than can be done with RDBMS software. But that's only the first part of the story. The capability to catch and hold so much data so cheaply means businesses can use all of their data to make more informed decisions.