Banking Domain Case Study in Hadoop and R

In this blog and the next few ones that will follow, we will analyze a banking domain dataset, which contains several files with details of its customers. This database was prepared by Petr Berka and Marta Sochorova.

The Berka dataset is a collection of financial information from a Czech bank. The dataset deals with over 5,300 bank clients with approximately 1,000,000 transactions. Additionally, the bank represented in the dataset has extended close to 700 loans and issued nearly 900 credit cards, all of which are represented in the data.

By the time you finish reading this blog,  you would have learned :

• How to analyze a bank’s data to predict a customer’s quality
• Using this analysis we can categorize a customer into three categories:

1. Excellent: Customers whose record is good with the bank
2. Good: Customers who have average earning with a good record till now
3. Risky: Customers who are under debt of bank or who has not paid the loan on time

• How to write PIG UDF
• How to connect Hadoop with R
• How to load data from Hadoop to R

How to analyze a bank’s data to predict the customer’s quality

Prerequisite

Software Technology

• Java installed Hadoop concepts
• Hadoop installed Java concepts
• Pig installed Pig concepts
• R-base
• Rstudio
• Ubuntu OS

View the detail case study here.

PIG Installation on Ubuntu

Step 1 : Install PIG from Cloudera repository

$ sudo apt-get install pig

Step 2 : For each user who will be submitting MapReduce jobs using MapReduce v1 (MRv1), or running Pig, Hive, or Sqoop in an MRv1 installation, set the HADOOP_MAPRED_HOME environment variable as follows: [In case it is not already updated]

$ sudo gedit .bashrc
 

export HADOOP_MAPRED_HOME=/usr/lib/hadoop-0.20-mapreduce

Step 3 : To start Pig in interactive mode (MRv1)

$ pig

Step 4 :  Examples

grunt> ls
hdfs://localhost/user/joe/input <dir>

grunt> A = LOAD 'input';
grunt> B = FILTER A BY $0 MATCHES '.*dfs[a-z.]+.*';
grunt> DUMP B; 

[For this example to run you need input directory to be created. Incase you

already have not created it in our previous mentioned steps of Hadoop Installation

 please create it:

$ sudo -u hdfs hadoop fs -mkdir -p /user/$USER

$ sudo -u hdfs hadoop fs -chown $USER /user/$USER

$ hadoop fs -mkdir input

$ hadoop fs -put /etc/hadoop/conf/*.xml input

$ hadoop fs -ls input ] 

Create your first apache pig script

0
inShare

As is the case with scripts in other programming languages such as SQL, Unix Shell, etc., Pig scripts are used to execute a set of Apache Pig commands collectively. This helps in reducing the time and effort invested in writing and executing each command manually while doing the Pig programming. This blog (Pig Programming: Create Your First Apache Pig script) is a step by step guide to help you create your first Apache Pig script.

Pig Programming: Create Your First Apache Pig Script

An Apache Pig script works in two modes:

Local Mode: In ‘local mode’, you can execute the pig script in local file system. In this case you don’t need to store the data in Hadoop HDFS file system, instead you can work with the data stored in local file system itself.
HDFS Mode: In ‘HDFS mode’, the data needs to be stored in HDFS file system and you can process the data with the help of pig script.

Pig Script in HDFS Mode:

Step1: Writing a script

Open an editor (e.g. gedit) in your Cloudera Demo VM environment:
Command:  gedit sample.pig

This command will create a ‘sample.pig’ file inside the home directory of cloudera user.

Let’s write few PIG commands in the sample script.
Let us say our task is to read data from a data file and to display the required contents on the as output.
The sample data file contains following data:
Shabbir           Khan             9314573259     Bangalore        Engineer
Manish            Sharma        8882148796     Gurgaon           Lecturer
Mahesh           Kumar          8521548932     Noida                Business
Sampath         Reddy           8547987412     Hyderabad         Engineer
Mohan            Reddy           9256458798     Hyderabad         Professor
Save the text file with the name ‘information.txt’

The sample data file contains five columns FirstName, LastName, MobileNo, City, and Profession separated by tab key. Our task is to read the content of this file in to HDFS and display First Name, Mobile Number and Profession of these contacts.
To process this data using Pig, this file should be present in Apache Hadoop HDFS.
Use  the following command:
Command: hadoop dfs –copyFromLocal information.txt hdfs:/

Edit the Pig script (sample.pig) to include following commands:
A = LOAD ‘/information.txt’ using PigStorage (‘\t’) as (FName: chararray, LName: chararray, MobileNo: chararray, City: chararray, Profession: chararray);
B = FOREACH A generate FName, MobileNo, Profession;
DUMP B;

Save and close the file.
The first command loads the file ‘information.txt’ into variable A with indirect schema (FName, LName, MobileNo, City, Profession).
The second command loads the required data from variable A to variable B.
The third line displays the content of variable B on the terminal/console.
Step 2: Execute the Pig Script
To execute the pig script in HDFS mode, run the following command:
Command: pig sample.pig

Review the result.

Congratulations on executing your first Pig script successfully!

Installing Pig in Hadoop

Installing Pig

To install Pig On Red Hat-compatible systems:

$ sudo yum install pig

To install Pig on SLES systems:

$ sudo zypper install pig

To install Pig on Ubuntu and other Debian systems:

$ sudo apt-get install pig

  Note:

Pig automatically uses the active Hadoop configuration (whether standalone, pseudo-distributed mode, or distributed). After installing the Pig package, you can start the grunt shell.

To start the Grunt Shell (MRv1):

$ export PIG_CONF_DIR=/usr/lib/pig/conf
$ export PIG_CLASSPATH=/usr/lib/hbase/hbase-0.94.2-cdh4.2.1-security.jar:

/usr/lib/zookeeper/zookeeper-3.4.5-cdh4.2.1.jar
$ pig 

grunt> 

To start the Grunt Shell (YARN):

  Important:

For each user who will be submitting MapReduce jobs using MapReduce v2 (YARN), or running Pig, Hive, or Sqoop in a YARN installation, set the HADOOP_MAPRED_HOME environment variable as follows:

$ export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce

$ export PIG_CONF_DIR=/usr/lib/pig/conf
$ export PIG_CLASSPATH=/usr/lib/hbase/hbase-0.94.2-cdh4.2.1-security.jar:

/usr/lib/zookeeper/zookeeper-3.4.5-cdh4.2.1.jar
$ pig 
...
grunt>

To verify that the input and output directories from the example grep job exist list an HDFS directory from the Grunt Shell:

grunt> ls
hdfs://localhost/user/joe/input <dir>
hdfs://localhost/user/joe/output <dir>

To run a grep example job using Pig for grep inputs:

grunt> A = LOAD 'input';
grunt> B = FILTER A BY $0 MATCHES '.*dfs[a-z.]+.*';
grunt> DUMP B;

To check the status of your job while it is running, look at the JobTracker web console http://localhost:50030/.