cloudera-spark

Developing Spark Applications
• If restrictions on HDFS encryption zones prevent files from being moved to the HDFS trashcan. This restriction
primarily applies to CDH 5.7 and lower. With CDH 5.8 and higher, each HDFS encryption zone has its own HDFS
trashcan, so the normal DROP TABLE behavior works correctly without the PURGE clause.
Using Spark MLlib
CDH 5.5 supports MLlib, Spark's machine learning library. For information on MLlib, see the Machine Learning Library
(MLlib) Guide.
Running a Spark MLlib Example
To try Spark MLlib using one of the Spark example applications, do the following:
1. Download MovieLens sample data and copy it to HDFS:
$ wget --no-check-certificate \
https://raw.githubusercontent.com/apache/spark/branch-1.6/data/mllib/sample_movielens_data.txt
$ hdfs dfs -copyFromLocal sample_movielens_data.txt /user/hdfs
2. Run the Spark MLlib MovieLens example application, which calculates recommendations based on movie reviews:
$ spark-submit --master local --class org.apache.spark.examples.mllib.MovieLensALS \
SPARK_HOME/lib/spark-examples.jar \
--rank 5 --numIterations 5 --lambda 1.0 --kryo sample_movielens_data.txt
Enabling Native Acceleration For MLlib
MLlib algorithms are compute intensive and benefit from hardware acceleration. To enable native acceleration for
MLlib, perform the following tasks.
Install Required Software
• Install the appropriate libgfortran 4.6+ package for your operating system. No compatible version is available
for RHEL 5 or 6.
OS
RHEL 7.1
SLES 11 SP3
Ubuntu 12.04
Ubuntu 14.04
Debian 7.1
Package Name
libgfortran
libgfortran3
libgfortran3
libgfortran3
libgfortran3
Package Version
4.8.x
4.7.2
4.6.3
4.8.4
4.7.2
• Install the GPL Extras parcel or package.
Verify Native Acceleration
You can verify that native acceleration is working by examining logs after running an application. To verify native
acceleration with an MLlib example application:
1. Do the steps in Running a Spark MLlib Example on page 20.
2. Check the logs. If native libraries are not loaded successfully, you see the following four warnings before the final
line, where the RMSE is printed:
15/07/12 12:33:01 WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeSystemBLAS
20 | Spark Guide