cloudera-spark

Developing Spark Applications
15/07/12 12:33:01 WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeRefBLAS
15/07/12 12:33:01 WARN LAPACK: Failed to load implementation from:
com.github.fommil.netlib.NativeSystemLAPACK
15/07/12 12:33:01 WARN LAPACK: Failed to load implementation from:
com.github.fommil.netlib.NativeRefLAPACK
Test RMSE = 1.5378651281107205.
You see this on a system with no libgfortran. The same error occurs after installing libgfortran on RHEL 6
because it installs version 4.4, not 4.6+.
After installing libgfortran 4.8 on RHEL 7, you should see something like this:
15/07/12 13:32:20 WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeSystemBLAS
15/07/12 13:32:20 WARN LAPACK: Failed to load implementation from:
com.github.fommil.netlib.NativeSystemLAPACK
Test RMSE = 1.5329939324808561.
Accessing External Storage from Spark
Spark can access all storage sources supported by Hadoop, including a local file system, HDFS, HBase, and Amazon S3.
Spark supports many file types, including text files, RCFile, SequenceFile, Hadoop InputFormat, Avro, Parquet,
and compression of all supported files.
For developer information about working with external storage, see External Storage in the Spark Programming Guide.
Accessing Compressed Files
You can read compressed files using one of the following methods:
• textFile(path)
• hadoopFile(path,outputFormatClass)
You can save compressed files using one of the following methods:
• saveAsTextFile(path, compressionCodecClass="codec_class")
• saveAsHadoopFile(path,outputFormatClass, compressionCodecClass="codec_class")
where codec_class is one of the classes in Compression Types.
For examples of accessing Avro and Parquet files, see Spark with Avro and Parquet.
Accessing Data Stored in Amazon S3 through Spark
To access data stored in Amazon S3 from Spark applications, you use Hadoop file APIs (SparkContext.hadoopFile,
JavaHadoopRDD.saveAsHadoopFile, SparkContext.newAPIHadoopRDD, and
JavaHadoopRDD.saveAsNewAPIHadoopFile) for reading and writing RDDs, providing URLs of the form
s3a://bucket_name/path/to/file. You can read and write Spark SQL DataFrames using the Data Source API.
You can access Amazon S3 by the following methods:
Without credentials:
Run EC2 instances with instance profiles associated with IAM roles that have the permissions you want. Requests
from a machine with such a profile authenticate without credentials.
With credentials:
• Specify the credentials in a configuration file, such as core-site.xml:
fs.s3a.access.key
Spark Guide | 21