cloudera-spark

Developing Spark Applications
|-- rating: double (nullable = true)
|-- year: integer (nullable = true)
|-- month: integer (nullable = true)
[Git,2.0,2011,7]
import com.databricks.spark.avro._
val sqlContext = new SQLContext(sc)
val df = sqlContext.read.avro("input_dir")
val name = "AvroTest"
val namespace = "com.cloudera.spark"
val parameters = Map("recordName" -> name, "recordNamespace" -> namespace)
df.write.options(parameters).avro("output_dir")
Figure 9: Specifying a Record Name
Java Example
Use the DataFrame API to query Avro files in Java. This example is almost identical to Figure 5: Scala Example with
Format on page 25:
import org.apache.spark.sql.*;
SQLContext sqlContext = new SQLContext(sc);
// Creates a DataFrame from a file
DataFrame df = sqlContext.read().format("com.databricks.spark.avro").load("input_dir");
// Saves the subset of the Avro records read in
df.filter("age > 5").write().format("com.databricks.spark.avro").save("output_dir");
Python Example
Use the DataFrame API to query Avro files in Python. This example is almost identical to Figure 5: Scala Example with
Format on page 25:
# Creates a DataFrame from a directory
df = sqlContext.read.format("com.databricks.spark.avro").load("input_dir")
# Saves the subset of the Avro records read in
df.where("age > 5").write.format("com.databricks.spark.avro").save("output_dir")
Accessing Parquet Files From Spark SQL Applications
Spark SQL supports loading and saving DataFrames from and to a variety of data sources and has native support for
Parquet. For information about Parquet, see Using Apache Parquet Data Files with CDH.
To read Parquet files in Spark SQL, use the SQLContext.read.parquet("path") method.
To write Parquet files in Spark SQL, use the DataFrame.write.parquet("path") method.
To set the compression type, configure the spark.sql.parquet.compression.codec property:
sqlContext.setConf("spark.sql.parquet.compression.codec","codec")
The supported codec values are: uncompressed, gzip, lzo, and snappy. The default is gzip.
Currently, Spark looks up column data from Parquet files by using the names stored within the data files. This is different
than the default Parquet lookup behavior of Impala and Hive. If data files are produced with a different physical layout
due to added or reordered columns, Spark still decodes the column data correctly. If the logical layout of the table is
changed in the metastore database, for example through an ALTER TABLE CHANGE statement that renames a column,
Spark Guide | 27