cloudera-spark

Developing Spark Applications
Writing Compressed Data Files
To set the compression type used on write, configure the spark.sql.avro.compression.codec property:
sqlContext.setConf("spark.sql.avro.compression.codec","codec")
The supported codec values are uncompressed, snappy, and deflate. Specify the level to use with deflate
compression in spark.sql.avro.deflate.level. For an example, see Figure 6: Writing Deflate Compressed
Records on page 26.
Accessing Partitioned Data Files
The spark-avro library supports writing and reading partitioned data. You pass the partition columns to the writer.
For examples, see Figure 7: Writing Partitioned Data on page 26 and Figure 8: Reading Partitioned Data on page 26.
Specifying Record Name and Namespace
Specify the record name and namespace to use when writing to disk by passing recordName and recordNamespace
as optional parameters. For an example, see Figure 9: Specifying a Record Name on page 27.
Spark SQL
You can write SQL queries to query a set of Avro files. First, create a temporary table pointing to the directory containing
the Avro files. Then query the temporary table:
sqlContext.sql("CREATE TEMPORARY TABLE table_name
USING com.databricks.spark.avro OPTIONS (path "input_dir"))
df = sqlContext.sql("SELECT * FROM table_name")
Avro to Spark SQL Conversion
The spark-avro library supports conversion for all Avro data types:
• boolean -> BooleanType
• int -> IntegerType
• long -> LongType
• float -> FloatType
• double -> DoubleType
• bytes -> BinaryType
• string -> StringType
• record -> StructType
• enum -> StringType
• array -> ArrayType
• map -> MapType
• fixed -> BinaryType
The spark-avro library supports the following union types:
• union(int, long) -> LongType
• union(float, double) -> DoubleType
• union(any, null) -> any
The library does not support complex union types.
All doc, aliases, and other fields are stripped when they are loaded into Spark.
Spark SQL to Avro Conversion
Every Spark SQL type is supported:
24 | Spark Guide