org.apache.avro.hadoop.io

Class AvroSequenceFile

java.lang.Object

org.apache.hadoop.io.SequenceFileBase

org.apache.avro.hadoop.io.AvroSequenceFile

public class AvroSequenceFile
extends SequenceFileBase

A wrapper around a Hadoop SequenceFile that also supports reading and writing Avro data.

The vanilla Hadoop SequenceFile contains a header followed by a sequence of records. A record consists of a key and a value. The key and value must either:

• implement the Writable interface, or
• be accepted by a Serialization registered with the SerializationFactory.

Since Avro data are Plain Old Java Objects (e.g., Integer for data with schema "int"), they do not implement Writable. Furthermore, a Serialization implementation cannot determine whether an object instance of type CharSequence that also implements Writable should be serialized using Avro or WritableSerialization.

The solution implemented in AvroSequenceFile is to:

• wrap Avro key data in an AvroKey object,
• wrap Avro value data in an AvroValue object,
• configure and register AvroSerialization with the SerializationFactory, which will accept only objects that are instances of either AvroKey or AvroValue, and
• store the Avro key and value schemas in the SequenceFile header.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroSequenceFile.BlockCompressWriter A Writer for Avro-enabled SequenceFiles using block-level compression.
static class	AvroSequenceFile.Reader A reader for SequenceFiles that may contain Avro data.
static class	AvroSequenceFile.RecordCompressWriter A Writer for Avro-enabled SequenceFiles using record-level compression.
static class	AvroSequenceFile.Writer A writer for an uncompressed SequenceFile that supports Avro data.

Nested classes/interfaces inherited from class org.apache.hadoop.io.SequenceFileBase

SequenceFileBase.BlockCompressWriterBase, SequenceFileBase.RecordCompressWriterBase

Field Summary

Fields

Modifier and Type	Field and Description
static Text	METADATA_FIELD_KEY_SCHEMA The SequencFile.Metadata field for the Avro key writer schema.
static Text	METADATA_FIELD_VALUE_SCHEMA The SequencFile.Metadata field for the Avro value writer schema.

Method Summary

Methods

Modifier and Type	Method and Description
static SequenceFile.Writer	createWriter(AvroSequenceFile.Writer.Options options) Creates a writer from a set of options.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

METADATA_FIELD_KEY_SCHEMA

public static final Text METADATA_FIELD_KEY_SCHEMA

The SequencFile.Metadata field for the Avro key writer schema.

METADATA_FIELD_VALUE_SCHEMA

public static final Text METADATA_FIELD_VALUE_SCHEMA

The SequencFile.Metadata field for the Avro value writer schema.

Method Detail

createWriter

public static SequenceFile.Writer createWriter(AvroSequenceFile.Writer.Options options)
                                        throws IOException

Creates a writer from a set of options.

Since there are different implementations of Writer depending on the compression type, this method constructs the appropriate subclass depending on the compression type given in the options.

Parameters:

options - The options for the writer.

Returns:

A new writer instance.

Throws:

IOException - If the writer cannot be created.