org.apache.avro.mapred

Class AvroOutputFormat<T>

java.lang.Object

org.apache.hadoop.mapred.FileOutputFormat<AvroWrapper<T>,NullWritable>

org.apache.avro.mapred.AvroOutputFormat<T>

All Implemented Interfaces:

OutputFormat<AvroWrapper<T>,NullWritable>

public class AvroOutputFormat<T>
extends FileOutputFormat<AvroWrapper<T>,NullWritable>

An OutputFormat for Avro data files.

You can specify various options using Job Configuration properties. Look at the fields in AvroJob as well as this class to get an overview of the supported options.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapred.FileOutputFormat

FileOutputFormat.Counter

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFLATE_LEVEL_KEY The configuration key for Avro deflate level.
static String	EXT The file name extension for avro data files.
static String	SYNC_INTERVAL_KEY The configuration key for Avro sync interval.
static String	XZ_LEVEL_KEY The configuration key for Avro XZ level.
static String	ZSTD_BUFFERPOOL_KEY The configuration key for Avro ZSTD buffer pool.
static String	ZSTD_LEVEL_KEY The configuration key for Avro ZSTD level.

Constructor Summary

Constructors

Constructor and Description
AvroOutputFormat()

Method Summary

Modifier and Type	Method and Description
RecordWriter<AvroWrapper<T>,NullWritable>	getRecordWriter(FileSystem ignore, JobConf job, String name, Progressable prog)
static void	setDeflateLevel(JobConf job, int level) Enable output compression using the deflate codec and specify its level.
static void	setSyncInterval(JobConf job, int syncIntervalInBytes) Set the sync interval to be used by the underlying DataFileWriter.

Methods inherited from class org.apache.hadoop.mapred.FileOutputFormat

checkOutputSpecs, getCompressOutput, getOutputCompressorClass, getOutputPath, getPathForCustomFile, getTaskOutputPath, getUniqueName, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputPath, setWorkOutputPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EXT

public static final String EXT

The file name extension for avro data files.

See Also:

Constant Field Values

DEFLATE_LEVEL_KEY

public static final String DEFLATE_LEVEL_KEY

The configuration key for Avro deflate level.

See Also:

Constant Field Values

XZ_LEVEL_KEY

public static final String XZ_LEVEL_KEY

The configuration key for Avro XZ level.

See Also:

Constant Field Values

ZSTD_LEVEL_KEY

public static final String ZSTD_LEVEL_KEY

The configuration key for Avro ZSTD level.

See Also:

Constant Field Values

ZSTD_BUFFERPOOL_KEY

public static final String ZSTD_BUFFERPOOL_KEY

The configuration key for Avro ZSTD buffer pool.

See Also:

Constant Field Values

SYNC_INTERVAL_KEY

public static final String SYNC_INTERVAL_KEY

The configuration key for Avro sync interval.

See Also:

Constant Field Values

Constructor Detail

AvroOutputFormat

public AvroOutputFormat()

Method Detail

setDeflateLevel

public static void setDeflateLevel(JobConf job,
                                   int level)

Enable output compression using the deflate codec and specify its level.

setSyncInterval

public static void setSyncInterval(JobConf job,
                                   int syncIntervalInBytes)

Set the sync interval to be used by the underlying DataFileWriter.

getRecordWriter

public RecordWriter<AvroWrapper<T>,NullWritable> getRecordWriter(FileSystem ignore,
                                                                 JobConf job,
                                                                 String name,
                                                                 Progressable prog)
                                                          throws IOException

Specified by:

getRecordWriter in interface OutputFormat<AvroWrapper<T>,NullWritable>

Specified by:

getRecordWriter in class FileOutputFormat<AvroWrapper<T>,NullWritable>

Throws:

IOException