Package org.apache.avro.mapred

Class AvroOutputFormat<T>

java.lang.Object

org.apache.hadoop.mapred.FileOutputFormat<AvroWrapper<T>,NullWritable>

org.apache.avro.mapred.AvroOutputFormat<T>

All Implemented Interfaces:

OutputFormat<AvroWrapper<T>,NullWritable>

public class AvroOutputFormat<T>
extends FileOutputFormat<AvroWrapper<T>,NullWritable>

An OutputFormat for Avro data files.

You can specify various options using Job Configuration properties. Look at the fields in AvroJob as well as this class to get an overview of the supported options.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapred.FileOutputFormat

FileOutputFormat.Counter

Field Summary

Fields

Modifier and Type	Field	Description
static final String	DEFLATE_LEVEL_KEY	The configuration key for Avro deflate level.
static final String	EXT	The file name extension for avro data files.
static final String	SYNC_INTERVAL_KEY	The configuration key for Avro sync interval.
static final String	XZ_LEVEL_KEY	The configuration key for Avro XZ level.
static final String	ZSTD_BUFFERPOOL_KEY	The configuration key for Avro ZSTD buffer pool.
static final String	ZSTD_LEVEL_KEY	The configuration key for Avro ZSTD level.

Constructor Summary

Constructors

Constructor	Description
AvroOutputFormat()

Method Summary

Modifier and Type	Method	Description
RecordWriter<AvroWrapper<T>,NullWritable>	getRecordWriter(FileSystem ignore, JobConf job, String name, Progressable prog)
static void	setDeflateLevel(JobConf job, int level)	Enable output compression using the deflate codec and specify its level.
static void	setSyncInterval(JobConf job, int syncIntervalInBytes)	Set the sync interval to be used by the underlying DataFileWriter.

Methods inherited from class org.apache.hadoop.mapred.FileOutputFormat

checkOutputSpecs, getCompressOutput, getOutputCompressorClass, getOutputPath, getPathForCustomFile, getTaskOutputPath, getUniqueName, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputPath, setWorkOutputPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

EXT

public static final String EXT

The file name extension for avro data files.

See Also:

• Constant Field Values

DEFLATE_LEVEL_KEY

public static final String DEFLATE_LEVEL_KEY

The configuration key for Avro deflate level.

See Also:

• Constant Field Values

XZ_LEVEL_KEY

public static final String XZ_LEVEL_KEY

The configuration key for Avro XZ level.

See Also:

• Constant Field Values

ZSTD_LEVEL_KEY

public static final String ZSTD_LEVEL_KEY

The configuration key for Avro ZSTD level.

See Also:

• Constant Field Values

ZSTD_BUFFERPOOL_KEY

public static final String ZSTD_BUFFERPOOL_KEY

The configuration key for Avro ZSTD buffer pool.

See Also:

• Constant Field Values

SYNC_INTERVAL_KEY

public static final String SYNC_INTERVAL_KEY

The configuration key for Avro sync interval.

See Also:

• Constant Field Values

Constructor Details

AvroOutputFormat

public AvroOutputFormat()

Method Details

setDeflateLevel

public static void setDeflateLevel(JobConf job,
 int level)

Enable output compression using the deflate codec and specify its level.

setSyncInterval

public static void setSyncInterval(JobConf job,
 int syncIntervalInBytes)

Set the sync interval to be used by the underlying DataFileWriter.

getRecordWriter

public RecordWriter<AvroWrapper<T>,NullWritable> getRecordWriter(FileSystem ignore,
 JobConf job,
 String name,
 Progressable prog)
                                                          throws IOException

Specified by:

getRecordWriter in interface OutputFormat<AvroWrapper<T>,NullWritable>

Specified by:

getRecordWriter in class FileOutputFormat<AvroWrapper<T>,NullWritable>

Throws:

IOException