Package org.apache.trevni.avro

Class AvroTrevniOutputFormat<T>

java.lang.Object

org.apache.hadoop.mapred.FileOutputFormat<AvroWrapper<T>,NullWritable>

org.apache.trevni.avro.AvroTrevniOutputFormat<T>

All Implemented Interfaces:

OutputFormat<AvroWrapper<T>,NullWritable>

public class AvroTrevniOutputFormat<T>
extends FileOutputFormat<AvroWrapper<T>,NullWritable>

An OutputFormat that writes Avro data to Trevni files.

Writes a directory of files per task, each comprising a single filesystem block. To reduce the number of files, increase the default filesystem block size for the job. Each task also requires enough memory to buffer a filesystem block.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapred.FileOutputFormat

FileOutputFormat.Counter

Field Summary

Fields

Modifier and Type	Field	Description
static final String	EXT	The file name extension for trevni files.
static final String	META_PREFIX

Constructor Summary

Constructors

Constructor	Description
AvroTrevniOutputFormat()

Method Summary

Modifier and Type	Method	Description
RecordWriter<AvroWrapper<T>,NullWritable>	getRecordWriter(FileSystem ignore, JobConf job, String name, Progressable prog)
static void	setMeta(JobConf job, String key, String value)	Add metadata to job output files.

Methods inherited from class org.apache.hadoop.mapred.FileOutputFormat

checkOutputSpecs, getCompressOutput, getOutputCompressorClass, getOutputPath, getPathForCustomFile, getTaskOutputPath, getUniqueName, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputPath, setWorkOutputPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

EXT

public static final String EXT

The file name extension for trevni files.

See Also:

• Constant Field Values

META_PREFIX

public static final String META_PREFIX

See Also:

• Constant Field Values

Constructor Details

AvroTrevniOutputFormat

public AvroTrevniOutputFormat()

Method Details

setMeta

public static void setMeta(JobConf job,
 String key,
 String value)

Add metadata to job output files.

getRecordWriter

public RecordWriter<AvroWrapper<T>,NullWritable> getRecordWriter(FileSystem ignore,
 JobConf job,
 String name,
 Progressable prog)
                                                          throws IOException

Specified by:

getRecordWriter in interface OutputFormat<AvroWrapper<T>,NullWritable>

Specified by:

getRecordWriter in class FileOutputFormat<AvroWrapper<T>,NullWritable>

Throws:

IOException