org.apache.trevni.avro

Class AvroTrevniOutputFormat<T>

java.lang.Object

org.apache.hadoop.mapred.FileOutputFormat<AvroWrapper<T>,org.apache.hadoop.io.NullWritable>

org.apache.trevni.avro.AvroTrevniOutputFormat<T>

All Implemented Interfaces:

org.apache.hadoop.mapred.OutputFormat<AvroWrapper<T>,org.apache.hadoop.io.NullWritable>

public class AvroTrevniOutputFormat<T>
extends org.apache.hadoop.mapred.FileOutputFormat<AvroWrapper<T>,org.apache.hadoop.io.NullWritable>

An OutputFormat that writes Avro data to Trevni files.

Writes a directory of files per task, each comprising a single filesystem block. To reduce the number of files, increase the default filesystem block size for the job. Each task also requires enough memory to buffer a filesystem block.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapred.FileOutputFormat

org.apache.hadoop.mapred.FileOutputFormat.Counter

Field Summary

Fields

Modifier and Type	Field and Description
static String	EXT The file name extension for trevni files.
static String	META_PREFIX

Constructor Summary

Constructors

Constructor and Description
AvroTrevniOutputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapred.RecordWriter<AvroWrapper<T>,org.apache.hadoop.io.NullWritable>	getRecordWriter(org.apache.hadoop.fs.FileSystem ignore, org.apache.hadoop.mapred.JobConf job, String name, org.apache.hadoop.util.Progressable prog)
static void	setMeta(org.apache.hadoop.mapred.JobConf job, String key, String value) Add metadata to job output files.

Methods inherited from class org.apache.hadoop.mapred.FileOutputFormat

checkOutputSpecs, getCompressOutput, getOutputCompressorClass, getOutputPath, getPathForCustomFile, getTaskOutputPath, getUniqueName, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputPath, setWorkOutputPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EXT

public static final String EXT

The file name extension for trevni files.

See Also:

Constant Field Values

META_PREFIX

public static final String META_PREFIX

See Also:

Constant Field Values

Constructor Detail

AvroTrevniOutputFormat

public AvroTrevniOutputFormat()

Method Detail

setMeta

public static void setMeta(org.apache.hadoop.mapred.JobConf job,
                           String key,
                           String value)

Add metadata to job output files.

getRecordWriter

public org.apache.hadoop.mapred.RecordWriter<AvroWrapper<T>,org.apache.hadoop.io.NullWritable> getRecordWriter(org.apache.hadoop.fs.FileSystem ignore,
                                                                                                               org.apache.hadoop.mapred.JobConf job,
                                                                                                               String name,
                                                                                                               org.apache.hadoop.util.Progressable prog)
                                                                                                        throws IOException

Specified by:

getRecordWriter in interface org.apache.hadoop.mapred.OutputFormat<AvroWrapper<T>,org.apache.hadoop.io.NullWritable>

Specified by:

getRecordWriter in class org.apache.hadoop.mapred.FileOutputFormat<AvroWrapper<T>,org.apache.hadoop.io.NullWritable>

Throws:

IOException