Package org.apache.trevni.avro.mapreduce

Class AvroTrevniKeyOutputFormat<T>

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<AvroKey<T>,NullWritable>

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<AvroKey<T>,NullWritable>

org.apache.trevni.avro.mapreduce.AvroTrevniKeyOutputFormat<T>

Type Parameters:

T - The (java) type of the Trevni data to write.

Writes a directory of files per task, each comprising a single filesystem block. To reduce the number of files, increase the default filesystem block size for the job. Each task also requires enough memory to buffer a filesystem block.

public class AvroTrevniKeyOutputFormat<T>
extends FileOutputFormat<AvroKey<T>,NullWritable>

An OutputFormat that writes Avro data to Trevni files. This implement was modeled off AvroKeyOutputFormat to allow for easy transition FileOutputFormat for writing Trevni container files.

Since Trevni container files only contain records (not key/value pairs), this output format ignores the value.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

FileOutputFormat.Counter

Field Summary

Fields inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

BASE_OUTPUT_NAME, COMPRESS, COMPRESS_CODEC, COMPRESS_TYPE, OUTDIR, PART

Constructor Summary

Constructors

Constructor	Description
AvroTrevniKeyOutputFormat()

Method Summary

Modifier and Type	Method	Description
RecordWriter<AvroKey<T>,NullWritable>	getRecordWriter(TaskAttemptContext context)

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

checkOutputSpecs, getCompressOutput, getDefaultWorkFile, getOutputCommitter, getOutputCompressorClass, getOutputName, getOutputPath, getPathForWorkFile, getUniqueFile, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputName, setOutputPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

AvroTrevniKeyOutputFormat

public AvroTrevniKeyOutputFormat()

Method Details

getRecordWriter

public RecordWriter<AvroKey<T>,NullWritable> getRecordWriter(TaskAttemptContext context)
                                                      throws IOException,
InterruptedException

Specified by:

getRecordWriter in class FileOutputFormat<AvroKey<T>,NullWritable>

Throws:

IOException

InterruptedException