org.apache.trevni.avro.mapreduce

Class AvroTrevniKeyValueOutputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<K,V>

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<AvroKey<K>,AvroValue<V>>

org.apache.trevni.avro.mapreduce.AvroTrevniKeyValueOutputFormat<K,V>

Type Parameters:

K - The type of key. If an Avro type, it must be wrapped in an AvroKey.

V - The type of value. If an Avro type, it must be wrapped in an AvroValue.

Writes a directory of files per task, each comprising a single filesystem block. To reduce the number of files, increase the default filesystem block size for the job. Each task also requires enough memory to buffer a filesystem block.

public class AvroTrevniKeyValueOutputFormat<K,V>
extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<AvroKey<K>,AvroValue<V>>

An OutputFormat that writes Avro data to Trevni files. This implement was modeled off AvroKeyValueOutputFormat to allow for easy transition * FileOutputFormat for writing Trevni container files of key/value pairs.

Since Trevni container files can only contain records (not key/value pairs), this output format puts the key and value into an Avro generic record with two fields, named 'key' and 'value'.

The keys and values given to this output format may be Avro objects wrapped in AvroKey or AvroValue objects. The basic Writable types are also supported (e.g., IntWritable, Text); they will be converted to their corresponding Avro types.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.Counter

Constructor Summary

Constructors

Constructor and Description
AvroTrevniKeyValueOutputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordWriter<AvroKey<K>,AvroValue<V>>	getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

checkOutputSpecs, getCompressOutput, getDefaultWorkFile, getOutputCommitter, getOutputCompressorClass, getOutputPath, getPathForWorkFile, getUniqueFile, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AvroTrevniKeyValueOutputFormat

public AvroTrevniKeyValueOutputFormat()

Method Detail

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter<AvroKey<K>,AvroValue<V>> getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                  throws IOException,
                                                                                         InterruptedException

Specified by:

getRecordWriter in class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<AvroKey<K>,AvroValue<V>>

Throws:

IOException

InterruptedException