K - The type of key. If an Avro type, it must be wrapped in an
AvroKey.V - The type of value. If an Avro type, it must be wrapped in an
AvroValue.
Writes a directory of files per task, each comprising a single filesystem block. To reduce the number of files, increase the default filesystem block size for the job. Each task also requires enough memory to buffer a filesystem block.
public class AvroTrevniKeyValueOutputFormat<K,V> extends FileOutputFormat<AvroKey<K>,AvroValue<V>>
OutputFormat that writes Avro data to
Trevni files.
This implement was modeled off
AvroKeyValueOutputFormat to allow for easy
transition
* FileOutputFormat for writing Trevni container files of key/value pairs.
Since Trevni container files can only contain records (not key/value pairs), this output format puts the key and value into an Avro generic record with two fields, named 'key' and 'value'.
The keys and values given to this output format may be Avro objects wrapped
in AvroKey or AvroValue objects. The basic Writable
types are also supported (e.g., IntWritable, Text); they will be converted to
their corresponding Avro types.
FileOutputFormat.CounterBASE_OUTPUT_NAME, COMPRESS, COMPRESS_CODEC, COMPRESS_TYPE, OUTDIR, PART| Constructor and Description |
|---|
AvroTrevniKeyValueOutputFormat() |
| Modifier and Type | Method and Description |
|---|---|
RecordWriter<AvroKey<K>,AvroValue<V>> |
getRecordWriter(TaskAttemptContext context) |
checkOutputSpecs, getCompressOutput, getDefaultWorkFile, getOutputCommitter, getOutputCompressorClass, getOutputName, getOutputPath, getPathForWorkFile, getUniqueFile, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputName, setOutputPathpublic RecordWriter<AvroKey<K>,AvroValue<V>> getRecordWriter(TaskAttemptContext context) throws IOException, InterruptedException
getRecordWriter in class FileOutputFormat<AvroKey<K>,AvroValue<V>>IOExceptionInterruptedExceptionCopyright © 2009–2020 The Apache Software Foundation. All rights reserved.