org.apache.trevni.avro.mapreduce

Class AvroTrevniRecordWriterBase<K,V,T>

java.lang.Object

org.apache.hadoop.mapreduce.RecordWriter<K,V>

org.apache.trevni.avro.mapreduce.AvroTrevniRecordWriterBase<K,V,T>

Type Parameters:

K - The type of key the record writer should generate.

V - The type of value the record wrtier should generate.

T - The type of the entries within the Trevni container file being writen.

Direct Known Subclasses:

AvroTrevniKeyRecordWriter, AvroTrevniKeyValueRecordWriter

public abstract class AvroTrevniRecordWriterBase<K,V,T>
extends org.apache.hadoop.mapreduce.RecordWriter<K,V>

Abstract base class for RecordWriters that writes Trevni container files.

Field Summary

Fields

Modifier and Type	Field and Description
static String	EXT trevni file extension
protected ColumnFileMetaData	meta meta data to be stored in the output file.
static String	META_PREFIX prefix of job configs that we care about
protected int	part Counter that increments as new trevni files are create because the current file has exceeded the block size
protected Schema	schema Provided avro schema from the context
protected AvroColumnWriter<T>	writer Trevni file writer

Constructor Summary

Constructors

Constructor and Description
AvroTrevniRecordWriterBase(org.apache.hadoop.mapreduce.TaskAttemptContext context) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
void	close(org.apache.hadoop.mapreduce.TaskAttemptContext arg0)
void	flush() A Trevni flush will close the current file and prep a new writer
protected abstract Schema	initSchema(org.apache.hadoop.mapreduce.TaskAttemptContext context) Use the task context to construct a schema for writing

Methods inherited from class org.apache.hadoop.mapreduce.RecordWriter

write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EXT

public static final String EXT

trevni file extension

See Also:

Constant Field Values

META_PREFIX

public static final String META_PREFIX

prefix of job configs that we care about

See Also:

Constant Field Values

part

protected int part

Counter that increments as new trevni files are create because the current file has exceeded the block size

writer

protected AvroColumnWriter<T> writer

Trevni file writer

schema

protected Schema schema

Provided avro schema from the context

meta

protected ColumnFileMetaData meta

meta data to be stored in the output file.

Constructor Detail

AvroTrevniRecordWriterBase

public AvroTrevniRecordWriterBase(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                           throws IOException

Constructor.

Parameters:

context - The TaskAttempContext to supply the writer with information form the job configuration

Throws:

IOException

Method Detail

initSchema

protected abstract Schema initSchema(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Use the task context to construct a schema for writing

Throws:

IOException

flush

public void flush()
           throws IOException

A Trevni flush will close the current file and prep a new writer

Throws:

IOException

close

public void close(org.apache.hadoop.mapreduce.TaskAttemptContext arg0)
           throws IOException,
                  InterruptedException

Specified by:

close in class org.apache.hadoop.mapreduce.RecordWriter<K,V>

Throws:

IOException

InterruptedException