Package org.apache.trevni.avro.mapreduce

Class AvroTrevniRecordWriterBase<K,V,T>

java.lang.Object

org.apache.hadoop.mapreduce.RecordWriter<K,V>

org.apache.trevni.avro.mapreduce.AvroTrevniRecordWriterBase<K,V,T>

Type Parameters:

K - The type of key the record writer should generate.

V - The type of value the record wrtier should generate.

T - The type of the entries within the Trevni container file being written.

Direct Known Subclasses:

AvroTrevniKeyRecordWriter, AvroTrevniKeyValueRecordWriter

public abstract class AvroTrevniRecordWriterBase<K,V,T>
extends RecordWriter<K,V>

Abstract base class for RecordWriters that writes Trevni container files.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	EXT	trevni file extension
protected ColumnFileMetaData	meta	meta data to be stored in the output file.
static final String	META_PREFIX	prefix of job configs that we care about
protected int	part	Counter that increments as new trevni files are create because the current file has exceeded the block size
protected Schema	schema	Provided avro schema from the context
protected AvroColumnWriter<T>	writer	Trevni file writer

Constructor Summary

Constructors

Constructor	Description
AvroTrevniRecordWriterBase(TaskAttemptContext context)	Constructor.

Method Summary

Modifier and Type	Method	Description
void	close(TaskAttemptContext arg0)
void	flush()	A Trevni flush will close the current file and prep a new writer
protected abstract Schema	initSchema(TaskAttemptContext context)	Use the task context to construct a schema for writing

Methods inherited from class org.apache.hadoop.mapreduce.RecordWriter

write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

EXT

public static final String EXT

trevni file extension

See Also:

• Constant Field Values

META_PREFIX

public static final String META_PREFIX

prefix of job configs that we care about

See Also:

• Constant Field Values

part

protected int part

Counter that increments as new trevni files are create because the current file has exceeded the block size

writer

protected AvroColumnWriter<T> writer

Trevni file writer

schema

protected Schema schema

Provided avro schema from the context

meta

protected ColumnFileMetaData meta

meta data to be stored in the output file.

Constructor Details

AvroTrevniRecordWriterBase

public AvroTrevniRecordWriterBase(TaskAttemptContext context)
                           throws IOException

Constructor.

Parameters:

context - The TaskAttempContext to supply the writer with information form the job configuration

Throws:

IOException

Method Details

initSchema

protected abstract Schema initSchema(TaskAttemptContext context)

Use the task context to construct a schema for writing

Throws:

IOException

flush

public void flush()
           throws IOException

A Trevni flush will close the current file and prep a new writer

Throws:

IOException

close

public void close(TaskAttemptContext arg0)
           throws IOException,
InterruptedException

Specified by:

close in class RecordWriter<K,V>

Throws:

IOException

InterruptedException