avro Namespace Reference

A bunch of templates and specializations for encoding and decoding specific types. More...

Namespaces
namespace	concepts
	The concept classes are used to simplify NodeImpl.
Classes
struct	is_serializable
struct	is_promotable
struct	type_to_avro
class	DataFileWriterBase
	Type-independent portion of DataFileWriter. More...
class	DataFileWriter
	An Avro datafile that can store objects of type T. More...
class	DataFileReaderBase
	The type independent portion of rader. More...
class	DataFileReader
	Reads the contents of data file one after another. More...
class	Decoder
	Decoder is an interface implemented by every decoder capable of decoding Avro data. More...
class	ResolvingDecoder
	ResolvingDecoder is derived from Decoder, with an additional function to obtain the field ordering of fiedls within a record. More...
class	Encoder
	The abstract base class for all Avro encoders. More...
class	Exception
	Wrapper for std::runtime_error that provides convenience constructor for boost::format objects. More...
class	GenericReader
	A utility class to read generic datum from decoders. More...
class	GenericWriter
	A utility class to write generic datum to encoders. More...
struct	codec_traits< std::pair< ValidSchema, GenericDatum > >
	Specialization of codec_traits for Generic datum along with its schema. More...
struct	codec_traits< GenericDatum >
	Specialization of codec_traits for GenericDatum. More...
class	GenericDatum
	Generic datum which can hold any Avro type. More...
class	GenericContainer
	The base class for all generic type for containers. More...
class	GenericUnion
	Generic container for unions. More...
class	GenericRecord
	The generic container for Avro records. More...
class	GenericArray
	The generic container for Avro arrays. More...
class	GenericMap
	The generic container for Avro maps. More...
class	GenericEnum
	Generic container for Avro enum. More...
class	GenericFixed
	Generic container for Avro fixed. More...
class	Layout
class	PrimitiveLayout
class	CompoundLayout
class	Name
class	Node
	Node is the building block for parse trees. More...
class	NodeImpl
	Implementation details for Node. More...
class	NodePrimitive
class	NodeSymbolic
class	NodeRecord
class	NodeEnum
class	NodeArray
class	NodeMap
class	NodeUnion
class	NodeFixed
class	Parser
	Class that wraps a reader or ValidatingReade with an interface that uses explicit get* names instead of getValue. More...
class	ReaderImpl
	Parses from an avro encoding to the requested type. More...
class	Resolver
class	ResolverSchema
class	ResolvingReader
class	Schema
	The root Schema object is a base class. Nobody constructs this class directly. More...
class	NullSchema
class	BoolSchema
class	IntSchema
class	LongSchema
class	FloatSchema
class	DoubleSchema
class	StringSchema
class	BytesSchema
class	RecordSchema
class	EnumSchema
class	ArraySchema
class	MapSchema
class	UnionSchema
class	FixedSchema
class	SymbolicSchema
class	Serializer
	Class that wraps a Writer or ValidatingWriter with an interface that uses explicit write* names instead of writeValue. More...
struct	codec_traits
	Codec_traits tells avro how to encode and decode an object of given type. More...
struct	codec_traits< bool >
	codec_traits for Avro boolean. More...
struct	codec_traits< int32_t >
	codec_traits for Avro int. More...
struct	codec_traits< int64_t >
	codec_traits for Avro long. More...
struct	codec_traits< float >
	codec_traits for Avro float. More...
struct	codec_traits< double >
	codec_traits for Avro double. More...
struct	codec_traits< std::string >
	codec_traits for Avro string. More...
struct	codec_traits< std::vector< uint8_t > >
	codec_traits for Avro bytes. More...
struct	codec_traits< boost::array< uint8_t, N > >
	codec_traits for Avro fixed. More...
struct	codec_traits< std::vector< T > >
	codec_traits for Avro arrays. More...
struct	codec_traits< std::map< std::string, T > >
	codec_traits for Avro maps. More...
class	InputStream
	A no-copy input stream. More...
class	OutputStream
	A no-copy output stream. More...
struct	StreamReader
	A convenience class for reading from an InputStream. More...
struct	StreamWriter
	A convinience class to write data into an OutputStream. More...
struct	Null
	define a type to identify Null in template functions More...
class	NullValidator
class	Validator
	This class is used by both the ValidatingSerializer and ValidationParser objects. More...
class	ValidSchema
	A ValidSchema is basically a non-mutable Schema that has passed some minumum of sanity checks. More...
class	WriterImpl
	Class for writing avro data to a stream. More...
Typedefs
typedef boost::array< uint8_t, 16 >	DataFileSync
	The sync value.
typedef boost::shared_ptr < Decoder >	DecoderPtr
	Shared pointer to Decoder.
typedef boost::shared_ptr < ResolvingDecoder >	ResolvingDecoderPtr
	Shared pointer to ResolvingDecoder.
typedef boost::shared_ptr < Encoder >	EncoderPtr
	Shared pointer to Encoder.
typedef boost::shared_ptr< Node >	NodePtr
typedef concepts::NoAttribute < Name >	NoName
typedef concepts::SingleAttribute < Name >	HasName
typedef concepts::NoAttribute < NodePtr >	NoLeaves
typedef concepts::SingleAttribute < NodePtr >	SingleLeaf
typedef concepts::MultiAttribute < NodePtr >	MultiLeaves
typedef concepts::NoAttribute < std::string >	NoLeafNames
typedef concepts::MultiAttribute < std::string >	LeafNames
typedef concepts::NoAttribute < int >	NoSize
typedef concepts::SingleAttribute< int >	HasSize
typedef NodeImpl< NoName, NoLeaves, NoLeafNames, NoSize >	NodeImplPrimitive
typedef NodeImpl< HasName, NoLeaves, NoLeafNames, NoSize >	NodeImplSymbolic
typedef NodeImpl< HasName, MultiLeaves, LeafNames, NoSize >	NodeImplRecord
typedef NodeImpl< HasName, NoLeaves, LeafNames, NoSize >	NodeImplEnum
typedef NodeImpl< NoName, SingleLeaf, NoLeafNames, NoSize >	NodeImplArray
typedef NodeImpl< NoName, MultiLeaves, NoLeafNames, NoSize >	NodeImplMap
typedef NodeImpl< NoName, MultiLeaves, NoLeafNames, NoSize >	NodeImplUnion
typedef NodeImpl< HasName, NoLeaves, NoLeafNames, HasSize >	NodeImplFixed
typedef ReaderImpl< NullValidator >	Reader
typedef ReaderImpl< Validator >	ValidatingReader
typedef WriterImpl< NullValidator >	Writer
typedef WriterImpl< Validator >	ValidatingWriter
Enumerations
enum	Codec { NULL_CODEC, DEFLATE_CODEC }
	Specify type of compression to use when writing data files. More...
enum	SchemaResolution {   RESOLVE_NO_MATCH, RESOLVE_MATCH, RESOLVE_PROMOTABLE_TO_LONG, RESOLVE_PROMOTABLE_TO_FLOAT,   RESOLVE_PROMOTABLE_TO_DOUBLE }
enum	Type {   AVRO_STRING, AVRO_BYTES, AVRO_INT, AVRO_LONG,   AVRO_FLOAT, AVRO_DOUBLE, AVRO_BOOL, AVRO_NULL,   AVRO_RECORD, AVRO_ENUM, AVRO_ARRAY, AVRO_MAP,   AVRO_UNION, AVRO_FIXED, AVRO_NUM_TYPES, AVRO_SYMBOLIC = AVRO_NUM_TYPES,   AVRO_UNKNOWN = -1 }
	The "type" for the schema. More...
Functions
template<typename Reader , typename T >
void	parse (Reader &p, T &val)
	The main parse entry point function.
template<typename T >
void	parse (ResolvingReader &p, T &val)
template<typename Reader , typename T >
void	translatingParse (Reader &p, T &val, const boost::false_type &)
template<typename Writer , typename T >
void	serialize (Writer &s, const T &val)
	The main serializer entry point function.
template<typename Writer , typename T >
void	serialize (Writer &s, const T &val, const boost::false_type &)
	Type trait should be set to is_serializable in otherwise force the compiler to complain.
AVRO_DECL void	compileJsonSchema (std::istream &is, ValidSchema &schema)
	Given a stream comtaining a JSON schema, compiles the schema to a ValidSchema object.
AVRO_DECL bool	compileJsonSchema (std::istream &is, ValidSchema &schema, std::string &error)
	Non-throwing version of compileJsonSchema.
AVRO_DECL ValidSchema	compileJsonSchemaFromStream (InputStream &is)
AVRO_DECL ValidSchema	compileJsonSchemaFromMemory (const uint8_t *input, size_t len)
AVRO_DECL ValidSchema	compileJsonSchemaFromString (const char *input)
AVRO_DECL ValidSchema	compileJsonSchemaFromString (const std::string &input)
AVRO_DECL ValidSchema	compileJsonSchemaFromFile (const char *filename)
AVRO_DECL DecoderPtr	binaryDecoder ()
	Returns an decoder that can decode binary Avro standard.
AVRO_DECL DecoderPtr	validatingDecoder (const ValidSchema &schema, const DecoderPtr &base)
	Returns an decoder that validates sequence of calls to an underlying Decoder against the given schema.
AVRO_DECL DecoderPtr	jsonDecoder (const ValidSchema &schema)
	Returns an decoder that can decode Avro standard for JSON.
AVRO_DECL ResolvingDecoderPtr	resolvingDecoder (const ValidSchema &writer, const ValidSchema &reader, const DecoderPtr &base)
	Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema.
AVRO_DECL EncoderPtr	binaryEncoder ()
	Returns an encoder that can encode binary Avro standard.
AVRO_DECL EncoderPtr	validatingEncoder (const ValidSchema &schema, const EncoderPtr &base)
	Returns an encoder that validates sequence of calls to an underlying Encoder against the given schema.
AVRO_DECL EncoderPtr	jsonEncoder (const ValidSchema &schema)
	Returns an encoder that can encode Avro standard for JSON.
std::ostream &	operator<< (std::ostream &os, const Name &n)
NodePtr	resolveSymbol (const NodePtr &node)
Type	nextType (Parser< ValidatingReader > &p)
bool	currentRecordName (Parser< ValidatingReader > &p, std::string &name)
bool	nextFieldName (Parser< ValidatingReader > &p, std::string &name)
Resolver *	constructResolver (const ValidSchema &rwriterSchema, const ValidSchema &readerSchema, const Layout &readerLayout)
template<typename T >
void	encode (Encoder &e, const T &t)
	Generic encoder function that makes use of the codec_traits.
template<typename T >
void	decode (Decoder &d, T &t)
	Generic decoder function that makes use of the codec_traits.
AVRO_DECL std::auto_ptr < OutputStream >	memoryOutputStream (size_t chunkSize=4 *1024)
	Returns a new OutputStream, which grows in memory chunks of specified size.
AVRO_DECL std::auto_ptr < InputStream >	memoryInputStream (const uint8_t *data, size_t len)
	Returns a new InputStream, with the data from the given byte array.
AVRO_DECL std::auto_ptr < InputStream >	memoryInputStream (const OutputStream &source)
	Returns a new InputStream with the contents written into an outputstream.
AVRO_DECL boost::shared_ptr < std::vector< uint8_t > >	snapshot (const OutputStream &source)
	Returns the contents written so far into the output stream, which should be a memory output stream.
AVRO_DECL std::auto_ptr < OutputStream >	fileOutputStream (const char *filename, size_t bufferSize=8 *1024)
	Returns a new OutputStream whose contents would be stored in a file.
AVRO_DECL std::auto_ptr < InputStream >	fileInputStream (const char *filename, size_t bufferSize=8 *1024)
	Returns a new InputStream whose contents come from the given file.
AVRO_DECL std::auto_ptr < OutputStream >	ostreamOutputStream (std::ostream &os, size_t bufferSize=8 *1024)
	Returns a new OutputStream whose contents will be sent to the given std::ostream.
AVRO_DECL std::auto_ptr < InputStream >	istreamInputStream (std::istream &in, size_t bufferSize=8 *1024)
	Returns a new InputStream whose contents come from the given std::istream.
void	copy (InputStream &in, OutputStream &out)
	A convenience function to copy all the contents of an input stream into an output stream.
bool	isPrimitive (Type t)
	Returns true if and only if the given type is a primitive.
bool	isCompound (Type t)
	Returns true if and only if the given type is a non primitive valid type.
bool	isAvroType (Type t)
	Returns true if and only if the given type is a valid avro type.
bool	isAvroTypeOrPseudoType (Type t)
	Returns true if and only if the given type is within the valid range of enumeration.
AVRO_DECL const std::string &	toString (Type type)
	Converts the given type into a string.
AVRO_DECL std::ostream &	operator<< (std::ostream &os, avro::Type type)
	Writes a string form of the given type into the given ostream.
std::ostream &	operator<< (std::ostream &os, const Null &null)
	Writes schema for null null type to os.
AVRO_DECL uint64_t	encodeZigzag64 (int64_t input)
AVRO_DECL int64_t	decodeZigzag64 (uint64_t input)
AVRO_DECL uint32_t	encodeZigzag32 (int32_t input)
AVRO_DECL int32_t	decodeZigzag32 (uint32_t input)
AVRO_DECL size_t	encodeInt32 (int32_t input, boost::array< uint8_t, 5 > &output)
AVRO_DECL size_t	encodeInt64 (int64_t input, boost::array< uint8_t, 10 > &output)
template<typename Reader , typename T >
void	parse (Reader &p, T &val, const boost::false_type &)
	Type trait should be set to is_serializable in otherwise force the compiler to complain.
template<typename Reader >
void	parse (Reader &p, std::vector< uint8_t > &val, const boost::true_type &)
template<typename T >
void	translatingParse (ResolvingReader &p, T &val, const boost::true_type &)
template<typename Writer , typename T >
void	serialize (Writer &s, T val, const boost::true_type &)
	The remainder of the file includes default implementations for serializable types.
template<typename Writer >
void	serialize (Writer &s, const std::vector< uint8_t > &val, const boost::true_type &)
Variables
class AVRO_DECL	InputStream
class AVRO_DECL	ValidSchema
	This class is used to implement an avro spec parser using a flex/bison compiler.
class AVRO_DECL	Schema

---

Detailed Description

A bunch of templates and specializations for encoding and decoding specific types.

Primitive AVRO types BOOLEAN, INT, LONG, FLOAT, DOUBLE, STRING and BYTES get decoded to and encoded from C++ types bool, int32_t, int64_t, float, double, std::string and std::vector<uint8_t> respectively. In addition, std::vector<T> for aribtrary type T gets encoded as an Avro array of T. Similarly, std::map<std::string, T> for arbitrary type T gets encoded as an Avro map with value type T.

Users can have their custom types encoded/decoded by specializing avro::codec_traits class for their types.

---

Enumeration Type Documentation

enum avro::Codec

Specify type of compression to use when writing data files.

enum avro::SchemaResolution

Enumerator:

RESOLVE_NO_MATCH	The schemas definitely do not match.
RESOLVE_MATCH	The schemas match at a cursory level. For records and enums, this means the name is the same, but it does not necessarily mean that every symbol or field is an exact match.
RESOLVE_PROMOTABLE_TO_LONG	For primitives, the matching may occur if the type is promotable. This means that the writer matches reader if the writer's type is promoted the specified type.

enum avro::Type

The "type" for the schema.

Enumerator:

AVRO_STRING	String
AVRO_BYTES	Sequence of variable length bytes data
AVRO_INT	32-bit integer
AVRO_LONG	64-bit integer
AVRO_FLOAT	Floating point number
AVRO_DOUBLE	Double precision floating point number
AVRO_BOOL	Boolean value
AVRO_NULL	Null
AVRO_RECORD	Record, a sequence of fields
AVRO_ENUM	Enumeration
AVRO_ARRAY	Homogeneous array of some specific type
AVRO_MAP	Homogeneous map from string to some specific type
AVRO_UNION	Union of one or more types
AVRO_FIXED	Fixed number of bytes
AVRO_NUM_TYPES	Marker
AVRO_SYMBOLIC	User internally to avoid circular references.
AVRO_UNKNOWN	Used internally.

---

Function Documentation

AVRO_DECL void avro::compileJsonSchema	(	std::istream &	is,
		ValidSchema &	schema
	)

Given a stream comtaining a JSON schema, compiles the schema to a ValidSchema object.

Throws if the schema cannot be compiled to a valid schema

AVRO_DECL bool avro::compileJsonSchema	(	std::istream &	is,
		ValidSchema &	schema,
		std::string &	error
	)

Non-throwing version of compileJsonSchema.

Returns:

True if no error, false if error (with the error string set)

AVRO_DECL std::auto_ptr<InputStream> avro::fileInputStream	(	const char *	filename,
		size_t	bufferSize = 8 *1024
	)

Returns a new InputStream whose contents come from the given file.

Data is read in chunks of given buffer size.

AVRO_DECL std::auto_ptr<OutputStream> avro::fileOutputStream	(	const char *	filename,
		size_t	bufferSize = 8 *1024
	)

Returns a new OutputStream whose contents would be stored in a file.

Data is written in chunks of given buffer size.

If there is a file with the given name, it is truncated and overwritten. If there is no file with the given name, it is created.

bool avro::isCompound

(

Type

t

)

[inline]

Returns true if and only if the given type is a non primitive valid type.

Primitive types are: string, bytes, int, long, float, double, boolean and null

References AVRO_NUM_TYPES, and AVRO_RECORD.

bool avro::isPrimitive

(

Type

t

)

[inline]

Returns true if and only if the given type is a primitive.

Primitive types are: string, bytes, int, long, float, double, boolean and null

References AVRO_RECORD, and AVRO_STRING.

AVRO_DECL std::auto_ptr<InputStream> avro::istreamInputStream	(	std::istream &	in,
		size_t	bufferSize = 8 *1024
	)

Returns a new InputStream whose contents come from the given std::istream.

The std::istream object should outlive the returned InputStream.

AVRO_DECL std::auto_ptr<InputStream> avro::memoryInputStream	(	const uint8_t *	data,
		size_t	len
	)

Returns a new InputStream, with the data from the given byte array.

It does not copy the data, the byte array should remain valid until the InputStream is used.

AVRO_DECL std::auto_ptr<InputStream> avro::memoryInputStream

(

const OutputStream &

source

)

Returns a new InputStream with the contents written into an outputstream.

The output stream must have been returned by an earlier call to memoryOutputStream(). The contents for the new input stream are the snapshot of the outputstream. One can construct any number of memory input stream from a single memory output stream.

std::ostream& avro::operator<<	(	std::ostream &	os,
		const Null &	null
	)

Writes schema for null null type to os.

Parameters:

os	The ostream to write to.
null	The value to be written.

AVRO_DECL std::auto_ptr<OutputStream> avro::ostreamOutputStream	(	std::ostream &	os,
		size_t	bufferSize = 8 *1024
	)

Returns a new OutputStream whose contents will be sent to the given std::ostream.

The std::ostream object should outlive the returned OutputStream.

template<typename Reader , typename T >

void avro::parse	(	Reader &	p,
		T &	val
	)

The main parse entry point function.

Takes a parser (either validating or plain) and the object that should receive the parsed data.

template<typename Reader , typename T >

void avro::parse	(	Reader &	p,
		T &	val,
		const boost::false_type &
	)

Type trait should be set to is_serializable in otherwise force the compiler to complain.

The remainder of the file includes default implementations for serializable types.

AVRO_DECL ResolvingDecoderPtr avro::resolvingDecoder	(	const ValidSchema &	writer,
		const ValidSchema &	reader,
		const DecoderPtr &	base
	)

Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema.

The client uses the decoder as if the data were written using readerSchema. // FIXME: Handle out of order fields.

template<typename Writer , typename T >

void avro::serialize	(	Writer &	s,
		const T &	val
	)

The main serializer entry point function.

Takes a serializer (either validating or plain) and the object that should be serialized.

AVRO_DECL boost::shared_ptr<std::vector<uint8_t> > avro::snapshot

(

const OutputStream &

source

)

Returns the contents written so far into the output stream, which should be a memory output stream.

That is it must have been returned by a pervious call to memoryOutputStream().

AVRO_DECL const std::string& avro::toString

(

Type

type

)

Converts the given type into a string.

Useful for generating messages.

---

Variable Documentation

class AVRO_DECL avro::ValidSchema

This class is used to implement an avro spec parser using a flex/bison compiler.

In order for the lexer to be reentrant, this class provides a lexer object for each parse. The bison parser also uses this class to build up an avro parse tree as the avro spec is parsed.