Avro C++
Namespaces | Classes | Typedefs | Enumerations | Functions | Variables
avro Namespace Reference

A bunch of templates and specializations for encoding and decoding specific types. More...

Namespaces

namespace  concepts
 

The concept classes are used to simplify NodeImpl.


Classes

struct  is_serializable
struct  is_promotable
struct  type_to_avro
class  DataFileWriterBase
 Type-independent portion of DataFileWriter. More...
class  DataFileWriter
 An Avro datafile that can store objects of type T. More...
class  DataFileReaderBase
 The type independent portion of rader. More...
class  DataFileReader
 Reads the contents of data file one after another. More...
class  Decoder
 Decoder is an interface implemented by every decoder capable of decoding Avro data. More...
class  ResolvingDecoder
 ResolvingDecoder is derived from Decoder, with an additional function to obtain the field ordering of fiedls within a record. More...
class  Encoder
 The abstract base class for all Avro encoders. More...
class  Exception
 Wrapper for std::runtime_error that provides convenience constructor for boost::format objects. More...
class  GenericReader
 A utility class to read generic datum from decoders. More...
class  GenericWriter
 A utility class to write generic datum to encoders. More...
struct  codec_traits< std::pair< ValidSchema, GenericDatum > >
 Specialization of codec_traits for Generic datum along with its schema. More...
struct  codec_traits< GenericDatum >
 Specialization of codec_traits for GenericDatum. More...
class  GenericDatum
 Generic datum which can hold any Avro type. More...
class  GenericContainer
 The base class for all generic type for containers. More...
class  GenericUnion
 Generic container for unions. More...
class  GenericRecord
 The generic container for Avro records. More...
class  GenericArray
 The generic container for Avro arrays. More...
class  GenericMap
 The generic container for Avro maps. More...
class  GenericEnum
 Generic container for Avro enum. More...
class  GenericFixed
 Generic container for Avro fixed. More...
class  Layout
class  PrimitiveLayout
class  CompoundLayout
class  Name
class  Node
 Node is the building block for parse trees. More...
class  NodeImpl
 Implementation details for Node. More...
class  NodePrimitive
class  NodeSymbolic
class  NodeRecord
class  NodeEnum
class  NodeArray
class  NodeMap
class  NodeUnion
class  NodeFixed
class  Parser
 Class that wraps a reader or ValidatingReade with an interface that uses explicit get* names instead of getValue. More...
class  ReaderImpl
 Parses from an avro encoding to the requested type. More...
class  Resolver
class  ResolverSchema
class  ResolvingReader
class  Schema
 The root Schema object is a base class. Nobody constructs this class directly. More...
class  NullSchema
class  BoolSchema
class  IntSchema
class  LongSchema
class  FloatSchema
class  DoubleSchema
class  StringSchema
class  BytesSchema
class  RecordSchema
class  EnumSchema
class  ArraySchema
class  MapSchema
class  UnionSchema
class  FixedSchema
class  SymbolicSchema
class  Serializer
 Class that wraps a Writer or ValidatingWriter with an interface that uses explicit write* names instead of writeValue. More...
struct  codec_traits
 Codec_traits tells avro how to encode and decode an object of given type. More...
struct  codec_traits< bool >
 codec_traits for Avro boolean. More...
struct  codec_traits< int32_t >
 codec_traits for Avro int. More...
struct  codec_traits< int64_t >
 codec_traits for Avro long. More...
struct  codec_traits< float >
 codec_traits for Avro float. More...
struct  codec_traits< double >
 codec_traits for Avro double. More...
struct  codec_traits< std::string >
 codec_traits for Avro string. More...
struct  codec_traits< std::vector< uint8_t > >
 codec_traits for Avro bytes. More...
struct  codec_traits< boost::array< uint8_t, N > >
 codec_traits for Avro fixed. More...
struct  codec_traits< std::vector< T > >
 codec_traits for Avro arrays. More...
struct  codec_traits< std::map< std::string, T > >
 codec_traits for Avro maps. More...
class  InputStream
 A no-copy input stream. More...
class  OutputStream
 A no-copy output stream. More...
struct  StreamReader
 A convenience class for reading from an InputStream. More...
struct  StreamWriter
 A convinience class to write data into an OutputStream. More...
struct  Null
 define a type to identify Null in template functions More...
class  NullValidator
class  Validator
 This class is used by both the ValidatingSerializer and ValidationParser objects. More...
class  ValidSchema
 A ValidSchema is basically a non-mutable Schema that has passed some minumum of sanity checks. More...
class  WriterImpl
 Class for writing avro data to a stream. More...

Typedefs

typedef boost::array< uint8_t, 16 > DataFileSync
 The sync value.
typedef boost::shared_ptr
< Decoder
DecoderPtr
 Shared pointer to Decoder.
typedef boost::shared_ptr
< ResolvingDecoder
ResolvingDecoderPtr
 Shared pointer to ResolvingDecoder.
typedef boost::shared_ptr
< Encoder
EncoderPtr
 Shared pointer to Encoder.
typedef boost::shared_ptr< NodeNodePtr
typedef concepts::NoAttribute
< Name
NoName
typedef
concepts::SingleAttribute
< Name
HasName
typedef concepts::NoAttribute
< NodePtr > 
NoLeaves
typedef
concepts::SingleAttribute
< NodePtr > 
SingleLeaf
typedef
concepts::MultiAttribute
< NodePtr > 
MultiLeaves
typedef concepts::NoAttribute
< std::string > 
NoLeafNames
typedef
concepts::MultiAttribute
< std::string > 
LeafNames
typedef concepts::NoAttribute
< int > 
NoSize
typedef
concepts::SingleAttribute< int > 
HasSize
typedef NodeImpl< NoName,
NoLeaves, NoLeafNames, NoSize
NodeImplPrimitive
typedef NodeImpl< HasName,
NoLeaves, NoLeafNames, NoSize
NodeImplSymbolic
typedef NodeImpl< HasName,
MultiLeaves, LeafNames, NoSize
NodeImplRecord
typedef NodeImpl< HasName,
NoLeaves, LeafNames, NoSize
NodeImplEnum
typedef NodeImpl< NoName,
SingleLeaf, NoLeafNames,
NoSize
NodeImplArray
typedef NodeImpl< NoName,
MultiLeaves, NoLeafNames,
NoSize
NodeImplMap
typedef NodeImpl< NoName,
MultiLeaves, NoLeafNames,
NoSize
NodeImplUnion
typedef NodeImpl< HasName,
NoLeaves, NoLeafNames, HasSize
NodeImplFixed
typedef ReaderImpl< NullValidatorReader
typedef ReaderImpl< ValidatorValidatingReader
typedef WriterImpl< NullValidatorWriter
typedef WriterImpl< ValidatorValidatingWriter

Enumerations

enum  Codec { NULL_CODEC, DEFLATE_CODEC }
 Specify type of compression to use when writing data files. More...
enum  SchemaResolution {
  RESOLVE_NO_MATCH, RESOLVE_MATCH, RESOLVE_PROMOTABLE_TO_LONG, RESOLVE_PROMOTABLE_TO_FLOAT,
  RESOLVE_PROMOTABLE_TO_DOUBLE
}
enum  Type {
  AVRO_STRING, AVRO_BYTES, AVRO_INT, AVRO_LONG,
  AVRO_FLOAT, AVRO_DOUBLE, AVRO_BOOL, AVRO_NULL,
  AVRO_RECORD, AVRO_ENUM, AVRO_ARRAY, AVRO_MAP,
  AVRO_UNION, AVRO_FIXED, AVRO_NUM_TYPES, AVRO_SYMBOLIC = AVRO_NUM_TYPES,
  AVRO_UNKNOWN = -1
}
 The "type" for the schema. More...

Functions

template<typename Reader , typename T >
void parse (Reader &p, T &val)
 The main parse entry point function.
template<typename T >
void parse (ResolvingReader &p, T &val)
template<typename Reader , typename T >
void translatingParse (Reader &p, T &val, const boost::false_type &)
template<typename Writer , typename T >
void serialize (Writer &s, const T &val)
 The main serializer entry point function.
template<typename Writer , typename T >
void serialize (Writer &s, const T &val, const boost::false_type &)
 Type trait should be set to is_serializable in otherwise force the compiler to complain.
AVRO_DECL void compileJsonSchema (std::istream &is, ValidSchema &schema)
 Given a stream comtaining a JSON schema, compiles the schema to a ValidSchema object.
AVRO_DECL bool compileJsonSchema (std::istream &is, ValidSchema &schema, std::string &error)
 Non-throwing version of compileJsonSchema.
AVRO_DECL ValidSchema compileJsonSchemaFromStream (InputStream &is)
AVRO_DECL ValidSchema compileJsonSchemaFromMemory (const uint8_t *input, size_t len)
AVRO_DECL ValidSchema compileJsonSchemaFromString (const char *input)
AVRO_DECL ValidSchema compileJsonSchemaFromString (const std::string &input)
AVRO_DECL ValidSchema compileJsonSchemaFromFile (const char *filename)
AVRO_DECL DecoderPtr binaryDecoder ()
 Returns an decoder that can decode binary Avro standard.
AVRO_DECL DecoderPtr validatingDecoder (const ValidSchema &schema, const DecoderPtr &base)
 Returns an decoder that validates sequence of calls to an underlying Decoder against the given schema.
AVRO_DECL DecoderPtr jsonDecoder (const ValidSchema &schema)
 Returns an decoder that can decode Avro standard for JSON.
AVRO_DECL ResolvingDecoderPtr resolvingDecoder (const ValidSchema &writer, const ValidSchema &reader, const DecoderPtr &base)
 Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema.
AVRO_DECL EncoderPtr binaryEncoder ()
 Returns an encoder that can encode binary Avro standard.
AVRO_DECL EncoderPtr validatingEncoder (const ValidSchema &schema, const EncoderPtr &base)
 Returns an encoder that validates sequence of calls to an underlying Encoder against the given schema.
AVRO_DECL EncoderPtr jsonEncoder (const ValidSchema &schema)
 Returns an encoder that can encode Avro standard for JSON.
std::ostream & operator<< (std::ostream &os, const Name &n)
NodePtr resolveSymbol (const NodePtr &node)
Type nextType (Parser< ValidatingReader > &p)
bool currentRecordName (Parser< ValidatingReader > &p, std::string &name)
bool nextFieldName (Parser< ValidatingReader > &p, std::string &name)
ResolverconstructResolver (const ValidSchema &rwriterSchema, const ValidSchema &readerSchema, const Layout &readerLayout)
template<typename T >
void encode (Encoder &e, const T &t)
 Generic encoder function that makes use of the codec_traits.
template<typename T >
void decode (Decoder &d, T &t)
 Generic decoder function that makes use of the codec_traits.
AVRO_DECL std::auto_ptr
< OutputStream
memoryOutputStream (size_t chunkSize=4 *1024)
 Returns a new OutputStream, which grows in memory chunks of specified size.
AVRO_DECL std::auto_ptr
< InputStream
memoryInputStream (const uint8_t *data, size_t len)
 Returns a new InputStream, with the data from the given byte array.
AVRO_DECL std::auto_ptr
< InputStream
memoryInputStream (const OutputStream &source)
 Returns a new InputStream with the contents written into an outputstream.
AVRO_DECL boost::shared_ptr
< std::vector< uint8_t > > 
snapshot (const OutputStream &source)
 Returns the contents written so far into the output stream, which should be a memory output stream.
AVRO_DECL std::auto_ptr
< OutputStream
fileOutputStream (const char *filename, size_t bufferSize=8 *1024)
 Returns a new OutputStream whose contents would be stored in a file.
AVRO_DECL std::auto_ptr
< InputStream
fileInputStream (const char *filename, size_t bufferSize=8 *1024)
 Returns a new InputStream whose contents come from the given file.
AVRO_DECL std::auto_ptr
< OutputStream
ostreamOutputStream (std::ostream &os, size_t bufferSize=8 *1024)
 Returns a new OutputStream whose contents will be sent to the given std::ostream.
AVRO_DECL std::auto_ptr
< InputStream
istreamInputStream (std::istream &in, size_t bufferSize=8 *1024)
 Returns a new InputStream whose contents come from the given std::istream.
void copy (InputStream &in, OutputStream &out)
 A convenience function to copy all the contents of an input stream into an output stream.
bool isPrimitive (Type t)
 Returns true if and only if the given type is a primitive.
bool isCompound (Type t)
 Returns true if and only if the given type is a non primitive valid type.
bool isAvroType (Type t)
 Returns true if and only if the given type is a valid avro type.
bool isAvroTypeOrPseudoType (Type t)
 Returns true if and only if the given type is within the valid range of enumeration.
AVRO_DECL const std::string & toString (Type type)
 Converts the given type into a string.
AVRO_DECL std::ostream & operator<< (std::ostream &os, avro::Type type)
 Writes a string form of the given type into the given ostream.
std::ostream & operator<< (std::ostream &os, const Null &null)
 Writes schema for null null type to os.
AVRO_DECL uint64_t encodeZigzag64 (int64_t input)
AVRO_DECL int64_t decodeZigzag64 (uint64_t input)
AVRO_DECL uint32_t encodeZigzag32 (int32_t input)
AVRO_DECL int32_t decodeZigzag32 (uint32_t input)
AVRO_DECL size_t encodeInt32 (int32_t input, boost::array< uint8_t, 5 > &output)
AVRO_DECL size_t encodeInt64 (int64_t input, boost::array< uint8_t, 10 > &output)
template<typename Reader , typename T >
void parse (Reader &p, T &val, const boost::false_type &)
 Type trait should be set to is_serializable in otherwise force the compiler to complain.
template<typename Reader >
void parse (Reader &p, std::vector< uint8_t > &val, const boost::true_type &)
template<typename T >
void translatingParse (ResolvingReader &p, T &val, const boost::true_type &)
template<typename Writer , typename T >
void serialize (Writer &s, T val, const boost::true_type &)
 The remainder of the file includes default implementations for serializable types.
template<typename Writer >
void serialize (Writer &s, const std::vector< uint8_t > &val, const boost::true_type &)

Variables

class AVRO_DECL InputStream
class AVRO_DECL ValidSchema
 This class is used to implement an avro spec parser using a flex/bison compiler.
class AVRO_DECL Schema

Detailed Description

A bunch of templates and specializations for encoding and decoding specific types.

Primitive AVRO types BOOLEAN, INT, LONG, FLOAT, DOUBLE, STRING and BYTES get decoded to and encoded from C++ types bool, int32_t, int64_t, float, double, std::string and std::vector<uint8_t> respectively. In addition, std::vector<T> for aribtrary type T gets encoded as an Avro array of T. Similarly, std::map<std::string, T> for arbitrary type T gets encoded as an Avro map with value type T.

Users can have their custom types encoded/decoded by specializing avro::codec_traits class for their types.


Enumeration Type Documentation

Specify type of compression to use when writing data files.

Enumerator:
RESOLVE_NO_MATCH 

The schemas definitely do not match.

RESOLVE_MATCH 

The schemas match at a cursory level.

For records and enums, this means the name is the same, but it does not necessarily mean that every symbol or field is an exact match.

RESOLVE_PROMOTABLE_TO_LONG 

For primitives, the matching may occur if the type is promotable.

This means that the writer matches reader if the writer's type is promoted the specified type.

enum avro::Type

The "type" for the schema.

Enumerator:
AVRO_STRING 

String

AVRO_BYTES 

Sequence of variable length bytes data

AVRO_INT 

32-bit integer

AVRO_LONG 

64-bit integer

AVRO_FLOAT 

Floating point number

AVRO_DOUBLE 

Double precision floating point number

AVRO_BOOL 

Boolean value

AVRO_NULL 

Null

AVRO_RECORD 

Record, a sequence of fields

AVRO_ENUM 

Enumeration

AVRO_ARRAY 

Homogeneous array of some specific type

AVRO_MAP 

Homogeneous map from string to some specific type

AVRO_UNION 

Union of one or more types

AVRO_FIXED 

Fixed number of bytes

AVRO_NUM_TYPES 

Marker

AVRO_SYMBOLIC 

User internally to avoid circular references.

AVRO_UNKNOWN 

Used internally.


Function Documentation

AVRO_DECL void avro::compileJsonSchema ( std::istream &  is,
ValidSchema &  schema 
)

Given a stream comtaining a JSON schema, compiles the schema to a ValidSchema object.

Throws if the schema cannot be compiled to a valid schema

AVRO_DECL bool avro::compileJsonSchema ( std::istream &  is,
ValidSchema &  schema,
std::string &  error 
)

Non-throwing version of compileJsonSchema.

Returns:
True if no error, false if error (with the error string set)
AVRO_DECL std::auto_ptr<InputStream> avro::fileInputStream ( const char *  filename,
size_t  bufferSize = 8 *1024 
)

Returns a new InputStream whose contents come from the given file.

Data is read in chunks of given buffer size.

AVRO_DECL std::auto_ptr<OutputStream> avro::fileOutputStream ( const char *  filename,
size_t  bufferSize = 8 *1024 
)

Returns a new OutputStream whose contents would be stored in a file.

Data is written in chunks of given buffer size.

If there is a file with the given name, it is truncated and overwritten. If there is no file with the given name, it is created.

bool avro::isCompound ( Type  t) [inline]

Returns true if and only if the given type is a non primitive valid type.

Primitive types are: string, bytes, int, long, float, double, boolean and null

References AVRO_NUM_TYPES, and AVRO_RECORD.

bool avro::isPrimitive ( Type  t) [inline]

Returns true if and only if the given type is a primitive.

Primitive types are: string, bytes, int, long, float, double, boolean and null

References AVRO_RECORD, and AVRO_STRING.

AVRO_DECL std::auto_ptr<InputStream> avro::istreamInputStream ( std::istream &  in,
size_t  bufferSize = 8 *1024 
)

Returns a new InputStream whose contents come from the given std::istream.

The std::istream object should outlive the returned InputStream.

AVRO_DECL std::auto_ptr<InputStream> avro::memoryInputStream ( const uint8_t *  data,
size_t  len 
)

Returns a new InputStream, with the data from the given byte array.

It does not copy the data, the byte array should remain valid until the InputStream is used.

AVRO_DECL std::auto_ptr<InputStream> avro::memoryInputStream ( const OutputStream &  source)

Returns a new InputStream with the contents written into an outputstream.

The output stream must have been returned by an earlier call to memoryOutputStream(). The contents for the new input stream are the snapshot of the outputstream. One can construct any number of memory input stream from a single memory output stream.

std::ostream& avro::operator<< ( std::ostream &  os,
const Null &  null 
)

Writes schema for null null type to os.

Parameters:
osThe ostream to write to.
nullThe value to be written.
AVRO_DECL std::auto_ptr<OutputStream> avro::ostreamOutputStream ( std::ostream &  os,
size_t  bufferSize = 8 *1024 
)

Returns a new OutputStream whose contents will be sent to the given std::ostream.

The std::ostream object should outlive the returned OutputStream.

template<typename Reader , typename T >
void avro::parse ( Reader &  p,
T &  val 
)

The main parse entry point function.

Takes a parser (either validating or plain) and the object that should receive the parsed data.

template<typename Reader , typename T >
void avro::parse ( Reader &  p,
T &  val,
const boost::false_type  
)

Type trait should be set to is_serializable in otherwise force the compiler to complain.

The remainder of the file includes default implementations for serializable types.

AVRO_DECL ResolvingDecoderPtr avro::resolvingDecoder ( const ValidSchema &  writer,
const ValidSchema &  reader,
const DecoderPtr &  base 
)

Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema.

The client uses the decoder as if the data were written using readerSchema. // FIXME: Handle out of order fields.

template<typename Writer , typename T >
void avro::serialize ( Writer &  s,
const T &  val 
)

The main serializer entry point function.

Takes a serializer (either validating or plain) and the object that should be serialized.

AVRO_DECL boost::shared_ptr<std::vector<uint8_t> > avro::snapshot ( const OutputStream &  source)

Returns the contents written so far into the output stream, which should be a memory output stream.

That is it must have been returned by a pervious call to memoryOutputStream().

AVRO_DECL const std::string& avro::toString ( Type  type)

Converts the given type into a string.

Useful for generating messages.


Variable Documentation

class AVRO_DECL avro::ValidSchema

This class is used to implement an avro spec parser using a flex/bison compiler.

In order for the lexer to be reentrant, this class provides a lexer object for each parse. The bison parser also uses this class to build up an avro parse tree as the avro spec is parsed.