Avro C++
Namespaces | Classes | Typedefs | Enumerations | Functions | Variables
avro Namespace Reference

A bunch of templates and specializations for encoding and decoding specific types. More...

Namespaces

 concepts
 The concept classes are used to simplify NodeImpl.
 

Classes

class  ArraySchema
 
class  BoolSchema
 
class  BytesSchema
 
struct  codec_traits
 Codec_traits tells avro how to encode and decode an object of given type. More...
 
struct  codec_traits< avro::null >
 codec_traits for Avro null. More...
 
struct  codec_traits< bool >
 codec_traits for Avro boolean. More...
 
struct  codec_traits< double >
 codec_traits for Avro double. More...
 
struct  codec_traits< float >
 codec_traits for Avro float. More...
 
struct  codec_traits< GenericDatum >
 Specialization of codec_traits for GenericDatum. More...
 
struct  codec_traits< int32_t >
 codec_traits for Avro int. More...
 
struct  codec_traits< int64_t >
 codec_traits for Avro long. More...
 
struct  codec_traits< std::array< uint8_t, N > >
 codec_traits for Avro fixed. More...
 
struct  codec_traits< std::conditional< avro::is_not_defined< bool_codec_traits >::value, std::vector< bool >::const_reference, void >::type >
 
struct  codec_traits< std::map< std::string, T > >
 codec_traits for Avro maps. More...
 
struct  codec_traits< std::pair< ValidSchema, GenericDatum > >
 Specialization of codec_traits for Generic datum along with its schema. More...
 
struct  codec_traits< std::string >
 codec_traits for Avro string. More...
 
struct  codec_traits< std::vector< T > >
 codec_traits for Avro arrays. More...
 
struct  codec_traits< std::vector< uint8_t > >
 codec_traits for Avro bytes. More...
 
class  CompoundLayout
 
class  DataFileReader
 Reads the contents of data file one after another. More...
 
class  DataFileReaderBase
 The type independent portion of rader. More...
 
class  DataFileWriter
 An Avro datafile that can store objects of type T. More...
 
class  DataFileWriterBase
 Type-independent portion of DataFileWriter. More...
 
class  Decoder
 Decoder is an interface implemented by every decoder capable of decoding Avro data. More...
 
class  DoubleSchema
 
class  Encoder
 The abstract base class for all Avro encoders. More...
 
class  EnumSchema
 
class  Exception
 Wrapper for std::runtime_error that provides convenience constructor for boost::format objects. More...
 
class  FixedSchema
 
class  FloatSchema
 
class  GenericArray
 The generic container for Avro arrays. More...
 
class  GenericContainer
 The base class for all generic type for containers. More...
 
class  GenericDatum
 Generic datum which can hold any Avro type. More...
 
class  GenericEnum
 Generic container for Avro enum. More...
 
class  GenericFixed
 Generic container for Avro fixed. More...
 
class  GenericMap
 The generic container for Avro maps. More...
 
class  GenericReader
 A utility class to read generic datum from decoders. More...
 
class  GenericRecord
 The generic container for Avro records. More...
 
class  GenericUnion
 Generic container for unions. More...
 
class  GenericWriter
 A utility class to write generic datum to encoders. More...
 
class  InputStream
 A no-copy input stream. More...
 
class  IntSchema
 
struct  is_defined
 Check if a T is a complete type i.e. More...
 
struct  is_not_defined
 Similar to is_defined, but used to check if T is not defined. More...
 
struct  is_promotable
 
struct  is_serializable
 Define an is_serializable trait for types we can serialize natively. More...
 
class  Layout
 
class  LogicalType
 
class  LongSchema
 
class  MapSchema
 
class  Name
 
class  Node
 Node is the building block for parse trees. More...
 
class  NodeArray
 
class  NodeEnum
 
class  NodeFixed
 
class  NodeImpl
 Implementation details for Node. More...
 
class  NodeMap
 
class  NodePrimitive
 
class  NodeRecord
 
class  NodeSymbolic
 
class  NodeUnion
 
struct  Null
 define a type to identify Null in template functions More...
 
class  NullSchema
 
class  NullValidator
 
class  OutputStream
 A no-copy output stream. More...
 
class  Parser
 Class that wraps a reader or ValidatingReade with an interface that uses explicit get* names instead of getValue. More...
 
class  PrimitiveLayout
 
class  ReaderImpl
 Parses from an avro encoding to the requested type. More...
 
class  RecordSchema
 
class  Resolver
 
class  ResolverSchema
 
class  ResolvingDecoder
 ResolvingDecoder is derived from Decoder, with an additional function to obtain the field ordering of fiedls within a record. More...
 
class  ResolvingReader
 
class  Schema
 The root Schema object is a base class. Nobody constructs this class directly. More...
 
class  SeekableInputStream
 An InputStream which also supports seeking to a specific offset. More...
 
class  Serializer
 Class that wraps a Writer or ValidatingWriter with an interface that uses explicit write* names instead of writeValue. More...
 
struct  StreamReader
 A convenience class for reading from an InputStream. More...
 
struct  StreamWriter
 A convinience class to write data into an OutputStream. More...
 
class  StringSchema
 
class  SymbolicSchema
 
struct  type_to_avro
 
class  UnionSchema
 
class  Validator
 This class is used by both the ValidatingSerializer and ValidationParser objects. More...
 
class  ValidSchema
 A ValidSchema is basically a non-mutable Schema that has passed some minumum of sanity checks. More...
 
class  WriterImpl
 Class for writing avro data to a stream. More...
 

Typedefs

typedef std::array< uint8_t, SyncSize > DataFileSync
 The sync value.
 
typedef std::shared_ptr< DecoderDecoderPtr
 Shared pointer to Decoder.
 
typedef std::shared_ptr< ResolvingDecoderResolvingDecoderPtr
 Shared pointer to ResolvingDecoder.
 
typedef std::shared_ptr< EncoderEncoderPtr
 Shared pointer to Encoder.
 
typedef std::shared_ptr< NodeNodePtr
 
typedef concepts::NoAttribute< NameNoName
 
typedef concepts::SingleAttribute< NameHasName
 
typedef concepts::SingleAttribute< std::string > HasDoc
 
typedef concepts::NoAttribute< NodePtr > NoLeaves
 
typedef concepts::SingleAttribute< NodePtr > SingleLeaf
 
typedef concepts::MultiAttribute< NodePtr > MultiLeaves
 
typedef concepts::NoAttribute< std::string > NoLeafNames
 
typedef concepts::MultiAttribute< std::string > LeafNames
 
typedef concepts::NoAttribute< int > NoSize
 
typedef concepts::SingleAttribute< int > HasSize
 
typedef NodeImpl< NoName, NoLeaves, NoLeafNames, NoSizeNodeImplPrimitive
 
typedef NodeImpl< HasName, NoLeaves, NoLeafNames, NoSizeNodeImplSymbolic
 
typedef NodeImpl< HasName, MultiLeaves, LeafNames, NoSizeNodeImplRecord
 
typedef NodeImpl< HasName, NoLeaves, LeafNames, NoSizeNodeImplEnum
 
typedef NodeImpl< NoName, SingleLeaf, NoLeafNames, NoSizeNodeImplArray
 
typedef NodeImpl< NoName, MultiLeaves, NoLeafNames, NoSizeNodeImplMap
 
typedef NodeImpl< NoName, MultiLeaves, NoLeafNames, NoSizeNodeImplUnion
 
typedef NodeImpl< HasName, NoLeaves, NoLeafNames, HasSizeNodeImplFixed
 
typedef ReaderImpl< NullValidatorReader
 
typedef ReaderImpl< ValidatorValidatingReader
 
typedef boost::blank null
 
typedef codec_traits< std::vector< bool >::const_reference > bool_codec_traits
 
typedef std::unique_ptr< InputStreamInputStreamPtr
 
typedef std::unique_ptr< SeekableInputStreamSeekableInputStreamPtr
 
typedef std::unique_ptr< OutputStreamOutputStreamPtr
 
typedef WriterImpl< NullValidatorWriter
 
typedef WriterImpl< ValidatorValidatingWriter
 

Enumerations

enum  Codec { NULL_CODEC, DEFLATE_CODEC }
 Specify type of compression to use when writing data files. More...
 
enum  SchemaResolution {
  RESOLVE_NO_MATCH, RESOLVE_MATCH, RESOLVE_PROMOTABLE_TO_LONG, RESOLVE_PROMOTABLE_TO_FLOAT,
  RESOLVE_PROMOTABLE_TO_DOUBLE
}
 
enum  Type {
  AVRO_STRING, AVRO_BYTES, AVRO_INT, AVRO_LONG,
  AVRO_FLOAT, AVRO_DOUBLE, AVRO_BOOL, AVRO_NULL,
  AVRO_RECORD, AVRO_ENUM, AVRO_ARRAY, AVRO_MAP,
  AVRO_UNION, AVRO_FIXED, AVRO_NUM_TYPES, AVRO_SYMBOLIC = AVRO_NUM_TYPES,
  AVRO_UNKNOWN = -1
}
 The "type" for the schema. More...
 

Functions

template<typename Reader , typename T >
void parse (Reader &p, T &val)
 The main parse entry point function. More...
 
template<typename T >
void parse (ResolvingReader &p, T &val)
 
template<typename Reader , typename T >
void parse (Reader &p, T &val, const std::false_type &)
 Type trait should be set to is_serializable in otherwise force the compiler to complain.
 
template<typename Reader , typename T >
void translatingParse (Reader &p, T &val, const std::false_type &)
 
template<typename Writer , typename T >
void serialize (Writer &s, const T &val)
 The main serializer entry point function. More...
 
template<typename Writer , typename T >
void serialize (Writer &s, const T &val, const std::false_type &)
 Type trait should be set to is_serializable in otherwise force the compiler to complain.
 
AVRO_DECL void compileJsonSchema (std::istream &is, ValidSchema &schema)
 Given a stream comtaining a JSON schema, compiles the schema to a ValidSchema object. More...
 
AVRO_DECL bool compileJsonSchema (std::istream &is, ValidSchema &schema, std::string &error)
 Non-throwing version of compileJsonSchema. More...
 
AVRO_DECL ValidSchema compileJsonSchemaFromStream (InputStream &is)
 
AVRO_DECL ValidSchema compileJsonSchemaFromMemory (const uint8_t *input, size_t len)
 
AVRO_DECL ValidSchema compileJsonSchemaFromString (const char *input)
 
AVRO_DECL ValidSchema compileJsonSchemaFromString (const std::string &input)
 
AVRO_DECL ValidSchema compileJsonSchemaFromFile (const char *filename)
 
AVRO_DECL DecoderPtr binaryDecoder ()
 Returns an decoder that can decode binary Avro standard.
 
AVRO_DECL DecoderPtr validatingDecoder (const ValidSchema &schema, const DecoderPtr &base)
 Returns an decoder that validates sequence of calls to an underlying Decoder against the given schema.
 
AVRO_DECL DecoderPtr jsonDecoder (const ValidSchema &schema)
 Returns an decoder that can decode Avro standard for JSON.
 
AVRO_DECL ResolvingDecoderPtr resolvingDecoder (const ValidSchema &writer, const ValidSchema &reader, const DecoderPtr &base)
 Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema. More...
 
AVRO_DECL EncoderPtr binaryEncoder ()
 Returns an encoder that can encode binary Avro standard.
 
AVRO_DECL EncoderPtr validatingEncoder (const ValidSchema &schema, const EncoderPtr &base)
 Returns an encoder that validates sequence of calls to an underlying Encoder against the given schema.
 
AVRO_DECL EncoderPtr jsonEncoder (const ValidSchema &schema)
 Returns an encoder that encodes Avro standard for JSON.
 
AVRO_DECL EncoderPtr jsonPrettyEncoder (const ValidSchema &schema)
 Returns an encoder that encodes Avro standard for pretty printed JSON.
 
std::ostream & operator<< (std::ostream &os, const Name &n)
 
NodePtr resolveSymbol (const NodePtr &node)
 
template<typename T >
std::string intToHex (T i)
 
Type nextType (Parser< ValidatingReader > &p)
 
bool currentRecordName (Parser< ValidatingReader > &p, std::string &name)
 
bool nextFieldName (Parser< ValidatingReader > &p, std::string &name)
 
std::unique_ptr< ResolverconstructResolver (const ValidSchema &rwriterSchema, const ValidSchema &readerSchema, const Layout &readerLayout)
 
template<typename T >
void encode (Encoder &e, const T &t)
 Generic encoder function that makes use of the codec_traits.
 
template<typename T >
void decode (Decoder &d, T &t)
 Generic decoder function that makes use of the codec_traits.
 
AVRO_DECL OutputStreamPtr memoryOutputStream (size_t chunkSize=4 *1024)
 Returns a new OutputStream, which grows in memory chunks of specified size.
 
AVRO_DECL InputStreamPtr memoryInputStream (const uint8_t *data, size_t len)
 Returns a new InputStream, with the data from the given byte array. More...
 
AVRO_DECL InputStreamPtr memoryInputStream (const OutputStream &source)
 Returns a new InputStream with the contents written into an outputstream. More...
 
AVRO_DECL std::shared_ptr< std::vector< uint8_t > > snapshot (const OutputStream &source)
 Returns the contents written so far into the output stream, which should be a memory output stream. More...
 
AVRO_DECL OutputStreamPtr fileOutputStream (const char *filename, size_t bufferSize=8 *1024)
 Returns a new OutputStream whose contents would be stored in a file. More...
 
AVRO_DECL InputStreamPtr fileInputStream (const char *filename, size_t bufferSize=8 *1024)
 Returns a new InputStream whose contents come from the given file. More...
 
AVRO_DECL SeekableInputStreamPtr fileSeekableInputStream (const char *filename, size_t bufferSize=8 *1024)
 
AVRO_DECL OutputStreamPtr ostreamOutputStream (std::ostream &os, size_t bufferSize=8 *1024)
 Returns a new OutputStream whose contents will be sent to the given std::ostream. More...
 
AVRO_DECL InputStreamPtr istreamInputStream (std::istream &in, size_t bufferSize=8 *1024)
 Returns a new InputStream whose contents come from the given std::istream. More...
 
AVRO_DECL InputStreamPtr nonSeekableIstreamInputStream (std::istream &is, size_t bufferSize=8 *1024)
 Returns a new InputStream whose contents come from the given std::istream. More...
 
void copy (InputStream &in, OutputStream &out)
 A convenience function to copy all the contents of an input stream into an output stream.
 
bool isPrimitive (Type t)
 Returns true if and only if the given type is a primitive. More...
 
bool isCompound (Type t)
 Returns true if and only if the given type is a non primitive valid type. More...
 
bool isAvroType (Type t)
 Returns true if and only if the given type is a valid avro type.
 
bool isAvroTypeOrPseudoType (Type t)
 Returns true if and only if the given type is within the valid range of enumeration.
 
AVRO_DECL const std::string & toString (Type type)
 Converts the given type into a string. More...
 
AVRO_DECL std::ostream & operator<< (std::ostream &os, avro::Type type)
 Writes a string form of the given type into the given ostream.
 
std::ostream & operator<< (std::ostream &os, const Null &null)
 Writes schema for null null type to os. More...
 
AVRO_DECL uint64_t encodeZigzag64 (int64_t input)
 
AVRO_DECL int64_t decodeZigzag64 (uint64_t input)
 
AVRO_DECL uint32_t encodeZigzag32 (int32_t input)
 
AVRO_DECL int32_t decodeZigzag32 (uint32_t input)
 
AVRO_DECL size_t encodeInt32 (int32_t input, std::array< uint8_t, 5 > &output)
 
AVRO_DECL size_t encodeInt64 (int64_t input, std::array< uint8_t, 10 > &output)
 
template<typename Reader , typename T >
void parse (Reader &p, T &val, const std::true_type &)
 The remainder of the file includes default implementations for serializable types.
 
template<typename Reader >
void parse (Reader &p, std::vector< uint8_t > &val, const std::true_type &)
 
template<typename T >
void translatingParse (ResolvingReader &p, T &val, const std::true_type &)
 
template<typename Writer , typename T >
void serialize (Writer &s, T val, const std::true_type &)
 The remainder of the file includes default implementations for serializable types.
 
template<typename Writer >
void serialize (Writer &s, const std::vector< uint8_t > &val, const std::true_type &)
 

Variables

class AVRO_DECL InputStream
 
class AVRO_DECL ValidSchema
 This class is used to implement an avro spec parser using a flex/bison compiler. More...
 
const int SyncSize = 16
 
class AVRO_DECL Schema
 

Detailed Description

A bunch of templates and specializations for encoding and decoding specific types.

Primitive AVRO types BOOLEAN, INT, LONG, FLOAT, DOUBLE, STRING and BYTES get decoded to and encoded from C++ types bool, int32_t, int64_t, float, double, std::string and std::vector<uint8_t> respectively. In addition, std::vector<T> for aribtrary type T gets encoded as an Avro array of T. Similarly, std::map<std::string, T> for arbitrary type T gets encoded as an Avro map with value type T.

Users can have their custom types encoded/decoded by specializing avro::codec_traits class for their types.

Enumeration Type Documentation

◆ Codec

Specify type of compression to use when writing data files.

◆ SchemaResolution

Enumerator
RESOLVE_NO_MATCH 

The schemas definitely do not match.

RESOLVE_MATCH 

The schemas match at a cursory level.

For records and enums, this means the name is the same, but it does not necessarily mean that every symbol or field is an exact match.

RESOLVE_PROMOTABLE_TO_LONG 

For primitives, the matching may occur if the type is promotable.

This means that the writer matches reader if the writer's type is promoted the specified type.

◆ Type

enum avro::Type

The "type" for the schema.

Enumerator
AVRO_STRING 

String

AVRO_BYTES 

Sequence of variable length bytes data

AVRO_INT 

32-bit integer

AVRO_LONG 

64-bit integer

AVRO_FLOAT 

Floating point number

AVRO_DOUBLE 

Double precision floating point number

AVRO_BOOL 

Boolean value

AVRO_NULL 

Null

AVRO_RECORD 

Record, a sequence of fields

AVRO_ENUM 

Enumeration

AVRO_ARRAY 

Homogeneous array of some specific type

AVRO_MAP 

Homogeneous map from string to some specific type

AVRO_UNION 

Union of one or more types

AVRO_FIXED 

Fixed number of bytes

AVRO_NUM_TYPES 

Marker

AVRO_SYMBOLIC 

User internally to avoid circular references.

AVRO_UNKNOWN 

Used internally.

Function Documentation

◆ compileJsonSchema() [1/2]

AVRO_DECL void avro::compileJsonSchema ( std::istream &  is,
ValidSchema schema 
)

Given a stream comtaining a JSON schema, compiles the schema to a ValidSchema object.

Throws if the schema cannot be compiled to a valid schema

◆ compileJsonSchema() [2/2]

AVRO_DECL bool avro::compileJsonSchema ( std::istream &  is,
ValidSchema schema,
std::string &  error 
)

Non-throwing version of compileJsonSchema.

Returns
True if no error, false if error (with the error string set)

◆ fileInputStream()

AVRO_DECL InputStreamPtr avro::fileInputStream ( const char *  filename,
size_t  bufferSize = 8 *1024 
)

Returns a new InputStream whose contents come from the given file.

Data is read in chunks of given buffer size.

Referenced by avro::OutputStream::~OutputStream().

◆ fileOutputStream()

AVRO_DECL OutputStreamPtr avro::fileOutputStream ( const char *  filename,
size_t  bufferSize = 8 *1024 
)

Returns a new OutputStream whose contents would be stored in a file.

Data is written in chunks of given buffer size.

If there is a file with the given name, it is truncated and overwritten. If there is no file with the given name, it is created.

Referenced by avro::OutputStream::~OutputStream().

◆ isCompound()

bool avro::isCompound ( Type  t)
inline

Returns true if and only if the given type is a non primitive valid type.

Primitive types are: string, bytes, int, long, float, double, boolean and null

References AVRO_NUM_TYPES, and AVRO_RECORD.

◆ isPrimitive()

bool avro::isPrimitive ( Type  t)
inline

Returns true if and only if the given type is a primitive.

Primitive types are: string, bytes, int, long, float, double, boolean and null

References AVRO_RECORD, and AVRO_STRING.

◆ istreamInputStream()

AVRO_DECL InputStreamPtr avro::istreamInputStream ( std::istream &  in,
size_t  bufferSize = 8 *1024 
)

Returns a new InputStream whose contents come from the given std::istream.

The std::istream object should outlive the returned InputStream.

Referenced by avro::OutputStream::~OutputStream().

◆ memoryInputStream() [1/2]

AVRO_DECL InputStreamPtr avro::memoryInputStream ( const uint8_t *  data,
size_t  len 
)

Returns a new InputStream, with the data from the given byte array.

It does not copy the data, the byte array should remain valid until the InputStream is used.

Referenced by avro::OutputStream::~OutputStream().

◆ memoryInputStream() [2/2]

AVRO_DECL InputStreamPtr avro::memoryInputStream ( const OutputStream source)

Returns a new InputStream with the contents written into an outputstream.

The output stream must have been returned by an earlier call to memoryOutputStream(). The contents for the new input stream are the snapshot of the outputstream. One can construct any number of memory input stream from a single memory output stream.

◆ nonSeekableIstreamInputStream()

AVRO_DECL InputStreamPtr avro::nonSeekableIstreamInputStream ( std::istream &  is,
size_t  bufferSize = 8 *1024 
)

Returns a new InputStream whose contents come from the given std::istream.

Use this instead of istreamInputStream if the istream does not support seekg (e.g. compressed streams). The returned InputStream would read off bytes instead of seeking. Of, course it has a performance penalty when reading instead of seeking; So, use this only when seekg does not work. The std::istream object should outlive the returned InputStream.

Referenced by avro::OutputStream::~OutputStream().

◆ operator<<()

std::ostream& avro::operator<< ( std::ostream &  os,
const Null null 
)

Writes schema for null null type to os.

Parameters
osThe ostream to write to.
nullThe value to be written.

◆ ostreamOutputStream()

AVRO_DECL OutputStreamPtr avro::ostreamOutputStream ( std::ostream &  os,
size_t  bufferSize = 8 *1024 
)

Returns a new OutputStream whose contents will be sent to the given std::ostream.

The std::ostream object should outlive the returned OutputStream.

Referenced by avro::OutputStream::~OutputStream().

◆ parse()

template<typename Reader , typename T >
void avro::parse ( Reader p,
T &  val 
)

The main parse entry point function.

Takes a parser (either validating or plain) and the object that should receive the parsed data.

Referenced by parse().

◆ resolvingDecoder()

AVRO_DECL ResolvingDecoderPtr avro::resolvingDecoder ( const ValidSchema writer,
const ValidSchema reader,
const DecoderPtr base 
)

Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema.

The client uses the decoder as if the data were written using readerSchema. // FIXME: Handle out of order fields.

◆ serialize()

template<typename Writer , typename T >
void avro::serialize ( Writer s,
const T &  val 
)

The main serializer entry point function.

Takes a serializer (either validating or plain) and the object that should be serialized.

Referenced by serialize().

◆ snapshot()

AVRO_DECL std::shared_ptr<std::vector<uint8_t> > avro::snapshot ( const OutputStream source)

Returns the contents written so far into the output stream, which should be a memory output stream.

That is it must have been returned by a pervious call to memoryOutputStream().

Referenced by avro::OutputStream::~OutputStream().

◆ toString()

AVRO_DECL const std::string& avro::toString ( Type  type)

Converts the given type into a string.

Useful for generating messages.

Referenced by isAvroTypeOrPseudoType().

Variable Documentation

◆ ValidSchema

class AVRO_DECL avro::ValidSchema

This class is used to implement an avro spec parser using a flex/bison compiler.

In order for the lexer to be reentrant, this class provides a lexer object for each parse. The bison parser also uses this class to build up an avro parse tree as the avro spec is parsed.