Avro C++
|
A bunch of templates and specializations for encoding and decoding specific types. More...
Namespaces | |
concepts | |
The concept classes are used to simplify NodeImpl. | |
Classes | |
class | ArraySchema |
class | BoolSchema |
class | BytesSchema |
struct | codec_traits |
Codec_traits tells avro how to encode and decode an object of given type. More... | |
struct | codec_traits< avro::null > |
codec_traits for Avro null. More... | |
struct | codec_traits< bool > |
codec_traits for Avro boolean. More... | |
struct | codec_traits< double > |
codec_traits for Avro double. More... | |
struct | codec_traits< float > |
codec_traits for Avro float. More... | |
struct | codec_traits< GenericDatum > |
Specialization of codec_traits for GenericDatum. More... | |
struct | codec_traits< int32_t > |
codec_traits for Avro int. More... | |
struct | codec_traits< int64_t > |
codec_traits for Avro long. More... | |
struct | codec_traits< std::array< uint8_t, N > > |
codec_traits for Avro fixed. More... | |
struct | codec_traits< std::conditional< avro::is_not_defined< bool_codec_traits >::value, std::vector< bool >::const_reference, void >::type > |
struct | codec_traits< std::map< std::string, T > > |
codec_traits for Avro maps. More... | |
struct | codec_traits< std::pair< ValidSchema, GenericDatum > > |
Specialization of codec_traits for Generic datum along with its schema. More... | |
struct | codec_traits< std::string > |
codec_traits for Avro string. More... | |
struct | codec_traits< std::vector< T > > |
codec_traits for Avro arrays. More... | |
struct | codec_traits< std::vector< uint8_t > > |
codec_traits for Avro bytes. More... | |
class | CompoundLayout |
class | CustomAttributes |
class | DataFileReader |
Reads the contents of data file one after another. More... | |
class | DataFileReaderBase |
The type independent portion of reader. More... | |
class | DataFileWriter |
An Avro datafile that can store objects of type T. More... | |
class | DataFileWriterBase |
Type-independent portion of DataFileWriter. More... | |
class | Decoder |
Decoder is an interface implemented by every decoder capable of decoding Avro data. More... | |
class | DoubleSchema |
class | Encoder |
The abstract base class for all Avro encoders. More... | |
class | EnumSchema |
class | Exception |
Wrapper for std::runtime_error that provides convenience constructor for boost::format objects. More... | |
class | FixedSchema |
class | FloatSchema |
class | GenericArray |
The generic container for Avro arrays. More... | |
class | GenericContainer |
The base class for all generic type for containers. More... | |
class | GenericDatum |
Generic datum which can hold any Avro type. More... | |
class | GenericEnum |
Generic container for Avro enum. More... | |
class | GenericFixed |
Generic container for Avro fixed. More... | |
class | GenericMap |
The generic container for Avro maps. More... | |
class | GenericReader |
A utility class to read generic datum from decoders. More... | |
class | GenericRecord |
The generic container for Avro records. More... | |
class | GenericUnion |
Generic container for unions. More... | |
class | GenericWriter |
A utility class to write generic datum to encoders. More... | |
class | InputStream |
A no-copy input stream. More... | |
class | IntSchema |
struct | is_defined |
Check if a T is a complete type i.e. More... | |
struct | is_not_defined |
Similar to is_defined, but used to check if T is not defined. More... | |
struct | is_promotable |
struct | is_serializable |
Define an is_serializable trait for types we can serialize natively. More... | |
class | Layout |
class | LogicalType |
class | LongSchema |
class | MapSchema |
class | Name |
class | Node |
Node is the building block for parse trees. More... | |
class | NodeArray |
class | NodeEnum |
class | NodeFixed |
class | NodeImpl |
Implementation details for Node. More... | |
class | NodeMap |
class | NodePrimitive |
class | NodeRecord |
class | NodeSymbolic |
class | NodeUnion |
struct | Null |
define a type to represent Avro Null in template functions More... | |
class | NullSchema |
class | NullValidator |
class | OutputStream |
A no-copy output stream. More... | |
class | Parser |
Class that wraps a reader or ValidatingReade with an interface that uses explicit get* names instead of getValue. More... | |
class | PrimitiveLayout |
class | ReaderImpl |
Parses from an avro encoding to the requested type. More... | |
class | RecordSchema |
class | Resolver |
class | ResolverSchema |
class | ResolvingDecoder |
ResolvingDecoder is derived from Decoder, with an additional function to obtain the field ordering of fields within a record. More... | |
class | ResolvingReader |
class | Schema |
The root Schema object is a base class. Nobody constructs this class directly. More... | |
class | SeekableInputStream |
An InputStream which also supports seeking to a specific offset. More... | |
class | Serializer |
Class that wraps a Writer or ValidatingWriter with an interface that uses explicit write* names instead of writeValue. More... | |
struct | StreamReader |
A convenience class for reading from an InputStream. More... | |
struct | StreamWriter |
A convenience class to write data into an OutputStream. More... | |
class | StringSchema |
class | SymbolicSchema |
struct | type_to_avro |
class | UnionSchema |
class | Validator |
This class is used by both the ValidatingSerializer and ValidationParser objects. More... | |
class | ValidSchema |
A ValidSchema is basically a non-mutable Schema that has passed some minimum of sanity checks. More... | |
class | WriterImpl |
Class for writing avro data to a stream. More... | |
Typedefs | |
typedef std::array< uint8_t, SyncSize > | DataFileSync |
The sync value. | |
using | DecoderPtr = std::shared_ptr< Decoder > |
Shared pointer to Decoder. | |
using | ResolvingDecoderPtr = std::shared_ptr< ResolvingDecoder > |
Shared pointer to ResolvingDecoder. | |
using | EncoderPtr = std::shared_ptr< Encoder > |
Shared pointer to Encoder. | |
using | NodePtr = std::shared_ptr< Node > |
using | NoName = concepts::NoAttribute< Name > |
using | HasName = concepts::SingleAttribute< Name > |
using | HasDoc = concepts::SingleAttribute< std::string > |
using | NoLeaves = concepts::NoAttribute< NodePtr > |
using | SingleLeaf = concepts::SingleAttribute< NodePtr > |
using | MultiLeaves = concepts::MultiAttribute< NodePtr > |
using | NoLeafNames = concepts::NoAttribute< std::string > |
using | LeafNames = concepts::MultiAttribute< std::string > |
using | MultiAttributes = concepts::MultiAttribute< CustomAttributes > |
using | NoAttributes = concepts::NoAttribute< CustomAttributes > |
using | NoSize = concepts::NoAttribute< int > |
using | HasSize = concepts::SingleAttribute< int > |
using | NodeImplPrimitive = NodeImpl< NoName, NoLeaves, NoLeafNames, MultiAttributes, NoSize > |
using | NodeImplSymbolic = NodeImpl< HasName, NoLeaves, NoLeafNames, NoAttributes, NoSize > |
using | NodeImplRecord = NodeImpl< HasName, MultiLeaves, LeafNames, MultiAttributes, NoSize > |
using | NodeImplEnum = NodeImpl< HasName, NoLeaves, LeafNames, NoAttributes, NoSize > |
using | NodeImplArray = NodeImpl< NoName, SingleLeaf, NoLeafNames, NoAttributes, NoSize > |
using | NodeImplMap = NodeImpl< NoName, MultiLeaves, NoLeafNames, NoAttributes, NoSize > |
using | NodeImplUnion = NodeImpl< NoName, MultiLeaves, NoLeafNames, NoAttributes, NoSize > |
using | NodeImplFixed = NodeImpl< HasName, NoLeaves, NoLeafNames, NoAttributes, HasSize > |
using | Reader = ReaderImpl< NullValidator > |
using | ValidatingReader = ReaderImpl< Validator > |
typedef boost::blank | null |
typedef codec_traits< std::vector< bool >::const_reference > | bool_codec_traits |
typedef std::unique_ptr< InputStream > | InputStreamPtr |
typedef std::unique_ptr< SeekableInputStream > | SeekableInputStreamPtr |
typedef std::unique_ptr< OutputStream > | OutputStreamPtr |
using | Writer = WriterImpl< NullValidator > |
using | ValidatingWriter = WriterImpl< Validator > |
Enumerations | |
enum | Codec { NULL_CODEC, DEFLATE_CODEC } |
Specify type of compression to use when writing data files. | |
enum | SchemaResolution { RESOLVE_NO_MATCH, RESOLVE_MATCH, RESOLVE_PROMOTABLE_TO_LONG, RESOLVE_PROMOTABLE_TO_FLOAT, RESOLVE_PROMOTABLE_TO_DOUBLE } |
enum | Type { AVRO_STRING, AVRO_BYTES, AVRO_INT, AVRO_LONG, AVRO_FLOAT, AVRO_DOUBLE, AVRO_BOOL, AVRO_NULL, AVRO_RECORD, AVRO_ENUM, AVRO_ARRAY, AVRO_MAP, AVRO_UNION, AVRO_FIXED, AVRO_NUM_TYPES, AVRO_SYMBOLIC = AVRO_NUM_TYPES, AVRO_UNKNOWN = -1 } |
The "type" for the schema. More... | |
Functions | |
template<typename Reader , typename T > | |
void | parse (Reader &p, T &val) |
The main parse entry point function. More... | |
template<typename T > | |
void | parse (ResolvingReader &p, T &val) |
template<typename Reader , typename T > | |
void | parse (Reader &p, T &val, const std::false_type &) |
Type trait should be set to is_serializable in otherwise force the compiler to complain. | |
template<typename Reader , typename T > | |
void | translatingParse (Reader &p, T &val, const std::false_type &) |
template<typename Reader , typename T > | |
void | parse (Reader &p, T &val, const std::true_type &) |
The remainder of the file includes default implementations for serializable types. | |
template<typename Reader > | |
void | parse (Reader &p, std::vector< uint8_t > &val, const std::true_type &) |
template<typename T > | |
void | translatingParse (ResolvingReader &p, T &val, const std::true_type &) |
template<typename Writer , typename T > | |
void | serialize (Writer &s, const T &val) |
The main serializer entry point function. More... | |
template<typename Writer , typename T > | |
void | serialize (Writer &s, const T &val, const std::false_type &) |
Type trait should be set to is_serializable in otherwise force the compiler to complain. | |
template<typename Writer , typename T > | |
void | serialize (Writer &s, T val, const std::true_type &) |
The remainder of the file includes default implementations for serializable types. | |
template<typename Writer > | |
void | serialize (Writer &s, const std::vector< uint8_t > &val, const std::true_type &) |
AVRO_DECL void | compileJsonSchema (std::istream &is, ValidSchema &schema) |
Given a stream containing a JSON schema, compiles the schema to a ValidSchema object. More... | |
AVRO_DECL bool | compileJsonSchema (std::istream &is, ValidSchema &schema, std::string &error) |
Non-throwing version of compileJsonSchema. More... | |
AVRO_DECL ValidSchema | compileJsonSchemaFromStream (InputStream &is) |
AVRO_DECL ValidSchema | compileJsonSchemaFromMemory (const uint8_t *input, size_t len) |
AVRO_DECL ValidSchema | compileJsonSchemaFromString (const char *input) |
AVRO_DECL ValidSchema | compileJsonSchemaFromString (const std::string &input) |
AVRO_DECL ValidSchema | compileJsonSchemaFromFile (const char *filename) |
AVRO_DECL DecoderPtr | binaryDecoder () |
Returns an decoder that can decode binary Avro standard. | |
AVRO_DECL DecoderPtr | validatingDecoder (const ValidSchema &schema, const DecoderPtr &base) |
Returns an decoder that validates sequence of calls to an underlying Decoder against the given schema. | |
AVRO_DECL DecoderPtr | jsonDecoder (const ValidSchema &schema) |
Returns an decoder that can decode Avro standard for JSON. | |
AVRO_DECL ResolvingDecoderPtr | resolvingDecoder (const ValidSchema &writer, const ValidSchema &reader, const DecoderPtr &base) |
Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema. More... | |
AVRO_DECL EncoderPtr | binaryEncoder () |
Returns an encoder that can encode binary Avro standard. | |
AVRO_DECL EncoderPtr | validatingEncoder (const ValidSchema &schema, const EncoderPtr &base) |
Returns an encoder that validates sequence of calls to an underlying Encoder against the given schema. | |
AVRO_DECL EncoderPtr | jsonEncoder (const ValidSchema &schema) |
Returns an encoder that encodes Avro standard for JSON. | |
AVRO_DECL EncoderPtr | jsonPrettyEncoder (const ValidSchema &schema) |
Returns an encoder that encodes Avro standard for pretty printed JSON. | |
std::ostream & | operator<< (std::ostream &os, const Name &n) |
NodePtr | resolveSymbol (const NodePtr &node) |
template<typename T > | |
std::string | intToHex (T i) |
Type | nextType (Parser< ValidatingReader > &p) |
bool | currentRecordName (Parser< ValidatingReader > &p, std::string &name) |
bool | nextFieldName (Parser< ValidatingReader > &p, std::string &name) |
std::unique_ptr< Resolver > | constructResolver (const ValidSchema &writerSchema, const ValidSchema &readerSchema, const Layout &readerLayout) |
template<typename T > | |
void | encode (Encoder &e, const T &t) |
Generic encoder function that makes use of the codec_traits. | |
template<typename T > | |
void | decode (Decoder &d, T &t) |
Generic decoder function that makes use of the codec_traits. | |
AVRO_DECL OutputStreamPtr | memoryOutputStream (size_t chunkSize=4 *1024) |
Returns a new OutputStream, which grows in memory chunks of specified size. | |
AVRO_DECL InputStreamPtr | memoryInputStream (const uint8_t *data, size_t len) |
Returns a new InputStream, with the data from the given byte array. More... | |
AVRO_DECL InputStreamPtr | memoryInputStream (const OutputStream &source) |
Returns a new InputStream with the contents written into an OutputStream. More... | |
AVRO_DECL std::shared_ptr< std::vector< uint8_t > > | snapshot (const OutputStream &source) |
Returns the contents written so far into the output stream, which should be a memory output stream. More... | |
AVRO_DECL OutputStreamPtr | fileOutputStream (const char *filename, size_t bufferSize=8 *1024) |
Returns a new OutputStream whose contents would be stored in a file. More... | |
AVRO_DECL InputStreamPtr | fileInputStream (const char *filename, size_t bufferSize=8 *1024) |
Returns a new InputStream whose contents come from the given file. More... | |
AVRO_DECL SeekableInputStreamPtr | fileSeekableInputStream (const char *filename, size_t bufferSize=8 *1024) |
AVRO_DECL OutputStreamPtr | ostreamOutputStream (std::ostream &os, size_t bufferSize=8 *1024) |
Returns a new OutputStream whose contents will be sent to the given std::ostream. More... | |
AVRO_DECL InputStreamPtr | istreamInputStream (std::istream &in, size_t bufferSize=8 *1024) |
Returns a new InputStream whose contents come from the given std::istream. More... | |
AVRO_DECL InputStreamPtr | nonSeekableIstreamInputStream (std::istream &is, size_t bufferSize=8 *1024) |
Returns a new InputStream whose contents come from the given std::istream. More... | |
void | copy (InputStream &in, OutputStream &out) |
A convenience function to copy all the contents of an input stream into an output stream. | |
constexpr bool | isPrimitive (Type t) noexcept |
Returns true if and only if the given type is a primitive. More... | |
constexpr bool | isCompound (Type t) noexcept |
Returns true if and only if the given type is a non primitive valid type. More... | |
constexpr bool | isAvroType (Type t) noexcept |
Returns true if and only if the given type is a valid avro type. | |
constexpr bool | isAvroTypeOrPseudoType (Type t) noexcept |
Returns true if and only if the given type is within the valid range of enumeration. | |
const AVRO_DECL std::string & | toString (Type type) noexcept |
Converts the given type into a string. More... | |
AVRO_DECL std::ostream & | operator<< (std::ostream &os, avro::Type type) |
Writes a string form of the given type into the given ostream. | |
std::ostream & | operator<< (std::ostream &os, const Null &null) |
Writes schema for null null type to os . More... | |
constexpr AVRO_DECL uint64_t | encodeZigzag64 (int64_t input) noexcept |
constexpr AVRO_DECL int64_t | decodeZigzag64 (uint64_t input) noexcept |
constexpr AVRO_DECL uint32_t | encodeZigzag32 (int32_t input) noexcept |
constexpr AVRO_DECL int32_t | decodeZigzag32 (uint32_t input) noexcept |
AVRO_DECL size_t | encodeInt32 (int32_t input, std::array< uint8_t, 5 > &output) noexcept |
AVRO_DECL size_t | encodeInt64 (int64_t input, std::array< uint8_t, 10 > &output) noexcept |
Variables | |
class AVRO_DECL | InputStream |
class AVRO_DECL | ValidSchema |
This class is used to implement an avro spec parser using a flex/bison compiler. More... | |
const int | SyncSize = 16 |
class AVRO_DECL | Schema |
A bunch of templates and specializations for encoding and decoding specific types.
Primitive AVRO types BOOLEAN, INT, LONG, FLOAT, DOUBLE, STRING and BYTES get decoded to and encoded from C++ types bool, int32_t, int64_t, float, double, std::string and std::vector<uint8_t> respectively. In addition, std::vector<T> for arbitrary type T gets encoded as an Avro array of T. Similarly, std::map<std::string, T> for arbitrary type T gets encoded as an Avro map with value type T.
Users can have their custom types encoded/decoded by specializing avro::codec_traits class for their types.
enum avro::Type |
The "type" for the schema.
AVRO_DECL void avro::compileJsonSchema | ( | std::istream & | is, |
ValidSchema & | schema | ||
) |
Given a stream containing a JSON schema, compiles the schema to a ValidSchema object.
Throws if the schema cannot be compiled to a valid schema
AVRO_DECL bool avro::compileJsonSchema | ( | std::istream & | is, |
ValidSchema & | schema, | ||
std::string & | error | ||
) |
Non-throwing version of compileJsonSchema.
AVRO_DECL InputStreamPtr avro::fileInputStream | ( | const char * | filename, |
size_t | bufferSize = 8 *1024 |
||
) |
Returns a new InputStream whose contents come from the given file.
Data is read in chunks of given buffer size.
AVRO_DECL OutputStreamPtr avro::fileOutputStream | ( | const char * | filename, |
size_t | bufferSize = 8 *1024 |
||
) |
Returns a new OutputStream whose contents would be stored in a file.
Data is written in chunks of given buffer size.
If there is a file with the given name, it is truncated and overwritten. If there is no file with the given name, it is created.
|
inlineconstexprnoexcept |
Returns true if and only if the given type is a non primitive valid type.
Primitive types are: string, bytes, int, long, float, double, boolean and null
References AVRO_NUM_TYPES, and AVRO_RECORD.
|
inlineconstexprnoexcept |
Returns true if and only if the given type is a primitive.
Primitive types are: string, bytes, int, long, float, double, boolean and null
References AVRO_RECORD, and AVRO_STRING.
AVRO_DECL InputStreamPtr avro::istreamInputStream | ( | std::istream & | in, |
size_t | bufferSize = 8 *1024 |
||
) |
Returns a new InputStream whose contents come from the given std::istream.
The std::istream object should outlive the returned InputStream.
AVRO_DECL InputStreamPtr avro::memoryInputStream | ( | const OutputStream & | source | ) |
Returns a new InputStream with the contents written into an OutputStream.
The output stream must have been returned by an earlier call to memoryOutputStream(). The contents for the new InputStream are the snapshot of the output stream. One can construct any number of memory input stream from a single memory output stream.
AVRO_DECL InputStreamPtr avro::memoryInputStream | ( | const uint8_t * | data, |
size_t | len | ||
) |
Returns a new InputStream, with the data from the given byte array.
It does not copy the data, the byte array should remain valid until the InputStream is used.
AVRO_DECL InputStreamPtr avro::nonSeekableIstreamInputStream | ( | std::istream & | is, |
size_t | bufferSize = 8 *1024 |
||
) |
Returns a new InputStream whose contents come from the given std::istream.
Use this instead of istreamInputStream if the istream does not support seekg (e.g. compressed streams). The returned InputStream would read off bytes instead of seeking. Of, course it has a performance penalty when reading instead of seeking; So, use this only when seekg does not work. The std::istream object should outlive the returned InputStream.
std::ostream& avro::operator<< | ( | std::ostream & | os, |
const Null & | null | ||
) |
Writes schema for null null
type to os
.
os | The ostream to write to. |
null | The value to be written. |
AVRO_DECL OutputStreamPtr avro::ostreamOutputStream | ( | std::ostream & | os, |
size_t | bufferSize = 8 *1024 |
||
) |
Returns a new OutputStream whose contents will be sent to the given std::ostream.
The std::ostream object should outlive the returned OutputStream.
void avro::parse | ( | Reader & | p, |
T & | val | ||
) |
The main parse entry point function.
Takes a parser (either validating or plain) and the object that should receive the parsed data.
AVRO_DECL ResolvingDecoderPtr avro::resolvingDecoder | ( | const ValidSchema & | writer, |
const ValidSchema & | reader, | ||
const DecoderPtr & | base | ||
) |
Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema.
The client uses the decoder as if the data were written using readerSchema. // FIXME: Handle out of order fields.
void avro::serialize | ( | Writer & | s, |
const T & | val | ||
) |
The main serializer entry point function.
Takes a serializer (either validating or plain) and the object that should be serialized.
AVRO_DECL std::shared_ptr<std::vector<uint8_t> > avro::snapshot | ( | const OutputStream & | source | ) |
Returns the contents written so far into the output stream, which should be a memory output stream.
That is it must have been returned by a previous call to memoryOutputStream().
|
noexcept |
Converts the given type into a string.
Useful for generating messages.
class AVRO_DECL avro::ValidSchema |
This class is used to implement an avro spec parser using a flex/bison compiler.
In order for the lexer to be reentrant, this class provides a lexer object for each parse. The bison parser also uses this class to build up an avro parse tree as the avro spec is parsed.