Introduction

KonFoo is a Python package for de-serializing byte streams into a meaningful representation. KonFoo helps you to deserialize a byte stream retrievable through a byte stream provider to any kind of data source into a meaningful representation by just declaring how the parts of a byte stream should be represented, respectively mapped to fields.

You can store the representation into an .ini file to analyse the byte stream data.

The built-in deserialize hook deserialize(buffer=bytes(), index=Index(), **option) available for all container and field classes allows you to adapt even expand or declare the representation during the de-serialization process on the fly.

The built-in deserializer provided by the pointer class (called through the Pointer.read_from() method) is able to follow even nested absolute or relative pointers to retrieve the byte stream from the byte stream provider necessary for its referenced data object and to de-serialize (map) it.

After de-serializing the byte stream provided by the byte stream provider the built-in serializer provided also by the pointer class (called through the Pointer.write_to() method) is able to transfer the manipulated values of any container or field in the representation back to the byte stream provider to write it into its data source.

Concept

KonFoo is based on defining or declaring a byte stream mapper (representation) through classes. KonFoo has two abstract base classes the container class and the field class.

A container contains field and/or container classes and knows how to view, save and load the values of the field items within the container.

A field represents the value of a content area in a byte stream which the field maps and knows how to unpack and pack its value from and to a byte stream.

The mixin pointer class has both features of the two base classes and has an interface to a byte stream provider to read and write byte streams from and back to the byte stream provider for its referenced data object respectively its byte stream mapper.

The built-in deserializer and serializer unpacks and packs the byte stream sequential to and from each field in the declared byte stream mapper.

Mapper

A byte stream mapper consists of a collection of container and field members, whereby the container members describe the structure and the field members describe the content of one or more memory areas in a data source. The mix-in pointer field serves in combination with a byte stream provider as an entry point to a data source for the byte stream mapper to deserialize and serialize its byte stream.

Containers

The role of a Container is to describe the structure of one or more memory areas in a data source. A container always needs one or more fields to describe the content of the memory area.

Overview

Here is an overview of the different available container classes.

• Structure
• Sequence, Array
• Pointer, StructurePointer, SequencePointer, ArrayPointer, StreamPointer, StringPointer, AutoStringPointer
• RelativePointer, StructureRelativePointer, SequenceRelativePointer, ArrayRelativePointer, StreamRelativePointer, StringRelativePointer

View Field Attributes

A container can view the attributes of each field nested in the container by calling its method view_fields(). Default attribute is the field value.

>>> # Create an empty container.
>>> container = Container()
>>> # View the field values in the container.
>>> container.view_fields()

Note

The attributes of each field for containers nested in the container are viewed as well (chained method call).

View as a JSON String

A container can view the attributes of each field nested in the container as a JSON formatted string by calling its method to_json(). Default attribute is the field value.

>>> container.to_json()
'null'

Note

The attributes of each field for containers nested in the container are viewed as well (chained method call).

List Field Items

A container can list all its field items nested in the container as a flatten list in the form of ('field path', field item) tuples by calling its method field_items().

>>> # List the field items in the container.
>>> container.field_items()
[]

List Field Attributes

A container can list the attributes of each field item nested in the container as a flatten list in the form of ('field path', attribute) or ('field path', list(attributes)) tuples by calling its method to_list(). Default attribute is the field value.

>>> # List the field values in the container.
>>> container.to_list()
[]

A container can list the attributes of each field item nested in the container as a flatten ordered dictionary in the form of {'field path': attribute} or {'field path': list(attributes)} pairs by calling its method to_dict(). Default attribute is the field value.

>>> # List the field values in the container.
>>> container.to_dict()
OrderedDict([('Container', OrderedDict())])

A container can list the attributes of each field item nested in the container as a flatten list of dictionaries containing the field path and the selected field attributes by calling its method to_csv(). Default attribute is the field value.

>>> # List the field values in the container.
>>> container.to_csv()
[]

Note

The class name of the instance is used for the root name as long as no name is given.

Write Field Attributes

A container can write the attributes of each field item nested in the container to a .csv file by calling its method write_csv(). Default attribute is the field value.

>>> # Save the field values to an '.csv' file.
>>> container.write_csv("_static/container.csv")

The generated .csv file for the container looks like this:

id,value

Note

The class name of the instance is used for the root name as long as no name is given.

Save Field Attributes

A container can save the attributes of each field item nested in the container to an .ini file by calling its method save(). Default attribute is the field value.

>>> # Save the field values to an '.ini' file.
>>> container.save("_static/container.ini")

The generated .ini file for the container looks like this:

[Container]

Note

The class name of the instance is used for the section name as long as no section is given.

Load Field Values

A container can load the value of each field item nested in the container from an .ini file by calling its method load().

>>> # Load the field values from an '.ini' file.
>>> container.load("_static/container.ini")
[Container]

Note

The class name of the instance is used for the section name as long as no section is given.

Fields

The role of a Field is to map a specific content area of a byte stream. A field is always placed in a container except from a pointer field which is the entry point for a mapper to connect the attached data object via a byte stream provider to a data source to retrieve the required byte stream for the mapper.

>>> # Create a field.
>>> field = Field()
>>> # Display the field.
>>> field
Field(index=Index(byte=0, bit=0,
                  address=0, base_address=0,
                  update=False),
      alignment=Alignment(byte_size=0, bit_offset=0),
      bit_size=0,
      value=None)

Overview

Here is an overview of the different available field classes.

• Stream, String
• Float, Double
• Decimal Bit, Byte, Char, Signed, Unsigned, Bitset, Bool, Enum, Scaled, Fraction, Bipolar, Unipolar, Datetime, IPv4Address
• Pointer, StructurePointer, SequencePointer, ArrayPointerStreamPointer, StringPointer, AutoStringPointer
• RelativePointer, StructureRelativePointer, SequenceRelativePointer, ArrayRelativePointer, StreamRelativePointer, StringRelativePointer

Name

A field has a type name. The field name consists of the name of the field base class and its field size to describe the kind of the field.

>>> # Field name.
>>> field.name
'Field0'

Size

A field has a bit_size. The field size defines the size of the content area of a byte stream that the field map.

>>> # Field bit size.
>>> field.bit_size
0

Value

A field has a value. The field value represents the content area of a byte stream that the field map.

>>> # Field value.
>>> field.value

Index

A field has an index. The field index contains the location of the field in a byte stream and in the providing data source. The field index is automatically calculated by the built-in deserializer and serializer from the start point of the byte stream and the start address of the byte stream in the providing data source.

>>> # Field index.
>>> field.index
Index(byte=0, bit=0, address=0, base_address=0, update=False)
>>> # Field index: byte offset of the field in the byte stream.
>>> field.index.byte
0
>>> # Field index: bit offset of the field relative to its byte offset.
>>> field.index.bit
0
>>> # Field index: memory address of the field in the data source.
>>> field.index.address
0
>>> # Field index: start address of the byte stream in the data source.
>>> field.index.base_address
0
>>> # Field index: update request for the byte stream.
>>> field.index.update
False

Alignment

A field has an alignment. The field alignment contains the location of the field within an aligned group of consecutive fields. The order how the consecutive fields are declared in a container defines the order how the consecutive fields are aligned to each other. The bit offset of the field alignment is automatically calculated by the built-in deserializer and serializer.

>>> # Field alignment.
>>> field.alignment
Alignment(byte_size=0, bit_offset=0)
>>> byte_size, bit_offset = field.alignment
>>> # Field alignment: byte size of the aligned field group.
>>> byte_size
0
>>> # Field alignment: bit offset of the field in its field group.
>>> bit_offset
0

A field can be aligned to a group of consecutive fields by using the align_to argument of the Field class to describe an atomic content part of a byte stream with more than one field.

>>> Decimal(15).alignment
Alignment(byte_size=2, bit_offset=0)
>>> Bool(1, align_to=2).alignment
Alignment(byte_size=2, bit_offset=0)

Note

A field aligns it self to the next matching byte size when the field size matches not full bytes and no field alignment is given.

For example to describe an atomic 16-bit value in a byte stream with more than one field can be achieved like this:

>>> # Create an empty structure for the atomic 16-bit value.
>>> atomic = Structure()
>>> # Add field for the first 15 bits of an atomic 16-bit value.
>>> atomic.size = Decimal(15, 2)
>>> # Add field for the last bit of an atomic 16-bit value.
>>> atomic.flag = Bool(1, 2)
>>> # Index the fields of the atomic 16-bit value.
>>> atomic.index_fields()
Index(byte=2, bit=0, address=2, base_address=0, update=False)
>>> # Display alignment of the size field.
>>> atomic.size.alignment
Alignment(byte_size=2, bit_offset=0)
>>> # Display alignment of the flag field.
>>> atomic.flag.alignment
Alignment(byte_size=2, bit_offset=15)

Note

The field alignment works only for the Decimal field classes.

Byte order

A field defines its own decoding/encoding byte_order. The default field byte order is auto it means that the field use the byte order which the byte stream mapper defines to unpack and pack the required bytes and bits for its field value from and to the byte stream.

>>> # Field byte order.
>>> field.byte_order
Byteorder.auto = 'auto'
>>> # Field byte order value.
>>> field.byte_order.value
'auto'

Enumeration

The name instead of the value of an enumeration can be displayed with the Enum field class by assigning an Enumeration class to the Enum field.

For example to describe a 2-bit ambivalent enumeration by an Enum field can be achieved like this:

>>> # Define the enumeration class.
>>> class Validity(Enumeration):
...     error = 0
...     correct = 1
...     forced = 2
...     undefined = 3
>>> # Create an enum field and assign an enumeration to the field.
>>> ambivalent = Enum(2, enumeration=Validity)
>>> # Display the value of the field.
>>> ambivalent.value
'error'
>>> # Returns the field value as an integer.
>>> int(ambivalent)
0
>>> # Display the field.
>>> ambivalent
Enum(index=Index(byte=0, bit=0,
                 address=0, base_address=0,
                 update=False),
     alignment=Alignment(byte_size=1, bit_offset=0),
     bit_size=2,
     value='error')