Document as Binary (DBN)
A considerable amount of time is spent to create and improve formats and protocols. There are likely more then a thousand which can be classify in two groups :
- specialized formats
Specialized formats have been made for a single purpose or application, they require some special decoding algorithm which will require more or less efforts to implement and constant work to keep up to date.
- derivated formats
Derivated formats are based on corner-stone formats like XML or JSON. They are easy to decode with the moddern tools we have and any developer can work with them, so they have a much lower maintenance cost. But they can never be a perfect for for all needs, a simple argument could be XML size and parsing time or json lack of data types.
There has been some attempts to achieve a new corner-stone format. The current best solutions ares ProtoBuffer and FlatBuffer, and yet they have not been made to give a solution for all needs, only for simple preformated needs.
What about DBN ?
DBN is the sum of all my experience in decoding/understanding over a hundred differents formats, syntaxes and protocols.
- Some really primitive like : INI,CSV ...
- some very old : BMP, XMAP, PBM ...
- some scientific : Grib, Netcdf ...
- some proprietary : DXF, MDB ...
- some military : IHO S-57, VPF ...
- some cryptic : IHO S-63, DWG ...
Old encodings, brain-fucking compressions, obsolete encryptions, half binary/half text. And so on...
All formats have unique caracteristics which can be resumed in key features :
- easy to decode and encode
- self describing
- programming language-free
- in place rewrite
DBN is a possible solution combining all the above requirements.
Document model : like Json and Yaml, DBN uses a document model, which define fields with types, attributes, arrays, sub-documents. Anything Json can do, so can DBN.
Self described : rare are the formats which can describe there structure whithin the file, XML with DTD/XSD or Blender DNA are among them. DBN use a similar approach as Blender, a header in the document defines the type of the following document, but DBN also inherit from XSD because the document structure definition is itself a DBN document just like XSD in written in XML. Like XSD the document type can be referenced with an URI, this allows to reduce the used space.
Binary : DBN aims to become a versatile storage and transfert format. The binary encoding allows a very compact form for high efficiency. In fact several todays format could be parsed using a DBN decoder with a pre-defined document type. The document model can be controled bit per bit to fit exactly the required needs.
Compressed : full or partial compression of the document is possible. This particularity is highly important for web transfer and storage. Unlike many formats the compression algorithm is not part of the format itself, both streaming or full-block compression modes are possible. The used algorithms are not finite and can be added without changing the format at all, this allows for smooth futur evolutions.
Encrypted : like compression, full or partial encryption of the document is possible, it is possible to stack several layers of encryption and compression in any order. Such capabilities are requiered for strategic exchange formats such as IHO S-63. And again the used algorithms are not finite and can be added without changing the format at all.
Streamed : writing very large files of several Gigabits require for formats to support streaming, which means writing in one flow, never going back in the stream to write block sizes, checksums or indexes. Audio and video formats, Grib and Netcdf science formats also have such capabilities, but are awfully complex compared to DBN.
Modifiable : most formats require to rewrite the complete file each time a modification is made. DBN thanks to it's binary and streaming capabilities can support direct file modification. The better the document type is defined to more efficient the DBN reader/writer will be able to modify the file without rewriting it. For example an indexation file could be written with DBN, records could be removed, modified, recycled without resizing/rewriting the full file, new records could be added at the end.
Easy to code : Yet, even with all those capabilities the format is straight forward and easy to code.