github.com/mh-cbon/go@v0.0.0-20160603070303-9e112a3fe4c0/src/encoding/gob/doc.go (about)

     1  // Copyright 2009 The Go Authors. All rights reserved.
     2  // Use of this source code is governed by a BSD-style
     3  // license that can be found in the LICENSE file.
     4  
     5  /*
     6  Package gob manages streams of gobs - binary values exchanged between an
     7  Encoder (transmitter) and a Decoder (receiver).  A typical use is transporting
     8  arguments and results of remote procedure calls (RPCs) such as those provided by
     9  package "net/rpc".
    10  
    11  The implementation compiles a custom codec for each data type in the stream and
    12  is most efficient when a single Encoder is used to transmit a stream of values,
    13  amortizing the cost of compilation.
    14  
    15  Basics
    16  
    17  A stream of gobs is self-describing.  Each data item in the stream is preceded by
    18  a specification of its type, expressed in terms of a small set of predefined
    19  types.  Pointers are not transmitted, but the things they point to are
    20  transmitted; that is, the values are flattened.  Recursive types work fine, but
    21  recursive values (data with cycles) are problematic.  This may change.
    22  
    23  To use gobs, create an Encoder and present it with a series of data items as
    24  values or addresses that can be dereferenced to values.  The Encoder makes sure
    25  all type information is sent before it is needed.  At the receive side, a
    26  Decoder retrieves values from the encoded stream and unpacks them into local
    27  variables.
    28  
    29  Types and Values
    30  
    31  The source and destination values/types need not correspond exactly.  For structs,
    32  fields (identified by name) that are in the source but absent from the receiving
    33  variable will be ignored.  Fields that are in the receiving variable but missing
    34  from the transmitted type or value will be ignored in the destination.  If a field
    35  with the same name is present in both, their types must be compatible. Both the
    36  receiver and transmitter will do all necessary indirection and dereferencing to
    37  convert between gobs and actual Go values.  For instance, a gob type that is
    38  schematically,
    39  
    40  	struct { A, B int }
    41  
    42  can be sent from or received into any of these Go types:
    43  
    44  	struct { A, B int }	// the same
    45  	*struct { A, B int }	// extra indirection of the struct
    46  	struct { *A, **B int }	// extra indirection of the fields
    47  	struct { A, B int64 }	// different concrete value type; see below
    48  
    49  It may also be received into any of these:
    50  
    51  	struct { A, B int }	// the same
    52  	struct { B, A int }	// ordering doesn't matter; matching is by name
    53  	struct { A, B, C int }	// extra field (C) ignored
    54  	struct { B int }	// missing field (A) ignored; data will be dropped
    55  	struct { B, C int }	// missing field (A) ignored; extra field (C) ignored.
    56  
    57  Attempting to receive into these types will draw a decode error:
    58  
    59  	struct { A int; B uint }	// change of signedness for B
    60  	struct { A int; B float }	// change of type for B
    61  	struct { }			// no field names in common
    62  	struct { C, D int }		// no field names in common
    63  
    64  Integers are transmitted two ways: arbitrary precision signed integers or
    65  arbitrary precision unsigned integers.  There is no int8, int16 etc.
    66  discrimination in the gob format; there are only signed and unsigned integers.  As
    67  described below, the transmitter sends the value in a variable-length encoding;
    68  the receiver accepts the value and stores it in the destination variable.
    69  Floating-point numbers are always sent using IEEE-754 64-bit precision (see
    70  below).
    71  
    72  Signed integers may be received into any signed integer variable: int, int16, etc.;
    73  unsigned integers may be received into any unsigned integer variable; and floating
    74  point values may be received into any floating point variable.  However,
    75  the destination variable must be able to represent the value or the decode
    76  operation will fail.
    77  
    78  Structs, arrays and slices are also supported. Structs encode and decode only
    79  exported fields. Strings and arrays of bytes are supported with a special,
    80  efficient representation (see below). When a slice is decoded, if the existing
    81  slice has capacity the slice will be extended in place; if not, a new array is
    82  allocated. Regardless, the length of the resulting slice reports the number of
    83  elements decoded.
    84  
    85  In general, if allocation is required, the decoder will allocate memory. If not,
    86  it will update the destination variables with values read from the stream. It does
    87  not initialize them first, so if the destination is a compound value such as a
    88  map, struct, or slice, the decoded values will be merged elementwise into the
    89  existing variables.
    90  
    91  Functions and channels will not be sent in a gob. Attempting to encode such a value
    92  at the top level will fail. A struct field of chan or func type is treated exactly
    93  like an unexported field and is ignored.
    94  
    95  Gob can encode a value of any type implementing the GobEncoder or
    96  encoding.BinaryMarshaler interfaces by calling the corresponding method,
    97  in that order of preference.
    98  
    99  Gob can decode a value of any type implementing the GobDecoder or
   100  encoding.BinaryUnmarshaler interfaces by calling the corresponding method,
   101  again in that order of preference.
   102  
   103  Encoding Details
   104  
   105  This section documents the encoding, details that are not important for most
   106  users. Details are presented bottom-up.
   107  
   108  An unsigned integer is sent one of two ways.  If it is less than 128, it is sent
   109  as a byte with that value.  Otherwise it is sent as a minimal-length big-endian
   110  (high byte first) byte stream holding the value, preceded by one byte holding the
   111  byte count, negated.  Thus 0 is transmitted as (00), 7 is transmitted as (07) and
   112  256 is transmitted as (FE 01 00).
   113  
   114  A boolean is encoded within an unsigned integer: 0 for false, 1 for true.
   115  
   116  A signed integer, i, is encoded within an unsigned integer, u.  Within u, bits 1
   117  upward contain the value; bit 0 says whether they should be complemented upon
   118  receipt.  The encode algorithm looks like this:
   119  
   120  	var u uint
   121  	if i < 0 {
   122  		u = (^uint(i) << 1) | 1 // complement i, bit 0 is 1
   123  	} else {
   124  		u = (uint(i) << 1) // do not complement i, bit 0 is 0
   125  	}
   126  	encodeUnsigned(u)
   127  
   128  The low bit is therefore analogous to a sign bit, but making it the complement bit
   129  instead guarantees that the largest negative integer is not a special case.  For
   130  example, -129=^128=(^256>>1) encodes as (FE 01 01).
   131  
   132  Floating-point numbers are always sent as a representation of a float64 value.
   133  That value is converted to a uint64 using math.Float64bits.  The uint64 is then
   134  byte-reversed and sent as a regular unsigned integer.  The byte-reversal means the
   135  exponent and high-precision part of the mantissa go first.  Since the low bits are
   136  often zero, this can save encoding bytes.  For instance, 17.0 is encoded in only
   137  three bytes (FE 31 40).
   138  
   139  Strings and slices of bytes are sent as an unsigned count followed by that many
   140  uninterpreted bytes of the value.
   141  
   142  All other slices and arrays are sent as an unsigned count followed by that many
   143  elements using the standard gob encoding for their type, recursively.
   144  
   145  Maps are sent as an unsigned count followed by that many key, element
   146  pairs. Empty but non-nil maps are sent, so if the receiver has not allocated
   147  one already, one will always be allocated on receipt unless the transmitted map
   148  is nil and not at the top level.
   149  
   150  In slices and arrays, as well as maps, all elements, even zero-valued elements,
   151  are transmitted, even if all the elements are zero.
   152  
   153  Structs are sent as a sequence of (field number, field value) pairs.  The field
   154  value is sent using the standard gob encoding for its type, recursively.  If a
   155  field has the zero value for its type (except for arrays; see above), it is omitted
   156  from the transmission.  The field number is defined by the type of the encoded
   157  struct: the first field of the encoded type is field 0, the second is field 1,
   158  etc.  When encoding a value, the field numbers are delta encoded for efficiency
   159  and the fields are always sent in order of increasing field number; the deltas are
   160  therefore unsigned.  The initialization for the delta encoding sets the field
   161  number to -1, so an unsigned integer field 0 with value 7 is transmitted as unsigned
   162  delta = 1, unsigned value = 7 or (01 07).  Finally, after all the fields have been
   163  sent a terminating mark denotes the end of the struct.  That mark is a delta=0
   164  value, which has representation (00).
   165  
   166  Interface types are not checked for compatibility; all interface types are
   167  treated, for transmission, as members of a single "interface" type, analogous to
   168  int or []byte - in effect they're all treated as interface{}.  Interface values
   169  are transmitted as a string identifying the concrete type being sent (a name
   170  that must be pre-defined by calling Register), followed by a byte count of the
   171  length of the following data (so the value can be skipped if it cannot be
   172  stored), followed by the usual encoding of concrete (dynamic) value stored in
   173  the interface value.  (A nil interface value is identified by the empty string
   174  and transmits no value.) Upon receipt, the decoder verifies that the unpacked
   175  concrete item satisfies the interface of the receiving variable.
   176  
   177  The representation of types is described below.  When a type is defined on a given
   178  connection between an Encoder and Decoder, it is assigned a signed integer type
   179  id.  When Encoder.Encode(v) is called, it makes sure there is an id assigned for
   180  the type of v and all its elements and then it sends the pair (typeid, encoded-v)
   181  where typeid is the type id of the encoded type of v and encoded-v is the gob
   182  encoding of the value v.
   183  
   184  To define a type, the encoder chooses an unused, positive type id and sends the
   185  pair (-type id, encoded-type) where encoded-type is the gob encoding of a wireType
   186  description, constructed from these types:
   187  
   188  	type wireType struct {
   189  		ArrayT  *ArrayType
   190  		SliceT  *SliceType
   191  		StructT *StructType
   192  		MapT    *MapType
   193  	}
   194  	type arrayType struct {
   195  		CommonType
   196  		Elem typeId
   197  		Len  int
   198  	}
   199  	type CommonType struct {
   200  		Name string // the name of the struct type
   201  		Id  int    // the id of the type, repeated so it's inside the type
   202  	}
   203  	type sliceType struct {
   204  		CommonType
   205  		Elem typeId
   206  	}
   207  	type structType struct {
   208  		CommonType
   209  		Field []*fieldType // the fields of the struct.
   210  	}
   211  	type fieldType struct {
   212  		Name string // the name of the field.
   213  		Id   int    // the type id of the field, which must be already defined
   214  	}
   215  	type mapType struct {
   216  		CommonType
   217  		Key  typeId
   218  		Elem typeId
   219  	}
   220  
   221  If there are nested type ids, the types for all inner type ids must be defined
   222  before the top-level type id is used to describe an encoded-v.
   223  
   224  For simplicity in setup, the connection is defined to understand these types a
   225  priori, as well as the basic gob types int, uint, etc.  Their ids are:
   226  
   227  	bool        1
   228  	int         2
   229  	uint        3
   230  	float       4
   231  	[]byte      5
   232  	string      6
   233  	complex     7
   234  	interface   8
   235  	// gap for reserved ids.
   236  	WireType    16
   237  	ArrayType   17
   238  	CommonType  18
   239  	SliceType   19
   240  	StructType  20
   241  	FieldType   21
   242  	// 22 is slice of fieldType.
   243  	MapType     23
   244  
   245  Finally, each message created by a call to Encode is preceded by an encoded
   246  unsigned integer count of the number of bytes remaining in the message.  After
   247  the initial type name, interface values are wrapped the same way; in effect, the
   248  interface value acts like a recursive invocation of Encode.
   249  
   250  In summary, a gob stream looks like
   251  
   252  	(byteCount (-type id, encoding of a wireType)* (type id, encoding of a value))*
   253  
   254  where * signifies zero or more repetitions and the type id of a value must
   255  be predefined or be defined before the value in the stream.
   256  
   257  Compatibility: Any future changes to the package will endeavor to maintain
   258  compatibility with streams encoded using previous versions.  That is, any released
   259  version of this package should be able to decode data written with any previously
   260  released version, subject to issues such as security fixes. See the Go compatibility
   261  document for background: https://golang.org/doc/go1compat
   262  
   263  See "Gobs of data" for a design discussion of the gob wire format:
   264  https://blog.golang.org/gobs-of-data
   265  */
   266  package gob
   267  
   268  /*
   269  Grammar:
   270  
   271  Tokens starting with a lower case letter are terminals; int(n)
   272  and uint(n) represent the signed/unsigned encodings of the value n.
   273  
   274  GobStream:
   275  	DelimitedMessage*
   276  DelimitedMessage:
   277  	uint(lengthOfMessage) Message
   278  Message:
   279  	TypeSequence TypedValue
   280  TypeSequence
   281  	(TypeDefinition DelimitedTypeDefinition*)?
   282  DelimitedTypeDefinition:
   283  	uint(lengthOfTypeDefinition) TypeDefinition
   284  TypedValue:
   285  	int(typeId) Value
   286  TypeDefinition:
   287  	int(-typeId) encodingOfWireType
   288  Value:
   289  	SingletonValue | StructValue
   290  SingletonValue:
   291  	uint(0) FieldValue
   292  FieldValue:
   293  	builtinValue | ArrayValue | MapValue | SliceValue | StructValue | InterfaceValue
   294  InterfaceValue:
   295  	NilInterfaceValue | NonNilInterfaceValue
   296  NilInterfaceValue:
   297  	uint(0)
   298  NonNilInterfaceValue:
   299  	ConcreteTypeName TypeSequence InterfaceContents
   300  ConcreteTypeName:
   301  	uint(lengthOfName) [already read=n] name
   302  InterfaceContents:
   303  	int(concreteTypeId) DelimitedValue
   304  DelimitedValue:
   305  	uint(length) Value
   306  ArrayValue:
   307  	uint(n) FieldValue*n [n elements]
   308  MapValue:
   309  	uint(n) (FieldValue FieldValue)*n  [n (key, value) pairs]
   310  SliceValue:
   311  	uint(n) FieldValue*n [n elements]
   312  StructValue:
   313  	(uint(fieldDelta) FieldValue)*
   314  */
   315  
   316  /*
   317  For implementers and the curious, here is an encoded example.  Given
   318  	type Point struct {X, Y int}
   319  and the value
   320  	p := Point{22, 33}
   321  the bytes transmitted that encode p will be:
   322  	1f ff 81 03 01 01 05 50 6f 69 6e 74 01 ff 82 00
   323  	01 02 01 01 58 01 04 00 01 01 59 01 04 00 00 00
   324  	07 ff 82 01 2c 01 42 00
   325  They are determined as follows.
   326  
   327  Since this is the first transmission of type Point, the type descriptor
   328  for Point itself must be sent before the value.  This is the first type
   329  we've sent on this Encoder, so it has type id 65 (0 through 64 are
   330  reserved).
   331  
   332  	1f	// This item (a type descriptor) is 31 bytes long.
   333  	ff 81	// The negative of the id for the type we're defining, -65.
   334  		// This is one byte (indicated by FF = -1) followed by
   335  		// ^-65<<1 | 1.  The low 1 bit signals to complement the
   336  		// rest upon receipt.
   337  
   338  	// Now we send a type descriptor, which is itself a struct (wireType).
   339  	// The type of wireType itself is known (it's built in, as is the type of
   340  	// all its components), so we just need to send a *value* of type wireType
   341  	// that represents type "Point".
   342  	// Here starts the encoding of that value.
   343  	// Set the field number implicitly to -1; this is done at the beginning
   344  	// of every struct, including nested structs.
   345  	03	// Add 3 to field number; now 2 (wireType.structType; this is a struct).
   346  		// structType starts with an embedded CommonType, which appears
   347  		// as a regular structure here too.
   348  	01	// add 1 to field number (now 0); start of embedded CommonType.
   349  	01	// add 1 to field number (now 0, the name of the type)
   350  	05	// string is (unsigned) 5 bytes long
   351  	50 6f 69 6e 74	// wireType.structType.CommonType.name = "Point"
   352  	01	// add 1 to field number (now 1, the id of the type)
   353  	ff 82	// wireType.structType.CommonType._id = 65
   354  	00	// end of embedded wiretype.structType.CommonType struct
   355  	01	// add 1 to field number (now 1, the field array in wireType.structType)
   356  	02	// There are two fields in the type (len(structType.field))
   357  	01	// Start of first field structure; add 1 to get field number 0: field[0].name
   358  	01	// 1 byte
   359  	58	// structType.field[0].name = "X"
   360  	01	// Add 1 to get field number 1: field[0].id
   361  	04	// structType.field[0].typeId is 2 (signed int).
   362  	00	// End of structType.field[0]; start structType.field[1]; set field number to -1.
   363  	01	// Add 1 to get field number 0: field[1].name
   364  	01	// 1 byte
   365  	59	// structType.field[1].name = "Y"
   366  	01	// Add 1 to get field number 1: field[1].id
   367  	04	// struct.Type.field[1].typeId is 2 (signed int).
   368  	00	// End of structType.field[1]; end of structType.field.
   369  	00	// end of wireType.structType structure
   370  	00	// end of wireType structure
   371  
   372  Now we can send the Point value.  Again the field number resets to -1:
   373  
   374  	07	// this value is 7 bytes long
   375  	ff 82	// the type number, 65 (1 byte (-FF) followed by 65<<1)
   376  	01	// add one to field number, yielding field 0
   377  	2c	// encoding of signed "22" (0x22 = 44 = 22<<1); Point.x = 22
   378  	01	// add one to field number, yielding field 1
   379  	42	// encoding of signed "33" (0x42 = 66 = 33<<1); Point.y = 33
   380  	00	// end of structure
   381  
   382  The type encoding is long and fairly intricate but we send it only once.
   383  If p is transmitted a second time, the type is already known so the
   384  output will be just:
   385  
   386  	07 ff 82 01 2c 01 42 00
   387  
   388  A single non-struct value at top level is transmitted like a field with
   389  delta tag 0.  For instance, a signed integer with value 3 presented as
   390  the argument to Encode will emit:
   391  
   392  	03 04 00 06
   393  
   394  Which represents:
   395  
   396  	03	// this value is 3 bytes long
   397  	04	// the type number, 2, represents an integer
   398  	00	// tag delta 0
   399  	06	// value 3
   400  
   401  */