github.com/viant/toolbox@v0.34.5/data/README.md (about)

     1  # Data utilities
     2  
     3  
     4  ### Expandable Map & Collection
     5  
     6  Expandable structure enable nested data structure to substitution source for other data structure or text.
     7  Data substitution expression starts with $ sign, you can use path where dot or [index] allows access data sub node.
     8  
     9  - [ExpandAsText](#ExpandAsText)
    10  - [Expand](#Expand)
    11  
    12  
    13  ### ExpandAsText
    14  
    15  ExpandAsText expands any expression that has satisfied dependencies, meaning only expression that path is present
    16  can be expanded, otherwise expression is left unchanged.
    17  In case when UDF is used, expression is expanded if UDF does not return an error.
    18  
    19  **Usage:**
    20  
    21  ```go
    22      aMap := Map(map[string]interface{}{
    23  		"key1": 1,
    24  		"key2": map[string]interface{}{
    25  			"subKey1":10,
    26  			"subKey2":20,
    27  		},
    28  		"key3": "subKey2",
    29  		"array": []interface{}{
    30  			111, 222, 333,
    31  		},
    32  		"slice": []interface{}{
    33  			map[string]interface{}{
    34  				"attr1":111,
    35  				"attr2":222,
    36  			},
    37  		},
    38  	})
    39  	expandedText := aMap.ExpandAsText(`1: $key1, 
    40  2: ${array[2]}  
    41  3: $key2.subKey1 
    42  4: $key2[$key3] ${slice[0].attr1}  
    43  5: ${(key1 + 1) * 3} 
    44  6: $abc
    45  7: end
    46  `)
    47  	
    48  	
    49  /* expands to 
    50  1: 1, 
    51  2: 333  
    52  3: 10 
    53  4: 20 111  
    54  5: 6 
    55  6: $abc
    56  7: end
    57  */
    58  ```
    59  
    60  ## Expand arbitrary data structure
    61  
    62  
    63  ```go
    64      aMap := Map(map[string]interface{}{
    65  		"key1": 1,
    66  		"key2": map[string]interface{}{
    67  			"subKey1":10,
    68  			"subKey2":20,
    69  		},
    70  		"key3": "subKey2",
    71  		"array": []interface{}{
    72  			111, 222, 333,
    73  		},
    74  		"slice": []interface{}{
    75  			map[string]interface{}{
    76  				"attr1":111,
    77  				"attr2":222,
    78  			},
    79  		},
    80  	})
    81      
    82      data := map[string]interface{}{
    83      	"k1": "$key1",
    84      	"k2":"$array",
    85      	"k3":"$key2",
    86      }
    87      expanded := aMap.Expand(data)
    88      /* expands to
    89          map[string]interface{}{
    90          	"k1": 1,
    91          	"k2": []interface{}{111, 222, 333},
    92          	"k3": map[string]interface{}{
    93                  "subKey1":10,
    94                  "subKey2":20,
    95              },
    96          }
    97      */
    98      
    99  ```
   100  
   101  
   102  # UDF expandable User defined function
   103  
   104  You can add dynamic data substitution by registering function in top level map.
   105  
   106  ```go
   107  type Udf func(interface{}, Map) (interface{}, error)
   108  ```
   109  
   110  ```go
   111          aMap: data.Map(map[string]interface{}{
   112              "dailyCap":   100,
   113              "overallCap": 2,
   114              "AsFloat": func(source interface{}, state Map) (interface{}, error) {
   115                  return toolbox.AsFloat(source), nil
   116              },
   117          })
   118  
   119          expanded := aMap.Expand("$AsFloat($dailyCap)")
   120          //expands to actual float: 100.0
   121  
   122  ```
   123  
   124  [Predefined UDF](udf)
   125  
   126  
   127  ### Compacted slice
   128  
   129  Using a generic data structure in a form []map[string]interface{} is extremely memory inefficient, 
   130  CompactedSlice addresses the memory inefficiency by storing the new item values in a slice 
   131  and by mapping corresponding fields to the item slice index positions. 
   132  On top of that any neighboring nil values can be compacted too. 
   133  
   134  
   135  **Usage**
   136  
   137  ```go
   138  
   139      collection := NewCompactedSlice(true, true)
   140  
   141      for i := 0;i<10;i++ {
   142          collection.Add(map[string]interface{}{
   143              "f1":  i+1,
   144              "f12": i+10,
   145              "f15": i*20,
   146              "f20": i+4,
   147              "f11": nil,
   148              "f12": nil,
   149              "f13": nil,
   150              "f14": "",
   151          })
   152  	}
   153      
   154      collection.Range(func(data interface{}) (bool, error) {
   155          actual = append(actual, toolbox.AsMap(data))
   156          return true, nil
   157      })
   158      
   159      
   160   
   161  ```