github.com/peterbourgon/diskv@v2.0.1+incompatible/README.md (about)

     1  # What is diskv?
     2  
     3  Diskv (disk-vee) is a simple, persistent key-value store written in the Go
     4  language. It starts with an incredibly simple API for storing arbitrary data on
     5  a filesystem by key, and builds several layers of performance-enhancing
     6  abstraction on top.  The end result is a conceptually simple, but highly
     7  performant, disk-backed storage system.
     8  
     9  [![Build Status][1]][2]
    10  
    11  [1]: https://drone.io/github.com/peterbourgon/diskv/status.png
    12  [2]: https://drone.io/github.com/peterbourgon/diskv/latest
    13  
    14  
    15  # Installing
    16  
    17  Install [Go 1][3], either [from source][4] or [with a prepackaged binary][5].
    18  Then,
    19  
    20  ```bash
    21  $ go get github.com/peterbourgon/diskv
    22  ```
    23  
    24  [3]: http://golang.org
    25  [4]: http://golang.org/doc/install/source
    26  [5]: http://golang.org/doc/install
    27  
    28  
    29  # Usage
    30  
    31  ```go
    32  package main
    33  
    34  import (
    35  	"fmt"
    36  	"github.com/peterbourgon/diskv"
    37  )
    38  
    39  func main() {
    40  	// Simplest transform function: put all the data files into the base dir.
    41  	flatTransform := func(s string) []string { return []string{} }
    42  
    43  	// Initialize a new diskv store, rooted at "my-data-dir", with a 1MB cache.
    44  	d := diskv.New(diskv.Options{
    45  		BasePath:     "my-data-dir",
    46  		Transform:    flatTransform,
    47  		CacheSizeMax: 1024 * 1024,
    48  	})
    49  
    50  	// Write three bytes to the key "alpha".
    51  	key := "alpha"
    52  	d.Write(key, []byte{'1', '2', '3'})
    53  
    54  	// Read the value back out of the store.
    55  	value, _ := d.Read(key)
    56  	fmt.Printf("%v\n", value)
    57  
    58  	// Erase the key+value from the store (and the disk).
    59  	d.Erase(key)
    60  }
    61  ```
    62  
    63  More complex examples can be found in the "examples" subdirectory.
    64  
    65  
    66  # Theory
    67  
    68  ## Basic idea
    69  
    70  At its core, diskv is a map of a key (`string`) to arbitrary data (`[]byte`).
    71  The data is written to a single file on disk, with the same name as the key.
    72  The key determines where that file will be stored, via a user-provided
    73  `TransformFunc`, which takes a key and returns a slice (`[]string`)
    74  corresponding to a path list where the key file will be stored. The simplest
    75  TransformFunc,
    76  
    77  ```go
    78  func SimpleTransform (key string) []string {
    79      return []string{}
    80  }
    81  ```
    82  
    83  will place all keys in the same, base directory. The design is inspired by
    84  [Redis diskstore][6]; a TransformFunc which emulates the default diskstore
    85  behavior is available in the content-addressable-storage example.
    86  
    87  [6]: http://groups.google.com/group/redis-db/browse_thread/thread/d444bc786689bde9?pli=1
    88  
    89  **Note** that your TransformFunc should ensure that one valid key doesn't
    90  transform to a subset of another valid key. That is, it shouldn't be possible
    91  to construct valid keys that resolve to directory names. As a concrete example,
    92  if your TransformFunc splits on every 3 characters, then
    93  
    94  ```go
    95  d.Write("abcabc", val) // OK: written to <base>/abc/abc/abcabc
    96  d.Write("abc", val)    // Error: attempted write to <base>/abc/abc, but it's a directory
    97  ```
    98  
    99  This will be addressed in an upcoming version of diskv.
   100  
   101  Probably the most important design principle behind diskv is that your data is
   102  always flatly available on the disk. diskv will never do anything that would
   103  prevent you from accessing, copying, backing up, or otherwise interacting with
   104  your data via common UNIX commandline tools.
   105  
   106  ## Adding a cache
   107  
   108  An in-memory caching layer is provided by combining the BasicStore
   109  functionality with a simple map structure, and keeping it up-to-date as
   110  appropriate. Since the map structure in Go is not threadsafe, it's combined
   111  with a RWMutex to provide safe concurrent access.
   112  
   113  ## Adding order
   114  
   115  diskv is a key-value store and therefore inherently unordered. An ordering
   116  system can be injected into the store by passing something which satisfies the
   117  diskv.Index interface. (A default implementation, using Google's
   118  [btree][7] package, is provided.) Basically, diskv keeps an ordered (by a
   119  user-provided Less function) index of the keys, which can be queried.
   120  
   121  [7]: https://github.com/google/btree
   122  
   123  ## Adding compression
   124  
   125  Something which implements the diskv.Compression interface may be passed
   126  during store creation, so that all Writes and Reads are filtered through
   127  a compression/decompression pipeline. Several default implementations,
   128  using stdlib compression algorithms, are provided. Note that data is cached
   129  compressed; the cost of decompression is borne with each Read.
   130  
   131  ## Streaming
   132  
   133  diskv also now provides ReadStream and WriteStream methods, to allow very large
   134  data to be handled efficiently.
   135  
   136  
   137  # Future plans
   138  
   139   * Needs plenty of robust testing: huge datasets, etc...
   140   * More thorough benchmarking
   141   * Your suggestions for use-cases I haven't thought of