github.com/sensu/afero@v1.1.1/README.md (about) 1 ![afero logo-sm](https://cloud.githubusercontent.com/assets/173412/11490338/d50e16dc-97a5-11e5-8b12-019a300d0fcb.png) 2 3 A FileSystem Abstraction System for Go 4 5 [![Build Status](https://travis-ci.org/spf13/afero.svg)](https://travis-ci.org/spf13/afero) [![Build status](https://ci.appveyor.com/api/projects/status/github/spf13/afero?branch=master&svg=true)](https://ci.appveyor.com/project/spf13/afero) [![GoDoc](https://godoc.org/github.com/spf13/afero?status.svg)](https://godoc.org/github.com/spf13/afero) [![Join the chat at https://gitter.im/spf13/afero](https://badges.gitter.im/Dev%20Chat.svg)](https://gitter.im/spf13/afero?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) 6 7 # Overview 8 9 Afero is an filesystem framework providing a simple, uniform and universal API 10 interacting with any filesystem, as an abstraction layer providing interfaces, 11 types and methods. Afero has an exceptionally clean interface and simple design 12 without needless constructors or initialization methods. 13 14 Afero is also a library providing a base set of interoperable backend 15 filesystems that make it easy to work with afero while retaining all the power 16 and benefit of the os and ioutil packages. 17 18 Afero provides significant improvements over using the os package alone, most 19 notably the ability to create mock and testing filesystems without relying on the disk. 20 21 It is suitable for use in a any situation where you would consider using the OS 22 package as it provides an additional abstraction that makes it easy to use a 23 memory backed file system during testing. It also adds support for the http 24 filesystem for full interoperability. 25 26 27 ## Afero Features 28 29 * A single consistent API for accessing a variety of filesystems 30 * Interoperation between a variety of file system types 31 * A set of interfaces to encourage and enforce interoperability between backends 32 * An atomic cross platform memory backed file system 33 * Support for compositional (union) file systems by combining multiple file systems acting as one 34 * Specialized backends which modify existing filesystems (Read Only, Regexp filtered) 35 * A set of utility functions ported from io, ioutil & hugo to be afero aware 36 37 38 # Using Afero 39 40 Afero is easy to use and easier to adopt. 41 42 A few different ways you could use Afero: 43 44 * Use the interfaces alone to define you own file system. 45 * Wrap for the OS packages. 46 * Define different filesystems for different parts of your application. 47 * Use Afero for mock filesystems while testing 48 49 ## Step 1: Install Afero 50 51 First use go get to install the latest version of the library. 52 53 $ go get github.com/spf13/afero 54 55 Next include Afero in your application. 56 ```go 57 import "github.com/spf13/afero" 58 ``` 59 60 ## Step 2: Declare a backend 61 62 First define a package variable and set it to a pointer to a filesystem. 63 ```go 64 var AppFs = afero.NewMemMapFs() 65 66 or 67 68 var AppFs = afero.NewOsFs() 69 ``` 70 It is important to note that if you repeat the composite literal you 71 will be using a completely new and isolated filesystem. In the case of 72 OsFs it will still use the same underlying filesystem but will reduce 73 the ability to drop in other filesystems as desired. 74 75 ## Step 3: Use it like you would the OS package 76 77 Throughout your application use any function and method like you normally 78 would. 79 80 So if my application before had: 81 ```go 82 os.Open('/tmp/foo') 83 ``` 84 We would replace it with: 85 ```go 86 AppFs.Open('/tmp/foo') 87 ``` 88 89 `AppFs` being the variable we defined above. 90 91 92 ## List of all available functions 93 94 File System Methods Available: 95 ```go 96 Chmod(name string, mode os.FileMode) : error 97 Chtimes(name string, atime time.Time, mtime time.Time) : error 98 Create(name string) : File, error 99 Mkdir(name string, perm os.FileMode) : error 100 MkdirAll(path string, perm os.FileMode) : error 101 Name() : string 102 Open(name string) : File, error 103 OpenFile(name string, flag int, perm os.FileMode) : File, error 104 Remove(name string) : error 105 RemoveAll(path string) : error 106 Rename(oldname, newname string) : error 107 Stat(name string) : os.FileInfo, error 108 ``` 109 File Interfaces and Methods Available: 110 ```go 111 io.Closer 112 io.Reader 113 io.ReaderAt 114 io.Seeker 115 io.Writer 116 io.WriterAt 117 118 Name() : string 119 Readdir(count int) : []os.FileInfo, error 120 Readdirnames(n int) : []string, error 121 Stat() : os.FileInfo, error 122 Sync() : error 123 Truncate(size int64) : error 124 WriteString(s string) : ret int, err error 125 ``` 126 In some applications it may make sense to define a new package that 127 simply exports the file system variable for easy access from anywhere. 128 129 ## Using Afero's utility functions 130 131 Afero provides a set of functions to make it easier to use the underlying file systems. 132 These functions have been primarily ported from io & ioutil with some developed for Hugo. 133 134 The afero utilities support all afero compatible backends. 135 136 The list of utilities includes: 137 138 ```go 139 DirExists(path string) (bool, error) 140 Exists(path string) (bool, error) 141 FileContainsBytes(filename string, subslice []byte) (bool, error) 142 GetTempDir(subPath string) string 143 IsDir(path string) (bool, error) 144 IsEmpty(path string) (bool, error) 145 ReadDir(dirname string) ([]os.FileInfo, error) 146 ReadFile(filename string) ([]byte, error) 147 SafeWriteReader(path string, r io.Reader) (err error) 148 TempDir(dir, prefix string) (name string, err error) 149 TempFile(dir, prefix string) (f File, err error) 150 Walk(root string, walkFn filepath.WalkFunc) error 151 WriteFile(filename string, data []byte, perm os.FileMode) error 152 WriteReader(path string, r io.Reader) (err error) 153 ``` 154 For a complete list see [Afero's GoDoc](https://godoc.org/github.com/spf13/afero) 155 156 They are available under two different approaches to use. You can either call 157 them directly where the first parameter of each function will be the file 158 system, or you can declare a new `Afero`, a custom type used to bind these 159 functions as methods to a given filesystem. 160 161 ### Calling utilities directly 162 163 ```go 164 fs := new(afero.MemMapFs) 165 f, err := afero.TempFile(fs,"", "ioutil-test") 166 167 ``` 168 169 ### Calling via Afero 170 171 ```go 172 fs := afero.NewMemMapFs() 173 afs := &afero.Afero{Fs: fs} 174 f, err := afs.TempFile("", "ioutil-test") 175 ``` 176 177 ## Using Afero for Testing 178 179 There is a large benefit to using a mock filesystem for testing. It has a 180 completely blank state every time it is initialized and can be easily 181 reproducible regardless of OS. You could create files to your heart’s content 182 and the file access would be fast while also saving you from all the annoying 183 issues with deleting temporary files, Windows file locking, etc. The MemMapFs 184 backend is perfect for testing. 185 186 * Much faster than performing I/O operations on disk 187 * Avoid security issues and permissions 188 * Far more control. 'rm -rf /' with confidence 189 * Test setup is far more easier to do 190 * No test cleanup needed 191 192 One way to accomplish this is to define a variable as mentioned above. 193 In your application this will be set to afero.NewOsFs() during testing you 194 can set it to afero.NewMemMapFs(). 195 196 It wouldn't be uncommon to have each test initialize a blank slate memory 197 backend. To do this I would define my `appFS = afero.NewOsFs()` somewhere 198 appropriate in my application code. This approach ensures that Tests are order 199 independent, with no test relying on the state left by an earlier test. 200 201 Then in my tests I would initialize a new MemMapFs for each test: 202 ```go 203 func TestExist(t *testing.T) { 204 appFS := afero.NewMemMapFs() 205 // create test files and directories 206 appFS.MkdirAll("src/a", 0755) 207 afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644) 208 afero.WriteFile(appFS, "src/c", []byte("file c"), 0644) 209 name := "src/c" 210 _, err := appFS.Stat(name) 211 if os.IsNotExist(err) { 212 t.Errorf("file \"%s\" does not exist.\n", name) 213 } 214 } 215 ``` 216 217 # Available Backends 218 219 ## Operating System Native 220 221 ### OsFs 222 223 The first is simply a wrapper around the native OS calls. This makes it 224 very easy to use as all of the calls are the same as the existing OS 225 calls. It also makes it trivial to have your code use the OS during 226 operation and a mock filesystem during testing or as needed. 227 228 ```go 229 appfs := afero.NewOsFs() 230 appfs.MkdirAll("src/a", 0755)) 231 ``` 232 233 ## Memory Backed Storage 234 235 ### MemMapFs 236 237 Afero also provides a fully atomic memory backed filesystem perfect for use in 238 mocking and to speed up unnecessary disk io when persistence isn’t 239 necessary. It is fully concurrent and will work within go routines 240 safely. 241 242 ```go 243 mm := afero.NewMemMapFs() 244 mm.MkdirAll("src/a", 0755)) 245 ``` 246 247 #### InMemoryFile 248 249 As part of MemMapFs, Afero also provides an atomic, fully concurrent memory 250 backed file implementation. This can be used in other memory backed file 251 systems with ease. Plans are to add a radix tree memory stored file 252 system using InMemoryFile. 253 254 ## Network Interfaces 255 256 ### SftpFs 257 258 Afero has experimental support for secure file transfer protocol (sftp). Which can 259 be used to perform file operations over a encrypted channel. 260 261 ## Filtering Backends 262 263 ### BasePathFs 264 265 The BasePathFs restricts all operations to a given path within an Fs. 266 The given file name to the operations on this Fs will be prepended with 267 the base path before calling the source Fs. 268 269 ```go 270 bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path") 271 ``` 272 273 ### ReadOnlyFs 274 275 A thin wrapper around the source Fs providing a read only view. 276 277 ```go 278 fs := afero.NewReadOnlyFs(afero.NewOsFs()) 279 _, err := fs.Create("/file.txt") 280 // err = syscall.EPERM 281 ``` 282 283 # RegexpFs 284 285 A filtered view on file names, any file NOT matching 286 the passed regexp will be treated as non-existing. 287 Files not matching the regexp provided will not be created. 288 Directories are not filtered. 289 290 ```go 291 fs := afero.NewRegexpFs(afero.NewMemMapFs(), regexp.MustCompile(`\.txt$`)) 292 _, err := fs.Create("/file.html") 293 // err = syscall.ENOENT 294 ``` 295 296 ### HttpFs 297 298 Afero provides an http compatible backend which can wrap any of the existing 299 backends. 300 301 The Http package requires a slightly specific version of Open which 302 returns an http.File type. 303 304 Afero provides an httpFs file system which satisfies this requirement. 305 Any Afero FileSystem can be used as an httpFs. 306 307 ```go 308 httpFs := afero.NewHttpFs(<ExistingFS>) 309 fileserver := http.FileServer(httpFs.Dir(<PATH>))) 310 http.Handle("/", fileserver) 311 ``` 312 313 ## Composite Backends 314 315 Afero provides the ability have two filesystems (or more) act as a single 316 file system. 317 318 ### CacheOnReadFs 319 320 The CacheOnReadFs will lazily make copies of any accessed files from the base 321 layer into the overlay. Subsequent reads will be pulled from the overlay 322 directly permitting the request is within the cache duration of when it was 323 created in the overlay. 324 325 If the base filesystem is writeable, any changes to files will be 326 done first to the base, then to the overlay layer. Write calls to open file 327 handles like `Write()` or `Truncate()` to the overlay first. 328 329 To writing files to the overlay only, you can use the overlay Fs directly (not 330 via the union Fs). 331 332 Cache files in the layer for the given time.Duration, a cache duration of 0 333 means "forever" meaning the file will not be re-requested from the base ever. 334 335 A read-only base will make the overlay also read-only but still copy files 336 from the base to the overlay when they're not present (or outdated) in the 337 caching layer. 338 339 ```go 340 base := afero.NewOsFs() 341 layer := afero.NewMemMapFs() 342 ufs := afero.NewCacheOnReadFs(base, layer, 100 * time.Second) 343 ``` 344 345 ### CopyOnWriteFs() 346 347 The CopyOnWriteFs is a read only base file system with a potentially 348 writeable layer on top. 349 350 Read operations will first look in the overlay and if not found there, will 351 serve the file from the base. 352 353 Changes to the file system will only be made in the overlay. 354 355 Any attempt to modify a file found only in the base will copy the file to the 356 overlay layer before modification (including opening a file with a writable 357 handle). 358 359 Removing and Renaming files present only in the base layer is not currently 360 permitted. If a file is present in the base layer and the overlay, only the 361 overlay will be removed/renamed. 362 363 ```go 364 base := afero.NewOsFs() 365 roBase := afero.NewReadOnlyFs(base) 366 ufs := afero.NewCopyOnWriteFs(roBase, afero.NewMemMapFs()) 367 368 fh, _ = ufs.Create("/home/test/file2.txt") 369 fh.WriteString("This is a test") 370 fh.Close() 371 ``` 372 373 In this example all write operations will only occur in memory (MemMapFs) 374 leaving the base filesystem (OsFs) untouched. 375 376 377 ## Desired/possible backends 378 379 The following is a short list of possible backends we hope someone will 380 implement: 381 382 * SSH 383 * ZIP 384 * TAR 385 * S3 386 387 # About the project 388 389 ## What's in the name 390 391 Afero comes from the latin roots Ad-Facere. 392 393 **"Ad"** is a prefix meaning "to". 394 395 **"Facere"** is a form of the root "faciō" making "make or do". 396 397 The literal meaning of afero is "to make" or "to do" which seems very fitting 398 for a library that allows one to make files and directories and do things with them. 399 400 The English word that shares the same roots as Afero is "affair". Affair shares 401 the same concept but as a noun it means "something that is made or done" or "an 402 object of a particular type". 403 404 It's also nice that unlike some of my other libraries (hugo, cobra, viper) it 405 Googles very well. 406 407 ## Release Notes 408 409 * **0.10.0** 2015.12.10 410 * Full compatibility with Windows 411 * Introduction of afero utilities 412 * Test suite rewritten to work cross platform 413 * Normalize paths for MemMapFs 414 * Adding Sync to the file interface 415 * **Breaking Change** Walk and ReadDir have changed parameter order 416 * Moving types used by MemMapFs to a subpackage 417 * General bugfixes and improvements 418 * **0.9.0** 2015.11.05 419 * New Walk function similar to filepath.Walk 420 * MemMapFs.OpenFile handles O_CREATE, O_APPEND, O_TRUNC 421 * MemMapFs.Remove now really deletes the file 422 * InMemoryFile.Readdir and Readdirnames work correctly 423 * InMemoryFile functions lock it for concurrent access 424 * Test suite improvements 425 * **0.8.0** 2014.10.28 426 * First public version 427 * Interfaces feel ready for people to build using 428 * Interfaces satisfy all known uses 429 * MemMapFs passes the majority of the OS test suite 430 * OsFs passes the majority of the OS test suite 431 432 ## Contributing 433 434 1. Fork it 435 2. Create your feature branch (`git checkout -b my-new-feature`) 436 3. Commit your changes (`git commit -am 'Add some feature'`) 437 4. Push to the branch (`git push origin my-new-feature`) 438 5. Create new Pull Request 439 440 ## Contributors 441 442 Names in no particular order: 443 444 * [spf13](https://github.com/spf13) 445 * [jaqx0r](https://github.com/jaqx0r) 446 * [mbertschler](https://github.com/mbertschler) 447 * [xor-gate](https://github.com/xor-gate) 448 449 ## License 450 451 Afero is released under the Apache 2.0 license. See 452 [LICENSE.txt](https://github.com/spf13/afero/blob/master/LICENSE.txt)