github.com/hbbio/afero@v1.2.3-0.20190412153849-bd67867be27b/README.md (about) 1  2 3 A FileSystem Abstraction System for Go 4 5 [](https://travis-ci.org/hbbio/afero) [](https://ci.appveyor.com/project/hbbio/afero) [](https://godoc.org/github.com/hbbio/afero) [](https://gitter.im/hbbio/afero?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) 6 7 # Overview 8 9 Afero is an filesystem framework providing a simple, uniform and universal API 10 interacting with any filesystem, as an abstraction layer providing interfaces, 11 types and methods. Afero has an exceptionally clean interface and simple design 12 without needless constructors or initialization methods. 13 14 Afero is also a library providing a base set of interoperable backend 15 filesystems that make it easy to work with afero while retaining all the power 16 and benefit of the os and ioutil packages. 17 18 Afero provides significant improvements over using the os package alone, most 19 notably the ability to create mock and testing filesystems without relying on the disk. 20 21 It is suitable for use in a any situation where you would consider using the OS 22 package as it provides an additional abstraction that makes it easy to use a 23 memory backed file system during testing. It also adds support for the http 24 filesystem for full interoperability. 25 26 ## Afero Features 27 28 - A single consistent API for accessing a variety of filesystems 29 - Interoperation between a variety of file system types 30 - A set of interfaces to encourage and enforce interoperability between backends 31 - An atomic cross platform memory backed file system 32 - Support for compositional (union) file systems by combining multiple file systems acting as one 33 - Specialized backends which modify existing filesystems (Read Only, Regexp filtered) 34 - A set of utility functions ported from io, ioutil & hugo to be afero aware 35 36 # Using Afero 37 38 Afero is easy to use and easier to adopt. 39 40 A few different ways you could use Afero: 41 42 - Use the interfaces alone to define you own file system. 43 - Wrap for the OS packages. 44 - Define different filesystems for different parts of your application. 45 - Use Afero for mock filesystems while testing 46 47 ## Step 1: Install Afero 48 49 First use go get to install the latest version of the library. 50 51 $ go get github.com/hbbio/afero 52 53 Next include Afero in your application. 54 55 ```go 56 import "github.com/hbbio/afero" 57 ``` 58 59 ## Step 2: Declare a backend 60 61 First define a package variable and set it to a pointer to a filesystem. 62 63 ```go 64 var AppFs = afero.NewMemMapFs() 65 66 or 67 68 var AppFs = afero.NewOsFs() 69 ``` 70 71 It is important to note that if you repeat the composite literal you 72 will be using a completely new and isolated filesystem. In the case of 73 OsFs it will still use the same underlying filesystem but will reduce 74 the ability to drop in other filesystems as desired. 75 76 ## Step 3: Use it like you would the OS package 77 78 Throughout your application use any function and method like you normally 79 would. 80 81 So if my application before had: 82 83 ```go 84 os.Open('/tmp/foo') 85 ``` 86 87 We would replace it with: 88 89 ```go 90 AppFs.Open('/tmp/foo') 91 ``` 92 93 `AppFs` being the variable we defined above. 94 95 ## List of all available functions 96 97 File System Methods Available: 98 99 ```go 100 Chmod(name string, mode os.FileMode) : error 101 Chtimes(name string, atime time.Time, mtime time.Time) : error 102 Create(name string) : File, error 103 Mkdir(name string, perm os.FileMode) : error 104 MkdirAll(path string, perm os.FileMode) : error 105 Name() : string 106 Open(name string) : File, error 107 OpenFile(name string, flag int, perm os.FileMode) : File, error 108 Remove(name string) : error 109 RemoveAll(path string) : error 110 Rename(oldname, newname string) : error 111 Stat(name string) : os.FileInfo, error 112 ``` 113 114 File Interfaces and Methods Available: 115 116 ```go 117 io.Closer 118 io.Reader 119 io.ReaderAt 120 io.Seeker 121 io.Writer 122 io.WriterAt 123 124 Name() : string 125 Readdir(count int) : []os.FileInfo, error 126 Readdirnames(n int) : []string, error 127 Stat() : os.FileInfo, error 128 Sync() : error 129 Truncate(size int64) : error 130 WriteString(s string) : ret int, err error 131 ``` 132 133 In some applications it may make sense to define a new package that 134 simply exports the file system variable for easy access from anywhere. 135 136 ## Using Afero's utility functions 137 138 Afero provides a set of functions to make it easier to use the underlying file systems. 139 These functions have been primarily ported from io & ioutil with some developed for Hugo. 140 141 The afero utilities support all afero compatible backends. 142 143 The list of utilities includes: 144 145 ```go 146 DirExists(path string) (bool, error) 147 Exists(path string) (bool, error) 148 FileContainsBytes(filename string, subslice []byte) (bool, error) 149 GetTempDir(subPath string) string 150 IsDir(path string) (bool, error) 151 IsEmpty(path string) (bool, error) 152 ReadDir(dirname string) ([]os.FileInfo, error) 153 ReadFile(filename string) ([]byte, error) 154 SafeWriteReader(path string, r io.Reader) (err error) 155 TempDir(dir, prefix string) (name string, err error) 156 TempFile(dir, prefix string) (f File, err error) 157 Walk(root string, walkFn filepath.WalkFunc) error 158 WriteFile(filename string, data []byte, perm os.FileMode) error 159 WriteReader(path string, r io.Reader) (err error) 160 ``` 161 162 For a complete list see [Afero's GoDoc](https://godoc.org/github.com/hbbio/afero) 163 164 They are available under two different approaches to use. You can either call 165 them directly where the first parameter of each function will be the file 166 system, or you can declare a new `Afero`, a custom type used to bind these 167 functions as methods to a given filesystem. 168 169 ### Calling utilities directly 170 171 ```go 172 fs := new(afero.MemMapFs) 173 f, err := afero.TempFile(fs,"", "ioutil-test") 174 175 ``` 176 177 ### Calling via Afero 178 179 ```go 180 fs := afero.NewMemMapFs() 181 afs := &afero.Afero{Fs: fs} 182 f, err := afs.TempFile("", "ioutil-test") 183 ``` 184 185 ## Using Afero for Testing 186 187 There is a large benefit to using a mock filesystem for testing. It has a 188 completely blank state every time it is initialized and can be easily 189 reproducible regardless of OS. You could create files to your heart’s content 190 and the file access would be fast while also saving you from all the annoying 191 issues with deleting temporary files, Windows file locking, etc. The MemMapFs 192 backend is perfect for testing. 193 194 - Much faster than performing I/O operations on disk 195 - Avoid security issues and permissions 196 - Far more control. 'rm -rf /' with confidence 197 - Test setup is far more easier to do 198 - No test cleanup needed 199 200 One way to accomplish this is to define a variable as mentioned above. 201 In your application this will be set to afero.NewOsFs() during testing you 202 can set it to afero.NewMemMapFs(). 203 204 It wouldn't be uncommon to have each test initialize a blank slate memory 205 backend. To do this I would define my `appFS = afero.NewOsFs()` somewhere 206 appropriate in my application code. This approach ensures that Tests are order 207 independent, with no test relying on the state left by an earlier test. 208 209 Then in my tests I would initialize a new MemMapFs for each test: 210 211 ```go 212 func TestExist(t *testing.T) { 213 appFS := afero.NewMemMapFs() 214 // create test files and directories 215 appFS.MkdirAll("src/a", 0755) 216 afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644) 217 afero.WriteFile(appFS, "src/c", []byte("file c"), 0644) 218 name := "src/c" 219 _, err := appFS.Stat(name) 220 if os.IsNotExist(err) { 221 t.Errorf("file \"%s\" does not exist.\n", name) 222 } 223 } 224 ``` 225 226 # Available Backends 227 228 ## Operating System Native 229 230 ### OsFs 231 232 The first is simply a wrapper around the native OS calls. This makes it 233 very easy to use as all of the calls are the same as the existing OS 234 calls. It also makes it trivial to have your code use the OS during 235 operation and a mock filesystem during testing or as needed. 236 237 ```go 238 appfs := afero.NewOsFs() 239 appfs.MkdirAll("src/a", 0755)) 240 ``` 241 242 ## Memory Backed Storage 243 244 ### MemMapFs 245 246 Afero also provides a fully atomic memory backed filesystem perfect for use in 247 mocking and to speed up unnecessary disk io when persistence isn’t 248 necessary. It is fully concurrent and will work within go routines 249 safely. 250 251 ```go 252 mm := afero.NewMemMapFs() 253 mm.MkdirAll("src/a", 0755)) 254 ``` 255 256 #### InMemoryFile 257 258 As part of MemMapFs, Afero also provides an atomic, fully concurrent memory 259 backed file implementation. This can be used in other memory backed file 260 systems with ease. Plans are to add a radix tree memory stored file 261 system using InMemoryFile. 262 263 ## Network Interfaces 264 265 ### SftpFs 266 267 Afero has experimental support for secure file transfer protocol (sftp). Which can 268 be used to perform file operations over a encrypted channel. 269 270 ## Filtering Backends 271 272 ### BasePathFs 273 274 The BasePathFs restricts all operations to a given path within an Fs. 275 The given file name to the operations on this Fs will be prepended with 276 the base path before calling the source Fs. 277 278 ```go 279 bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path") 280 ``` 281 282 ### ReadOnlyFs 283 284 A thin wrapper around the source Fs providing a read only view. 285 286 ```go 287 fs := afero.NewReadOnlyFs(afero.NewOsFs()) 288 _, err := fs.Create("/file.txt") 289 // err = syscall.EPERM 290 ``` 291 292 # RegexpFs 293 294 A filtered view on file names, any file NOT matching 295 the passed regexp will be treated as non-existing. 296 Files not matching the regexp provided will not be created. 297 Directories are not filtered. 298 299 ```go 300 fs := afero.NewRegexpFs(afero.NewMemMapFs(), regexp.MustCompile(`\.txt$`)) 301 _, err := fs.Create("/file.html") 302 // err = syscall.ENOENT 303 ``` 304 305 ### HttpFs 306 307 Afero provides an http compatible backend which can wrap any of the existing 308 backends. 309 310 The Http package requires a slightly specific version of Open which 311 returns an http.File type. 312 313 Afero provides an httpFs file system which satisfies this requirement. 314 Any Afero FileSystem can be used as an httpFs. 315 316 ```go 317 httpFs := afero.NewHttpFs(<ExistingFS>) 318 fileserver := http.FileServer(httpFs.Dir(<PATH>))) 319 http.Handle("/", fileserver) 320 ``` 321 322 ## Composite Backends 323 324 Afero provides the ability have two filesystems (or more) act as a single 325 file system. 326 327 ### CacheOnReadFs 328 329 The CacheOnReadFs will lazily make copies of any accessed files from the base 330 layer into the overlay. Subsequent reads will be pulled from the overlay 331 directly permitting the request is within the cache duration of when it was 332 created in the overlay. 333 334 If the base filesystem is writeable, any changes to files will be 335 done first to the base, then to the overlay layer. Write calls to open file 336 handles like `Write()` or `Truncate()` to the overlay first. 337 338 To writing files to the overlay only, you can use the overlay Fs directly (not 339 via the union Fs). 340 341 Cache files in the layer for the given time.Duration, a cache duration of 0 342 means "forever" meaning the file will not be re-requested from the base ever. 343 344 A read-only base will make the overlay also read-only but still copy files 345 from the base to the overlay when they're not present (or outdated) in the 346 caching layer. 347 348 ```go 349 base := afero.NewOsFs() 350 layer := afero.NewMemMapFs() 351 ufs := afero.NewCacheOnReadFs(base, layer, 100 * time.Second) 352 ``` 353 354 ### CopyOnWriteFs() 355 356 The CopyOnWriteFs is a read only base file system with a potentially 357 writeable layer on top. 358 359 Read operations will first look in the overlay and if not found there, will 360 serve the file from the base. 361 362 Changes to the file system will only be made in the overlay. 363 364 Any attempt to modify a file found only in the base will copy the file to the 365 overlay layer before modification (including opening a file with a writable 366 handle). 367 368 Removing and Renaming files present only in the base layer is not currently 369 permitted. If a file is present in the base layer and the overlay, only the 370 overlay will be removed/renamed. 371 372 ```go 373 base := afero.NewOsFs() 374 roBase := afero.NewReadOnlyFs(base) 375 ufs := afero.NewCopyOnWriteFs(roBase, afero.NewMemMapFs()) 376 377 fh, _ = ufs.Create("/home/test/file2.txt") 378 fh.WriteString("This is a test") 379 fh.Close() 380 ``` 381 382 In this example all write operations will only occur in memory (MemMapFs) 383 leaving the base filesystem (OsFs) untouched. 384 385 ## Desired/possible backends 386 387 The following is a short list of possible backends we hope someone will 388 implement: 389 390 - SSH 391 - ZIP 392 - TAR 393 - S3 394 395 # About the project 396 397 ## What's in the name 398 399 Afero comes from the latin roots Ad-Facere. 400 401 **"Ad"** is a prefix meaning "to". 402 403 **"Facere"** is a form of the root "faciō" making "make or do". 404 405 The literal meaning of afero is "to make" or "to do" which seems very fitting 406 for a library that allows one to make files and directories and do things with them. 407 408 The English word that shares the same roots as Afero is "affair". Affair shares 409 the same concept but as a noun it means "something that is made or done" or "an 410 object of a particular type". 411 412 It's also nice that unlike some of my other libraries (hugo, cobra, viper) it 413 Googles very well. 414 415 ## Release Notes 416 417 - **0.10.0** 2015.12.10 418 - Full compatibility with Windows 419 - Introduction of afero utilities 420 - Test suite rewritten to work cross platform 421 - Normalize paths for MemMapFs 422 - Adding Sync to the file interface 423 - **Breaking Change** Walk and ReadDir have changed parameter order 424 - Moving types used by MemMapFs to a subpackage 425 - General bugfixes and improvements 426 - **0.9.0** 2015.11.05 427 - New Walk function similar to filepath.Walk 428 - MemMapFs.OpenFile handles O_CREATE, O_APPEND, O_TRUNC 429 - MemMapFs.Remove now really deletes the file 430 - InMemoryFile.Readdir and Readdirnames work correctly 431 - InMemoryFile functions lock it for concurrent access 432 - Test suite improvements 433 - **0.8.0** 2014.10.28 434 - First public version 435 - Interfaces feel ready for people to build using 436 - Interfaces satisfy all known uses 437 - MemMapFs passes the majority of the OS test suite 438 - OsFs passes the majority of the OS test suite 439 440 ## Contributing 441 442 1. Fork it 443 2. Create your feature branch (`git checkout -b my-new-feature`) 444 3. Commit your changes (`git commit -am 'Add some feature'`) 445 4. Push to the branch (`git push origin my-new-feature`) 446 5. Create new Pull Request 447 448 ## Contributors 449 450 Names in no particular order: 451 452 - [spf13](https://github.com/spf13) 453 - [jaqx0r](https://github.com/jaqx0r) 454 - [mbertschler](https://github.com/mbertschler) 455 - [xor-gate](https://github.com/xor-gate) 456 457 ## License 458 459 Afero is released under the Apache 2.0 license. See 460 [LICENSE.txt](https://github.com/hbbio/afero/blob/master/LICENSE.txt)