github.com/djmaze/goofys@v0.24.2/README.md (about) 1 <img src="doc/goofys.png" height="32" width="32" align="middle" /> Goofys is a high-performance, POSIX-ish [Amazon S3](https://aws.amazon.com/s3/) file system written in Go 2 3 [](https://travis-ci.org/kahing/goofys) 4 [](https://github.com/kahing/goofys/releases/) 5 [](https://twitter.com/s3goofys) 6 7 # Overview 8 9 Goofys allows you to mount an S3 bucket as a filey system. 10 11 It's a Filey System instead of a File System because goofys strives 12 for performance first and POSIX second. Particularly things that are 13 difficult to support on S3 or would translate into more than one 14 round-trip would either fail (random writes) or faked (no per-file 15 permission). Goofys does not have an on disk data cache (checkout 16 [catfs](https://github.com/kahing/catfs)), and consistency model is 17 close-to-open. 18 19 # Installation 20 21 * On Linux, install via [pre-built binaries](https://github.com/kahing/goofys/releases/latest/download/goofys). You may also need to install fuse-utils first. 22 23 * On macOS, install via [Homebrew](https://brew.sh/): 24 25 ```ShellSession 26 $ brew cask install osxfuse 27 $ brew install goofys 28 ``` 29 30 * Or build from source with Go 1.10 or later: 31 32 ```ShellSession 33 $ export GOPATH=$HOME/work 34 $ go get github.com/kahing/goofys 35 $ go install github.com/kahing/goofys 36 ``` 37 38 # Usage 39 40 ```ShellSession 41 $ cat ~/.aws/credentials 42 [default] 43 aws_access_key_id = AKID1234567890 44 aws_secret_access_key = MY-SECRET-KEY 45 $ $GOPATH/bin/goofys <bucket> <mountpoint> 46 $ $GOPATH/bin/goofys <bucket:prefix> <mountpoint> # if you only want to mount objects under a prefix 47 ``` 48 49 Users can also configure credentials via the 50 [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) 51 or the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. 52 53 To mount an S3 bucket on startup, make sure the credential is 54 configured for `root`, and can add this to `/etc/fstab`: 55 56 ``` 57 goofys#bucket /mnt/mountpoint fuse _netdev,allow_other,--file-mode=0666,--dir-mode=0777 0 0 58 ``` 59 60 See also: [Instruction for Azure Blob Storage, Azure Data Lake Gen1, and Azure Data Lake Gen2](https://github.com/kahing/goofys/blob/master/README-azure.md). 61 62 Got more questions? Check out [questions other people asked](https://github.com/kahing/goofys/issues?utf8=%E2%9C%93&q=is%3Aissue%20label%3Aquestion%20) 63 64 # Benchmark 65 66 Using `--stat-cache-ttl 1s --type-cache-ttl 1s` for goofys 67 `-ostat_cache_expire=1` for s3fs to simulate cold runs. Detail for the 68 benchmark can be found in 69 [bench.sh](https://github.com/kahing/goofys/blob/master/bench/bench.sh). [Raw data](https://github.com/kahing/goofys/blob/master/bench/) 70 is available as well. The test was run on an EC2 m5.4xlarge in us-west-2a 71 connected to a bucket in us-west-2. Units are seconds. 72 73  74 75 To run the benchmark, configure EC2's instance role to be able to write to `$TESTBUCKET`, and then do: 76 ```ShellSession 77 $ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v /tmp/cache:/tmp/cache kahing/goofys-bench 78 # result will be written to $TESTBUCKET 79 ``` 80 81 See also: [cached benchmark result](https://github.com/kahing/goofys/blob/master/bench/cache/README.md) and [result on Azure](https://github.com/kahing/goofys/blob/master/bench/azure/README.md). 82 83 # License 84 85 Copyright (C) 2015 - 2019 Ka-Hing Cheung 86 87 Licensed under the Apache License, Version 2.0 88 89 # Current Status 90 91 goofys has been tested under Linux and macOS. 92 93 List of non-POSIX behaviors/limitations: 94 * only sequential writes supported 95 * does not store file mode/owner/group 96 * use `--(dir|file)-mode` or `--(uid|gid)` options 97 * does not support symlink or hardlink 98 * `ctime`, `atime` is always the same as `mtime` 99 * cannot `rename` directories with more than 1000 children 100 * `unlink` returns success even if file is not present 101 * `fsync` is ignored, files are only flushed on `close` 102 103 In addition to the items above, the following are supportable but not yet implemented: 104 * creating files larger than 1TB 105 106 ## Compatibility with non-AWS S3 107 108 goofys has been tested with the following non-AWS S3 providers: 109 110 * Amplidata / WD ActiveScale 111 * Ceph (ex: Digital Ocean Spaces, DreamObjects, gridscale) 112 * EdgeFS 113 * EMC Atmos 114 * Google Cloud Storage 115 * Minio (limited) 116 * OpenStack Swift 117 * S3Proxy 118 * Scaleway 119 * Wasabi 120 121 Additionally, goofys also works with the following non-S3 object stores: 122 123 * Azure Blob Storage 124 * Azure Data Lake Gen1 125 * Azure Data Lake Gen2 126 127 # References 128 129 * Data is stored on [Amazon S3](https://aws.amazon.com/s3/) 130 * [Amazon SDK for Go](https://github.com/aws/aws-sdk-go) 131 * Other related fuse filesystems 132 * [catfs](https://github.com/kahing/catfs): caching layer that can be used with goofys 133 * [s3fs](https://github.com/s3fs-fuse/s3fs-fuse): another popular filesystem for S3 134 * [gcsfuse](https://github.com/googlecloudplatform/gcsfuse): 135 filesystem for 136 [Google Cloud Storage](https://cloud.google.com/storage/). Goofys 137 borrowed some skeleton code from this project. 138 * [S3Proxy](https://github.com/andrewgaul/s3proxy) is used for `go test` 139 * [fuse binding](https://github.com/jacobsa/fuse), also used by `gcsfuse`