github.com/adrianjagielak/goofys@v0.24.1-0.20230810095418-94919a5d2254/README.md (about) 1 <img src="doc/goofys.png" height="32" width="32" align="middle" /> Goofys is a high-performance, POSIX-ish [Amazon S3](https://aws.amazon.com/s3/) file system written in Go 2 3 [](https://travis-ci.org/kahing/goofys) 4 [](https://github.com/kahing/goofys/releases/) 5 [](https://twitter.com/s3goofys) 6 [](https://stackoverflow.com/search?q=%5Bgoofys%5D+is%3Aquestion) 7 8 # Overview 9 10 Goofys allows you to mount an S3 bucket as a filey system. 11 12 It's a Filey System instead of a File System because goofys strives 13 for performance first and POSIX second. Particularly things that are 14 difficult to support on S3 or would translate into more than one 15 round-trip would either fail (random writes) or faked (no per-file 16 permission). Goofys does not have an on disk data cache (checkout 17 [catfs](https://github.com/kahing/catfs)), and consistency model is 18 close-to-open. 19 20 # Installation 21 22 * On Linux, install via [pre-built binaries](https://github.com/kahing/goofys/releases/latest/download/goofys). 23 You may also need to install fuse too if you want to mount it on startup. 24 25 * On macOS, install via [Homebrew](https://brew.sh/): 26 27 ```ShellSession 28 $ brew cask install osxfuse 29 $ brew install goofys 30 ``` 31 32 * Or build from source with Go 1.10 or later: 33 34 ```ShellSession 35 $ export GOPATH=$HOME/work 36 $ go get github.com/kahing/goofys 37 $ go install github.com/kahing/goofys 38 ``` 39 40 # Usage 41 42 ```ShellSession 43 $ cat ~/.aws/credentials 44 [default] 45 aws_access_key_id = AKID1234567890 46 aws_secret_access_key = MY-SECRET-KEY 47 $ $GOPATH/bin/goofys <bucket> <mountpoint> 48 $ $GOPATH/bin/goofys <bucket:prefix> <mountpoint> # if you only want to mount objects under a prefix 49 ``` 50 51 Users can also configure credentials via the 52 [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) 53 or the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. 54 55 To mount an S3 bucket on startup, make sure the credential is 56 configured for `root`, and can add this to `/etc/fstab`: 57 58 ``` 59 goofys#bucket /mnt/mountpoint fuse _netdev,allow_other,--file-mode=0666,--dir-mode=0777 0 0 60 ``` 61 62 See also: [Instruction for Azure Blob Storage, Azure Data Lake Gen1, and Azure Data Lake Gen2](https://github.com/kahing/goofys/blob/master/README-azure.md). 63 64 Got more questions? Check out [questions other people asked](https://github.com/kahing/goofys/issues?utf8=%E2%9C%93&q=is%3Aissue%20label%3Aquestion%20) 65 66 # Benchmark 67 68 Using `--stat-cache-ttl 1s --type-cache-ttl 1s` for goofys 69 `-ostat_cache_expire=1` for s3fs to simulate cold runs. Detail for the 70 benchmark can be found in 71 [bench.sh](https://github.com/kahing/goofys/blob/master/bench/bench.sh). [Raw data](https://github.com/kahing/goofys/blob/master/bench/) 72 is available as well. The test was run on an EC2 m5.4xlarge in us-west-2a 73 connected to a bucket in us-west-2. Units are seconds. 74 75  76 77 To run the benchmark, configure EC2's instance role to be able to write to `$TESTBUCKET`, and then do: 78 ```ShellSession 79 $ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v /tmp/cache:/tmp/cache kahing/goofys-bench 80 # result will be written to $TESTBUCKET 81 ``` 82 83 See also: [cached benchmark result](https://github.com/kahing/goofys/blob/master/bench/cache/README.md) and [result on Azure](https://github.com/kahing/goofys/blob/master/bench/azure/README.md). 84 85 # License 86 87 Copyright (C) 2015 - 2019 Ka-Hing Cheung 88 89 Licensed under the Apache License, Version 2.0 90 91 # Current Status 92 93 goofys has been tested under Linux and macOS. 94 95 List of non-POSIX behaviors/limitations: 96 * only sequential writes supported 97 * does not store file mode/owner/group 98 * use `--(dir|file)-mode` or `--(uid|gid)` options 99 * does not support symlink or hardlink 100 * `ctime`, `atime` is always the same as `mtime` 101 * cannot `rename` directories with more than 1000 children 102 * `unlink` returns success even if file is not present 103 * `fsync` is ignored, files are only flushed on `close` 104 105 ## Compatibility with non-AWS S3 106 107 goofys has been tested with the following non-AWS S3 providers: 108 109 * Amplidata / WD ActiveScale 110 * Ceph (ex: Digital Ocean Spaces, DreamObjects, gridscale) 111 * EdgeFS 112 * EMC Atmos 113 * Google Cloud Storage 114 * Minio (limited) 115 * OpenStack Swift 116 * S3Proxy 117 * Scaleway 118 * Wasabi 119 120 Additionally, goofys also works with the following non-S3 object stores: 121 122 * Azure Blob Storage 123 * Azure Data Lake Gen1 124 * Azure Data Lake Gen2 125 126 # References 127 128 * Data is stored on [Amazon S3](https://aws.amazon.com/s3/) 129 * [Amazon SDK for Go](https://github.com/aws/aws-sdk-go) 130 * Other related fuse filesystems 131 * [catfs](https://github.com/kahing/catfs): caching layer that can be used with goofys 132 * [s3fs](https://github.com/s3fs-fuse/s3fs-fuse): another popular filesystem for S3 133 * [gcsfuse](https://github.com/googlecloudplatform/gcsfuse): 134 filesystem for 135 [Google Cloud Storage](https://cloud.google.com/storage/). Goofys 136 borrowed some skeleton code from this project. 137 * [S3Proxy](https://github.com/andrewgaul/s3proxy) is used for `go test` 138 * [fuse binding](https://github.com/jacobsa/fuse), also used by `gcsfuse`