github.com/sagansystems/goofys-app@v0.19.1-0.20180410053237-b2302fdf5af9/README.md (about) 1 Goofys is a high-performance, POSIX-ish [Amazon S3](https://aws.amazon.com/s3/) file system written in Go 2 3 [![Build Status](https://travis-ci.org/kahing/goofys.svg?branch=master)](https://travis-ci.org/kahing/goofys) 4 [![Github All Releases](https://img.shields.io/github/downloads/kahing/goofys/total.svg)](https://github.com/kahing/goofys/releases/) 5 [![Twitter Follow](https://img.shields.io/twitter/follow/s3goofys.svg?style=social&label=Follow)](https://twitter.com/s3goofys) 6 7 # Overview 8 9 Goofys allows you to mount an S3 bucket as a filey system. 10 11 It's a Filey System instead of a File System because goofys strives 12 for performance first and POSIX second. Particularly things that are 13 difficult to support on S3 or would translate into more than one 14 round-trip would either fail (random writes) or faked (no per-file 15 permission). Goofys does not have a on disk data cache (checkout 16 [catfs](https://github.com/kahing/catfs)), and consistency model is 17 close-to-open. 18 19 # Installation 20 21 * On Linux, install via [pre-built binaries](http://bit.ly/goofys-latest). You may also need to install fuse-utils first. 22 23 * On Mac OS X, install via [Homebrew](http://brew.sh/): 24 25 ```ShellSession 26 $ brew cask install osxfuse 27 $ brew install goofys 28 ``` 29 30 * Or build from source: 31 32 ```ShellSession 33 $ export GOPATH=$HOME/work 34 $ go get github.com/kahing/goofys 35 $ go install github.com/kahing/goofys 36 ``` 37 38 # Usage 39 40 ```ShellSession 41 $ cat ~/.aws/credentials 42 [default] 43 aws_access_key_id = AKID1234567890 44 aws_secret_access_key = MY-SECRET-KEY 45 $ $GOPATH/bin/goofys <bucket> <mountpoint> 46 $ $GOPATH/bin/goofys <bucket:prefix> <mountpoint> # if you only want to mount objects under a prefix 47 ``` 48 49 Users can also configure credentials via the 50 [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) 51 or the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. 52 53 To mount an S3 bucket on startup, make sure the credential is 54 configured for `root`, and can add this to `/etc/fstab`: 55 56 ``` 57 goofys#bucket /mnt/mountpoint fuse _netdev,allow_other,--file-mode=0666 0 0 58 ``` 59 60 Got more questions? Check out [questions other people asked](https://github.com/kahing/goofys/issues?utf8=%E2%9C%93&q=is%3Aissue%20label%3Aquestion%20) 61 62 # Benchmark 63 64 Using `--stat-cache-ttl 1s --type-cache-ttl 1s` for goofys 65 `-ostat_cache_expire=1` for s3fs to simulate cold runs. Detail for the 66 benchmark can be found in 67 [bench.sh](https://github.com/kahing/goofys/blob/master/bench/bench.sh). [Raw data](https://github.com/kahing/goofys/blob/master/bench/) 68 is available as well. Test was run on an EC2 m4.16xlarge in us-west-2a 69 connecting to a bucket in us-west-2. Units are seconds. 70 71 ![Benchmark result](/bench/bench.png?raw=true "Benchmark") 72 73 (†) riofs does not wait for HTTP response before returning from `release()`, so the create files benchmarks do not measure the right thing for it 74 75 ## Benchmark with caching enabled 76 77 Enabling `--cache` has little impact on write speed (since `catfs` 78 implements a write-through cache) but read has a large variance. Time 79 to first byte is competitive with `s3fs` which suggests layering fuse 80 filesystems can be a viable approach. 81 82 ![Cached Benchmark result](/bench/bench-cached.png?raw=true "Cached Benchmark") 83 84 85 <a name="runbenchmark"></a> 86 To run the benchmark, do: 87 88 ```ShellSession 89 $ cat > ~/.passwd-riofs 90 export AWS_ACCESS_KEY_ID=AKID1234567890 91 export AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY 92 $ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v ~/.passwd-riofs:/root/.passwd-riofs -v /tmp/cache:/tmp/cache kahing/goofys-bench 93 # result will be written to $TESTBUCKET 94 ``` 95 96 if `CACHE` is set to `true`, the read benchmarks ('Read 1GB' and 'Time to 1st byte') will be cached read. 97 98 # License 99 100 Copyright (C) 2015 - 2017 Ka-Hing Cheung 101 102 Licensed under the Apache License, Version 2.0 103 104 # Current Status 105 106 goofys has been tested under Linux and OS X. 107 108 List of non-POSIX behaviors/limitations: 109 * only sequential writes supported 110 * does not store file mode/owner/group 111 * use `--(dir|file)-mode` or `--(uid|gid)` options 112 * does not support symlink or hardlink 113 * `ctime`, `atime` is always the same as `mtime` 114 * cannot rename non-empty directories 115 * `unlink` returns success even if file is not present 116 * `fsync` is ignored, files are only flushed on `close` 117 118 In addition to the items above, the following supportable but not yet implemented: 119 * creating files larger than 1TB 120 121 ## Compatibility with non-AWS S3 122 123 goofys has been tested with the following non-AWS providers: 124 125 * Amplidata 126 * DreamObjects (Ceph) 127 * EMC Atmos 128 * Google Cloud Storage 129 * OpenStack Swift 130 * S3Proxy 131 * Minio (limited) 132 133 # References 134 135 * Data is stored on [Amazon S3](https://aws.amazon.com/s3/) 136 * [Amazon SDK for Go](https://github.com/aws/aws-sdk-go) 137 * Other related fuse filesystems 138 * [catfs](https://github.com/kahing/catfs): caching layer that can be used with goofys 139 * [s3fs](https://github.com/s3fs-fuse/s3fs-fuse): another popular filesystem for S3 140 * [gcsfuse](https://github.com/googlecloudplatform/gcsfuse): 141 filesystem for 142 [Google Cloud Storage](https://cloud.google.com/storage/). Goofys 143 borrowed some skeleton code from this project. 144 * [S3Proxy](https://github.com/andrewgaul/s3proxy) is used for `go test` 145 * [fuse binding](https://github.com/jacobsa/fuse), also used by `gcsfuse`