github.com/sagansystems/goofys-app@v0.19.1-0.20180410053237-b2302fdf5af9/README.md (about)

     1  Goofys is a high-performance, POSIX-ish [Amazon S3](https://aws.amazon.com/s3/) file system written in Go
     2  
     3  [![Build Status](https://travis-ci.org/kahing/goofys.svg?branch=master)](https://travis-ci.org/kahing/goofys)
     4  [![Github All Releases](https://img.shields.io/github/downloads/kahing/goofys/total.svg)](https://github.com/kahing/goofys/releases/)
     5  [![Twitter Follow](https://img.shields.io/twitter/follow/s3goofys.svg?style=social&label=Follow)](https://twitter.com/s3goofys)
     6  
     7  # Overview
     8  
     9  Goofys allows you to mount an S3 bucket as a filey system.
    10  
    11  It's a Filey System instead of a File System because goofys strives
    12  for performance first and POSIX second. Particularly things that are
    13  difficult to support on S3 or would translate into more than one
    14  round-trip would either fail (random writes) or faked (no per-file
    15  permission). Goofys does not have a on disk data cache (checkout
    16  [catfs](https://github.com/kahing/catfs)), and consistency model is
    17  close-to-open.
    18  
    19  # Installation
    20  
    21  * On Linux, install via [pre-built binaries](http://bit.ly/goofys-latest). You may also need to install fuse-utils first.
    22  
    23  * On Mac OS X, install via [Homebrew](http://brew.sh/):
    24  
    25  ```ShellSession
    26  $ brew cask install osxfuse
    27  $ brew install goofys
    28  ```
    29  
    30  * Or build from source:
    31  
    32  ```ShellSession
    33  $ export GOPATH=$HOME/work
    34  $ go get github.com/kahing/goofys
    35  $ go install github.com/kahing/goofys
    36  ```
    37  
    38  # Usage
    39  
    40  ```ShellSession
    41  $ cat ~/.aws/credentials
    42  [default]
    43  aws_access_key_id = AKID1234567890
    44  aws_secret_access_key = MY-SECRET-KEY
    45  $ $GOPATH/bin/goofys <bucket> <mountpoint>
    46  $ $GOPATH/bin/goofys <bucket:prefix> <mountpoint> # if you only want to mount objects under a prefix
    47  ```
    48  
    49  Users can also configure credentials via the
    50  [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)
    51  or the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.
    52  
    53  To mount an S3 bucket on startup, make sure the credential is
    54  configured for `root`, and can add this to `/etc/fstab`:
    55  
    56  ```
    57  goofys#bucket   /mnt/mountpoint        fuse     _netdev,allow_other,--file-mode=0666    0       0
    58  ```
    59  
    60  Got more questions? Check out [questions other people asked](https://github.com/kahing/goofys/issues?utf8=%E2%9C%93&q=is%3Aissue%20label%3Aquestion%20)
    61  
    62  # Benchmark
    63  
    64  Using `--stat-cache-ttl 1s --type-cache-ttl 1s` for goofys
    65  `-ostat_cache_expire=1` for s3fs to simulate cold runs. Detail for the
    66  benchmark can be found in
    67  [bench.sh](https://github.com/kahing/goofys/blob/master/bench/bench.sh). [Raw data](https://github.com/kahing/goofys/blob/master/bench/)
    68  is available as well. Test was run on an EC2 m4.16xlarge in us-west-2a
    69  connecting to a bucket in us-west-2. Units are seconds.
    70  
    71  ![Benchmark result](/bench/bench.png?raw=true "Benchmark")
    72  
    73  (†) riofs does not wait for HTTP response before returning from `release()`, so the create files benchmarks do not measure the right thing for it
    74  
    75  ## Benchmark with caching enabled
    76  
    77  Enabling `--cache` has little impact on write speed (since `catfs`
    78  implements a write-through cache) but read has a large variance. Time
    79  to first byte is competitive with `s3fs` which suggests layering fuse
    80  filesystems can be a viable approach.
    81  
    82  ![Cached Benchmark result](/bench/bench-cached.png?raw=true "Cached Benchmark")
    83  
    84  
    85  <a name="runbenchmark"></a>
    86  To run the benchmark, do:
    87  
    88  ```ShellSession
    89  $ cat > ~/.passwd-riofs
    90  export AWS_ACCESS_KEY_ID=AKID1234567890
    91  export AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY
    92  $ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v  ~/.passwd-riofs:/root/.passwd-riofs -v /tmp/cache:/tmp/cache kahing/goofys-bench
    93  # result will be written to $TESTBUCKET
    94  ```
    95  
    96  if `CACHE` is set to `true`, the read benchmarks ('Read 1GB' and 'Time to 1st byte') will be cached read.
    97  
    98  # License
    99  
   100  Copyright (C) 2015 - 2017 Ka-Hing Cheung
   101  
   102  Licensed under the Apache License, Version 2.0
   103  
   104  # Current Status
   105  
   106  goofys has been tested under Linux and OS X.
   107  
   108  List of non-POSIX behaviors/limitations:
   109    * only sequential writes supported
   110    * does not store file mode/owner/group
   111      * use `--(dir|file)-mode` or `--(uid|gid)` options
   112    * does not support symlink or hardlink
   113    * `ctime`, `atime` is always the same as `mtime`
   114    * cannot rename non-empty directories
   115    * `unlink` returns success even if file is not present
   116    * `fsync` is ignored, files are only flushed on `close`
   117  
   118  In addition to the items above, the following supportable but not yet implemented:
   119    * creating files larger than 1TB
   120  
   121  ## Compatibility with non-AWS S3
   122  
   123  goofys has been tested with the following non-AWS providers:
   124  
   125  * Amplidata
   126  * DreamObjects (Ceph)
   127  * EMC Atmos
   128  * Google Cloud Storage
   129  * OpenStack Swift
   130  * S3Proxy
   131  * Minio (limited)
   132  
   133  # References
   134  
   135    * Data is stored on [Amazon S3](https://aws.amazon.com/s3/)
   136    * [Amazon SDK for Go](https://github.com/aws/aws-sdk-go)
   137    * Other related fuse filesystems
   138      * [catfs](https://github.com/kahing/catfs): caching layer that can be used with goofys
   139      * [s3fs](https://github.com/s3fs-fuse/s3fs-fuse): another popular filesystem for S3
   140      * [gcsfuse](https://github.com/googlecloudplatform/gcsfuse):
   141        filesystem for
   142        [Google Cloud Storage](https://cloud.google.com/storage/). Goofys
   143        borrowed some skeleton code from this project.
   144    * [S3Proxy](https://github.com/andrewgaul/s3proxy) is used for `go test`
   145    * [fuse binding](https://github.com/jacobsa/fuse), also used by `gcsfuse`