github.com/t2y/goofys@v0.19.1-0.20190123053037-27053313e616/README.md (about)

     1  <img src="doc/goofys.png" height="32" width="32" align="middle" /> Goofys is a high-performance, POSIX-ish [Amazon S3](https://aws.amazon.com/s3/) file system written in Go
     2  
     3  [![Build Status](https://travis-ci.org/kahing/goofys.svg?branch=master)](https://travis-ci.org/kahing/goofys)
     4  [![Github All Releases](https://img.shields.io/github/downloads/kahing/goofys/total.svg)](https://github.com/kahing/goofys/releases/)
     5  [![Twitter Follow](https://img.shields.io/twitter/follow/s3goofys.svg?style=social&label=Follow)](https://twitter.com/s3goofys)
     6  
     7  # Overview
     8  
     9  Goofys allows you to mount an S3 bucket as a filey system.
    10  
    11  It's a Filey System instead of a File System because goofys strives
    12  for performance first and POSIX second. Particularly things that are
    13  difficult to support on S3 or would translate into more than one
    14  round-trip would either fail (random writes) or faked (no per-file
    15  permission). Goofys does not have an on disk data cache (checkout
    16  [catfs](https://github.com/kahing/catfs)), and consistency model is
    17  close-to-open.
    18  
    19  # Installation
    20  
    21  * On Linux, install via [pre-built binaries](https://bit.ly/goofys-latest). You may also need to install fuse-utils first.
    22  
    23  * On macOS, install via [Homebrew](https://brew.sh/):
    24  
    25  ```ShellSession
    26  $ brew cask install osxfuse
    27  $ brew install goofys
    28  ```
    29  
    30  * Or build from source with Go 1.9 or later:
    31  
    32  ```ShellSession
    33  $ export GOPATH=$HOME/work
    34  $ go get github.com/kahing/goofys
    35  $ go install github.com/kahing/goofys
    36  ```
    37  
    38  # Usage
    39  
    40  ```ShellSession
    41  $ cat ~/.aws/credentials
    42  [default]
    43  aws_access_key_id = AKID1234567890
    44  aws_secret_access_key = MY-SECRET-KEY
    45  $ $GOPATH/bin/goofys <bucket> <mountpoint>
    46  $ $GOPATH/bin/goofys <bucket:prefix> <mountpoint> # if you only want to mount objects under a prefix
    47  ```
    48  
    49  Users can also configure credentials via the
    50  [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)
    51  or the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.
    52  
    53  To mount an S3 bucket on startup, make sure the credential is
    54  configured for `root`, and can add this to `/etc/fstab`:
    55  
    56  ```
    57  goofys#bucket   /mnt/mountpoint        fuse     _netdev,allow_other,--file-mode=0666    0       0
    58  ```
    59  
    60  Got more questions? Check out [questions other people asked](https://github.com/kahing/goofys/issues?utf8=%E2%9C%93&q=is%3Aissue%20label%3Aquestion%20)
    61  
    62  # Benchmark
    63  
    64  Using `--stat-cache-ttl 1s --type-cache-ttl 1s` for goofys
    65  `-ostat_cache_expire=1` for s3fs to simulate cold runs. Detail for the
    66  benchmark can be found in
    67  [bench.sh](https://github.com/kahing/goofys/blob/master/bench/bench.sh). [Raw data](https://github.com/kahing/goofys/blob/master/bench/)
    68  is available as well. The test was run on an EC2 m4.16xlarge in us-west-2a
    69  connected to a bucket in us-west-2. Units are seconds.
    70  
    71  ![Benchmark result](/bench/bench.png?raw=true "Benchmark")
    72  
    73  (†) riofs does not wait for HTTP response before returning from `release()`, so the create files benchmarks do not measure the right thing for it
    74  
    75  ## Benchmark with caching enabled
    76  
    77  Enabling `--cache` has little impact on write speed (since `catfs`
    78  implements a write-through cache) but read has a large variance. Time
    79  to first byte is competitive with `s3fs` which suggests layering fuse
    80  filesystems can be a viable approach.
    81  
    82  ![Cached Benchmark result](/bench/bench-cached.png?raw=true "Cached Benchmark")
    83  
    84  
    85  <a name="runbenchmark"></a>
    86  To run the benchmark, do:
    87  
    88  ```ShellSession
    89  $ cat > ~/.passwd-riofs
    90  export AWS_ACCESS_KEY_ID=AKID1234567890
    91  export AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY
    92  $ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v  ~/.passwd-riofs:/root/.passwd-riofs -v /tmp/cache:/tmp/cache kahing/goofys-bench
    93  # result will be written to $TESTBUCKET
    94  ```
    95  
    96  if `CACHE` is set to `true`, the read benchmarks ('Read 1GB' and 'Time to 1st byte') will be cached read.
    97  
    98  # License
    99  
   100  Copyright (C) 2015 - 2018 Ka-Hing Cheung
   101  
   102  Licensed under the Apache License, Version 2.0
   103  
   104  # Current Status
   105  
   106  goofys has been tested under Linux and macOS.
   107  
   108  List of non-POSIX behaviors/limitations:
   109    * only sequential writes supported
   110    * does not store file mode/owner/group
   111      * use `--(dir|file)-mode` or `--(uid|gid)` options
   112    * does not support symlink or hardlink
   113    * `ctime`, `atime` is always the same as `mtime`
   114    * cannot rename non-empty directories
   115    * `unlink` returns success even if file is not present
   116    * `fsync` is ignored, files are only flushed on `close`
   117  
   118  In addition to the items above, the following are supportable but not yet implemented:
   119    * creating files larger than 1TB
   120  
   121  ## Compatibility with non-AWS S3
   122  
   123  goofys has been tested with the following non-AWS providers:
   124  
   125  * Amplidata
   126  * Digital Ocean Spaces (Ceph)
   127  * DreamObjects (Ceph)
   128  * EMC Atmos
   129  * Google Cloud Storage
   130  * OpenStack Swift
   131  * S3Proxy
   132  * Minio (limited)
   133  * Wasabi
   134  
   135  # References
   136  
   137    * Data is stored on [Amazon S3](https://aws.amazon.com/s3/)
   138    * [Amazon SDK for Go](https://github.com/aws/aws-sdk-go)
   139    * Other related fuse filesystems
   140      * [catfs](https://github.com/kahing/catfs): caching layer that can be used with goofys
   141      * [s3fs](https://github.com/s3fs-fuse/s3fs-fuse): another popular filesystem for S3
   142      * [gcsfuse](https://github.com/googlecloudplatform/gcsfuse):
   143        filesystem for
   144        [Google Cloud Storage](https://cloud.google.com/storage/). Goofys
   145        borrowed some skeleton code from this project.
   146    * [S3Proxy](https://github.com/andrewgaul/s3proxy) is used for `go test`
   147    * [fuse binding](https://github.com/jacobsa/fuse), also used by `gcsfuse`