github.com/swiftstack/ProxyFS@v0.0.0-20210203235616-4017c267d62f/pfsagentd/README.md (about)

     1  # PFSAgent
     2  
     3  FUSE Driver for presenting ProxyFS Volumes as locally mounted file systems
     4  
     5  ## Synopsis
     6  
     7  PFSAgent is a program that remotely accesses a ProxyFS Volume through
     8  a new `PROXYFS` HTTP Method targetted at any Swift Proxy Server in a
     9  Swift cluster that has contains the `pfs_middleware` filter. All metadata
    10  operations tunnel through `pfs_middleware` on their way to the `ProxyFS`
    11  instance currently managing the specified Volume (Swift Account). To
    12  allow file reads and writes to scale out, however, PFSAgent employs
    13  a `bypass_mode` instructing `pfs_middleware` to pass through an Object
    14  GET or PUT HTTP Method rather than redirect it over to `ProxyFS`.
    15  
    16  ## Setup
    17  
    18  There are a few distinct entities to be configured.
    19  
    20  ### OpenStack Swift Proxy
    21  
    22  Each Swift Proxy Server configuration file (`proxy-server.conf`) is assumed
    23  to already contain a `[filter:pfs]` section pointing the `pfs_middleware` at
    24  one or more ProxyFS instances. This is what enables Swift API and S3 API
    25  "BiModal" access to ProxyFS Volumes. A new Key:Value that is one of:
    26  * bypass_mode = off
    27  * bypass_mode = read-only
    28  * bypass_mode = read-write
    29  The Value `off` is assumed if the `bypass_mode` Key is not specified.
    30  
    31  ### PFSAgent Daemon (pfsagentd)
    32  
    33  The program is supplied a configuration file in .INI format. Here is an example:
    34  ```
    35  [Agent]
    36  FUSEVolumeName:                                    CommonVolume
    37  FUSEMountPointPath:                             AgentMountPoint
    38  FUSEUnMountRetryDelay:                                    100ms
    39  FUSEUnMountRetryCap:                                        100
    40  PlugInPath:                         pfsagentd-swift-auth-plugin
    41  PlugInEnvName:                                    SwiftAuthBlob
    42  PlugInEnvValue: {"AuthURL":"http://localhost:8080/auth/v1.0"\u002C"AuthUser":"test:tester"\u002C"AuthKey":"testing"\u002C"Account":"AUTH_test"}
    43  SwiftTimeout:                                               10m
    44  SwiftRetryLimit:                                             10
    45  SwiftRetryDelay:                                             1s
    46  SwiftRetryDelayVariance:                                     25
    47  SwiftRetryExpBackoff:                                       1.4
    48  SwiftConnectionPoolSize:                                    200
    49  FetchExtentsFromFileOffset:                                  32
    50  FetchExtentsBeforeFileOffset:                                 0
    51  ReadCacheLineSize:                                      1048576
    52  ReadCacheLineCount:                                        1000
    53  LeaseRetryLimit:                                             10
    54  LeaseRetryDelay:                                             1s
    55  LeaseRetryDelayVariance:                                     25
    56  LeaseRetryExpBackoff:                                       1.4
    57  SharedLeaseLimit:                                          1000
    58  ExclusiveLeaseLimit:                                        100
    59  ExtentMapEntryLimit:                                    1048576
    60  DirtyLogSegmentLimit:                                        50
    61  DirtyFileLimit:                                              50 # TODO - obsolete this
    62  MaxFlushSize:                                          10485760
    63  MaxFlushTime:                                             200ms
    64  LogFilePath:                             /var/log/pfsagentd.log
    65  LogToConsole:                                              true
    66  TraceEnabled:                                             false
    67  HTTPServerIPAddr:                                       0.0.0.0
    68  HTTPServerTCPPort:                                         9090
    69  ReadDirPlusEnabled:                                       false
    70  XAttrEnabled:                                             false
    71  EntryDuration:                                               0s
    72  AttrDuration:                                                0s
    73  AttrBlockSize:                                            65536
    74  ReaddirMaxEntries:                                         1024
    75  FUSEMaxBackground:                                          100
    76  FUSECongestionThreshhold:                                     0
    77  FUSEMaxWrite:                                            131072
    78  RetryRPCDeadlineIO:                                         60s
    79  RetryRPCKeepAlivePeriod:                                    60s
    80  ```
    81  
    82  In the above example, some important fields are as follows:
    83  * FUSEVolumeName should be set to the ProxyFS Volume being mounted
    84  * FUSEMountPointPath should be set to where the FUSE presentation of the Volume should appear (must pre-exist)
    85  * PlugInPath should be set to point to the desired Swift Authorization PlugIn
    86  * PlugInEnvName should be set to the name of the ENV variable used to pass Swift Auth secrets blob to the plug-in
    87  * PlugInEnvValue should be set to the value of the Swift Auth secrets blob if PFSAgent should set it
    88  * HTTPServerIPAddr should be set to the IP Address where PFSAgent should present its embedded HTTP Server
    89  * HTTPServerTCPPort should be set to the TCP Port upon which the PFSAgent should present its embedded HTTP Server
    90  
    91  The balance of the settings are more related to tuning choices. Among those, the most pertinent are:
    92  * ReadCacheLineSize specifies how much of a Swift Object is read when a read cache miss occurs
    93  * ReadCacheLineCount specifies how many such read cache lines will be used
    94  * MaxFlushSize specifies how frequently in terms of byte count writes are sent to new Swift Objects
    95  * MaxFlushTime specifies how frequently in terms of time writes are sent to new Swift Objects
    96  
    97  Note the use of `\u002C` in the example PlugInEnvValue line above. This avoids the .INI
    98  parser from interpreting the commas as value separators and incorrectly assuming there
    99  are multiple values being specified. Similarly, space would be interpreted as a list
   100  element separator so `\u0020` should be used instead.
   101  
   102  Each mounted ProxyFS Volume requires an instance of PFSAgent (`pfsagentd`) to run.
   103  Hence, each ProxyFS Volume must be described by a unique configuration file as described above.
   104  
   105  It is convenient to employ `systemd` to launch each PFSAgent instance via unique service files.
   106  An example service file that can be used to allow launching multiple such PFSAgent instances
   107  (by specifying the configuration file name in the `systemd` invocation) is:
   108  ```
   109  [Unit]
   110  Description=ProxyFS Agent %i
   111  Wants=network-online.target
   112  After=network-online.target
   113  ConditionFileNotEmpty=/opt/ss/etc/pfsagent/%i.conf
   114  
   115  [Service]
   116  ExecStart=/opt/ss/bin/pfsagentd /opt/ss/etc/pfsagent/%i.conf
   117  
   118  Type=simple
   119  
   120  # Restart the ssnoded daemon after a 2 seconds delay, in case it crashes.
   121  Restart=always
   122  RestartSec=2
   123  StartLimitInterval=11s
   124  
   125  [Install]
   126  WantedBy=network-online.target
   127  ```
   128  
   129  ### Authentication Plug-In
   130  
   131  While PFSAgent targets OpenStack Swift clusters that have provisioned
   132  ProxyFS and placed the pfs_middleware in the Swift Proxy pipeline, it
   133  must use the normal Swift Proxy to gain access to ProxyFS LogSegments
   134  that host FileInode data. Further, via the pfs_middleware in the Swift
   135  Proxy pipeline, at least the Mount JSON RPC will travel through the
   136  normal Swift Proxy to get to ProxyFS. To perform either these GETs,
   137  PUTs, or Mount JSON RPC, the request must contain a valid X-Auth-Token.
   138  Rather than assume normal Swift Authentifiction is used to compute
   139  this X-Auth-Token, PFSAgent implements a plug-in mechanism whereby an
   140  external process, the plug-in, performs the actual authentication step
   141  and returns the X-Auth-Token to be provided as proof in each subsequent
   142  Swift API request. Such tokens may expire, resulting in a `401 Unauthorized`
   143  response code. Thus, this plug-in must also support token renewal or
   144  replacement. In cases where a full re-authorization is required, the
   145  plug-in will already have the long-lived credentials with which to repeat
   146  the authentication as performed originally.
   147  
   148  #### Authentication-specific Setup
   149  
   150  The configuration supplied to PFSAgent (`pfsagentd`) includes, among
   151  other keys, the following:
   152  
   153  ```
   154  [Agent]
   155  ...
   156  PlugInPath:   # The path to the plug-in
   157  PlugInEnvName: # Specifies the name of an ENV variable the plug-in should load
   158  PlugInEnvValue: # If supplied, PFSAgent will set the ENV variable to this value
   159  ...
   160  ```
   161  
   162  #### Plug-In Operation
   163  
   164  PFSAgent will, at start-up, launch the program indicated by [Agent]PlugInPath,
   165  sending the value of [Agent]PlugInEnvName as Arg1. If [Agent]PlugInEnvValue is
   166  specified, it will also ensure the ENV variable named will be set to this
   167  value in the plug-in's process.
   168  
   169  At that point, and at any time subsequently receiving a byte via os.Stdin, the
   170  plug-in will perform the authentication (or renewal) step. The result will
   171  be used to generate two pieces of information:
   172  
   173  * X-Auth-Token - to be used by PFSAgent in all subsequent Swift API calls (GETs, PUTs, and Mouunt JSON RPC requests)
   174  * StorageURL - to form the first portion of any such Swift API path
   175  
   176  The response (either at start-up or following reception of a
   177  byte via os.Stdin) is sent to os.Stdout as a UTF-8 encoded JSON object:
   178  
   179  ```
   180  {
   181      "AuthToken"  : "<valid Swift Auth Token (i.e. to be passed via `X-Auth-Token`)>"
   182      "StorageURL" : "<path including correct transport ("http{|s}"), version ("proxyfs"), and SwiftAccount>",
   183  }
   184  ```
   185  
   186  At PFSAgent termination, the plug-in should see os.Stdin close.
   187  This should trigger the plug-in to also exit (perhaps after
   188  cleaning up any not-to-be-persisted details of its execution).