github.com/rohankumardubey/proxyfs@v0.0.0-20210108201508-653efa9ab00e/CONFIGURING.md (about) 1 # Configuring ProxyFS 2 3 Two programs are produced by running `make` in the top-level directory of this repository: `mkproxyfs` and `proxyfsd`. This file endeavors to describe how to specify a configuration that is consumed by these two programs. In addition, two libraries are also produced that work in concert with Samba to provide SMB support: `libproxyfs.so` and `proxyfs.so`. The two Samba libraries, like the rest of Samba, are typically configured with a `smb.conf` file. This Samba configuration specifies what SMB Shares to expose and how to route requests to ProxyFS Volumes. This file will defer explanation of the contents of `smb.conf` to other documentation but it is best practice that all of the sections needed to be added to `smb.conf` would be derivable from the configuration provided to `mkproxyfs` and `proxyfsd`. In addition, NFS is served via the normal `nfsd` service via local FUSE exports of ProxyFS Volumes. As such, their exportation is provided to `nfsd` via the normal `/etc/exports` mechanism with info that is, again by best practice, also derivable from the configuration provided to `mkproxyfs` and `proxyfsd`. 4 5 ## ProxyFS Programs 6 7 * mkproxyfs 8 9 Used to format a ProxyFS Volume 10 11 One of the following command line options must be provided: 12 13 * -N indicates that the underlying Swift Account must be empty 14 * -I indicates that the underlying Swift Account should only be formatted if necessary 15 * -F indicates that the underlying Swift Account will first be emptied before formatting 16 17 Invocation: `mkproxyfs -N|-I|-F VolumeNameToFormat ConfFile [ConfFileOverrides]*` 18 19 * proxyfsd 20 21 Mounts all ProxyFS Volumes to be served by this instance presented via two mechanisms: 22 23 * FUSE (consumed by `nfsd`) 24 * JSON RPC (consumed by Samba, Swift Proxy, and PFSAgent) 25 26 Invocation: `proxyfsd ConfFile [ConfFileOverrides]*` 27 28 ## ConfFile and ConfFileOverrides Format 29 30 The file supplied to both `mkproxyfs` and `proxyfsd` (they should be the same) are in .INI format. As an example: 31 ``` 32 # This is a comment 33 [SectionName] # This is a comment as well 34 Key0: 35 Key1: Value1 36 Key2: Value2,Value3 37 .include included.conf 38 ``` 39 There is not hard limit to the number of values specified for a given key. 40 41 Note the `.include` directive that enables assembling a set of configuration sections, keys, and values from multiple files even when only a single ConfFile is specified on the command line to either `mkproxyfs` or `proxyfsd`. 42 43 It is permissible to repeat sections. All keys supplied in the subsequent mention of a particular section are logically appended (i.e. it's as if all keys specified in both mentions of a section were together). If a key is repeated, it replaces the previous definition of that key. Zero or more values for a given key are allowed. 44 45 While the `.include` mechanism is convenient for constructing configurations from a combination of files, there is another mechanism to amend a configuration. Note the `[ConfFileOverrides]*` optional portion of either the `mkproxyfs` or `proxyfsd` command line invocations. Each space-separated `ConfFileOverride` is of one of the forms: 46 ``` 47 SectionName.Key0= 48 SectionName.Key1=Value1 49 SectionName.Key2: Value2,Value3 50 ``` 51 As before, there is no hard limit to the number of values specified for a given key. Also, repeated keys merely replace the prior set of values for that key. 52 53 ## Key Requirements 54 55 The following table will list what keys are required or optional. While it is the convention of ProxyFS that keys are not defaulted when not provided, the need to support graceful upgrades necessitates some optional keys to be defaulted until such time as they are considered required. 56 57 There are three points in time when the configuration is consumed: 58 * At the time of `mkproyxfs` invocation to format a volume 59 * At the time when `proxyfsd` is started 60 * At the time when `proxyfsd` receives a SIGHUP 61 62 Given that a key might want to be modified between each of the above points in time, it is important to specify which ones are modifiable and under which circumstance such a modification will be honored. 63 64 The SIGHUP mechanism is provided to ask `proxyfsd` to reconsume the ConfFile without requiring a restart. Note that the ConfFileOverrides (if any) will still override any key settings when the ConfFile is re-read. 65 66 For each of the keys supported, the following table will list whether or not its required, any default if it is not, and when changes to it will be allowed, ignored, and/or honored: 67 68 | Section<br>Name | Key<br>Name | <br>Required | <br>Default | Changeable<br>On Restart | Changeable<br>On SIGHUP | 69 | ----------------------------------------- | ---------------------------------------- | ------------ | ------------------ | ------------------------ | ---------------------------- | 70 | Cluster | WhoAmI | Yes | | Yes | No | 71 | | Peers | Yes | | Yes | Yes but WhoAmI should remain | 72 | | PrivateClusterUDPPort | No | 8123 | Yes | No | 73 | | UDPPacketSendSize | No | 1400 | Yes | No | 74 | | UDPPacketRecvSize | No | 1500 | Yes | No | 75 | | UDPPacketCapPerMessage | No | 255 | Yes | No | 76 | | HeartBeatDuration | No | 1s | Yes | No | 77 | | HeartBeatMissLimit | No | 3 | Yes | No | 78 | | MessageQueueDepthPerPeer | No | 4 | Yes | No | 79 | | MaxRequestDuration | No | 1s | Yes | No | 80 | | LivenessCheckRedundancy | No | 2 | Yes | No | 81 | | LogLevel | No | <i>None</i> | Yes | No | 82 | Peer:<i>PeerName</i> | PublicIPAddr | Yes | | Yes | Yes but WhoAmI should remain | 83 | | PrivateIPAddr | Yes | | Yes | Yes but WhoAmI should remain | 84 | | ReadCacheQuotaFraction | Yes | | Yes | Yes | 85 | SwiftClient | NoAuthIPAddr | No | 127.0.0.1 | Yes | No | 86 | | NoAuthTCPPort | Yes | | Yes | No | 87 | | RetryLimit | Yes | | Yes | Yes | 88 | | RetryLimitObject | Yes | | Yes | Yes | 89 | | RetryDelay | Yes | | Yes | Yes | 90 | | RetryDelayObject | Yes | | Yes | Yes | 91 | | RetryExpBackoff | Yes | | Yes | Yes | 92 | | RetryExpBackoffObject | Yes | | Yes | Yes | 93 | | ChunkedConnectionPoolSize | Yes | | Yes | No | 94 | | NonChunkedConnectionPoolSize | Yes | | Yes | No | 95 | | ChecksumChunkedPutChunks | No | false | Yes | Yes | 96 | | SwiftReconNoWriteThreshold | No | 80 | Yes | Yes | 97 | | SwiftReconNoWriteErrno | No | ENOSPC | Yes | Yes | 98 | | SwiftReconReadOnlyThreshold | No | 90 | Yes | Yes | 99 | | SwiftReconReadOnlyErrno | No | EROFS | Yes | Yes | 100 | | SwiftConfDir | No | /etc/swift | Yes | Yes | 101 | | SwiftReconChecksPerConfCheck | No | 10 | Yes | Yes | 102 | FSGlobals | VolumeGroupList | Yes | | Yes | Yes | 103 | | CheckpointHeaderConsensusAttempts | No | 5 | Yes | No | 104 | | MountRetryLimit | No | 6 | Yes | No | 105 | | MountRetryDelay | No | 1s | Yes | No | 106 | | MountRetryExpBackoff | No | 2 | Yes | No | 107 | | LogCheckpointHeaderPosts | No | true | Yes | No | 108 | | TryLockBackoffMin | No | 10ms | Yes | No | 109 | | TryLockBackoffMax | No | 50ms | Yes | No | 110 | | TryLockSerializationThreshhold | No | 5 | Yes | No | 111 | | SymlinkMax | No | 32 | Yes | No | 112 | | CoalesceElementChunkSize | No | 16 | Yes | No | 113 | | InodeRecCacheEvictLowLimit | Yes | | Yes | No | 114 | | InodeRecCacheEvictHighLimit | Yes | | Yes | No | 115 | | LogSegmentRecCacheEvictLowLimit | Yes | | Yes | No | 116 | | LogSegmentRecCacheEvictHighLimit | Yes | | Yes | No | 117 | | BPlusTreeObjectCacheEvictLowLimit | Yes | | Yes | No | 118 | | BPlusTreeObjectCacheEvictHighLimit | Yes | | Yes | No | 119 | | CreatedDeletedObjectsCacheEvictLowLimit | No | as for LogSegments | Yes | No | 120 | | CreatedDeletedObjectsCacheEvictHighLimit | No | as for LogSegments | Yes | No | 121 | | DirEntryCacheEvictLowLimit | Yes | | Yes | No | 122 | | DirEntryCacheEvictHighLimit | Yes | | Yes | No | 123 | | FileExtentMapEvictLowLimit | Yes | | Yes | No | 124 | | FileExtentMapEvictHighLimit | Yes | | Yes | No | 125 | | EtcdEnabled | No | false | Yes but don't re-enable | No | 126 | | EtcdEndpoints | If enabled | | Yes | No | 127 | | EtcdAutoSyncInterval | If enabled | | Yes | No | 128 | | EtcdDialTimeout | If enabled | | Yes | No | 129 | | EtcdOpTimeout | If enabled | | Yes | No | 130 | | MetadataRecycleBin | No | false | Yes | No | 131 | | SMBUserList | No | <i>None</i> | Yes | Yes | 132 | VolumeGroup:<i>VolumeGroupName</i> | VolumeList | Yes | | Yes | Yes | 133 | | VirtualIPAddr | Yes | | Yes | Yes | 134 | | PrimaryPeer | Yes | | Yes | Yes but WhoAmI should remain | 135 | | ReadCacheLineSize | Yes | | Yes | No | 136 | | ReadCacheWeight | Yes | | Yes | No - though it should be | 137 | | SMBWorkgroup | No | WORKGROUP | Yes | Yes | 138 | | SMBActiveDirectoryEnabled | Yes | | Yes | Yes | 139 | | SMBActiveDirectoryRealm | Yes | | Yes | Yes | 140 | | SMBActiveDirectoryIDMapDefaultMin | Yes | | Yes | Yes | 141 | | SMBActiveDirectoryIDMapDefaultMax | Yes | | Yes | Yes | 142 | | SMBActiveDirectoryIDMapWorkgroupMin | Yes | | Yes | Yes | 143 | | SMBActiveDirectoryIDMapWorkgroupMax | Yes | | Yes | Yes | 144 | Volume:<i>VolumeName</i> | FSID | Yes | | Yes | No | 145 | | FUSEMountPointName | Yes | | Yes | Yes for newly served volume | 146 | | NFSExportClientMapList | No | <i>None</i> | Yes | Yes | 147 | | NFSExportName | No | <i>None</i> | Yes | Yes | 148 | | SMBShareName | No | <i>None</i> | Yes | Yes | 149 | | AccountName | Yes | | No | No | 150 | | AutoFormat | No | false | No | No | 151 | | NonceValuesToReserve | Yes | | Yes | Yes for newly served volume | 152 | | MaxEntriesPerDirNode | Yes | | Yes for new directories | Yes for newly served volume | 153 | | MaxExtentsPerFileNode | Yes | | Yes for new files | Yes for newly served volume | 154 | | MaxInodesPerMetadataNode | Yes | | No | No | 155 | | MaxLogSegmentsPerMetadataNode | Yes | | No | No | 156 | | MaxDirFileNodesPerMetadataNode | Yes | | No | No | 157 | | MaxCreatedDeletedObjectsPerMetadataNode | Yes | | Yes | Yes for newly served volume | 158 | | CheckpointEtcdKeyName | If enabled | | No | No | 159 | | CheckpointContainerName | Yes | | No | No | 160 | | CheckpointContainerStoragePolicy | Yes | | No | No | 161 | | CheckpointInterval | Yes | | Yes | Yes for newly served volume | 162 | | ReplayLogFileName | No | <i>None</i> | No | No | 163 | | DefaultPhysicalContainerLayout | Yes | | Yes | Yes for newly served volume | 164 | | MaxFlushSize | Yes | | Yes | Yes for newly served volume | 165 | | MaxFlushTime | Yes | | Yes | Yes for newly served volume | 166 | | FileDefragmentChunkSize | No | 10485760 | Yes | Yes for newly served volume | 167 | | FileDefragmentChunkDelay | No | 10ms | Yes | Yes for newly served volume | 168 | | ReportedBlockSize | No | 64Kibi | Yes | Yes for newly served volume | 169 | | ReportedFragmentSize | No | 64Kibi | Yes | Yes for newly served volume | 170 | | ReportedNumBlocks | No | 100Tebi/64Kibi | Yes | Yes for newly served volume | 171 | | ReportedNumInodes | No | 100Gibi | Yes | Yes for newly served volume | 172 | | SnapShotIDNumBits | No | 10 | No | No | 173 | | MaxBytesInodeCache | No | 10485760 | Yes | Yes for newly served volume | 174 | | InodeCacheEvictInterval | No | 1s | Yes | Yes for newly served volume | 175 | | SnapShotPolicy | No | <i>None</i> | Yes | Yes | 176 | | SMBValidUserList | Yes | | Yes | Yes | 177 | | SMBBrowseable | Yes | | Yes | Yes | 178 | | SMBStrictSync | Yes | | Yes | Yes | 179 | | SMBAuditLogging | Yes | | Yes | Yes | 180 | | SMBEncryptionRequired | Yes | | Yes | Yes | 181 | | ActiveLeaseEvictLowLimit | No | 5000 | Yes | Yes for newly served volume | 182 | | ActiveLeaseEvictHighLimit | No | 5010 | Yes | Yes for newly served volume | 183 | NFSClientMap:<i>MapName</i> | ClientPattern | Yes | | Yes | Yes | 184 | | AccessMode | Yes | | Yes | Yes | 185 | | RootSquash | Yes | | Yes | Yes | 186 | | Secure | Yes | | Yes | Yes | 187 | SMBUsers | <i>SMBUserName</i> | Yes | | Yes | Yes | 188 | PhysicalContainerLayout:<i>LayoutName</i> | ContainerStoragePolicy | Yes | | Yes | Yes for newly served volume | 189 | | ContainerNamePrefix | Yes | | Yes | Yes for newly served volume | 190 | | ContainersPerPeer | Yes | | Yes | Yes for newly served volume | 191 | | MaxObjectsPerContainer | Yes | | Yes | Yes for newly served volume | 192 | SnapShotPolicy:<i>PolicyName</i> | ScheduleList | Yes | | Yes | Yes | 193 | | TimeZone | Yes | | Yes | Yes | 194 | SnapShotSchedule:<i>ScheduleName</i> | CronTab | Yes | | Yes | Yes | 195 | | Keep | Yes | | Yes | Yes | 196 | JSONRPCServer | TCPPort | Yes | | Yes | No | 197 | | FastTCPPort | Yes | | Yes | No | 198 | | DataPathLogging | Yes | | Yes | No | 199 | | RetryRPCPort | No | <i>None</i> | Yes | No | 200 | | RetryRPCTTLCompleted | No | 10m | Yes | No | 201 | | RetryRPCAckTrim | No | 100ms | Yes | No | 202 | | RetryRPCDeadlineIO | No | 60s | Yes | No | 203 | | RetryRPCKeepAlivePeriod | No | 60s | Yes | No | 204 | | MinLeaseDuration | No | 250ms | Yes | No | 205 | | LeaseInterruptInterval | No | 250ms | Yes | No | 206 | | LeaseInterruptLimit | No | 20 | Yes | No | 207 | Logging | LogFilePath | No | <i>None</i> | Yes | No | 208 | | LogToConsole | No | false | Yes | No | 209 | | TraceLevelLogging | No | <i>None</i> | Yes | No | 210 | | DebugLevelLogging | No | <i>None</i> | Yes | No | 211 | EventLog | Enabled | No | false | Yes | Yes | 212 | | BufferKey | If enabled | | Yes | Yes | 213 | | BufferLength | If enabled | | Yes | Yes | 214 | | MinBackoff | If enabled | | Yes | Yes | 215 | | MaxBackoff | If enabled | | Yes | Yes | 216 | | DaemonPollDelay | No | | N/A | N/A | 217 | | DaemonOutputPath | No | os.Stdout | N/A | N/A | 218 | Stats | UDPPort | Unless TCP | | Yes | No | 219 | | TCPPort | Unless UDP | | Yes | No | 220 | | BufferLength | Yes | | Yes | No | 221 | | MaxLatency | Yes | | Yes | No | 222 | HTTPServer | TCPPort | Yes | | Yes | No | 223 | | JobHistoryMaxSize | Yes | | Yes | No | 224 | StatsLogger | Period | Yes | | Yes | Yes | 225 | | Verbose | Yes | | Yes | Yes | 226 | ProxyfsDebug | ProfileType | Yes | | Yes | No | 227 | | DebugServerPort | Yes | | Yes | No | 228 | TrackedLock | LockHoldTimeLimit | No | 0s | Yes | Yes | 229 | | LockCheckPeriod | No | 0s | Yes | Yes |