github.com/rohankumardubey/proxyfs@v0.0.0-20210108201508-653efa9ab00e/docs/source/api/index.rst (about) 1 ============================================= 2 ProxyFS Bimodal Support of Object Storage API 3 ============================================= 4 5 This document covers how ProxyFS supports the Swift Object Storage API listed at 6 http://developer.openstack.org/api-ref-objectstorage-v1.html 7 8 9 Bimodal access 10 ============== 11 12 A primary of the motivation for ProxyFS was to have the ability to read and 13 write the same data in the same account whether accessing it via the object 14 API or file system API. Such access is possible via the ``pfs`` `proxy middleware 15 <https://github.com/swiftstack/ProxyFS/tree/development/pfs_middleware/>`__, 16 but there are numerous caveats. 17 18 Writing via object API 19 ---------------------- 20 21 - Containers in Swift map to top-level directories in the volume. 22 Directory names are limited to 255 bytes (to match most filesystems); as a 23 result, container names are limited to the lower of 255 or 24 ``max_container_name_length``. 25 - When creating containers, the ``X-Storage-Policy`` header is ignored. 26 - Object names correspond to path components in the volume. 27 28 - When creating objects that have slashes in their name (such as 29 ``foo/bar/baz``), the leading sub-directories (in this case, 30 ``foo/bar``) will be automatically created if they do not already 31 exist. These are *not* automatically deleted when all objects below 32 them are deleted. 33 - You cannot delete or overwrite a directory object when there are 34 still objects "within" it. Attempting to do so will result in a 35 ``409 Conflict`` error response. This complicates deleting entire 36 containers with ``swift delete <container>`` or ``swift delete --all``. 37 - All "parent directories" in an object's name must either not 38 already exist or already exist as directories. You cannot create an 39 object if one of its parent directories is in fact an object. 40 Attempting to do so results in a ``409 Conflict`` error response. 41 - Again, directory names are limited to 255 bytes, so no parent 42 directory component may be longer. Attempting to create paths with 43 longer components results in a ``400 Bad Request`` error response. 44 - Ordinarily, object names are opaque strings, and the Swift paths 45 ``/v1/AUTH_test/foo/bar``, ``/v1/AUTH_test/foo/./bar``, 46 ``/v1/AUTH_test/foo//bar``, and ``/v1/AUTH_test/foo/baz/../bar`` are 47 all distinct. In a filesystem, ``.`` and ``..`` have special meanings, 48 which would lead these to all map to the same file. To prevent 49 unintentional overwrites, only the first name is valid in a ProxyFS 50 account. Attempting to create an object at the other names results in 51 a ``400 Bad Request`` error response. 52 53 - Object ETags (MD5 digests) are written as sysmeta, similar to what was 54 done for Swift's encryption and SLO features. If the client provides an 55 ETag header in a PUT request, it will be verified before finalizing the 56 write. 57 - Object metadata must be valid UTF-8. Ordinarily, Swift only requires 58 UTF-8 for account and container metadata. 59 - Object creation does not support the ``If-None-Match:*`` header. 60 - Objects do not support expiration (though the ``X-Delete-At`` and 61 ``X-Delete-After`` headers will be written down). 62 63 64 Reading objects via the object API 65 ---------------------------------- 66 67 - Account-level container counts, object counts, and total bytes used 68 reflect the storage of the underlying ProxyFS log segments and 69 checkpoints. As such, the counts are not particularly useful and the 70 bytes-used is only a crude approximation. 71 - When listing an account, ``delimiter``, ``prefix``, and ``reverse`` 72 query params are ignored. 73 - When querying containers, the ``X-Storage-Policy`` header returns the 74 default storage policy for the cluster, which is not necessarily the 75 storage policy being used for log segment containers or the checkpoint 76 container. 77 - Container-level object counts and total bytes used are always 0. 78 - When listing a container, the ``reverse`` query parameter is ignored. 79 Only ``/`` is supported for ``delimiter``. 80 - Auto-created directory objects are presented as zero-byte objects 81 with ``Content-Type: application/directory``. 82 83 84 Reading objects via the file API 85 -------------------------------- 86 87 - Account, container, and object metadata are inaccessible via the file API. 88 - Large objects are only assembled via the object API. DLOs or SLOs, when 89 accessed through the filesystem API, will be zero-byte files or JSON 90 (respectively), similar to what happens when including a 91 ``?multipart-manifest=get`` query parameter via the object API. 92 - Swift "symlink" objects are only followed when accessed through the object 93 API. When accessed through the filesystem API, these will be zero-byte 94 files. 95 96 Reading files via the object API 97 -------------------------------- 98 99 - Since containers map to directories, files written at the root of the 100 volume are not accessible via the object API. 101 - Since files may not have been written sequentially, the ETag *will 102 not* be the MD5 digest of the contents. It should be considered "an 103 opaque validator for differentiating between multiple representations 104 of the same resource" as `RFC 2616 105 <https://tools.ietf.org/html/rfc2616.html#section-13.3.3>`__ and 106 `RFC 7232 <https://tools.ietf.org/html/rfc7232#section-2.3>`__ 107 intended. 108 109 .. note:: 110 Filesystem access to SLO segment data can invalidate the 111 manifest, as the segment ETag may change. 112 113 114 Swift functionality broken as of 1.3 115 ==================================== 116 117 - Account and container quotas are mostly worthless. 118 - No CORS support. 119 - No support for expiring objects. 120 121 122 Swift functionality bypassed by filesystem access 123 ================================================= 124 125 - Account and container ACLs are not enforced. ProxyFS uses an entirely 126 separate authentication and authorization system. 127 - Object versioning is only enforced when overwrites and deletes happen 128 through the object API. ProxyFS is not `VMS 129 <https://en.wikipedia.org/wiki/Files-11>`__. 130 131 Swift functionality not thoroughly tested 132 ========================================= 133 134 - Form POST 135 - Bulk upload/delete 136 - Static web 137 - Versioning 138 - Encryption 139 - Symlinks