github.com/rohankumardubey/proxyfs@v0.0.0-20210108201508-653efa9ab00e/docs/source/api/index.rst (about)

     1  =============================================
     2  ProxyFS Bimodal Support of Object Storage API
     3  =============================================
     4  
     5  This document covers how ProxyFS supports the Swift Object Storage API listed at
     6  http://developer.openstack.org/api-ref-objectstorage-v1.html
     7  
     8  
     9  Bimodal access
    10  ==============
    11  
    12  A primary of the motivation for ProxyFS was to have the ability to read and
    13  write the same data in the same account whether accessing it via the object
    14  API or file system API. Such access is possible via the ``pfs`` `proxy middleware
    15  <https://github.com/swiftstack/ProxyFS/tree/development/pfs_middleware/>`__,
    16  but there are numerous caveats.
    17  
    18  Writing via object API
    19  ----------------------
    20  
    21  - Containers in Swift map to top-level directories in the volume.
    22    Directory names are limited to 255 bytes (to match most filesystems); as a
    23    result, container names are limited to the lower of 255 or
    24    ``max_container_name_length``.
    25  - When creating containers, the ``X-Storage-Policy`` header is ignored.
    26  - Object names correspond to path components in the volume.
    27  
    28    - When creating objects that have slashes in their name (such as
    29      ``foo/bar/baz``), the leading sub-directories (in this case,
    30      ``foo/bar``) will be automatically created if they do not already
    31      exist. These are *not* automatically deleted when all objects below
    32      them are deleted.
    33    - You cannot delete or overwrite a directory object when there are
    34      still objects "within" it. Attempting to do so will result in a
    35      ``409 Conflict`` error response.  This complicates deleting entire
    36      containers with ``swift delete <container>`` or ``swift delete --all``.
    37    - All "parent directories" in an object's name must either not
    38      already exist or already exist as directories. You cannot create an
    39      object if one of its parent directories is in fact an object.
    40      Attempting to do so results in a ``409 Conflict`` error response.
    41    - Again, directory names are limited to 255 bytes, so no parent
    42      directory component may be longer. Attempting to create paths with
    43      longer components results in a ``400 Bad Request`` error response.
    44    - Ordinarily, object names are opaque strings, and the Swift paths
    45      ``/v1/AUTH_test/foo/bar``, ``/v1/AUTH_test/foo/./bar``,
    46      ``/v1/AUTH_test/foo//bar``, and ``/v1/AUTH_test/foo/baz/../bar`` are
    47      all distinct. In a filesystem, ``.`` and ``..`` have special meanings,
    48      which would lead these to all map to the same file. To prevent
    49      unintentional overwrites, only the first name is valid in a ProxyFS
    50      account. Attempting to create an object at the other names results in
    51      a ``400 Bad Request`` error response.
    52  
    53  - Object ETags (MD5 digests) are written as sysmeta, similar to what was
    54    done for Swift's encryption and SLO features. If the client provides an
    55    ETag header in a PUT request, it will be verified before finalizing the
    56    write.
    57  - Object metadata must be valid UTF-8. Ordinarily, Swift only requires
    58    UTF-8 for account and container metadata.
    59  - Object creation does not support the ``If-None-Match:*`` header.
    60  - Objects do not support expiration (though the ``X-Delete-At`` and
    61    ``X-Delete-After`` headers will be written down).
    62  
    63  
    64  Reading objects via the object API
    65  ----------------------------------
    66  
    67  - Account-level container counts, object counts, and total bytes used
    68    reflect the storage of the underlying ProxyFS log segments and
    69    checkpoints. As such, the counts are not particularly useful and the
    70    bytes-used is only a crude approximation.
    71  - When listing an account, ``delimiter``, ``prefix``, and ``reverse``
    72    query params are ignored.
    73  - When querying containers, the ``X-Storage-Policy`` header returns the
    74    default storage policy for the cluster, which is not necessarily the
    75    storage policy being used for log segment containers or the checkpoint
    76    container.
    77  - Container-level object counts and total bytes used are always 0.
    78  - When listing a container, the ``reverse`` query parameter is ignored.
    79    Only ``/`` is supported for ``delimiter``.
    80  - Auto-created directory objects are presented as zero-byte objects
    81    with ``Content-Type: application/directory``.
    82  
    83  
    84  Reading objects via the file API
    85  --------------------------------
    86  
    87  - Account, container, and object metadata are inaccessible via the file API.
    88  - Large objects are only assembled via the object API. DLOs or SLOs, when
    89    accessed through the filesystem API, will be zero-byte files or JSON
    90    (respectively), similar to what happens when including a
    91    ``?multipart-manifest=get`` query parameter via the object API.
    92  - Swift "symlink" objects are only followed when accessed through the object
    93    API. When accessed through the filesystem API, these will be zero-byte
    94    files.
    95  
    96  Reading files via the object API
    97  --------------------------------
    98  
    99  - Since containers map to directories, files written at the root of the
   100    volume are not accessible via the object API.
   101  - Since files may not have been written sequentially, the ETag *will
   102    not* be the MD5 digest of the contents. It should be considered "an
   103    opaque validator for differentiating between multiple representations
   104    of the same resource" as `RFC 2616
   105    <https://tools.ietf.org/html/rfc2616.html#section-13.3.3>`__ and
   106    `RFC 7232 <https://tools.ietf.org/html/rfc7232#section-2.3>`__
   107    intended.
   108  
   109  .. note::
   110    Filesystem access to SLO segment data can invalidate the
   111    manifest, as the segment ETag may change.
   112  
   113  
   114  Swift functionality broken as of 1.3
   115  ====================================
   116  
   117  - Account and container quotas are mostly worthless.
   118  - No CORS support.
   119  - No support for expiring objects.
   120  
   121  
   122  Swift functionality bypassed by filesystem access
   123  =================================================
   124  
   125  - Account and container ACLs are not enforced. ProxyFS uses an entirely
   126    separate authentication and authorization system.
   127  - Object versioning is only enforced when overwrites and deletes happen
   128    through the object API. ProxyFS is not `VMS
   129    <https://en.wikipedia.org/wiki/Files-11>`__.
   130  
   131  Swift functionality not thoroughly tested
   132  =========================================
   133  
   134  - Form POST
   135  - Bulk upload/delete
   136  - Static web
   137  - Versioning
   138  - Encryption
   139  - Symlinks