github.com/treeverse/lakefs@v1.24.1-0.20240520134607-95648127bfb0/design/accepted/template-server/template-server.md (about)

     1  # Template Server
     2  
     3  ## About
     4  
     5  The template server adds a simple generic templating capability to the
     6  lakeFS web server.  It adds server-side support for administrators and
     7  developers to serve templated web objects securely.
     8  
     9  Among the planned applications:
    10  
    11  * Serve capabilities to clients.  Serving capabilities from a template
    12    lets developers add them easily but allows administrators to control
    13    what is actually served.
    14  * Serve prepopulated config files for popular setups.  For instance we
    15    can serve `.lakectl` and Hadoop/Spark config files.  These sometimes
    16    need to be modified, for instance when lakeFS does not know the name
    17    of its endpoint.
    18  
    19    This can be an important part of for time-to-value.  Using templates
    20    we offer admins some flexibility in configuration: for instance they
    21    might define templates with different values, and clients can select
    22    which to receive by path.
    23    
    24  ## Template format
    25  
    26  We use Golang [text/template][text/template] to expand most templates.
    27  For browsers (based on filename, e.g. `*.html`) we instead must use
    28  [html/template][html/template] for safety.
    29  
    30  Accesses to http://<lakefs>/api/v1/templates/some/path read the template
    31  from `some/path`.
    32  
    33  The template is read by lakeFS from a specific (configured) prefix, by
    34  default `lakefs://lakefs-templates/`.  We might decide to support
    35  various object storage types; `lakefs` gives the best management and lineage
    36  capabilities, but possible supporting the production blockstore types `s3`,
    37  `azure` and `gcs` will simplify deployment of some lakeFS deployments.
    38  
    39  If storing on lakeFS, IAM authorizes users to see this template by
    40  `fs:ReadObject`, at which point expansion can start.  If we want to store
    41  templates on the blockstore types, we will probably need to add a new IAM
    42  action type for template expansion, or alternatively overload
    43  `fs:ReadObject` to support off-lakeFS URIs.  We will supply the following
    44  replacements (at least):
    45  
    46  | template function | value source                        | IAM permissions                              |
    47  |:------------------|:------------------------------------|:---------------------------------------------|
    48  | config            | exportable lakeFS configuration[^1] | `arn:lakefs:template:::config/path/to/field` |
    49  | object            | (small) object on lakeFS            | IAM read permission for that object          |
    50  | contenttype       | none; set `Content-Type:`           | (none)                                       |
    51  | new_credentials   | new credentials added to user       | `auth:CreateCredentials`                     |
    52  
    53  A configuration variable will be "exportable" if its struct field is tagged
    54  `` `export:yes` ``, or (if implementing that is too hard) if its type
    55  implements
    56  
    57  ```go
    58  interface Exportable {
    59  	Export() string
    60  }
    61  ```
    62  
    63  IAM is still checked for exportable configuration variables (which will be
    64  ON by default for users, but can be removed).
    65  
    66  Adding a freeform dictionary to our config will allow admins to set up
    67  any needed configuration.
    68  
    69  We shall supply the user object in `.user` and the parsed query args in
    70  `.query`[^2], to allow conditional operations.  We will probably add new
    71  functions or objects.
    72  
    73  ## Template expansion flow
    74  
    75  #### _:warning: This flow assumes templates stored on lakeFS. :warning:_
    76  
    77  1. User accesses the template
    78     http://<lakefs>/api/v1/templates/main/expand/me.
    79  1. lakeFS expands to a lakeFS path (by default this is
    80     lakefs://lakefs-templates/main/expand/me).
    81  1. lakeFS uses IAM to verify user has `fs:ReadObject` permission on this
    82     object.
    83  1. Object is parsed into a text/template.
    84  1. lakeFS creates template
    85     [`Funcs`](https://pkg.go.dev/text/template#FuncMap) that will check IAM
    86     permissions as required for expansion.
    87  1. Template gets expanded and returned to user.
    88  
    89  ### Credentials
    90  
    91  We supply a function `new_credentials` that creates a new set of credential
    92  for the user, registers them, and returns them.  See this [example of template
    93  functions returning multiple values][example_template_multiple].
    94  
    95  This lets us implement the requirement to create and return new credentials
    96  as part of a downloaded configuration.
    97  
    98  ## Examples
    99  
   100  (All examples subject to change as we define object format, and decide
   101  what functions are allowed.)
   102  
   103  Set a template that looks like this:
   104  
   105  ```conf
   106  spark.hadoop.fs.lakefs.impl=io.lakefs.LakeFSFileSystem
   107  {{with $creds := new_credentials}}
   108  spark.hadoop.fs.lakefs.access_key={{$creds.ID}}
   109  spark.hadoop.fs.lakefs.secret_key={{$creds.Secret}}
   110  {{end}}
   111  spark.hadoop.fs.lakefs.endpoint={{"local.templates.lakefs_url" | config}}
   112  # Or pass something from "local" to give users a different key.  In any case 
   113  # requires IAM authorization to fetch.
   114  spark.hadoop.fs.s3a.api.access_key={{"blockstore.s3.credentials.access_key_id" | config}}
   115  spark.hadoop.fs.s3a.api.secret_key={{"blockstore.s3.credentials.secret_access_key" | config}}
   116  ```
   117  
   118  This is a complete ready configuration for using lakeFSFS.  In future,
   119  we could leverage it to provide more configuration, but it immediately
   120  allows a single well-known point to set up.
   121  
   122  A future lakeFSFS version might read from the configured endpoint when
   123  it is loaded, allowing that to be the only required configuration.
   124  
   125  
   126  [text/template]: https://pkg.go.dev/text/template
   127  [html/template]: https://pkg.go.dev/html/template
   128  [example_template_multiple]: https://go.dev/play/p/1sfcodvxNvr
   129  
   130  [^1]: note that this effectively includes environment variables!
   131  [^2]: might instead be a function.