github.com/NVIDIA/aistore@v1.3.23-0.20240517131212-7df6609be51d/docs/tensorflow.md (about)

     1  ---
     2  layout: post
     3  title: TENSORFLOW
     4  permalink: /docs/tensorflow
     5  redirect_from:
     6   - /tensorflow.md/
     7   - /docs/tensorflow.md/
     8  ---
     9  
    10  ## Table of Contents
    11  
    12  - [Overview](#overview)
    13  - [Examples](#examples)
    14  
    15  ## Overview
    16  
    17  AIS cluster provides out-of-the-box integration with TensorFlow TFRecord format
    18  
    19  - Creating TensorFlow datasets from TFRecords stored in AIS cluster with `tf.data.TFRecordDataset` API. See [S3 compatibility docs](s3compat.md)
    20  - Creating TensorFlow datasets from *TAR* files stored in AIS cluster with `tf.data.TFRecordDataset` API.
    21  The conversion is executed remotely, on the fly in the cluster.
    22  
    23  ## Examples
    24  
    25  ### Create TensorFlow dataset from TFRecords stored in AIS
    26  
    27  ```python
    28  import tensorflow as tf
    29  import os
    30  
    31  os.environ["S3_ENDPOINT"] = CLUSTER_ENDPOINT
    32  
    33  # (...)
    34  
    35  train_dataset = tf.data.TFRecordDataset(filenames=[
    36      "s3://tf/train-1.tfrecord",
    37      "s3://tf/train-2.tfrecord",
    38  ]).map(record_parser).batch(BATCH_SIZE)
    39  
    40  # (...)
    41  
    42  model.fit(train_dataset, ...)
    43  ```
    44  
    45  ### Create TensorFlow dataset from TARs stored in AIS
    46  
    47  ```python
    48  import tensorflow as tf
    49  import os
    50  
    51  os.environ["S3_ENDPOINT"] = CLUSTER_ENDPOINT
    52  
    53  # (...)
    54  
    55  # ?uuid query param to convert TAR to a transformed data.
    56  
    57  train_dataset = tf.data.TFRecordDataset(filenames=[
    58      "s3://tf/train-1.tar?uuid=<uuid of tensorflow transformer>",
    59      "s3://tf/train-2.tar?uuid=<uuid of tensorflow transformer>",
    60  ]).map(record_parser).batch(BATCH_SIZE)
    61  
    62  # (...)
    63  
    64  model.fit(train_dataset, ...)
    65  ```