github.com/NVIDIA/aistore@v1.3.23-0.20240517131212-7df6609be51d/python/examples/sdk/uploading-objects.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Uploading objects to AIS with the SDK"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "pip install aistore"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from aistore import Client\n",
    "\n",
    "# Use the client class to get a reference to a named bucket:\n",
    "ais_url = \"http://localhost:8080\"\n",
    "client = Client(ais_url)\n",
    "\n",
    "# All object operations are done within the context of a bucket\n",
    "bucket = client.bucket(\"my-bck\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## There are 4 primary ways to create objects in AIS with the SDK\n",
    "1. object.put_content creates an object with raw bytes\n",
    "2. object.put_file creates an object from file contents\n",
    "3. object.promote takes files the AIS storage targets can access and promotes them to objects\n",
    "4. bucket.put_files puts an entire directory as multiple objects in AIS"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Object.put_data"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "data_obj = bucket.object(\"my-data-object\")\n",
    "data_obj.put_content(b\"raw bytes content\")\n",
    "bucket.list_objects()"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Object.put_file"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import tempfile\n",
    "\n",
    "# Create a temporary file to demonstrate put_file\n",
    "with tempfile.NamedTemporaryFile() as f:\n",
    "    f.write(b\"content inside of a local file\")\n",
    "    f.flush()\n",
    "    bucket.object(\"my-file-object\").put_file(f.name)\n",
    "bucket.list_objects()"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Object.promote\n",
    "\n",
    "This method only works if the filepath provided can be accessed by the AIS storage targets. See [the aiatscale.org blog](https://aiatscale.org/blog/2022/03/17/promote) for details on promoting.\n",
    "It will work in a local example if AIS is running on the same machine, shown below."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import shutil\n",
    "\n",
    "# Set up an example folder with multiple files\n",
    "example_dir = Path().absolute().joinpath(\"promote-example\")\n",
    "example_dir.mkdir(exist_ok=True)\n",
    "filenames = [f\"file_{i}\" for i in range(10)]\n",
    "for filename in filenames:\n",
    "    filepath = example_dir.joinpath(filename)\n",
    "    with open(filepath, \"w\") as file:\n",
    "        file.write(\"content in each test file\")\n",
    "# Promote the entire directory (more options available in the docs)\n",
    "bucket.object(\"promoted-object\").promote(str(example_dir))\n",
    "\n",
    "# Delete the example folder\n",
    "shutil.rmtree(example_dir)\n",
    "\n",
    "bucket.list_objects()"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Bucket.put_files"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Again, set up an example folder with multiple files\n",
    "example_dir = Path().absolute().joinpath(\"put_files-example\")\n",
    "example_dir.mkdir(exist_ok=True)\n",
    "filenames = [f\"file_{i}\" for i in range(10)]\n",
    "for filename in filenames:\n",
    "    filepath = example_dir.joinpath(filename)\n",
    "    with open(filepath, \"w\") as file:\n",
    "        file.write(\"content in each test file\")\n",
    "# Put each file in the entire directory (more options available in the docs, including recursive put)\n",
    "bucket.put_files(str(example_dir), obj_prefix=\"example-put-\")\n",
    "\n",
    "# Delete the example folder\n",
    "shutil.rmtree(example_dir)\n",
    "\n",
    "bucket.list_objects()"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}