github.com/NVIDIA/aistore@v1.3.23-0.20240517131212-7df6609be51d/python/examples/sdk/accessing-objects.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Accessing AIStore objects with the SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pip install aistore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Use the client class to get a reference to a bucket:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from aistore import Client\n",
    "from pathlib import Path\n",
    "\n",
    "ais_url = \"http://localhost:8080\"\n",
    "client = Client(ais_url)\n",
    "bucket = client.bucket(\"my-bck\").create(exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### First, create a few objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "object_names = [f\"example_obj_{i}\" for i in range(10)]\n",
    "for name in object_names:\n",
    "    bucket.object(name).put_content(b\"object content\")\n",
    "# Create one with a different prefix\n",
    "bucket.object(\"prefix-example\").put_content(b\"prefixed object content\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## There are 3 ways to list the objects inside a bucket\n",
    "1. list_objects provides a page of objects and the data to get the next page\n",
    "2. list_objects_iter returns an iterator over all objects\n",
    "3. list_all_objects returns a single list of all objects (calling list_objects until exhausted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### list_objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "objects = []\n",
    "response = bucket.list_objects(page_size=3)\n",
    "objects.extend(response.entries)\n",
    "# Use the info from the previous response to get the next page of objects\n",
    "bucket.list_objects(\n",
    "    uuid=response.uuid, continuation_token=response.continuation_token, page_size=3\n",
    ")\n",
    "objects.extend(response.entries)\n",
    "# Now we should have the first 2 pages of size 3 each\n",
    "print(objects)\n",
    "print(\"Total number of objects:\", len(objects))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### list_objects_iter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "iterator = bucket.list_objects_iter(prefix=\"example_\", props=\"name,size,copies\")\n",
    "for bucket_entry in iterator:\n",
    "    print(bucket_entry)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### list_all_objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bucket.list_all_objects(prefix=\"example_\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Call head() on an object to view properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bucket.object(object_names[0]).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Object.get returns an ObjectReader object with a few ways to access the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "object_reader = bucket.object(object_names[0]).get()\n",
    "print(\"Read all from the stream:\", object_reader.read_all())\n",
    "print(\"Raw response:\", object_reader.raw())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### For larger objects, you can read in chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(\"Chunks:\")\n",
    "for chunk in bucket.object(object_names[0]).get(chunk_size=3):\n",
    "    print(chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Or you can provide your own writer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Pass a writer that appends to a file\n",
    "filename = Path().absolute().joinpath(\"example-obj-writer\")\n",
    "with open(filename, \"ab\") as writer:\n",
    "    bucket.object(object_names[0]).get(writer=writer)\n",
    "\n",
    "# Read from the file to see the output\n",
    "with open(filename, \"rb\") as reader:\n",
    "    print(reader.read())\n",
    "\n",
    "filename.unlink()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Working with multiple objects\n",
    "AIS supports multi-object operations on groups of objects. For examples of working with groups of objects see [here](https://github.com/NVIDIA/aistore/blob/master/python/examples/sdk/multi-object-operations.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "3.11.1 (main, Dec  7 2022, 01:11:34) [GCC 11.3.0]"
  },
  "vscode": {
   "interpreter": {
    "hash": "ead1b95f633dc9c51826328e1846203f51a198c6fb5f2884a80417ba131d4e82"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}