github.com/alwaysproblem/mlserving-tutorial@v0.0.0-20221124033215-121cfddbfbf4/TFserving/CustomOp/README.md (about)

     1  # Custom Operations for Tensorflow and Tensorflow serving
     2  
     3  This is an example for integration a custom operation to tensorflow and tensorflow serving. For simplification, we take the operation system with ubuntu 20.04 and python 3.8 and gcc/g++ 9.4. The basic tools for bazel is `bazelisk`. For tensorflow version, we take version 2.10 and tfserving 2.10 since the tensorflow APIs are intend to be stable. The custom operation will be the CPU add_index operation. In order to integrate custom operation, we need to set up a environment of building source code of tensorflow and tensorflow server. The procedures of this are listed here:
     4  
     5  1. Setup environment successfully (can build tensorflow and tensorflow serving from source code)
     6  2. Create an operation for tensorflow. (from repo [custom-op](https://github.com/tensorflow/custom-op))
     7  3. Integrate the custom op into tensorflow serving
     8  4. initialize server and test the custom op.
     9  
    10  ## Create an op for Tensorflow
    11  
    12  ### Environment setup tutorial
    13  
    14  - clone this repo
    15  
    16  ```bash
    17  $ git clone https://github.com/Alwaysproblem/MLserving-tutorial.git
    18  $ cd MLserving-tutorial/TFserving/CustomOp
    19  ```
    20  
    21  - The code from `custom-op` is modified from the [custom-op](https://github.com/tensorflow/custom-op). you can also clone the template and change the code.
    22  
    23  ```bash
    24  $ git clone https://github.com/tensorflow/custom-op.git
    25  ```
    26  
    27  - install gcc/g++
    28  
    29  ```bash
    30  apt update -y
    31  apt install g++-9 gcc-9 -y
    32  update-alternatives --install /usr/local/bin/g++ g++ /usr/bin/g++-9 90
    33  update-alternatives --install /usr/local/bin/gcc gcc /usr/bin/gcc-9 90
    34  apt install rsync -y
    35  ```
    36  
    37  - Install python
    38  
    39  ```bash
    40  $ sudo apt install python3-dev python3-pip
    41  $ pip install -U --user pip numpy wheel packaging requests opt_einsum
    42  $ pip install -U --user keras_preprocessing --no-deps
    43  ```
    44  
    45  - create an virtual environment (optional)
    46  
    47  ```bash
    48  $ conda create -n build python=3.8 -y
    49  $ conda activate build
    50  $ pip install -U --user pip numpy wheel packaging requests opt_einsum
    51  $ pip install -U --user keras_preprocessing --no-deps
    52  ```
    53  
    54  - Install Bazel
    55  
    56  ```bash
    57  $ wget https://github.com/bazelbuild/bazelisk/releases/download/v1.15.0/bazelisk-linux-amd64
    58  $ mv bazelisk-linux-amd64 /usr/local/bin/bazel
    59  $ bazel help
    60  
    61  # WARNING: Invoking Bazel in batch mode since it is not invoked from within a workspace (below a directory having a WORKSPACE file).
    62  #                                                            [bazel release 5.3.2]
    63  # Usage: bazel <command> <options> ...
    64  
    65  # Available commands:
    66  #   analyze-profile     Analyzes build profile data.
    67  #   aquery              Analyzes the given targets and queries the action graph.
    68  #   build               Builds the specified targets.
    69  #   canonicalize-flags  Canonicalizes a list of bazel options.
    70  #   clean               Removes output files and optionally stops the server.
    71  #   coverage            Generates code coverage report for specified test targets.
    72  #   cquery              Loads, analyzes, and queries the specified targets w/ configurations.
    73  #   dump                Dumps the internal state of the bazel server process.
    74  #   fetch               Fetches external repositories that are prerequisites to the targets.
    75  #   help                Prints help for commands, or the index.
    76  #   info                Displays runtime info about the bazel server.
    77  #   license             Prints the license of this software.
    78  #   mobile-install      Installs targets to mobile devices.
    79  #   print_action        Prints the command line args for compiling a file.
    80  #   query               Executes a dependency graph query.
    81  #   run                 Runs the specified target.
    82  #   shutdown            Stops the bazel server.
    83  #   sync                Syncs all repositories specified in the workspace file
    84  #   test                Builds and runs the specified test targets.
    85  #   version             Prints version information for bazel.
    86  
    87  # Getting more help:
    88  #   bazel help <command>
    89  #                    Prints help and options for <command>.
    90  #   bazel help startup_options
    91  #                    Options for the JVM hosting bazel.
    92  #   bazel help target-syntax
    93  #                    Explains the syntax for specifying targets.
    94  #   bazel help info-keys
    95  #                    Displays a list of keys used by the info command.
    96  ```
    97  
    98  - Enter the custom-op dir
    99  
   100  ```bash
   101  cd custom-op
   102  ```
   103  
   104  - configure the application
   105  
   106  ```bash
   107  $ bash ./configure.sh # all yes!
   108  ```
   109  
   110  - test for env
   111  
   112  ```bash
   113  bazel test //tensorflow_zero_out:zero_out_ops_py_test
   114  ```
   115  
   116  ### Create an Op
   117  
   118  - Build wheel
   119  
   120  ```bash
   121  bazel build //:build_pip_pkg
   122  bazel-bin/build_pip_pkg artifacts
   123  ls artifacts
   124  # tensorflow_custom_ops-*.whl
   125  ```
   126  
   127  - Install custom ops wheel to the virtual env
   128  
   129  ```bash
   130  python -m pip install artifacts/tensorflow_custom_ops-*.whl
   131  ```