Inference of Caffe* and TensorFlow* Trained Models with Intel’s Deep Learning Deployment Toolkit

Install Deployment Toolkit

First, download Deployment Toolkit.

Then, install the Deployment Toolkit.

The application by default is installed in /opt/intel/deeplearning_XXX/(for root user) or in ~/intel/deeplearning_XXX/(for non-root user). Further in the tutorial we will refer to it as IE_INSTALL.

E.g. ~/intel/deeplearning_deploymenttoolkit_2017.1.0.4463/.

Read introduction to the tool.

Configure Model Optimizer (MO) for Caffe

Configure the Model Optimizer (MO) for Caffe.

Comments for MO configuration steps:

1. ModelOptimizer folder is

    IE_INSTALL/deployment_tools/model_optimizer

2. When copying files to Caffe, you can use the following commands, assuming you are in
   

    IE_INSTALL/deployment_tools/model_optimizer

   Example command could be:

   cp -R adapters/include/caffe/* CAFFE_HOME/include/caffe/
   cp -R adapters/src/caffe/* CAFFE_HOME/src/caffe

3. To make sure that rebuild of Caffe catched new adapters, in console output you will see something like:

   CXX src/caffe/LayerDescriptor.cpp
   CXX src/caffe/CaffeCustomLayer.cpp
   CXX src/caffe/CaffeAPI.cpp
   CXX src/caffe/NetworkDescriptor.cpp

4. If you get a problem with hdf5 library on (at least on Ubuntu 16.04), use the following fix. By default the built library is in
   

   CAFFE_HOME/build/lib

You need to set a variable FRAMEWORK_HOME pointing to it.

E.g. write in ~/.bashrc:

export FRAMEWORK_HOME="CAFFE_HOME/build/lib"
export LD_LIBRARY_PATH="CAFFE_HOME/build/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH="IE_INSTALL/deployment_tools/model_optimizer/bin:$LD_LIBRARY_PATH"

After you have configured MO on the machine, you need to convert existing Caffe model to the IR.

Configure Model Optimizer (MO) for TensorFlow (TF)

Configure the Model Optimizer (MO) for Tensorflow.

Comments for MO configuration steps:

Install TensorFlow from sources. Further in this tutorial we will refer to the name of framework asTF.

Follow this link for full instruction.

Further in this tutorial, we will refer the directory, containing TF sources (be default it istensorflow) as TF_HOME.

Comments for TensorFlow installation steps:

1. After cloning and going into the tensorflow(by default) directory, checkout tor1.2 branch:

   git checkout r1.2

2. If you have built TF with python 3.5, it might be useful to run pip installation of TF wheel with specifying that it is the pip of python version 3+.

   E.g.

   sudo pip3 install /tmp/tensorflow_pkg/tensorflow-1.2.1-cp35-cp35m-linux_x86_64.whl

3. If you decide to check TF installation leave tensorflow directory. Unless, you will most likely get error while importing TF from python console.

Build Graph Transform Tool, here are instructions.

Comments for Graph Transform Tool installation steps:

bazel build tensorflow/tools/graph_transforms:transform_graph

After that you need to go the ModelOptimizerForTensorFlow in MO folder. E.g.

cd IE_INSTALL/deployment_tools/model_optimizer/ModelOptimizerForTensorFlow

Run the configuration script with necessary parameters. For instance, it should look like:

python configure.py --framework_path TF_HOME

Note: Make sure that there are two dashes before the argument. Sometimes the page is rendered with single dash and you will get an error in terminal.

Install modeloptimizer package:

sudo python3 setup.py install

After you have configured MO on the machine, you need to convert existing Tensorflow model to the IR.

Convert Caffe model to IR

In this tutorial we will take already trained model. In particular, AlexNet from the Caffe ModelZoo.

We will refer to the directory with Caffe model as MODEL_DIR. It should contain a .prototxt (deploy-ready topology structure) and .caffemodel (trained weights) files.

To run MO, use tutorial.

Comments for MO running steps:

1. To just generate IR, you need to execute it only with options: precision, weights, deploy proto file and that it will be the input for IE (-i).

   ./ModelOptimizer -p FP32
   -w MODEL_DIR/bvlc_alexnet.caffemodel
   -d MODEL_DIR/deploy.prototxt -i -b 1

It should output something like "Writing binary data to: Artifacts/AlexNet/AlexNet.bin".

Meaning that AlexNet IR was generated and both (.xml - topology description and .bin - weights of the model) are placed in Artifacts.

After you have generated IR, let's run it via the appropriate sample. In this case, as AlexNet is a classification model, you need to refer to instructions for building IE samples and running the classification sample.

Convert TensorFlow model to IR

In this tutorial we will take already trained model. In particular, Inception V1 Model from TF ModelZoo.

We will refer to the directory with TF model as MODEL_DIR. It should contain a .pb - a binary protobuf file with deploy-ready topology structure and trained weights.

How to get model from the Model Zoo and convert it to IR

If you have already a model in a .pb format, skip this section and go directly to IR generation.

Originally, you get model checkpoint from the given link. Unzip it to the folder. We will refer to it as CKPT_DIR.

To get the desired .pb file, which is required to generate valid IR, you need to make the .pb file first.

To do that, go to the specific folder:

cd IE_INSTALL/deployment_tools/model_optimizer/ModelOptimizerForTensorFlow/modeloptimizer/frameworks/tensorflow

After that you need to run freeze_checkpoint.py:

python3 freeze_checkpoint.py --checkpoint CKPT_DIR/inception_v1.ckpt --output_pb / MODEL_DIR/inception.pb --output InceptionV1/Logits/Predictions/Reshape_1 --net inception_v1

If there is no error in terminal, you will find the .pb file in a MODEL_DIR.

Now move back to the root folder:

cd IE_INSTALL/deployment_tools/model_optimizer/ModelOptimizerForTensorFlow

To run MO, use tutorial.

Comments for MO running steps:

1. When specifying a path to the protobuf file, use absolute paths only. Relative ones like ~/models/inception woould not work for you.

2. To just generate IR, you need first to know the input and the outpuit layers names of the model you are going to convert. If you don't know that, you can use summarize_graph util from the TF, otherwise skip this step. To use it:

   2.1 Go to TF directory. E.g.:

   cd TF_HOME

   2.2 Build it and run, using the instruction.

   In the output, you will get something like:

   Found 1 possible inputs: (name=Placeholder, type=float(1), shape=[1,224,224,3])
   No variables spotted.
   Found 1 possible outputs: (name=InceptionV1/Logits/Predictions/Reshape_1, op=Reshape)
   Found 6633279 (6.63M) const parameters, 0 (0) variable parameters, and 0 control_edges
   Op types used: 298 Const, 231 Identity, 114 Add, 114 Mul, 58 Conv2D, 57 Relu, 57 Rsqrt, 57 Sub, 13 MaxPool, 9 ConcatV2, 2 Reshape, 1 AvgPool, 1 BiasAdd, 1 Placeholder, 1 Softmax, 1 Squeeze

   In this case input layer name is input, output layer is InceptionV3/Predictions/Reshape_1.

3. To generate IR, you need execute MO with necessary options: path to protobuf model, output path, graph input file, input layer name, output layer name.

   E.g.:

   python3 modeloptimizer/scripts/model_optimizer.py \
   --input_model=MODEL_DIR/inception.pb \
   --output_model=IE_INSTALL/deployment_tools/model_optimizer/ModelOptimizerForTensorFlow/gen/v1.pb \
   --input=Placeholder \
   --output=InceptionV1/Logits/Predictions/Reshape_1 \
   --transforms="\
   strip_unused_nodes(type=float;shape=1,224,224,3) \
   remove_nodes(op=Identity) \
   remove_nodes(op=CheckNumerics) \
   fold_constants(ignore_errors=true) \
   fold_batch_norms \
   strip_unused_nodes(type=float; shape=1,224,224,3) \
   remove_nodes(op=Identity) \
   remove_nodes(op=CheckNumerics) \
   fold_constants(ignore_errors=true) \
   fold_batch_norms \
   strip_unused_nodes(type=float; shape=1,224,224,3) \
   remove_nodes(op=Identity) \
   remove_nodes(op=CheckNumerics) \
   fold_constants(ignore_errors=true) \
   fold_batch_norms  \
   calc_shapes(input_types=float; input_shapes=1,224,224,3) \
   create_ir(model_name=v1; output_dir=gen)"

   It is very important to set the name of the input and output layer in --input and--output argument respectively.

   Note that, this execution influences the original model, so make sure you have made a copy of it in advance. If execution of IR generation fails, it would be better to use the clean model copy.

If you don't see any error message, it means that Inception V1 IR was generated and both (.xml - topology description and .bin - weights of the model) are placed in the folder with output model that you specified in the output_model property when running MO (step 3). In this tutorial it is /home/temp/inception.

After you have generated IR, let's run it via the appropriate sample. In this case, as Inception V1 is a classification model, you need to refer to instructions for building IE samples and then running classification sample.

Build IE samples

General info about IE is here.

More specifically samples information and description can be found here.

Follow these instructions and build samples.

Comments for IE samples building steps:

1. From the build folder in samples directory:

   cmake -DCMAKE_BUILD_TYPE=Release ..
   make

2. After that you need to go to the applications binaries:

   cd IE_INSTALL/deployment_tools/inference_engine/bin/intel64/Release

You will find there binaries for all samples in IE.

IE provides an opportunity to infer the IR on a specific device. Currently, it supports CPU and GPU.

To make the IE recognize required libraries availability, use the setvars.sh script, which will set all necessary environment variables.

What is left is running the required sample with appropriate commands, providing IR information.

Run IE classification sample

If your model is a classification one, it should be infered using the classification sample of IE. To run it, you can refer to the following instructions page.

We will refer to the directory which contains IR (.xml and .bin files) as IR_DIR.

For that, run (expect that you are in a Release folder):

source ../../setvars.sh

For IR you have generated with MO, the command to run inference can be the following:

./classification_sample \
-i ~/Downloads/cat3.bmp \
-m IR_DIR/AlexNet.xml \
-d CPU

Please note, that it IE assumes that weights are in the same folder as .xml file.

You will see top-10 predictions output for the given image.