Download PDF [PDF 1.39 MB]

• Introduction
• Release Notes
• Installing Intel® Deep Learning SDK Training Tool
• Introducing the Intel® Deep Learning SDK Training Tool
• Uploads Tab
• Datasets Tab
• Models Tab
• Uploading Images
• Creating a Training Dataset
• Adding Source Images to a Dataset
• Data augmentation
• Preprocessing Input Image Data
• Selecting Image Chroma type
• Resizing Images
• Setting Database options
• Generating a Dataset
• Saving, Editing and Reviewing a Dataset.
• Creating a Model
• Assigning a Dataset.
• Configuring Model Topology.
• Transforming Input Images.
• Configuring Training Parameters.
• Running the Training Process.
• Saving, Editing and Reviewing a Model
• Additional Resources

Introduction

Release Notes

Please find Release Notes at https://software.intel.com/en-us/articles/deep-learning-sdk-release-notes

Installing Intel® Deep Learning SDK Training Tool

For installation steps please see the Intel® Deep Learning SDK Training Tool Installation Guide.

Introducing the Intel® Deep Learning SDK Training Tool

The Intel® Deep Learning SDK Training Tool is a feature of the Intel Deep Learning SDK, which is a free set of tools for data scientists, researchers and software developers to develop, train, and deploy deep learning solutions. The Deployment Tool is currently unavailable.

With the Intel Deep Learning SDK Training Tool, you can:

• Easily prepare training data, design models, and train models with automated experiments and advanced visualizations
• Simplify the installation and usage of popular deep learning frameworks optimized for Intel platforms

The Training Tool is a web application running on a Linux* server and provides a user-friendly, intuitive interface for building and training deep learning models.

When you start the Training Tool and login, you are presented with a workspace displaying the home page and a series of the main tabs in the blue panel on the left side. These tabs provide access to a set of features that enable you to upload source images, create training datasets, and build and train your deep learning models.

Uploads Tab

Before you build and train your model, use this tab to upload an archive of images that will form a dataset for training the model. For details see the Uploading Images topic.

Datasets Tab

Create datasets from previously uploaded images using the Datasets panel. A dataset is not just a bunch of images, but it is a database of a specific format that holds all images arranged in categories.

While creating a dataset you can change image color mode (i.e. grayscale or RGB color), encoding format, or multiply the initial image set by data augmentation, i.e. applying different transformations to original images to create modified versions. All changes are stored in the database and do not physically affect the original files.

You can use the entire image set for training and validation (by assigning a percentage for each subset) or use separate folders for each procedure.

For more information see the Creating a Training Dataset section.

Models Tab

Use this tab to create and train your model with an existing dataset. There are three available pre-defined topologies you can use for your model:

• LeNet
• AlexNet
• GoogLeNet

While configuring the model, you can apply transformations to images from the selected dataset without changing the original image files.

While each of pre-defined models already have a set of default parameters values that are optimal for general use, the Models tab enables you to efficiently configure the learning process for specific use cases.

For more information see the Creating a Model section.

Uploading Images

Before creating training datasets, you need to upload images you intend to use for training your model.

Use the Uploads tab to upload input images as a RAR archive with the strictly predefined structure.

All images inside the uploaded archive must be divided into separate subfolders which are named after desired training labels/categories. For example, the structure of a sample archive for 0-9 digits, which could be used for training a LeNet model, may look like the following:

digits.rar/

    0/

        0_01.png

        0_02.png

         …

    1/

        1_01.png

        1_02.png

         …

       …



    9/

        9_01.png

        9_02.png

         …

Choose the archive located on your computer or on the web and specify the root directory that will hold all extracted images.

The directory path is relative to the Docker installation directory you specified while installing the Training Tool. For the installation steps, see Installing Intel® Deep Learning SDK Training Tool.

The table under the Upload button provides the information about current uploads and upload history:

Creating a Training Dataset

You can easily create training datasets using the Datasets tab. Once you click the tab, the panel comes up with the New Dataset icon and a list of previously saved datasets:

You can look up saved datasets by searching them via the name, edit, rename, delete them or complete their generation process. For more information see Saving, Editing and Reviewing a Dataset.

To start creating a dataset, click New Dataset that launches the wizard. A wizard screen contains the following elements:

1. Dataset Name field – Sets the name of the dataset
2. Dataset Description field – Sets the description for the dataset
3. Dataset Manage panel – Enables saving, running or deleting the current dataset at any step
4. Navigation panel – indicates the current step and switches between dataset creation steps.

The wizard divides the workflow of creating training image dataset into three separate steps indicated as tabs on the navigation bar in the wizard screen:

1. Define the folder that contains source images, number of files to use for training and validation and other settings in the Data folder tab.
2. Make preprocessing settings in the Image Preprocessing tab for input images.
3. Choose the image database options in the Database option tab.

Whenever you need to modify the settings you can switch over the steps using the Next and Back buttons or by clicking a tab on the navigation bar directly.

To abort creating the dataset, click the Delete icon in the toolbar in the upper right corner.

Adding Source Images to a Dataset

Start creating a dataset with setting its name and the source folder.

Set the dataset name using the Dataset Name field to identify the dataset in the dataset collection. Using meaningful names can help you find the dataset in the list when you are creating a model.

You can add annotations for the dataset if needed using the Description field.

Use the Source folder field to specify the path to the root folder that holds contents of the extracted RAR archive that you previously uploaded to the system. If you have not completed this step, see Uploading Images to learn about image archives used for datasets.

From the entire set of training images you can define the image groups for each phase of the model training process:

• Training – trains the model using a set of image samples.
• Validation – could be used for model selection and hyper-parameter fine-tuning.

To define the validation subset, choose a percentage of images for validation in the Validation percentage field. The default value is 10%.

Alternatively, you can also use separate folder for validation. You can specify this folder once you select the Use other folder option.

NOTE: If you are using other folder for validation, the respective percentage field resets to zero value.

Data augmentation

You can extend your dataset using the Training Tool augmentation feature. It enables you to enlarge the set by creating copies of existing images and applying a number of transformations such as rotating, shifting, zooming and reflecting.

You can simply specify the maximum number of transformations to be applied to each image in the dataset using the Max number of transformations per image field.

Alternatively you can use the Advanced section to additionally define which types of transformations to apply, transformation parameters and weights. Weight here is the percentage of the selected augmentation type in the total number of performed augmentations, in percent. The higher the specified weight, the more augmentations of the selected type are performed. The total weights of all selected augmentation types must be 100%.

Sometimes transformations result in exposing undefined parts of the image. For example, after zooming out, an image might have blank areas in the border area. Choose a method to fill those blank areas in augmented images using the Fill Method section:

• Constant - Fills the missing pixels with a certain hexadecimal color code value in RGB format
• Nearest - Fills the missing pixels with the values of neighboring pixels
• Wrap - Fills the missing pixels by tiling the image
• Reflect– Fills the missing pixels by reflecting a region of the image.

Preprocessing Input Image Data

You can pre-process images included in a dataset using the Image Processing tab.

Selecting Image Chroma type

The tool enables you to use color or grayscale modes for images. Choose the desired option in the Type group.

• If you select the grayscale option in creating a dataset with color images in RGB format, the tool automatically perform the pixel wise RGB-to-grayscale conversion according to the formula:
  Y = 0.299*Red + 0.587*Green + 0.114*Blue,
  where Y is the intensity/grayscale value.
• If you use the Color option for grayscale images, the algorithm uses the intensity value as the values of red, green, blue channels: R = Y, G = Y, B = Y.

Resizing Images

By default, the resize dimensions are set to 28x28 pixels but you can resize the images to arbitrary sixe with one of the available resize options:

• Squash
• Crop
• Fill
• Half crop, half fill.

The table below demonstrates the results of resizing an example image of original size 128x174 pixels, to a 100x100 pixels square image using each resizing method.

Setting Database options

The Training Tool stores all input images in a database file. At this step you can set desired database options such as database type and image encoding.

You can choose a certain type of the database using the DB backend drop-down list:

• LMDB
• LevelDB

To find more information on both types, please see LMDB and LevelDB home pages.

Choose an image encoding format from the Image Encoding drop-down list if needed. Using PNG or JPEG formats can save disk space, but may increase the training time.

Generating a Dataset

After you complete configuring the dataset, you can launch the dataset generation process by clicking the Run icon in the dataset manage panel . The Training Tool starts the process and shows the progress bar and the details of the dataset generation process:

Once the generating completes the dataset changes its status in the right tab list to Complete.

For more information about saving datasets and dataset statuses see Saving, Editing and Reviewing a Dataset. 

Saving, Editing and Reviewing a Dataset

When you click the Dataset tab, the Dataset panel comes up with the New Dataset icon and the list of previously saved datasets. Datasets in the list can be in one of three states: Draft, Ready or Completed.

You can save a dataset as a Draft at any moment before all of its mandatory fields are set. The Ready status indicates that you have set all mandatory fields for the saved dataset and it is ready to be generated. The Completed status identifies already generated datasets.

To find a dataset by its unique name, use the Search field.

You can rename, edit or delete a dataset in the Draft state using the toolbar in the right upper corner .

For a dataset in the Ready state, the Run operation is additionally available: .

To view or edit datasets in Draft or Ready state, select it from the list or click the Edit icon in the toolbar.

To view the details of a Completed dataset, select it from the list.

Creating a Model

After images were uploaded and a dataset was generated using those images, you are ready to create and train a deep learning model. To begin the model creation process, choose the Models tab in the vertical blue panel. The panel comes up with the New Model icon and the list of previously trained and drafted models displayed under the Search text field.

You can look up existing models by searching them by the given model name and rename, edit, build or delete them. For more information see Saving, Editing and Reviewing a Model.

To create a new model, use the New Model icon. Once you click it, you are presented with a wizard screen.

Wizard screen contains the following elements:

1. Model Name field – A mandatory field that sets the name of the model
2. Model description field – Adds an optional description about the model
3. Model manage panel – Enables saving, running training or deleting the current model at any step
4. Navigation panel – indicates the current step and switches between model creation steps

The new model creation process consists of four stages as illustrated in the navigation pane.

1. Select a dataset from the list of generated datasets in the Dataset Selection tab.
2. Choose and tune a pre-defined model topology in the Topology tab.
3. Transform images from the dataset if needed in the Data Transformation tab.
4. Configure default parameters to tune the learning process in the Parameters tab.

Assigning a Dataset

The first stage of creating a model is the dataset selection stage.

As the first step in this stage, enter a unique name for the model in the Model Name text field. Using meaningful names can help you find the model in the model list.

In order to quickly recognize the model among other existing models in the future, the Training Tool provides an option of entering a descriptive text about the model in the Description field.

Every model should be linked to a particular dataset at any given time. The next step is choosing a dataset that provides the model with training and validation images. Select an existing dataset from the listed datasets or search for one by the name and select it. Press Next to move on to the second stage of the process.

Configuring Model Topology

In the second stage you need to configure the topology of the model.

First step is to select a specific model topology from the pre-loaded three topologies listed in the Topology name list. These pre-loaded topologies come configured with the optimal training/validation settings for that specific topology under general use conditions. However you can customize it to match specific requirements via checking the Fine tine topology check-box. There are two levels of fine tuning available - light and medium - you may pick one as desired. Configuration options in the following stages will change upon selecting the fine tuning option and the level fine tuning chosen. Back/Next buttons in the bottom blue pane allows you to move between the four stages as needed.

Transforming Input Images

The third stage allows you to add pre-processing to the images in before they are fed to the model for training or validation.

You may add three optional pre-processing operations to the training data. Two of them, cropping and horizontal mirroring, add some degree of randomness to the training process, by applying those operations to randomly chosen training images. In image classification tasks with large datasets, these types of random pre-processing are used to enhance the performance of the learned model by making it robust to deviations of input images which may  not be covered in the training set.

The mean subtracting, if selected will be done for each and every image, and there are two options: subtract the mean image or pixel.

Configuring Training Parameters

In the fourth and final stage, training parameters (i.e. hyper-parameters) are configured to tune the training process. Pre-loaded models in the Training Tool come with a set of default values for each of the parameter fields. These values are the optimal parameter values for the given module in its general use case.

Typical training of a deep learning module involves hundreds of thousands of parameters (aka weights), hence a module is trained over-and-over with a given training set. One complete pass of the total training dataset is called an epoch. At the end of one epoch every image in the training dataset has passed exactly once through the module. You can adjust the number of epochs using the Training epochs field. This number depends on the module topology, parameter estimation algorithm (solver type), initial learning rate and the learning rate decay curve, required final accuracy, and the size of the training dataset.

Within an epoch, images in the training dataset are partitioned in to batches and the module is trained with one batch at a time. Once a batch of images passe through the module, the parameters of the module are updated, and then a next batch is used. One such pass is called an iteration. In general, a larger batch size reduces the variance in the parameter update process and may leads to a faster conversion. However, larger the batch size higher the memory usage will be during the training.

By specifying the Validation interval value, you can define how often validations should take place in  number of epochs. For example setting the value to 1 will lead to validations taking place at the end of each epoch. Use the Validation batch size value to define the size of the batch of validation images .

Training a deep learning module is a lengthy and complex task, therefore the tool  regularly takes snapshots to backup the status of the module being trained and the status of the solver. To set the frequency of backups, use the Snapshot intervals field.

Parameter (or weight) estimation is not only about optimizing the loss/error function for the training datasetas the estimated weights should be able to generalize the model to new unseen data. Using the Weight decay setting, you can adjust the regularization term of the model to avoid overfitting.

Learning rate determines the degree an update step influences the current values of the weights of the module. Larger learning rates will cause drastic changes at updates and could lead to either oscillations around the minima or missing the minima all together, while unreasonably smaller rate would lead to very slow convergence. The Base learning rate is the initial learning rate at the start of the learning process.

Momentum captures the direction of the last weiht update and helps to reduce oscilations and the possibility of getting stuck in a local minima. Momentum ranges from 0 to 1 and typically higher value such as 0.9 is used. However, it is important to use a lower learning rate when using a higher momentum to avoid drastic weight updates.

You may choose a solver type from a list of available types, the default one is the stochastic gradient descent.

Use Advanced learning rate options to further specify how the learning rate changes during the training.

There are several learning rate update policies (or curves) to choose from.

Step size determines how often the learning rate should be adjusted (in number of iterations).

Gamma controls the amount of change in the learning rate (determines the learning rate function shape) at every adjustment step.

By checking the Visualize LR box you can visualize the learning rate change as a curve.

Running the Training Process

After you complete configuring the model, you can launch the model training process by clicking the Run icon in the model manage panel . The Training Tool starts the process and shows the progress bar and the status of the model as it is being trained:

Once the training completes the model changes its status in the list to Trainingcompleted.

For more information about saving models and model statuses see Saving, Editing and Reviewing a Model.

Saving, Editing and Reviewing a Model

When you click the Dataset icon, the Dataset panel comes up with the New Dataset icon and the list of previously saved datasets. A dataset in the list can be in one of three states: Draft, Ready or TrainingCompleted.

You can only save the model as a Draft at any moment prior to setting all mandatory fields. The Ready status indicates that you have set all mandatory fields for the saved model and it is ready to be trained. The Train Completed status is achieved when the model has completed training with the associated dataset.

To find a model by the given unique name, use the Search field.

You can Rename, Edit or Delete a model in the Draft state using the toolbar in the right upper corner for that model.

For a model which is in the Ready state the Run operation is additionally available: .

To view or edit a model in either Draft or Ready state, select it from the list or click the Edit icon in the toolbar.

For a model in the TrainingCompleted state, Rename, Duplicate and Delete operations are available: .

To view the details of a completed model, select it from the list.

Additional Resources

To ask questions and share information with other users of the Intel® Deep Learning SDK Training Tool, visit Intel® Deep Learning SDK forum.