Executive Summary
Intel's Software and Services Group (SSG) engineers recently collaborated with Honeywell Corporation to launch a joint systematic project, which is the first commercial unmanned aerial vehicle inspection service for solar farms. The Intel SSG team provided the automatic defect inspection system adopting the Intel® Distribution of Caffe*-based1 deep learning technology. The results proved that deep learning technology can be applied as a general solution for numerous inspection services in the markets. They also showed that the Intel® Xeon® Scalable processors along with Intel optimized deep learning frameworks, as well as functions in the Intel® Math Kernel Library (Intel® MKL)2 can provide sufficiently competitive performance for deep learning training and inference workloads.
Background
A solar farm, which is referred to as a photovoltaic power station, is a large-scale photovoltaic system designed for the supply of merchant power into the electricity grid. To ensure the efficiency of power output, solar farms are usually far away from cities, and sited in agricultural areas with complex terrains. Routine inspection and maintenance is a herculean task for solar farms. The traditional manual inspection method can only support the inspection frequency of once in three months. Because of the hostile environment, solar panels may have defects; broken solar panel units reduce the power output efficiency. Moreover, according to the U.S. Department of Labor, utility inspection worker is one of the top ten most dangerous jobs in the United States. So, how to improve the inspection efficiency and keep workers safe at the same time is the big challenge.
Figure 1: Location shooting of solar farm inspection by Intel Falco™ 8+ UAV.
The collaboration project combines the advanced commercial Intel® Falcon™ 8+ UAV system3 and Intel® optimized deep learning platforms and technologies with Honeywell's leadership in aerospace safety. The combined solution uses multiple Intel Falcon 8+ UAVs to patrol and inspect solar farm infrastructure. The automatic defect inspection system based on the Intel Distribution of Caffe deep learning framework developed by the Intel SSG team can greatly improve the efficiency for data analysis, and reduce the workload of skilled workers. Usually, most of the deep learning workloads are accelerated with graphics processing units. Initially, Honeywell colleagues doubted whether the Intel® architecture platform could provide the capability to support their requirements. Eventually, with optimization provided by the Intel SSG team, this collaboration project provided a positive answer to this question and convinced the Honeywell colleagues of the Intel architecture platform’s computation power for deep learning tasks.
Solutions
In the solar farm panel defect inspection system, the SSG team uses the Faster R-CNN*4 object detection model, using the Intel Distribution of Caffe that is optimized for Intel® architecture platforms. There is no such topology support by original BVLC/Caffe (Berkeley Vision and Learning Center)5, and the Intel Distribution of Caffe enabled and accelerated it for Intel® processors. Faster R-CNN can detect an object's location, size, and type. We used the ZFNet* based6 Faster R-CNN model in our solution.
The training process is supervised learning. The number of sample images for training is very limited due to Honeywell lacking real captured images from UAVs. In order to use a limited training set to ensure the high accuracy and robustness of the Faster R-CNN model, we used the dataset augmentation and ensemble method in our solution. We took about 300 original solar panel images captured by thermal imaging cameras on UAVs as input, labeled them as either passed or had defects, and also marked the positions of the defects, then rotated each image 36 times, 10 degrees each time, for the purpose of data augmentation. The image size is 640 x 480. We did not augment the thermal images with different scales, since the defect detection was size- and position-sensitive. We fed the augmented input into a Faster R-CNN model for training. For the output result, we classified each original image by the results of the 36 augmented images ensemble. The workflow is as follows:
Figure 2: Workflow chart of the automatic defect inspection system.
Results
Intel Xeon Platinum 8180 processor can complete the training within six hours; the detection accuracy for solar panels is 96.3 percent, and the mean Average Precision (mAP) is 68.9 percent, given some adverse influences from environments. To illustrate the inspection effect of this system, the following sample images are shown. The image shown on the left is the original thermal image captured by UAVs. The image shown on the right is the detection result obtained from this inspection system. We can see that the classification and detection results are very accurate in this sample. Meanwhile, the confidence ratio of inspection results also provides valuable information for other decision-making systems in the subsequent process.
Figure 3: The inspection effect of the automatic defect inspection system.
In this application case, utility workers would use the inspection report to guide the engineering team to change the bad units of solar panels. We cannot replace only the exact broken part of the solar panel; the solar panel should usually be replaced as a whole. That’s why end users care more about the detection rate—whether there are defects in this solar panel. The inference performance is 188 images per second. Both the training and inference performance met the throughput requirements from Honeywell and utility workers.
Our automatic defect inspection system is very efficient for collected image analysis, and reduces manual workloads. The traditional manual inspection solution can only support the inspection frequency of once in three months. With the UAV-based inspection solution, we can support the inspection frequency of once a week if we still ask skilled utility workers to deal with the raw data, and we can support the inspection frequency of three times a week if adopting the automatic defect inspection system to deal with the raw data collected by UAVs.
HW | TFlops | SW | Batch Size | Images/Second | Time to Train |
|---|---|---|---|---|---|
Intel® Xeon® processor E5-2699 (b) | 3.1 | Caffe* optimized for Intel® architecture with the Intel® Math Kernel Library (Intel® MKL) | 1 | 0.60 | 9.3Hrs |
Intel Xeon Platinum 8180 processor (2.5 GHz 8180) (a) | 8.2 | Caffe optimized for Intel architecture with the Intel MKL | 1 | 0.93 | 6 Hrs |
Conclusion
This automatic defect inspection application for solar farms demonstrates that deep learning technology can be applied to solve real-world problems, such as unmanned inspection in harsh or dangerous environments. The architecture of Faster R-CNN can learn the sophisticated features from the input images for classification and detection tasks. This is a general solution for numerous inspection services in the markets, which can be used for oil and gas inspection, such as pipeline seepage and leakage; utilities inspection, like transmission lines and substations; and even for crisis response to emergencies. The UAVs can fly high up for close-up inspections without putting workers in danger. And the automatic defect inspection system can greatly improve the efficiency of mass data analysis without the help of skilled workers.
This project also proved that the newly launched Intel Xeon Scalable processors can provide the necessary computation capability for deep learning training and inference workloads (in addition to classical machine learning and some other artificial intelligence algorithms). Caffe optimized for Intel architecture platforms takes advantage of optimized deep learning functions in the Intel MKL. This has increased the performance in a single server over 100 times. The detailed configuration can refer to “Configuration Details” part. Recent advances in distributed algorithms have enabled the use of hundreds of servers to further reduce the time to train. These significant performance improvements made heavy, deep learning workload training on Intel architecture a reality.
Acknowledgements
A special thanks to the I2R (Ideas2Reality) program7 to support this project with funding, training, coaching and business connections: Kapil Kane, Rebecca Gu, Martin Daffner, Jerry Hu; and Yan Yin, Zhenyu Tang for supports on UAV area, and Young Wang, Frank Zhang, Jiong Gong, Haihao Shen for supports on deep learning area.
References
- Fork of BVLC/Caffe
- Intel® Math Kernel Library
- Intel® Falcon™ 8+ UAV system
- S. Ren, K. He, R. Girshick, and J. Sun, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Advances in neural information processing systems. 2015
- BVLC / Caffe
- M.D. Zeiler, R. Fergus, Visualizing and understanding Convolutional Networks. In European conference on computer vision. 2014.
- I2R (Ideas2Reality) program.
Configuration Details
a. Platform: 2S Intel Xeon Platinum 8180 CPU @ 2.50GHz (28 cores), HT disabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 384GB DDR4-2666 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.10.2.el7.x86_64. SSD: Intel® SSD DC S3700 Series (800GB, 2.5in SATA 6Gb/s, 25nm, MLC).
Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact‘, OMP_NUM_THREADS=56, CPU Freq set with cpupower frequency-set -d 2.5G -u 3.8G -g performance
Intel® Distribution of Caffe:, revision c7ed32772affaf1d9951e2a93d986d22a8d14b88. Intel C++ compiler ver. 17.0.2 20170213, Intel® Math Kernel Library version 2018.0.20170425. Intel® Distribution of Caffe run with “numactl -l“.
BVLC-Caffe: revision 91b09280f5233cafc62954c98ce8bc4c204e7475. BLAS: atlas ver. 3.10.1.
b. Platform: 2S Intel Xeon CPU E5-2699 v3 @ 2.30GHz (18 cores), HT enabled, turbo disabled, scaling governor set to “performance” via intel_pstate driver, 256GB DDR4-2133 ECC RAM. CentOS Linux release 7.3.1611 (Core), Linux kernel 3.10.0-514.el7.x86_64. OS drive: Seagate* Enterprise ST2000NX0253 2 TB 2.5" Internal Hard Drive.
Performance measured with: Environment variables: KMP_AFFINITY='granularity=fine, compact,1,0‘, OMP_NUM_THREADS=36, CPU Freq set with cpupower frequency-set -d 2.3G -u 2.3G -g performance
Intel® Distribution of Caffe: revision c7ed32772affaf1d9951e2a93d986d22a8d14b88. Intel C++ compiler ver. 17.0.2 20170213, Intel® Math Kernel Library version 2018.0.20170425. Intel® Distribution of Caffe run with “numactl -l“.
BVLC-Caffe: revision 91b09280f5233cafc62954c98ce8bc4c204e7475. BLAS: atlas ver. 3.10.1.