By Dr. Shahram Tafazoli, Founder & CEO of Motion Metrics
A little over two years ago, after early field trials of the initial LoaderMetrics™ prototypes, we gathered our engineering team to review three major challenges in developing missing tooth detection for wheel loaders:
The first challenge had to do with the missing tooth detection algorithm. Using a regular optical camera, we noticed that the bucket teeth are only visible from where the camera is located for at most one or two seconds in each cycle. Hence, our traditional image processing algorithm, used on ShovelMetrics™, did not have enough time to find the bucket and analyze the condition of the teeth. This resulted in a very poor capture rate of at most five captured image logs generated per hour. It was simply unacceptable.
The second challenge involved developing a lens cleaning system for the LoaderMetrics™ camera. On a cable shovel, the bucket viewing camera is mounted high up on the shovel boom and is therefore looking down whereas on a hydraulic shovel, the camera is installed on the stick and looks almost horizontally at the bucket teeth. But for loaders, we only had one option. The camera needs to be installed below the bucket looking up! This precarious position guarantees that the camera will get dirty. We quickly learned that on rainy days, mud often gets thrown at the camera, completely obstructing visibility. This video shows why we call this the “pie in the face” effect:
The third challenge had to do with the fact that because the camera is installed under the bucket, it is subject to impact from flying rocks and dirt. Our early camera bracket designs experienced significant abuse. Even with a strong bracket, the camera lens as well as the light source beside it were often in harm’s way and would not last too long in the harsh mining environment unless properly guarded.
I remember clearly that we discussed these 3 challenges and were close to calling it quits. However, as we brainstormed further, we thought of some solutions. Also, being hard-headed we told ourselves “good engineers don’t give up easily”. This is a nasty safety and efficiency problem that 3,000+ loaders operating around the globe are subject to, with no good solution out there. Here is what we came up with for each of the 3 challenges:
- Around the same time, we were forming our Artificial Intelligence team with particular expertise in Deep Learning. The idea here is rather than designing logical algorithms or traditional image processing routines, let the computer learn from humans. This in particular is referred to as Supervised Machine Learning. Our AI team was confident that if they stayed focused on this problem, they could come up with a good algorithm to automatically monitor tooth health in a matter of months. This meant that we (humans) needed to start labeling thousands of loader bucket images. We started right away. Time was of the essence and our existing customers were losing their patience. The schematic diagram below shows our devised architecture for the deep learning layers for detecting a missing tooth.
Simplified architecture of our deep learning network
- Regarding the challenge of the camera and light not staying clean, we looked into the commercial market for various lens cleaning options. We did find some traditional solutions but we needed to modify them for our application as well as enhance them to enable internet-based control of the function. We designed the LCS (Lens Cleaning System), which uses the combination of high pressure washer-fluid and high-pressure compressed air. We tested the early prototype and it was promising. This meant taking a commercially available product and upgrading it to make it an IIOT (Industrial Internet of Things) device and making it rugged enough to withstand harsh mine conditions. During this process, we also added a level sensing mechanism to inform mine personnel if the fluid level was getting too low.
- We came up with three ideas to address the third challenge. First, we further enhanced the bracket design by using thicker steel and a blocking plate on top of the camera. Second, we worked with an industrial manufacturer of thermal vision solutions to custom design a rugged thermal camera for us with compatible field of view of 78 degrees. Remember that loader teeth get quite hot as soon as the digging operation starts. This allows us to eliminate the light source as the thermal camera can easily see the teeth 24/7. Finally, our third idea came from an observation: Unlike visible range imaging, thermal vision can see beyond small (narrow) occlusions. Hence, we were free to design a strong metal mesh to be installed in front of the camera glass to guard it from falling rocks. We now call this removable guard the Motion Metrics Camera Guard (MCG).
Our camera bracket installed on the loader
These ideas got us excited. We did not give up and continued improving our system to address the above 3 challenges. Today LoaderMetrics™ is a great success thanks to this team effort.
We have developed a deep learning algorithm which almost never misses the bucket provided it can get a good view right after dumping a load, regardless of the duration. It works even if the bucket teeth are visible for just a fraction of a second. Indeed, we formed 3 teams back then to develop three parallel approaches (one with traditional image processing, one combining imaging and motion sensing, and one with deep learning) and the decision was quite easy: deep learning beat the other two approaches hands down!
During this period, our hard-working engineers managed to develop a rugged LCS solution and we tested it many times and it worked nicely. The exciting part about LCS is that with good internet connectivity at the mine, we can clean the dirty camera lens from 1000’s of kilometers away at our own facilities. Welcome to the age of connectivity.
Lens Cleaning System In Action
Our initial tests with the redesigned bracket, thermal camera, and the protective mesh guard proved very effective. We already got some results after months of testing at various mine sites. We observed some fully smashed mesh guards that got sacrificed but protected our precious thermal camera.
The camera mesh before and after being hit
The new patent-pending thermal vision LoaderMetricsTM system has recently been installed in different mines around the world on different types of loaders including LeTourneau L1850 and L2350 as well as CAT 993 and CAT 994. I was very proud when we recently received an update from our hard-working Motion Metrics LatAm SpA team accompanied by our senior field support engineer from Motion Metrics International. In the early morning on Nov. 19th, 2016, a recently commissioned LoaderMetrics™ system at a large copper mine in Chile successfully detected a missing tooth in the middle of the bucket two minutes and six seconds after the incident. Here are the processed image logs from the system:
Missing tooth images
This is a good lesson for all of us. By not giving up and breaking down the original challenge to 3 hurdles, we managed to overcome all three. I congratulate all of our hard-working engineers at Motion Metrics. This detection alone probably saved the mine a million dollars in damages while more importantly ensuring the safety of their operations.
Finally, it would be amiss if I did not point out a fun fact. After several installations of our thermal camera-based LoaderMetrics™ systems at various mines, we have noticed a strange looking object in some of our captured images. We referred to this object as a UFO originally as we had no idea what it was. You can see this in the right hand side of the images near the bucket. Is it an airplane, a light, the moon, or a faraway hot object? We had no idea about this until most recently our Support Manager, Mr. Iman Masoum, reported from the mine site in Chile that the UFO is indeed the Sun. I recall in our discussions with the thermal camera manufacturer we did bring up the so call “sun burn” issue which has been reported in the literature for thermal cameras and the manufacturer assured us it will not be an issue and that their product will survive exposure to the Sun. Here, we have the proof of that. And we thought it was a UFO!