AN INNOVATIVE PERSON DETECTION SYSTEM BASED ON THERMAL IMAGING CAMERAS DEDICATE FOR UNDERGROUND BELT CONVEYORS

: Within the INESI-project (Increasing Efficiency and Safety Improvement in Underground Mining Transportation Routes) long-wavelength infrared (LWIR) cameras are used for detecting persons on underground belt conveyors or within hazardous areas e.g. in front of crusher or skip vessels by the project partners KOMAG and the Institute for Advanced Mining Technologies (AMT). The test case for evaluating the performance of thermal imaging regarding these applications is the Polish Sobieski under-ground coal mine operated by Tauron mining company. By the development of thermal image processing algorithms, an automated detection of persons and classification of different objects was achieved. This may allow implementing smart services for person detection on underground belt conveyors as well as material characterization between coal, rock and disturbing objects on belt conveyors.


INTRODUCTION
Underground Coal Mining is a very challenging environment regarding the safety of the miners, dealing with heavy machinery in complex situations and harsh conditions.The digitization and automation of specific mining operations allows the development _________ of technical solutions to increase the safety and productivity in underground mining conditions (Jasiulek et al. 2019).Various research and development projects as well as the INESI-project started in July deal with digitization and automation of the different processes within mining operations to increase resource efficiency, productivity and safety in underground mining.The fundamental aim of the project is the development of a person detection system for underground belt conveyors operated in hazardous areas.
Therefore the KOMAG Institute of Mining Technology and the Institute for Advanced Mining Technologies (AMT) of the RWTH Aachen University validate thermal imaging as a method for person detection in underground mining transportation processes.For this purpose, measurements in laboratory and in situ underground coal mining conditions are taken.The aim of these tests is to assess thermal imaging regarding its general usability in harsh underground mining conditions.Basing on this, the feasibility to detect persons on an operating underground belt conveyor will be investigated.The measurements will also enable to such validate and include the overall differentiation among persons in different positions, different materials such as rock or coal and in the case of an empty belt conveyor.

PRINCIPLES OF THERMAL IMAGING
To illustrate and explain the technical qualities of thermal imaging in harsh underground conditions, the functional principal will be explained in comparison to the visible radiation spectrum of light.Typical RGB cameras can capture images of reflected visual light, which is located within the visible radiation spectrum of roughly 0.38 µm (violet) to 0.78 µm (red) (see Fig. 1).The infrared radiation spectrum is located between a wavelength range from 0.78 µm to 1 mm.Cameras, which are developed to capture images within this spectrum, are designated as infrared cameras.To ensure a reliable use of the person detection system in underground mining conditions, the applied camera must be resistant to ambient light and especially specific absorptions of electromagnetic radiation.Atmospheric absorptions which can occur in underground coal mining conditions are caused by the presence of for example methane, oxygen, carbon dioxide, water or humidity.
Because of that absorptions occurring in the underground mine atmosphere, cameras which can operate in the long-wavelength infrared (LWIR) spectrum from 8 µm to 15 µm wavelength fulfil the requirements for a reliable person detection (Fig. 1).This kind of cameras are often referred to as thermal cameras, LWIR-cameras or thermal imaging systems.Thermal cameras are designed to visualize thermal radiation, which is emitted by each object with a temperature above 0 K and work with a wavelength spectrum which is not influenced significantly either by atmospheric absorption or electromagnetic radiation.The gathered radiation is visualized in grey-scaled images, so called thermal images.Within the thermal image a higher radiation intensity is illustrated with a brighter image according to 32 × 32 pixels (Bernhard 2014).There are several extrinsic and intrinsic factors, which influence the amount of radiation emitted by an object.The amount of radiation is for example dependent on temperature, material characteristics and surface conditions.Reflections and interfering radiations can maximize radiation intensity.In contrast to this, radiation can also be reduced by atmospheric absorptions, entry lens and filter materials (Vollmer and Möllmann, 2010) (Polnik, 2015).

THERMAL CAMERA SYSTEM FOR PERSON DETECTION SYSTEM
ON UNDERGROUND BELT CONVEYOR The hardware equipment, which is used for the person detection system is illustrated in Table 1.To ensure a safe operation in underground coal mines, the thermalimaging person detection system is developed and constructed to operate in an explosive environment.For this purpose, an explosion-proof ATEX-certified computer (BbCM-05eS.1101) is used as a finished product by the KOMAG Division of Mechatronic Systems.In Figure 2 on the left side the BbCM-05eS.1101explosion-proof computer with the intrinsically safe keyboard type PCK001 is illustrated at the testing station of the KOMAG Institute of Mining Technology.The picture on the right side presents the same computer after opening the main chamber and two connection chambers.Figure 3 illustrates a close-up of the intrinsically safe keyboard type PCK001.The computer is connected to the keyboard with a wire equipped with a 9-pin D-SUB connector shown in Fig. 4. Since the thermal imaging system for a detection of people on a belt conveyor is to be used in underground mining conditions, a four-fiber single-mode cable was used as a transmission means.Single-mode fiber was chosen basing on the understanding of the transmission networks at hard coal mines.Table 2 shows the technical parameters of the computer in explosion-proof make.
The thermal imaging camera consists of the following electronic components located at ITG KOMAG, such as an infrared detector (Fig. 5, left) and lens (Fig. 5, right) and a video camera (Fig. 6, left).The power supply with the appropriate current efficiency was used for supply (Fig. 6, right).As the fiber transmission is planned to be used as a transmission means, the camera is additionally equipped with a fiber optic converter.The complete solution, due to its operational safety underground where an explosion hazard occurs, was enclosed in a flameproof enclosure (Fig. 7).The technical data of the used explosion-proof thermal imaging camera are listed in Table 3.The individual electronic components are installed in an explosion-proof housing type Bb.The individual components of the thermal imaging camera are mounted on a stand made of a PVC plate together with a power supply and a protection circuit.The signal converter together with the alarm relay is installed above the thermal imaging camera as shown in Fig. 7 on the left side.On the right side of Fig. 7 the completed thermal imaging camera system in explosion-proof housing is presented (Polnik 2016).
The thermal imaging camera is equipped with a mounting system enabling the lens to be set in the x, y, z coordinates, with a possibility of fixing to any steel structure with screws.
Possibility of using a magnetic holder with neodymium magnet is also assumed.These neodymium magnets have a lifting capacity of 20 kg each, so in the case of the four for a single handle gives the total capacity is 80 kg.

MEASUREMENTS AND PRELIMINARY TESTS
As regards the presented advantages it can be seen that thermal imaging is a promising sensor technology, investigated in the INESI-project as a reliable sensor in underground coal mining conditions for a person detection on a belt conveyor.For developing the person detection system several measurements were taken in defined conditions followed by measurements taken in in situ mining conditions using a belt conveyor in the Polish Sobieski underground coal mine operated by the Tauron mining company.

DATA ACQUISITION IN LABORATORY
With the aim to gather data under defined conditions for developing a database of the following image categories, the laboratory tests were conducted.During all the laboratory measurements the thermal imaging camera was positioned orthogonally in the 2-metre distance to the conveyor.The following image data categories were gathered: 1. Person: a. Different standing positions, b.Different sitting positions, c.Different lying positions; 2. Materials: d.Coal, e. Rock, f.Various objects (e.g., pneumatic drilling equipment, etc.); 3. Scenery.All the measurements in the category Person were conducted at the conveyor static position and at the person's motion.For the image category Material coal, rock and various objects were heated up for a period of 30 seconds with a halogenic lamp.The aim was to simulate the temperature of extracted coal and rock in an underground mine which is about 30 °C-40 °C. Figure 8 shows images from each image category.With this measurement set-up all the laboratory tests were conducted.The measurements were taken with the frame rate of 20 Hz.

DATA ACQUISITION IN SOBIESKI UNDERGROUND COAL MINE
To examine the usability of the thermal imaging camera in in-situ underground mining conditions some measurements were conducted in the Sobieski underground coal mine, operated by the Tauron mining company in 2018.For the field tests two different locations were chosen.The first location was in the transportation route with a belt conveyor, transporting the run-of-mine to the processing plant.The ambient temperature in this location was about 0 °C with an air humidity of 88%.Men-riding on this belt conveyor was not possible due to the feeding process of the processing plant and nearby power plant, so the measurements were taken in the second location of the underground mine.The belt conveyor in this location was designed for men-riding.The ambient temperature in this location was about 6 °C with an air humidity of 89%.In both locations the thermal imaging camera was positioned orthogonally to the transportation direction of the belt conveyor.
The static position of the camera gave a homogenous background for each image category.The thermal image camera was connected to the explosion-proof computer, which ensured data transmission, recording and data storage (Fig. 9).
In both locations the static and dynamic measurements were taken to collect data for developing the database including all the image categories.The dynamic meas-urements were conducted at the speed of 2.5 m/s.On this basis the thermal images for the following image categories could be gathered: 1  The measurements of c. categories were conducted in the case of different persons in different positions (e.g., sitting, lying, etc.). Figure 10 shows images from each image category.When comparing the image categories in Fig. 10, it becomes apparent that different ambient temperatures in both locations have an impact on the gathered data.Due to this fact a direct comparison of both locations is given.For a detection of persons on the belt conveyor the thermal images of a. and c. categories are considered (Uth 2019).

APPROACH TO PERSON DETECTION ON UNDERGROUND BELT CONVEYOR
Thermal images represent the thermal radiation of an observed area in a grey-scale.To increase the range of information several digital image-processing techniques are applied to the thermal images.In the following part of the article different steps of image processing are mentioned.They were developed using thermal images obtained in laboratory conditions.The steps of digital image processing are a combination of standard and specific image processing operations, which were especially developed at the Institute for Advanced Mining Technologies (AMT).The person detection system mainly consists of the image processing steps: • Pre-processing for noise reduction and detail enhancement; • Feature extraction for database creation; • Machine learning approach for person detection.
A fundamental requirement for the person detection in hazardous areas of an underground mine, for example in front of a crusher or skip vessels, is the static position of the thermal imaging camera.The images are tailored to their most important region, also known as the region of interest (ROI) to reduce unimportant information and noise in the image.Finally, the contrast of images is spread over the entire image area in order to visualize differences in radiation.
Subsequent to the pre-processing step the thermal images are deposited in the database.The following extraction of the contained features is performed using MATLAB © (Uth, Polnik 2018).Within the extraction of features in thermal images, specific interest points are characterized by: • X, Y-Coordinates; • Size; • Orientation.
Figure 11 illustrates the extracted interest points of a person on a belt conveyor, which characterize the contained features.
For each category (e.g., person, coal, empty belt) the specific interest points are deposited in the database, building a respective summarized describing feature vector.Using the extracted features a classification model was trained for the compiled database of the person detection.The model based on Support Vector Machines was compared to other models using MATLAB © Classification Learner.The thermal images were segmented into blocks of 32 × 32 pixels.This is needed to increase the developed database, being an important step of the machine learning approaches (Bishop 2009).Using this trained model a prediction of unknown new data e.g. in a livestream of the thermal imaging camera can be performed in real time.Figure 12 illustrates the results of the person detection.The results of the trained model are shown in Table 4.  Basing on the conducted measurements, a person in different positions (e.g., standing, sitting, lying etc.) can be detected on a belt conveyor with 97% effectiveness.Coal can be detected with 100%, which can be explained by other surrounding conditions in the location, where the measurements take place.The developed method proved, that a person's detection on an underground belt conveyor, basing on thermal imaging, is possible.As regards its implementation in hazardous areas of underground mining e.g. in front of a crusher or skip vessels, the system should be a redundant system in combination with other detection systems.

CONCLUSIONS AND OUTLOOK
Within the INESI-project, thermal imaging was investigated as a sensor technology for a person detection on an underground belt conveyor.Thermal imaging was identified to have a big potential regarding its unique properties in underground mining conditions compared to conventional RGB cameras.On the basis of laboratory tests, the general usability of thermal imaging was assessed.In the result of measurements, which were conducted in relevant in situ underground mining conditions by KOMAG and AMT the performance of the persons detection was shown.To achieve this objective KOMAG developed the ATEX-certified thermal imaging camera in close cooperation with AMT, which was responsible for a development of the person detection system.Its fundamental part included the pre-processing with standardized and specific developed algorithms in the laboratory conditions as well as the measurements taken in underground mining conditions.To develop the person detection system a database including different image categories was established.For each image category a summarized describing feature vector was configured, with the aim to detect persons on an operating belt conveyor.The generated results for a differentiation between persons and coal on the moving belt conveyor as well as an empty belt conveyor enable a person detection.The person detection system should be developed further to achieve a more precise result and a disturbing object detection.The person detection system should be a redundant system in combination with other detection systems to improve the reliability especially in the case of a failure of one of the used detection systems.In conclusion it can be stated that the AMT developed a person detection system with a thermal imaging camera, which was developed by KOMAG.Both partners confirm the fact that thermal imaging and digital image processing are suitable sensor technologies for the mining industry and that the person detection by thermal imaging on an operating underground belt conveyor with a speed of 2.5 m/s was successfully demonstrated.

Fig. 4 .
Fig. 4. D-SUB 9 connector for connecting a computer with an intrinsically safe keyboard

Fig. 8 .
Fig. 8.Samples of each image category . Static measurements: a. Empty belt conveyor, b.Belt conveyor with coal, c.Person on the belt conveyor; 2. Dynamic measurements: a. Empty belt conveyor in motion, b.Coal on moving belt conveyor, c.Person on moving belt conveyor.

Fig. 11 .
Fig. 11.Extracted features of a person on a belt conveyor characterized by presented interest points(Uth, Polnik 2018)

Table 1 .
Equipment list of person detection system

Table 4 .
Results of person detection system