Keywords:
Digital Twin, deep learning, 3D photogrammetry, damage inspection, artificial intelligence, image segmentation
Introduction
The world is currently in the fourth industrial revolution, this revolution has been taking place since the turn of the millennium. Industry 4.0 combines the physical world with the digital and virtual worlds. This can lead to more efficient production, smarter products, and more efficient use of resources. The Digital Twin is an important element of Industry 4.0. Digital Twins are often used together with Internet of Things (IoT) and Artificial Intelligence (AI) is frequently an integrated part of the data processing section. AI can here be both machine learning and deep learning. The application of Digital Twins has been reported in several different areas including healthcare [1] and [2], smart cities [3] and [4] and manufacturing, see the review paper [5]. The latter is the area where most Digital Twin applications have been reported. There are many examples of small-scale Digital Twins but a lack of very large-scale Digital Twin projects in the literature. The reason is a lack of specific domain knowledge of how successful upscaling is done.
References [6] and [7] discuss the application of Digital Twins in the wind power industry. A cloud-based Digital Twin monitoring and analysis system is discussed in [7]. A working prototype Digital Twin of a wind farm is presented where data are fed to the model and both technical and business parameters are generated for the wind farm. This can be used to evaluate the wind farm both technically and economically. A case study for wind farm use and smart grid energy consumption is discussed in [6].
A key enabling technology in the wind energy industry and many other industries is digitalization. Sensors placed on wind turbines will generate large amounts of data that when combined with other digital technologies such as Big Data, Internet of Things, and Cloud Computing will open up many new possibilities. The data from sensors and wind turbine inspection measurements can be combined in a Digital Twin. Many different definitions of a Digital Twin are used in the literature but we will use the definition of Bolton et al. [8], where a Digital Twin is a ‘dynamic virtual representation of a physical object or system across its lifecycle, using real-time data to enable understanding, learning and reasoning’. A Digital Twin can be both data-driven and based on physical models. Sensor data from the operating access for example a wind turbine are used to update the state of the Digital Twin. Advanced numerical tools must be further developed to better understand sensor signals and inspection data and simulate the consequence of structural and material deterioration. Some examples can be found in [9, 10]. The structural performance of each component of the wind turbine running in the field can be evaluated using its counterpart Digital Twin in a control center.
The Digital Twin for the wind turbine can be used as a prognostic health management system. This will lead to a change in the maintenance strategies from fixed schedules and intervals towards predictive maintenance. The Digital Twin will make it possible to economically find the balance between wind turbine utilization and lifetime reserve and maintenance.
Currently, wind turbines and their components are often inspected manually using lifts, and sometimes it is needed that a person climbs onto the structure. This type of inspection is expensive, time-consuming, and can potentially also be dangerous. Drones can be used to collect a huge amount of images. The drone can be programmed to autonomously follow a predetermined route and acquire images at specific waypoints. This approach significantly reduces the time needed for the inspection. It is very time-consuming for a person and sometimes also inaccurate to go through massive amounts of images and make reports where for example the defects found in the images are documented. Combining AI with the Digital Twin concept produces a more efficient approach. Deep learning can find the defects in the images in near real-time and is therefore much faster than a human. Defects from multiple images can be categorized by AI and the 3D location can be calculated and mapped onto the Digital Twin. One specific defect that is observed in several images can then be seen on the Digital Twin from different angles and distances. AI is also cable of finding defects too small to be observed by humans. Reference [11] provides an example of wind turbine surface damage detection using AI where drone inspection was used.
This study builds on the findings reported in [12], in which a transition piece was selected as an example of a large-scale structure. In the current study, both a transition piece and a rotor blade are used to demonstrate newly developed functionalities. As such, novel contributions of this study to the current knowledge base are:
  1. Image segmentation to isolate the structure of interest from complex image background. In the previous study [12], AI can occasionally find damage in the images on distant structures in the background and report the damages that do not belong to the structure of interest. These damages should not be mapped to the Digital Twin of the concerned structure in the center of the images.
  2. Real geometric features of defects/damages mapped to the Digital Twin. To achieve this, image segmentation is performed on the small section of the total image that is inside the bounding box calculated by deep learning algorithm. The pixels from the image segmentation, representing realistic damage shape and size, are mapped to the structure.
  3. A more precise new algorithm using surface normals for mapping the damage pixels is presented in this study, enabling realistic representation of unique defects/damage of the structure.
This paper is organized as follows. Section 2 presents the visual Digital Twin and how it has been used in the study. Section 3 describes in detail the methodology of the visual Digital Twin. This includes algorithms for pre-processing drone images and algorithms that map damages found in images to a 3D model of a structure. The image pre-processing and 2D to 3D damage mapping techniques are applied in section 4 which demonstrates the techniques on a set of images from one drone flight using a transition piece as demonstration. Section 5 apply the mapping algorithms to a composite wind turbine blade where subsurface delamination damage is visually inspected. The main conclusions of the study are summarized in section 6.
The visual Digital Twin
The arrival of powerful and cost-effective drones has opened up many new applications. Drone inspection of very large structures is an example of a new application type that gives better results and is more cost-effective than previously used methods, see references [13] and [14]. Wind turbine transition pieces are currently inspected at the factory for different types of damages. This is accomplished by an inspector from a crane. The objectives of this study are to develop and demonstrate fully automatic intelligent drone inspections based on RGB images and to find and recognize paint damages and defects of wind turbine transition pieces TPs, see [15] and [16]. The starting point is a description of the physical position of the TP in a georeferenced coordinate system. A detailed meshed CAD model of the TP is moved to the same georeferenced coordinate system used during drone flights. This makes it easier to compare it to the reconstructed 3D model. An AI algorithm is applied to the RGB images to detect and classify paint imperfections and damages, respectively. The RGB images from the drone flight are used to generate a 3D model of the TP. This reconstruction model is produced with the use of photogrammetry and is an important part of the digital twin. Information from the AI algorithm is used in the pre-processing of the images. The 3D Digital Twin is updated with the positions, types, and sizes of the identified paint imperfections and damages identified in the images by AI. These different processing steps are summarized in the flowchart shown in Figure 1 and explained in detail in the next section. Information from the Digital Twin is used to update the physical structure. Information on the position and type of paint damage from the Digital Twin can be used to determine if maintenance is needed on the physical structure. After the offshore installation of the TPs, drones can be used to inspect the transition pieces at regular time intervals. The Digital Twin will then be updated based on the new images, providing essential information for asset management. Green boxes are used in the flowchart to indicate the new algorithms and processing steps compared to the reference [12]. The image segmentation steps and the 2D to 3D mapping technique that uses the projection along the surface normals method are all introduced in this paper. A major difference in the mapping algorithm is that the paint damage pixels will be mapped to the tower instead of the bounding box corners which was the case in reference [12].