Vision Transformers meet Quality Control

In industrial manufacturing, visual quality control is an extremely important process, for example to ensure that customers do not receive a car with a faulty weld seam or that defective components inside an electric toothbrush do not result in a shorter service life. It essentially involves visually inspecting finished products for defects. From a technical perspective, these are referred to as anomalies. Because customer confidence in products is vital, yet the inspection process is very monotonous, there are already numerous approaches to automating and thereby optimising this process using machine learning algorithms [0].

The machine learning algorithms used here originate from the field of image processing, known in English as computer vision (CV). In recent years, a new type of model has emerged in this field – so-called Vision Transformer models. This development raises an intriguing question: what are vision transformer models, and what potential do they offer in the field of industrial quality control?

CONTENTS

1. What are Vision Transformer models?
2. How does anomaly detection work?
3. What does this mean for visual quality control?
4. Conclusion

What are Vision Transformer models?

To answer this question, we need to take a brief detour into the field of natural language processing. There, Transformer models have been regarded as state-of-the-art for several years now, and since the advent of ChatGPT at the very latest, this technology has become widely known to the general public. Of particular interest for this article is a specific type of Transformer model – the Vision Transformer. Vision Transformer models are models that utilise the same mechanism as ‘classic’ Transformer models – the attention mechanism [1] – but are specialised for image processing [2]. At this point, a brief technical deep dive:

• As Transformers have their origins in natural language processing, they are specialised in processing sequences. Images, however, are represented as matrices of pixel values (0–255). To enable the processing of images, these are divided into so-called patches. The model then receives a vector of ‘flattened’ patches as input (Figure 1).
• During processing, the Transformer model enriches the input vector with additional information. The result of this process is a vector or matrix containing information in the form of numbers. This result is referred to as the latent space.
• In contrast to classical computer vision models (CNNs), the input vector is not compressed during processing. The disadvantage of this lack of compression is that Vision Transformers are very large compared to CNNs. A very well-known and widely used CNN is ResNet [3].
• The advantage of Vision Transformer models over CNNs is that they are particularly good at capturing global relationships within images [2]. This means that if a piece of information in the top-left corner of an image is related to a piece of information in the bottom-right corner, this tends to be more clearly evident from the latent space generated by a Vision Transformer model than from the latent space of a CNN model.
• As images can have any resolution, the size of Vision Transformers is quite a problem. This makes them unsuitable, particularly for production scenarios where hardware with limited capacity is available. This problem is being addressed through the development of so-called hierarchical Vision Transformers. These models use patch-merging layers to reduce the size of the processed images during the training process [4]. Figure 1 shows the architecture of the hierarchical Vision Transformer model EsViT. This model is comparable in size (number of parameters) to ResNet.

Fig. 1: Hierarchical Vision Transformer architecture, illustrated using the EsViT model as an example

How does anomaly detection work?

Data sets for anomaly detection are often highly imbalanced – there is a very high proportion of normal data and a very low proportion of anomalous data. For this reason, and because it can sometimes be difficult to predict which anomalies might occur, it can be challenging to train a model to learn the appearance of anomalies directly. Instead, approaches that learn the appearance of normal data are often preferred [5]. With these unsupervised learning methods, only normal data is used during training. The model thereby learns the distribution of the normal data. When the trained model is presented with anomalous data, this lies outside or on the edge of the learnt distribution and can thus be identified. Figure 2 shows a high-level architecture of the training and anomaly detection process. The first model in the process is referred to as the encoder. It is responsible for extracting knowledge from the image data and representing it in a compressed form. This model can be a Vision Transformer or a CNN. It is used pre-trained and is not fine-tuned. The second model, the decoder, is the actual anomaly detection model. It is responsible for learning the distribution of normal data. During anomaly detection, it is then able to determine at image level whether a defect is present and to mark at pixel level where the defect is located. As these models calculate a probability for each entry in the latent space, the model size depends heavily on the size of the latent space – the more compressed the information, the better.

Fig. 2: High-level architecture of the training and anomaly detection process

What does this mean for visual quality control?

1. Improved performance

Various types of anomalies can occur in visual quality control [5]:

• Textural anomalies relate to the integrity of a product and may include, for example, contaminants, cracks, holes or scratches. Detecting them requires little semantic information.
• Functional anomalies relate to the correctness of a product in relation to a specific use case. This could, for example, be an incorrectly connected cable or a misplaced transistor. Detecting these anomalies requires a great deal of semantic information.

Fig. 3: on the left, an intact metal gear with a scratch (textual anomaly); on the right, a correctly and incorrectly placed transistor (functional anomaly). Images from the MVTecAD dataset [6].

By recognising global relationships using the attention mechanism, Vision Transformers are very well suited to extracting semantic information from an image [2]. The resulting latent space contains more information whilst still being smaller than when using ResNet. In the experiments carried out, the latent space generated using ResNet had 2,048 entries (features), whilst the latent space generated using a large Vision Transformer had only 768. This means that, despite the encoder being three times larger, the size of the overall model is slightly smaller. Table 1 shows a comparison of the results from the various models. The metric used is the Area Under the Receiver Operating Characteristics Curve (AUROC) at image level (AD) and at pixel level (AL). The ROC curve illustrates the range between the true positive rate and the false positive rate. Despite its smaller size, the anomaly detection results are better when using the large Transformer model, DeiT, than when using the ResNet model. The EsViT model also generates a latent space of size 768. The entire model is therefore not even half the size of the one using ResNet. However, the results are significantly worse.

Table 1: Comparison of the performance of the various encoder models. DeiT is a large Transformer; the following three models are hierarchical Transformers; the final model, ResNet, is a CNN.

2. More flexible models

Companies often have a wide range of products that is constantly changing – new products are added or existing ones are modified. Furthermore, there are varying requirements as to what is considered an anomaly in a product and what is not – small scratches or dust particles may be unproblematic for some products, whilst for others they may be critical. As a result, a quality control model must be individually tailored to each product. In a company with many products and a high rate of change, this can lead to considerable effort being required to develop such models.

The use of pre-trained Vision Transformer models offers two major advantages in this regard:

• Reduced training time by minimising the size of the model to be trained
• Simpler data preparation for new or modified products
  => No need to synthetically generate images with potential defects

Conclusion

Large Vision Transformer models improve the performance of anomaly detection methods and have the potential to reduce the overall model size. Hierarchical Vision Transformers produce smaller models, but do not yet deliver the required quality of results. Nevertheless, they represent promising approaches and development in this area is continuing. The latest hierarchical models are so small that they can even run on smartphones [7]. This exciting development shows that Transformer models can be of great interest not only for Generative AI but also for the field of industrial manufacturing.

The results described are findings from my Master’s thesis. The thesis was written at Karlsruhe University of Applied Sciences, in the Faculty of Computer Science and Business Informatics, and is entitled ‘Evaluating the use of pre-trained Vision Transformer models for visual quality control in industrial manufacturing’.

Further information on the technical details can also be found in our accompanying paper https://pabair.github.io/assets/ECML2024.pdf

Sources

[0] T. Wang, Y. Chen, M. Qiao, and H. Snoussi, “A fast and robust convolutional neural network-based defect detection model in product quality control,” The International Journal of Advanced Manufacturing Technology, vol. 94, no. 9–12, pp. 3465–3471, 2018, ISSN: 0268-3768. DOI: 10.1007/s00170-017-0882-0. [Online]. Available: https://link.springer.com/content/pdf/10.1007/s00170-017-0882-0.pdf?pdf=inlinelink.

[1] A. Vaswani, N. Shazeer, N. Parmar, et al., Attention is all you need. [Online]. Available: https://arxiv.org/pdf/1706.03762.

[2] A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al., ‘An image is worth 16x16 words: Transformers for image recognition at scale’, 2020. [Online]. Available: https://arxiv.org/pdf/2010.11929.

[3] K. He, X. Zhang, S. Ren, and J. Sun, ‘Deep residual learning for image recognition’, 2015. [Online]. Available: https://arxiv.org/pdf/1512.03385.

[4] C. Li, J. Yang, P. Zhang, et al., Efficient self-supervised vision transformers for representation learning. [Online]. Available: https://arxiv.org/pdf/2106.09785.pdf.

[5] X. Tao, X. Gong, X. Zhang, S. Yan, and C. Adak, “Deep learning for unsupervised anomaly localisation in industrial images: A survey,” IEEE Transactions on Instrumentation and Measurement, vol. 71, pp. 1–21, 2022, ISSN: 0018-9456. DOI: 10.1109/TIM.2022.3196436.

[6] Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger and Carsten Steger, “The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection”, International Journal of Computer Vision, 2021

[7] Y. Li, J. Hu, Y. Wen, et al., “Rethinking vision transformers for MobileNet size and speed,” arXiv preprint arXiv:2212.08059, 2022.

Do you have any questions or need assistance?

You can find out more about the possibilities of artificial intelligence for your business on our website. Feel free to get in touch with us.