To train and validate the YOLO-v8 model for the smoke detection, a dataset comprising 31,100 images depicting various fire scenes including flame and smoke objects was collected. The dataset includes images of outdoor fires such as vehicle fires, urban fires, and wildfires, as well as indoor fires in various buildings. Non-fire images were gathered from diverse environments, including buildings, mountains, urban areas, sunsets, and night scenes. For training and validating the YOLO model for the object detection, 30,500 images were used, consisting of 29,423 fire images and 1,077 non-fire images. These images were divided into training data and validation data in 9:1 ratio to train the model and evaluate its object detection performance during the training phase. The test data for evaluating the performance of the proposed smoke segmentation model consisted of 600 images extracted from six fire incident videos. At the preprocessing step for training the object detection model and evaluating the performance of the proposed smoke segmentation model, bounding boxes were added to the flame and smoke objects in each 640 x 640 RGB input image using Roboflow[14], and ground truth smoke regions on each images were created for the performance evaluation. Figure 1 shows example images from the dataset used for training, including fire and non-fire scenes.

Several researches[15-18] have been conducted to detect smoke region in the fire images by using the color spaces. Since CIELAB and RGB color space-based models were good for the separation between smoke and non-smoke regions reported by Lee and Kim[15], these color space-based models were chosen as the reference models for the comparison purpose.

As one of the reference color space models for the smoke segmentation, the CIELAB color space model suggested by Zhao et al.[16] was used in the present study. The image conversion from RGB to the CIELAB color space is described from Eq. (1) to Eq. (8), and the rule to detect smoke regions in the CIELAB color space is given by Eq. (9).

where, X_n, Y_n and Z_n are 0.950456, 1 and 1.088754, respectively

where, L, C, a and b represent lightness, chroma, green-red axis and blue-yellow axis, respectively, in the CIELAB color space.

As another reference color space model, the RGB color space model proposed by Toreyin et al.[17] and Kim[18] was used and the rules are described from Eq. (10) to Eq. (13).

where, R, G and B represent the red, green, and blue channels in the RGB color space, respectively. C_max and C_min are the maximum and minimum pixel values among the R, G, and B channels. Th₁ to Th₄ are the respective thresholds.

In addition to the reference models, the YUV color space model suggested by Prema et al.[11] was considered and used to investigate the smoke segmentation characteristics of the color space-based models, and the rules to detect the smoke regions are defined from Eq. (14) to Eq. (17).

where, Y, U and V are luminance and chrominance components of each pixel in the RGB image.

The results of the rule-based smoke region detection using the CIELAB, RGB and YUV color spaces are shown in Figure 2. These models could detect smoke regions from the fire images, but the prediction performance was poor. Especially, it was limited to detect the black-gray colored smoke and the smoke region in the fire images containing the objects having the color features similar to smoke such as white building, sky, etc. It was found that the smoke segmentation performance of the conventional color space-based models is strongly affected by environment color features.

In this study, a rule-based smoke region detection technique using color spaces was used and the rules were developed to properly detect the smoke region by considering the smoke color features derived from various fire occurrence environments. Deep learning object detection model, YOLO-v8, was applied to crop the bounding box containing the smoke object before doing the smoke segmentation. It could minimize the area of other images inside the bounding box and significantly reduce the erroneous smoke segmentation in the images containing the objects with color information similar to smoke. To optimize the YOLO-v8 model, various hyperparameters were adjusted during training, with the learning rate ranging from 0.01 to 0.001, batch sizes from 16 to 128, and momentum values between 0.9 and 0.999. Additionally, different optimizers, including SGD, Adam, AdamW, Radam, and Nadam, were tested to analyze their impact on model performance. Figure 3 further illustrates the YOLO-YCbCr based smoke segmentation procedure, demonstrating how the trained YOLO-v8 model is utilized to accurately identify and extract smoke regions in various environments.

As shown in Figure 4, the optimized YOLO-v8 model detects the smoke objects on RGB input fire images, the selected regions are cropped and converted into YCbCr image. The conversion formula from RGB to YCbCr is given by Eq. (18).

where, Y denotes the brightness, Cb and Cr represent the blue-difference chroma component and red-difference chroma component, respectively.

From YCbCr information of the image in the bounding box cropped by YOLO-v8, the YCbCr rules suggested in the present study were applied for the smoke segmentation. These rules were derived by analyzing the Y, Cb, and Cr color characteristics of smoke in various fire occurrence environments to distinguish smoke regions more effectively. Three steps of the YCbCr rules were proposed by considering the color features of smoke occurred at various environments. The proposed rules are shown in Figure 4 and explained as below.

•Step 1: Determine the main color (white-light gray or black-gray) of the smoke in the detected smoke object regions based on the brightness difference between the background image and the bounding box.

•Step 2: Detect the white-light gray smoke region by differentiating other objects having color similar to smoke based on the Y, Cb, and Cr values of the pixels, if the step 1 determines the smoke color to be white-light gray.

•Step 3: Detect the black-gray smoke region by using the Y value of the pixels within the bounding box, if the step 1 determines the smoke color to be black-gray.

After detecting the smoke region in the bounding box, the detected smoke regions are finally marked on the original image.

To evaluate the smoke segmentation performance, mean IoU was used, which represents the degree of the similarity between the actual smoke region and the predicted smoke region in an image, as determine by Eq. (19).

where, P_i is the predicted segmentation result of the i-th image, and G_i is corresponding ground truth, and N denotes the number of images in a test dataset.

In addition, a confusion matrix analysis was conducted as shown in Table 1. The IoU values were used to determine true positives (TP) and false negatives (FN); the IoU value of 50% or higher was treated as TP, while the IoU value below 50% was treated as FN. If non-smoke regions were detected as smoke region, it was treated as FP. If non-smoke regions were not detected as smoke region, it was treated as TN. Based on the confusion matrix results, precision, recall, and F1-score were determined to quantitatively evaluate the smoke detection performance, and defined from Eq. (20) to Eq. (22), respectively. The precision represents the ratio of the smoke region predicted correctly to the total smoke regions predicted, and the recall means the correctly predicted smoke region to the total actual smoke regions. The F1-score indicates the harmonic mean of the precision and the recall.

In this study, the YOLO-v8 was used to enhance the smoke segmentation for the color space-based models. A pre-trained YOLO-v8 model was employed, and transfer learning was conducted by using a fire image dataset. Hyperparameters were optimized for the YOLO-v8 model to well detect smoke objects in the fire images. The object detection performance was evaluated by using 600 images extracted from six fire images. As shown in Figure 5, the optimized YOLO-v8 well detected smoke objects and the values of the precision and the recall were approximately 92.4% and 91.9%, respectively. The value of the mean Average Precision (mAP), a performance metric for object detection algorithms, was about 96.7%, indicating high object detection performance. The precision-recall (PR) curve is presented in Figure 5(b).

The performance evaluation of the proposed smoke segmentation model was conducted by using 600 fire images extracted from six different fire incident videos. The segmentation results of the proposed model were compared with those of the YOLO-RGB model and the YOLO-CIELAB model considered as the reference models. To assess the smoke segmentation performance, the IoU values were determined as shown in Table 2. The IoU values of the proposed model range from 48.6% to 80.3%, which are greater than those for the reference models in all test video cases. It was observed that the performance of the color space-based smoke segmentation models is still affected by the fire occurrence environment although the YOLO minimizes the effects of the background color features. Additionally, the mean IoU value about all test images of the proposed model is 68.6%, which is similar to the values of deep learning-based smoke segmentation models conducted by Yuan et al.[12]. The mean IoU values for the YOLO-CIELAB model and the YOLO-RGB model are 37.3% and 44.1%, respectively. When compared with the IoU values for the YOLO-CIELAB model and the YOLO-RGB model, the mean IoU improvements by using the proposed model are 31.3% and 24.5%, respectively. In addition, applying the YOLO to the color space-based models such as CIELAB model and RGB model increased the mean IoU values by 16.9% and 21%, respectively. The performance evaluation results show that identifying the smoke color features characterized by considering various fire occurrence environments is very important for the smoke segmentation. It was also found that applying the YOLO is significantly useful to enhance the smoke segmentation performance for the color space-based segmentation models.

Smoke detection performance was evaluated based on the confusion matrix analysis, and the values of the precision, recall, and F1-score are presented in Figure 6. The results show that the YOLO-YCbCr model outperforms the reference models across all performance metrics. Specifically, the values of the precision, recall, and F1-score for the proposed model are 81%, 79.7%, and 79.8%, respectively. When compared to the YOLO-CIELAB model and the YOLO-RGB model, using the YOLO-YCbCr leads to the improvements in precision by 23.9% and 20.5%, in recall by 27% and 16%, and in F1-score by 28.6% and 20.5%, respectively. It was expected that the proposed smoke segmentation can be applicable in real-time fire detection system because it has good smoke segmentation performance and low computational costs such as low preprocessing time, small model size, and fast inference time. This efficiency, combined with its small size, minimal preprocessing, and low memory usage, makes the YOLO-YCbCr model well-suited for real-time applications. With an FPS of 49.8, it enables fast and reliable smoke detection, ensuring practical deployment in real-time fire detection systems.

The smoke segmentation results for each image were thoroughly analyzed to identify the segmentation characteristics of the proposed model and its problems. Representative segmentation results in each fire incident video are shown in Figure 7 and Figure 8. Figure 7 shows the smoke segmentation results for the white-light gray colored smokes and the segmentation performance of the proposed model performed better than the reference models according to mean IoU values. The mean IoU values of the proposed model, the YOLO-CIELAB model, and the YOLO-RGB model are 53.6%, 56.1%, and 69.1% respectively. In the case of white-light gray smokes presented in Figure 7(a), three smoke segmentation models detected the smoke region with the high IoU value because the smoke in the image is well distinguished from the background objects (or colors). However, in the white-light gray smoke images including other objects having similar color to smoke shown in Figures 7(b) and 7(c), both the YOLO-CIELAB model and YOLO-RGB model detected non-smoke region as smoke. In addition, too bright region of the center of smoke was not detected in the reference models. On the other hand, the proposed smoke segmentation model relatively well predicted the smoke region because the rules by considering both the luminance and the chrominances (blue-difference, red-difference) were used to distinguish the smoke region from the smoke-like object regions after identifying the mean luminance of images as described in Figure 4. Figures 8(a)~8(c) show the smoke segmentation results for the black-gray colored smokes. The YOLO-CIELAB model and the YOLO-RGB model used in this study had poor smoke segmentation performance, and the mean IoU values were 21.1% and 32.2%, respectively. The proposed model well detected the black-gray smoke region by considering the optimized luminance thresholds. The mean IoU value of the proposed model for the fire incidents producing the black-gray smoke was 68.1%, which is greater than that of the reference models. Despite the improved performance, the proposed YOLO-YCbCr model has certain limitations. One major challenge is distinguishing smoke from bounding box containing objects with color similar to smoke. This can result in false positives, where white or gray objects such as buildings, clouds, or bright reflections are misclassified as smoke. Additionally, the proposed model is inherently dependent on YOLO’s detection performance. Since the segmentation process is applied only to regions detected by YOLO, any misdetection or failure to detect smoke at this stage directly impacts the accuracy of the final segmentation. To address these limitations, future improvements should focus on integrating texture-based filtering, motion analysis, or hybrid color space approaches to enhance the robustness of smoke segmentation. Furthermore, improving YOLO’s object detection performance is essential, which can be achieved by training the model on diverse fire incident datasets that include various fire scenarios. While the YOLO-YCbCr model shows higher smoke segmentation performance compared to the reference models, its prediction performance and robustness should be increased to enhance the applicability in the real-time fire detection system. Since the model will be applied to fire surveillance systems, including CCTV and drone-based monitoring systems, it is essential to enhance its detection accuracy, prediction reliability, and overall system robustness. Improving these aspects will ensure more effective and reliable smoke detection, enabling early fire detection and minimizing false alarms in diverse real-world environments. In future studies, the YCbCr rules should be optimized by considering various fire incident images and combined with other novel methods to cope the shortcomings of the color space-based smoke segmentation model. Additionally, the proposed model under practical deployment scenarios, such as integration with intelligent monitoring systems or IoT-based fire detection platforms, should be tested and improved to strengthen its applicability in real-world environments.