Ship Bridge OOW Activity Status Detection Using Wi-Fi Beamforming Feedback Information (2024)

1. Introduction

Waterway transport is an important part of world trade and plays an important role in promoting international economic development and energy transportation security. However, compared with land and air transport, waterway transport is relatively lagging in the progress of intelligent regulation [1,2]. Among all the causes of ship accidents, the unsafe state of OOWs, such as fatigued driving, negligence of lookout, and insufficient manpower, remains one of the main causes of water transportation accidents [3]. The use of intelligent technology for real-time perception and state assessment of ship drivers is of great significance in reducing accidents caused by driving human factors.

Traditional methods for monitoring ship driving status, such as video surveillance [4] and crew self-reporting [5], have obvious limitations, though they can provide basic information on ship driving behavior to some extent. Although the video surveillance system can continuously record the situation in the ship’s co*ckpit, it is often difficult to capture the specific details of the crew’s maneuvers in detail due to field of view limitations and picture quality issues. This is particularly the case in bad weather or poorly lit conditions, where the video quality may further deteriorate, affecting the effectiveness of the surveillance. In addition, data processing and analysis of video surveillance systems requires a large amount of human resources for video content playback and review, which is both time-consuming and inefficient. Crew self-reporting, another traditional monitoring tool, relies on the subjective evaluation and review of crew members’ own behavior. However, this method is susceptible to personal bias and memory accuracy, and crew members may omit or distort information either intentionally or unintentionally for a variety of reasons, resulting in discrepancies between the reported content and the actual situation. More importantly, self-reporting does not provide real-time data and does little for rapid response in emergencies and accident prevention.

In recent years, with the rapid development of the integration of communication and sensing technologies, the use of ubiquitous Wi-Fi signals to sense driving behavior has received increasing attention [6]. This technique provides fine-grained perception of people’s indoor activities by analyzing changes in Wi-Fi signal characteristics, which enables real-time monitoring and recognition of driving behaviors. Compared with traditional monitoring methods such as machine vision and wearable sensors, Wi-Fi signal-based behavioral recognition has several significant advantages. In addition to being flexible and inexpensive to deploy, is can effectively protect personal privacy [7,8]. Moreover, Wi-Fi technology is robust and remains unaffected by ambient light. Its ability to perceive paths in non-line-of-sight (NLOS) environments opens up new perspectives in the field of ship-driving behavior monitoring, greatly expanding the scope, application, and effectiveness of monitoring technologies.

Although Wi-Fi sensing technology has demonstrated good accuracy in several application scenarios, it faces a number of challenges when deployed on a ship’s bridge for motion sensing and state assessment. Specifically, these challenges include:

Hull deformation and vibrations significantly impact the propagation of Wi-Fi signals indoors, affecting the reliability of signal-feature-based estimation [9]. Designing robust Wi-Fi sensing methodologies for indoor ship environments, particularly under low signal-to-noise ratios, is crucial for accurate and stable sensing of ship driver states.
Current driving behavior research lacks in-depth understanding and comprehensive modeling of the subtle behavioral patterns of ship drivers [10]. How to accurately interpret and recognize the mechanism of driving state and its modeling in ship environments has not been adequately addressed, representing a second challenge in improving the accuracy and effectiveness of ship driving safety monitoring.
Current research on the state of OOWs mostly focuses on the analysis of single indicators, such as fatigue level or attention concentration [11,12]. This single-dimension research method fails to comprehensively reveal the complexity and dynamic changes of OOW status, making how to comprehensively perceive and evaluate OOW driving status a third challenge.

To tackle the challenges outlined above, this paper proposes a driving state detection method based on Wi-Fi Beamforming Feedback Information (BFI) [13]. By comprehensively evaluating multiple driving state features, this method can effectively meet the needs of ship driving state detection. The main contributions of this research can be summarized as follows:

To address the problem of traditional Wi-Fi CSI sensing methods being insufficiently accurate in indoor ship environments, this study uses BFI-based wireless sensing technology for co*ckpit watchman detection. Thanks to the directional advantages of BFI, the impact of the complex indoor ship environment on the accuracy of wireless sensing is reduced.
To address the problem of only a single index being considered in current approaches to detecting the state of OOWs, this study introduces multiple indexes, such as the number of active people, action entropy, action time, etc., which can comprehensively assess the activity degree of OOWs from multiple perspectives while providing a more scientific and comprehensive evaluation method to ensure the safety of ship driving.
Finally, this study conducts extensive experiments of the proposed method on a real-world ship bridge, with the experimental results showing that the overall recognition accuracy of the method reaches 92.4%.

The remainder of this paper is structured as follows: the related work and background of BFI sensing technology are introduced in Section 2; the core method of this article is described in detail in Section 3; the experiments conducted to test and verify the method are presented in Section 4; finally, Section 5 provides conclusions and a discussion.

2. Related Work and Background

2.1. Related Work

In recent years, several methods have been proposed for monitoring the status and activities of officers on watch (OOWs) on ship bridges. Bergasa et al. [14] developed a real-time monitoring system based on infrared cameras to detect driver drowsiness by analyzing eye closure frequency and yawning. However, these systems are challenged by poor lighting conditions, and require constant maintenance. Lützhöft et al. [15] used eye-tracking devices to study navigation safety; however, their reliance on continuous visual data makes them vulnerable to occlusion and environmental interference. Youn et al. [16] implemented wearable devices to distinguish lookout behavior using depth cameras; however, the need for constant wearability and user compliance pose significant challenges. Li et al. [17] designed a fatigue monitoring system based on facial recognition, which, despite its high accuracy, can be affected by user cooperation and environmental conditions. Zhao et al. [18] applied machine learning to detect hazardous behaviors such as leaving the post or falling; however, the system’s effectiveness in dynamic environments is yet to be fully validated.

The limitations of these traditional methods highlight the need for more robust and adaptable solutions. To address these challenges, our study proposes a novel method based on Wi-Fi Beamforming Feedback Information (BFI) for detecting and assessing the activity levels of OOWs on ship bridges. This method leverages the ubiquity and cost-effectiveness of Wi-Fi technology to provide accurate and privacy-preserving monitoring. By integrating indices such as the number of active individuals, action entropy, and activity time, our approach offers a comprehensive evaluation of OOW activity to improve detection accuracy in dynamic environments and provide a reliable assessment of OOW performance, thereby enhancing navigational safety.

Channel state information (CSI) is a source of radio frequency information that is widely used in Wi-Fi sensing [19]. CSI exhibits good sensing capabilities, and has relatively low implementation costs along with high sensing accuracy. Currently, the next-generation Wi-Fi standard task group IEEE 802.11bf [20] is actively embedding Wi-Fi awareness functions into the Wi-Fi standard. In IEEE 802.11bf, legacy devices are made to comply with traditional Wi-Fi standards such as IEEE 802.11ac/ax to enable Wi-Fi sensing. In recent years, BFI has attracted attention as an alternative to CSI for radio frequency information [21]. In MIMO systems [22], beamforming technology significantly improves the signal gain in a specific direction by optimizing the phase and amplitude distribution of signals among multiple transmit antennas, thereby enhancing signal transmission efficiency and quality.

Wi-BFI [13] is the first open-source tool for extracting and decoding Wi-Fi BFI, supporting multiple networks and bandwidths to enable real-time and offline data processing and visual display of channel status. BeamSense [23] is a new wireless sensing system that utilizes Compressed Beamforming Report, which enables CSI-based sensing algorithms to run with high accuracy on Wi-Fi devices that widely support transmit beamforming, improving the perception universality and generalization ability. Regarding wireless sensing research in indoor ship environments, Liu et al. [24] studied Wi-Fi location fingerprint modeling in the dynamic environment of ships and proposed an indoor positioning method for ship personnel based on spatiotemporal location fingerprint features. Chen et al. [25] used the amplitude and phase information of Wi-Fi signals to construct a subcarrier sensitivity assessment model for ship factors and human behavior, and designed a driver number identification and off-duty detection method on this basis.

2.2. Wi-Fi BFI Sensing

When OOWs move within the ship’s bridge, their body movement can have a blocking effect on the propagation path of the wireless signals, which can cause significant changes in signal strength. For example, when an OOW is close to the wireless transmitting source, the strength of the signal may be enhanced due to the near-field effect; on the contrary, when an OOW is located between the transmitting source and the receiver, the signal strength may be weakened due to body blocking. These variations in signal strength carry critical information about the people’s location and movement trajectory. By analyzing the BFI signal features, it is possible to correlate the features with the specific status or behaviors of the personnel on the ship’s bridge.

The Wi-Fi human sensing system mainly consists of an Access Point (AP) and Station (STA) [21], which can be a mobile device such as a smartphone or a smart bracelet or a stationary device such as a personal computer or a router. The goal of the system is to monitor the physical movement of a human subject (the Subject, S), as shown in Figure 1. In this configuration, the wireless signal propagation between the AP and STA is directly interfered with by the motion of the person [26]. At time point t, the distance between AP and S is denoted as $d_{A, S} (t)$ , while the distance between S and STA is denoted as $d_{S, S T A} (t)$ . The variation of these distances directly affects the propagation path and signal strength of the wireless signal. The wireless channel gain modeling can be expressed as follows:

$h_{A, E} (t) = h_{A, S, E} (t) + h_{S, A, E} + h_{D, A, E} (t) .$

(1)

This model consists of three parts: $h_{A, S, E} (t)$ represents the channel gain from the AP to the STA via the reflection of S, $h_{S, A, E}$ represents the static channel gain for the direct communication path, and $h_{D, A, E} (t)$ represents the dynamic channel gain due to interfering motion (e.g., movement of people) along that path. The channel gain from AP to STA via reflection of S can be specifically expressed as follows:

$h_{A, S, E} (t) = \frac{λ}{2 \sqrt{G_{A, S, E}}} exp (- i 2 π \frac{d_{A, S} (t) + d_{S, E} (t)}{λ}) {(\frac{4 π}{d_{A, S} (t) d_{S, E} (t)})}^{α / 2}$

(2)

where $λ$ is the carrier wavelength, $G_{A, S, E}$ represents the product of the $T_{x}$ and $R_{x}$ antenna gains and the reflection coefficient of S, and $α$ is the path loss exponent. This expression reveals how the signal strength varies with the distances between AP and S and between S and STA. The working principle of person perception based on Wi-Fi is based on the change in channel gain due to physical motion. The motion of the human subject S leads to changes in $d_{A, S} (t)$ and $d_{S, E} (t)$ , which in turn leads to changes in the channel gain $h_{A, S, E} (t)$ over time. By analyzing the time series of $h_{A, S, E} (t)$ obtained from the channel state information of the Wi-Fi frames, the AP and STA can sense the motion of S.

3. Methodology

This section details the methodology adopted for detecting and evaluating the activity levels of OOWs on ship bridges using Wi-Fi BFI. The proposed approach involves a comprehensive evaluation framework that integrates multiple indices to provide accurate and reliable monitoring of OOW activities. The methodology is designed to address the limitations of traditional monitoring systems and improve navigational safety through advanced wireless sensing techniques. Figure 2 presents the comprehensive framework used for detecting and evaluating the activity levels of OOWs on ship bridges, encompassing data collection, feature extraction, activity detection, and activity level evaluation.

3.1. Overview of the Methodology

The proposed methodology consists of data collection, feature extraction, activity detection, and activity level evaluation. Wi-Fi BFI data are collected from multiple access points strategically placed within the ship bridge. The BFI data capture the physical and directional properties of the Wi-Fi signals, which are influenced by the movements and activities of the OOWs. The collected BFI data are processed to extract relevant features such as the signal strength, angle of arrival, and movement patterns. These features are critical for identifying and distinguishing different types of activities performed by the OOWs. Machine learning algorithms are employed to analyze the extracted features and detect specific activities. The algorithms are trained to recognize patterns associated with various OOW activities, ensuring accurate identification even in dynamic environments. The detected activities are further analyzed to evaluate the overall activity level of the OOWs. Indices such as the number of active individuals, action entropy, and activity time are used to provide a comprehensive assessment of OOW performance. This evaluation helps in identifying periods of inactivity or low engagement which could indicate potential safety risks.

3.2. Data Collection and Preprocessing

The BFI data collection process begins with the STA receiving probe frames from the AP. Under the IEEE 802.11ac/ax standard, this is typically an empty packet. The STA uses these frames to estimate the CSI. The CSI estimate provides detailed information about the propagation of the signal through the wireless channel, including signal attenuation and phase changes. Based on the CSI, the STA computes the beamforming feedback BFI. This process involves converting the CSI data to another format that can effectively reflect the characteristics of the channel and provide the necessary information for the AP’s beamforming strategy.

After the BFI has been computed, the STA transmits it back to the AP. It is worth noting that no encryption is used in this transmission process, allowing the data to be captured by network analysis tools such as Wireshark 4.2.0. This in turn allows us to extract the BFI data directly from the network traffic. After the BFI has been transmitted, a sniffer in the system captures the data. The sniffer’s task is to decode the BFIs and extract key information such as the right singularity matrix of each subcarrier $V_{k}^{H}$ and the average subcarrier stream gain $\bar{Λ}$ . This information is crucial for understanding and analyzing the characteristics of the wireless channel.

When preprocessing the BFI data, our primary concerns were data integrity and outlier handling. To process missing values in the BFI data, a filling based on the local average of the surrounding data points was used. Specifically, the arithmetic mean of n data points before and after each missing data point was calculated as a replacement value. This maintains the continuity of the data in the time series and reduces the effect of outliers. The treatment can be expressed by the following equation:

$B F I_{t} = \{\begin{matrix} \frac{1}{2 n} \sum_{k = - n}^{n} B F I_{t + k} & if B F I_{t} is missing \\ B F I_{t} & otherwise \end{matrix}$

(3)

where $B F I_{t}$ is the Beamforming Feedback lnformation value at time t and n is the chosen size of the neighborhood, which determines the number of points considered for calculating the average; for example, if n = 5, then for each missing value the average of five data points before and after (for a total of eleven data points, including the missing point itself) are calculated.

To handle outliers, a method based on the Z-score [27] is used to identify and handle outlier data points. Specifically, any data point that exceeds three times the standard deviation is considered an outlier and is replaced with the mean. The mathematical expression of the method is

$B F I_{t} = \{\begin{matrix} μ_{BFI} & if |Z (B F I_{t})| > 3 \\ B F I_{t} & otherwise \end{matrix},$

(4)

where $μ_{BFI}$ is the mean value of the BFl data over a given time period and $Z (B F I_{t})$ is the Z-score of $B F I_{t}$ , calculated as follows:

$Z (B F I_{t}) = \frac{B F I_{t} - μ_{BFI}}{σ_{BFI}}$

(5)

where $σ_{BFI}$ denotes the standard deviation of all BFI values. The data obtained after the steps outlined above are shown in Figure 3.

3.3. Activity Feature Extraction

3.3.1. Activty Entropy

Using machine learning models, in particular classification algorithms based on time series analysis, it is possible to classify BFI data streams into datasets representing two states: motion and stationary. A key step in the classification process is the construction of feature vectors, which are mainly statistical properties of the data packets over time. For example, significant fluctuations in BFI can be observed for packets in motion, which can be visualized by calculating the rate of change between packets [28]:

$Δ B F I = \sqrt{{(B F I_{t} - B F I_{t - 1})}^{2}}$

(6)

where $B F I_{t}$ and $B F I_{t - 1}$ represent the respective BFI values at two consecutive time points. By analyzing these differences, feature vectors reflecting the motion and stationary states can be constructed. Then, an SVM model [29] is used as a classifier to distinguish between motion and stationary states. The SVM distinguishes different categories by constructing one or more hyperplanes. During the training process, the following objective function are optimized to maximize the boundary between categories:

$min_{w, b} \frac{1}{2} {∥ w ∥}^{2} + C \sum_{i = 1}^{n} ξ_{i}$

3.3.2. Number of Active OOWs

The number of OOWs can be accurately estimated from BFI data by establishing a mathematical model between the signal variation characteristics and the number of OOWs. In this paper, we adopt an innovative approach that utilizes the Euclidean distances of BFIs at different points in time as the underlying characteristics. The core of this approach is to capture the changes in signal characteristics caused by the movement of people by quantifying the degree of variation between BFI data. The BFI data can be represented as

$F_{1} = \frac{{\dot{h}}_{ȷ}^{k}}{m ({\dot{h}}_{ȷ}^{k})},$

(10)

where ${\dot{h}}_{ȷ}^{k}$ is the BFI value of the ship’s bridge watch situation received at the kth sampling time in the jth Wi-Fi link and $m ({\dot{h}}_{ȷ}^{k})$ is the sample mean of the BFI value of ${\dot{h}}_{ȷ}^{k}$ . For each time point of BFI data $B F I (t)$ , its mean $μ_{B F I}$ and standard deviation $σ_{B F I}$ are first calculated; then, the normalized BFI data $B F I_{norm}$ can be calculated by the following equation:

$B F I_{norm} (t) = \frac{F_{t} - μ_{B F I}}{σ_{B F I}} .$

(11)

Next, the BFI dispersion between $t_{1}$ and $t_{2}$ at different time points is calculated. The dispersion $D_{B F I} (t_{1}, t_{2})$ is obtained by calculating the Euclidean distance between the normalized BFI data at two time points:

$D_{B F I} (t_{1}, t_{2}) = \sqrt{\sum {(B F I_{norm} (t_{1}) - B F I_{norm} (t_{2}))}^{2}} .$

(12)

In order to further improve the accuracy and robustness of the detection algorithm, the features calculated based on the Euclidean distance are normalized. Specifically, the features are processed using the min–max normalization method, which scales all feature values to the range between 0 and 1. The normalized eigenvalue calculation formula is as follows:

$D_{norm} = \frac{D - D_{min}}{D_{max} - D_{min}}$

(13)

among which D is the original eigenvalue, $D_{min}$ and $D_{max}$ are the minimum and maximum values among all eigenvalues, respectively, and $D_{norm}$ is the normalized eigenvalue. In this way, it is ensured that all feature values are on a unified scale, thereby improving the efficiency and accuracy of subsequent SVM model processing data for classification. Figure 6a shows the associated distribution of BFI to the number of active OOWs.

3.3.3. Total Number of People on Duty

According to the principle of the Poisson process [31], the behavioral patterns of different crew members are superimposed to form an overall behavioral pattern, the frequency of which is the sum of the behavioral frequencies of each individual; therefore, in order to analyze the collective behavior pattern on the ship’s bridge, two key time periods are defined: the Duty Action Period (DAP) and the Micro-Adjustment Action Period (MAAP). The DAP refers to the time during the observation period when at least one crew member is performing obvious duty activities, such as operating equipment, patrolling, etc. The duration of each DAP is denoted by $g_{i}$ and follows a specific probability distribution $p_{G} (g)$ . In contrast, the MAAP refers to a continuous period of time in which all crew members are in a relatively static state, making only minor adjustments or no significant activity. The duration of each MAAP is represented by $s_{i}$ and follows a probability distribution $p_{S} (s)$ . The existence of these two cycles intuitively reflects the dynamic changes in the activities of OOWs on the ship’s bridge. The schematic diagram of the on-duty active time is shown in Figure 6b.

In order to describe the probability distribution of the DAP and MAAP, Poisson process modeling is adopted. Assuming that N is the total number of people on the ship’s bridge, the behavior pattern of each person can be regarded as an independent Poisson process. The overall behavioral pattern, that is, the occurrence of the DAP and MAAP, follows an exponential distribution with a parameter that is the sum of all individual behavioral frequencies. Specifically, the probability density function of the DAP interval time is $p_{T} (D A P) = N λ_{D A P} e^{- N λ_{D A P} T}$ , where $λ_{D A P}$ T is the average frequency of duty activities performed by a single crew member. In the same way, the probability density function of the MAAP interval time is $p_{T} (M A A P) = N λ_{M A A P} e^{- N λ_{M A A P} T}$ , where $λ_{M A A P} T$ is the average frequency of fine-tuning actions by a single crew member.

The DAP and MAAP duration can be mathematically characterized using M/G/∞ queuing theory [32]. According to the memoryless property of exponential distribution, for a population of N people, the probability distribution of an MAAP duration is expressed as

$p_{S} (s ∣ N, \bar{γ}) = N \bar{γ} e^{- s N \bar{γ}},$

(14)

among which ${\bar{γ}}^{- 1} = \frac{1}{N} \sum_{n = 1}^{N} γ_{n}^{- 1}$ is the average frequency of individual duty behaviors in the bridge. Similarly, given a group of N people on duty, the probability density function for the duration of a DAP can be expressed as

$p_{G} (g ∣ N, \bar{γ}) = \frac{d}{d g} (- N {\bar{γ}}^{- 1} \prod_{i = 1}^{\infty} m_{i}^{*} (g ∣ N)),$

(15)

where $m_{i}^{*}$ is the i-time convolution of m and itself. Based on the obtained bridge crowd duty action period probability distribution $p_{G} (g)$ and fine-tuning period probability distribution $p_{S} (s)$ , this mathematical representation is then used to propose a maximum posterior. The empirical probability (MAP) estimation rule is used to estimate the number of people N for all DAP and MAAP durations before a given time t. Let $g_{1}$ , $g_{2}$ , ⋯, $g_{N_{f} (t)}$ represent the duration of all DAPs before time t, where $N_{f} (t)$ is the total number of these duty action cycles until time t, and let $s_{1}$ , $s_{2}$ , ⋯, $s_{N_{s} (t)}$ represent the duration of all MAAPs [33] before time t, where $N_{s} (t)$ is the total number of these duty action cycles until time t. Then,

$\begin{matrix} N (t), \hat{\bar{γ}} = arg max_{N, \bar{γ}} p (N, \bar{γ} ∣ g_{1}, \dots, g_{N_{f} (t)}, s_{1}, \dots, s_{N_{S} (t)}) \\ = arg max_{N, \bar{γ}} p (g_{1}, \dots, g_{N_{f} (t)}, s_{1}, \dots, s_{N_{S} (t)} ∣ N, \bar{γ}) p (N) p (\bar{γ}) \\ = arg max_{N, \bar{γ}} p_{Γ} (\bar{γ}) \prod_{i = 1}^{N_{f} (t)} p_{G} (g_{i} ∣ N, \bar{γ}) \prod_{j = 1}^{N_{S} (t)} p_{S} (s_{j} ∣ N, \bar{γ}), \end{matrix}$

(16)

where the left-hand side of the equation, that is, $N (t), \hat{\bar{γ}}$ , represents the estimated number of people $N (t)$ on duty at time t, and the average frequency of individual duty behaviors $\hat{\bar{γ}}$ , $p (.)$ represents the probability of the parameter. This step is based on the independence of the duration of the DAP and MAAP, and does not make any a priori assumptions about the number of people N; that is, it is assumed that $p (N)$ is uniformly distributed. Here, $p_{Γ}$ and $\bar{γ}$ are a priori data which need to be extracted from the bridge duty video in advance.

3.4. Driving Activity Evaluation

Based on the comprehensive consideration of the number of personnel performing operations $N_{o p}$ , the total number of bridge personnel $N_{t o t a l}$ , the total time of performing operations $T_{o p}$ , and the action entropy $E_{m o t i o n}$ as key indicators, the interaction between nonlinear factors and indicators is integrated to quantify the behavioral dynamics of the bridge personnel and its potential impact on navigation safety. The calculation formula for the ship driving activity rate (ACR) can be expressed as

$A C R = f (N_{o p}, N_{total}, T_{o p}, E_{motion}) .$

(17)

Considering the nonlinear characteristics of operation time and the impact of action entropy on activity evaluation, logarithmic and exponential functions are introduced to deal with these nonlinear factors:

$\begin{matrix} T_{Non} = log {(T_{o p} + 1)}^{w_{2}} \\ E_{Non} = e^{w_{3} E_{motion}} \end{matrix}$

4. Experiments

The experiment used two commercial off-the-shelf Wi-Fi devices, Tx and Rx, to collect Wi-Fi BFI data to implement the proposed method. A laptop connected to the Wi-Fi receiver acted as the sniffing end to process the BFI data. A Kinect depth camera was used to capture real benchmarks for comparison.

4.1. Experimental Setup

(1) Scenario Setting. To evaluate the performance of the proposed method in depth, this study constructed and tested two different experimental scenarios. First, a closed-space six-degrees-of-freedom ship driving simulation platform with a size of 2.62 m × 3.34 m was used to conduct a comprehensive system performance evaluation (referred to as test platform 1). Second, a “Golden Six” cruise ship cab that actually sailed on the Chongqing to Yichang route in China was selected as the second test platform. The size of the bridge used for actual system data collection and performance testing (referred to as test platform 2) was 12.85 m × 4.85 m, with the specific situation shown in Figure 7. In these two test scenarios, Tx and Rx respectively indicate the specific locations of the transmitter and receiver antennas of the Wi-Fi device.

(2) Hardware. Each experimental scenario setup used a 3 × 3 IEEE 802.11ac MU-MIMO system operating on channel 153 with a center frequency of $f_{c}$ = 5.77 GHz and a bandwidth of 80 MHz. Each system contained one AP (beamformer) and three STAs (beamformers), enabling M = 3 and N = 1 antennas, respectively. We set up the network using an off-the-shelf Netgear Nighthawk X4S AC2600 router (San Jose, CA, USA) for the AP and STAs. At the same time, a Kinect device produced by Microsoft was used to capture visual data with depth information on the driver’s behavior in real time. The captured video data were then used for calibration and verification. By testing these two scenarios, this study ensured a comprehensive evaluation of the applicability and effectiveness of our proposed method in different environments. The simulated driving platform provided a highly controllable experimental environment for indexed comprehensive performance evaluation testing, while the real cruise cab test further tested the stability and accuracy of the system under actual sailing conditions.

(3) Ground Truth. We used the Kinect camera [34] to record detailed driving activities at a frame rate of 30 fps and manually analyzed the recorded video clips to generate ground truth. At the same time, Network Time Protocol was used to synchronize the time recorded by the camera and the collected Wi-Fi BFI data.

(4) Test Subjects. The test recruited nine volunteers between the ages of 21 and 34 with ship-driving qualifications or related knowledge to participate in the experimental evaluation. Table 1 summarizes the details of all volunteers. The experiments were intentionally conducted to ensure that the age, weight, and height of the test subjects varied significantly; specifically, the weight of the test subjects ranged from 52 kg to 75 kg and their height ranged from 165 cm to 186 cm.

4.2. Experimental Procedures

(1) We tested the overall performance of the OOW activity rate on test platform 1. We asked all volunteers to complete the ship driving test program on the simulated driving platform. The system simultaneously collected Wi-Fi BFI data and the sniffer PC executed the algorithm of this article for processing. When the driving test program was terminated, BFI data collection was stopped simultaneously, and the activity evaluation index of the current system was calculated and compared with the driving test results displayed on the simulation platform.

(2) We conducted indicator testing in a real ship scenario on test platform 2. The test indicators included activity time, number of active drivers, and total number of people on duty. The experiment collected a total of 8.3 h of data during three days of actual sailing. After data calibration and segmentation, a total of 6000 BFI datasets, each with a length of 2 min, were extracted. The dataset contained BFI data from situations ranging from no active OOWs to a maximum of four active OOWs.

4.3. Detection Performance

Activity Rate Accuracy. The evaluation criterion for the activity rate accuracy was the Pearson Correlation Coefficient (PCC) between the simulated driving test scores and the calculated Activity Rate. An ascending trajectory of PCC values over the assessment period denotes an intensified correlation between the algorithmic detection outcomes and the driver’s actual activity rate. As depicted in Figure 8a, this escalation in PCC values over time affirms the robustness and accuracy of the activity level detection method. The increased duration correlates with higher PCC values, suggesting that the algorithm progressively mirrors the driver’s activity level with greater precision in real-time contexts.

Activity Time. Table 2 shows a comparison of the accuracy of movement time estimated based on BFI technology using the actual movement time recorded by video surveillance as the true value. In scenarios with different numbers of people moving, the method in this paper demonstrates an accuracy of more than 90%, verifying the effectiveness of BFI technology in motion time detection. At the same time, the corresponding system delays are kept at a low level, showing the sensitivity of this method to human activities in the scene.

Number of Active Drivers. The confusion matrix (Figure 8b) showcases the performance of our activity detection method within a maritime setting. High accuracy is indicated along the matrix’s diagonal, with values of 99.2% for zero people, 98.1% for one person, and over 83% for up to four people, evidencing the method’s precision in identifying the correct number of active individuals. The minimal values along the diagonal reflect the system’s occasional misclassifications, which are relatively infrequent. Overall, the system proves highly effective for personnel monitoring on a ship’s bridge.

Overall Number of OOWs. Figure 8c depicts the accuracy of our method for detecting the number of personnel on duty within the challenging environment of a ship’s bridge. The accuracy is highest when detecting a single individual, at approximately 95%, and generally decreases as the number of people increases, showing the method’s high reliability for smaller groups. There is a notable trend of declining accuracy with more individuals present, reaching around 75% for seven people. The error bars suggest greater variance in accuracy as the number of individuals increases, indicating challenges in detecting larger groups with the same precision as smaller ones. Despite the complexity of the environment, the proposed method exhibits commendable accuracy, particularly for smaller crew sizes, indicating its effectiveness for monitoring maritime watchkeeping.

Different Number of STAs. Due to the physical locations of various station terminals, there may be significant channel attenuation between specific STAs and the beamformer, potentially impairing model performance. To further investigate the impact of the number of sites on performance, in this study we experimentally validated the effect of varying site counts on the accuracy of detection results. The graph displays the accuracy of duty status detection using different numbers of sites. The data are organized according to three indicators, namely, number of personnel, movement time, and action entropy, with each category corresponding to the detection accuracy when using one, two, and three sites, respectively. As observed in Figure 8d, there is a notable improvement in detection accuracy across all data groups as the number of sites increases. In particular, when the number of sites reaches three the accuracy for all indicators approaches or exceeds 90%, demonstrating the effectiveness of a multi-site configuration in enhancing the performance of detection systems. These results emphasize the importance of deploying multiple detection sites in complex environments to ensure higher data accuracy and system reliability.

Different Antenna Configurations. Different antenna types significantly influence the outcomes of activity detection due to inherent disparities in their reception sensitivity, signal coverage area, and signal stability. This paper provides a detailed analysis of the application effects of 2 × 2 and 2 × 4 antenna configurations, which care both commonly used in activity detection systems. As depicted in Figure 8e, the detection results using the 2 × 4 antenna combination are markedly superior to those using the 2 × 2 combination. This improvement is attributed to the data transmission protocols, which allow for the extraction of four azimuth angles in the 2 × 2 configuration and ten azimuth angles in the 2 × 4 configuration, thereby increasing both the volume of data and the accuracy of detection.

Baseline Comparison. The methods compared in this study included our developed technique alongside two established techniques: MAIS [35] and FCC [36], with the results shown in Figure 8f. As is evident from the data, the performance of our method consistently surpasses that of FCC and MAIS across varying numbers of individuals present in the environment. This enhancement in precision highlights the adaptability and robustness of our method, particularly within the complex and dynamically changing conditions prevalent on a ship’s bridge. In scenarios with an increasing count of personnel, our method showcases higher accuracy rates and demonstrates notable stability in performance, with the standard deviation bars indicating less variability compared to the other methods. This suggests that our method is less susceptible to the ship’s environmental interference and movement dynamics. Furthermore, the graph illustrates that while FCC and MAIS offer reasonable accuracy under static or less dynamic conditions, their performance does not translate as effectively to the more variable and challenging environment of a ship’s bridge. This is indicated by the lower accuracy percentages and larger error bars for these methods compared to our method.

5. Conclusions

This study introduces a method for detecting the active status of OOWs on ship bridges by implementing wireless sensing technology for human activities in ship cabins. To achieve this goal, we have designed a detection method for ship cab OOW activity sensing based on Wi-Fi BFI data. In addition, the BFI perception model has been improved to better adapt to the complex ship cabin background, thereby improving the generalization ability of the existing sensing model. Our experimental results lead to the following conclusions:

The proposed method showcases high accuracy, especially in scenarios with fewer personnel, highlighting its potential for monitoring duties typically involving a small number of crew members.
While the system maintains commendable accuracy across varying group sizes, a decrease in performance is noted as the number of individuals rises, reflecting the intrinsic challenges of dense and active maritime environments.
Despite the complexities inherent to ship bridges, such as restricted spaces and the presence of operational equipment, the proposed method demonstrates significant effectiveness in distinguishing between different levels of activity.
The results suggest the applicability of the proposed approach for enhancing operational monitoring and safety management in maritime navigation, providing a foundation for real-time assessment of bridge activities.

The sensing method proposed in this article can be successfully applied to OOW monitoring application scenarios. However, there are a number of issues urgent that need to be solved. We intend to improve the following aspects in future work: (1) In response to the challenges that the current system may encounter in high-density environments, especially in multi-person monitoring scenarios, we plan to develop more advanced data processing algorithms. These algorithms will focus on improving the distinction between people, reducing the impact of occlusion, and improving the accuracy and stability of the system when monitoring multiple dynamic targets. (2) Further in-depth research will be conducted on the use of BFI technology for real-time behavior recognition of ship bridge personnel, including but not limited to the recognition of behavioral patterns such as walking, work operations, and first aid response. More powerful support can be provided for the ship’s emergency response and daily safety management through the accurate identification and analysis of specific behavioral patterns. (3) We will focus on improving the applicability of the system under different types of ships and environmental conditions, including bridges of different sizes and layouts as well as different sea and weather conditions. At the same time, we will explore the application of machine learning and deep learning technologies to improve the system’s generalization capabilities to ensure that the system can adapt to a wider range of monitoring scenarios and complex maritime activities.

Author Contributions

Conceptualization, M.C. (Mozi Chen), M.C. (Mengda Chen) and Y.Z.; methodology, M.C. (Mengda Chen), C.L. and Y.Z.; software and experiments, Y.L.; validation, M.C. (Mengda Chen), Y.Z. and Y.L.; investigation, Y.Z. and C.L.; resources, M.C. (Mozi Chen) and L.Z.; data curation, Y.L.; writing—original draft preparation, M.C. (Mengda Chen); writing—review and editing, M.C. (Mozi Chen); visualization, L.Z. and C.L.; supervision, M.C. (Mozi Chen); project administration, M.C. (Mengda Chen); funding acquisition, M.C. (Mozi Chen). All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the National Natural Science Foundation of China (NSFC) under grant No. 2021CFA001 and by the Natural Science Foundation of Hubei Province Youth Program under grant No. 20221J0059.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Processed data cannot be shared at this time as they are part of ongoing research.

Acknowledgments

The experimental environment and examination data of test platform 1 were provided by the Hubei Key Laboratory of Inland Shipping Technology. We thank them for supporting this study. We are also grateful for the support of the National Institute of Advanced Industrial Science and Technology (AIST).

Conflicts of Interest

Liang Zhang and Yang Liu were employed by Wuhan Zhongyuan Electronic Information Limited Corporation. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

Papadimitriou, E.; Schneider, C.; Tello, J.; Damen, W.; Vrouenraets, M.; Ten Broeke, A. Transport safety and human factors in the era of automation: What can transport modes learn from each other? Accid. Anal. Prev. 2020, 144, 105656. [Google Scholar] [CrossRef] [PubMed]
European Railway Agency. Railway Safety Performance in the European Union; European Commission: Luxembourg, 2011.
Cao, Y.; Wang, X.; Yang, Z.; Wang, J.; Wang, H.; Liu, Z. Research in marine accidents: A bibliometric analysis, systematic review and future directions. Ocean. Eng. 2023, 284, 115048. [Google Scholar] [CrossRef]
Huang, Y.; Chen, L.; Chen, P.; Negenborn, R.; Van Gelder, P. Ship collision avoidance methods: State-of-the-art. Saf. Sci. 2020, 121, 451–473. [Google Scholar] [CrossRef]
Kaplan, S.; Guvensan, M.; Yavuz, A.; Karalurt, Y. Driver behavior analysis for safe driving: A survey. IEEE Trans. Intell. Transp. Syst. 2015, 16, 3017–3032. [Google Scholar] [CrossRef]
Restuccia, F. IEEE 802.11 bf: Toward ubiquitous Wi-Fi sensing. arXiv 2021, arXiv:2103.14918. [Google Scholar]
Paik, B.; Cho, S.; Park, B.; Lee, D.; Bae, B. Development of real-time monitoring system using wired and wireless networks in a full-scale ship. Int. J. Nav. Archit. Ocean. Eng. 2010, 2, 132–138. [Google Scholar]
Heo, S.; Kim, J. Measurements Data-Based Location Determination for Characterizing Indoor Signal Propagations of the Shipboard Sensor Nodes. Wirel. Pers. Commun. 2016, 90, 1405–1412. [Google Scholar] [CrossRef]
Chen, M.; Liu, K.; Ma, J.; Gu, Y.; Dong, Z.; Liu, C. SWIM: Speed-aware WiFi-based passive indoor localization for mobile ship environment. IEEE Trans. Mob. Comput. 2019, 20, 765–779. [Google Scholar] [CrossRef]
Wang, W.; Liu, A.; Shahzad, M.; Ling, K.; Lu, S. Understanding and Modeling of WiFi Signal Based Human Activity Recognition. In Proceedings of the 21th Annual International Conference on Mobile Computing and Networking (MobiCom’15), Paris, France, 7–11 September 2015. [Google Scholar]
Zheng, T.; Chen, Z.; Cai, C.; Luo, J.; Zhang, X. V2iFi: In-vehicle vital sign monitoring via compact RF sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2020, 4, 70. [Google Scholar] [CrossRef]
Bai, Y.; Wang, Z.; Zheng, K.; Wang, X.; Wang, J. WiDrive: Adaptive WiFi-based recognition of driver activity for real-time and safe takeover. In Proceedings of the 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), Dallas, TX, USA, 7–10 July 2019; pp. 901–911. [Google Scholar]
Haque, K.; Meneghello, F.; Restuccia, F. Wi-BFI: Extracting the IEEE 802.11 Beamforming Feedback Information from Commercial Wi-Fi Devices. In Proceedings of the 17th ACM Workshop on Wireless Network Testbeds, Experimental Evaluation & Characterization, Madrid, Spain, 6 October 2023; pp. 104–111. [Google Scholar]
Bergasa, L.M.; Nuevo, J.; Sotelo, M.A.; Barea, R.; Lopez, M.E. Real-time system for monitoring driver vigilance. IEEE Trans. Intell. Transp. Syst. 2006, 7, 63–77. [Google Scholar] [CrossRef]
Lützhöft, M.; Dukic, T. Show me where you look and I’ll tell you if you’re safe: Eye tracking of maritime watchkeepers. In Proceedings of the 39th Nordic Ergonomics Society Conference, Lysekil, Sweden, 1–3 October 2007; pp. 75–78. [Google Scholar]
Youn, I.H.; Park, D.J.; Yim, J.B. Analysis of lookout activity in a simulated environment to investigate maritime accidents caused by human error. Appl. Sci. 2018, 9, 4. [Google Scholar] [CrossRef]
Li, K.; Gong, Y.; Ren, Z. A fatigue driving detection algorithm based on facial multi-feature fusion. IEEE Access 2020, 8, 101244–101259. [Google Scholar] [CrossRef]
Zhao, C.; Zhang, W.; Chen, C.; Yang, X.; Yue, J.; Han, B. Recognition of unsafe onboard mooring and unmooring operation behavior based on improved YOLO-v4 algorithm. J. Mar. Sci. Eng. 2023, 11, 291. [Google Scholar] [CrossRef]
Ma, Y.; Zhou, G.; Wang, S. WiFi sensing with channel state information: A survey. ACM Comput. Surv. (CSUR) 2019, 52, 1–36. [Google Scholar] [CrossRef]
Chen, C.; Song, H.; Li, Q.; Meneghello, F.; Restuccia, F.; Cordeiro, C. Wi-Fi sensing based on IEEE 802.11 bf. IEEE Commun. Mag. 2020, 61, 121–127. [Google Scholar] [CrossRef]
Kanda, T.; Sato, T.; Awano, H.; Kondo, S.; Yamamoto, K. Respiratory rate estimation based on WiFi frame capture. In Proceedings of the 2022 IEEE 19th Annual Consumer Communications & Networking Conference (CCNC), Las Vegas, NV, USA, 8–11 January 2022; pp. 881–884. [Google Scholar]
He, Y.; Chen, Y.; Hu, Y.; Zeng, B. WiFi vision: Sensing, recognition, and detection with commodity MIMO-OFDM WiFi. IEEE Internet Things J. 2020, 7, 8296–8317. [Google Scholar] [CrossRef]
Haque, K.; Zhang, M.; Meneghello, F.; Restuccia, F. BeamSense: Rethinking Wireless Sensing with MU-MIMO Wi-Fi Beamforming Feedback. arXiv 2023, arXiv:2303.09687. [Google Scholar]
Liu, K.; Chen, M.; Cai, E.; Ma, J.; Liu, S. Indoor localization strategy based on fault-tolerant area division for shipboard surveillance. Autom. Constr. 2018, 95, 206–218. [Google Scholar] [CrossRef]
Chen, M.; Ma, J.; Zeng, X.; Liu, K.; Chen, M.; Zheng, K.; Wang, K. MD-alarm: A novel manpower detection method for ship bridge watchkeeping using Wi-Fi signals. IEEE Trans. Instrum. Meas. 2022, 71, 1–13. [Google Scholar] [CrossRef]
Itahara, S.; Kondo, S.; Yamash*ta, K.; Nishio, T.; Yamamoto, K.; Koda, Y. Beamforming feedback-based model-driven angle of departure estimation toward legacy support in WiFi sensing: An experimental study. IEEE Access 2022, 10, 59737–59747. [Google Scholar] [CrossRef]
Habib, U.; Zucker, G.; Blochle, M.; Judex, F.; Haase, J. Outliers detection method using clustering in buildings data. In Proceedings of the IECON 2015—41st Annual Conference of the IEEE Industrial Electronics Society, Yokohama, Japan, 9–12 November 2015; pp. 694–700. [Google Scholar]
Di Domenico, S.; De Sanctis, M.; Cianca, E.; Bianchi, G. A trained-once crowd counting method using differential wifi channel state information. In Proceedings of the 3rd International on Workshop on Physical Analytics, Singapore, 26 June 2016; pp. 37–42. [Google Scholar]
Huang, S.; Cai, N.; Pacheco, P.; Narrandes, S.; Wang, Y.; Xu, W. Applications of support vector machine (SVM) learning in cancer genomics. Cancer Genom. Proteom. 2018, 15, 41–51. [Google Scholar]
Qi, X.; Wang, H.; Chu, J.; Chiam, K. Effect of autocorrelation function model on spatial prediction of geological interfaces. Can. Geotech. J. 2022, 59, 583–600. [Google Scholar] [CrossRef]
Last, G.; Penrose, M. Lectures on the Poisson Process; Cambridge University Press: Cambridge, UK, 2017. [Google Scholar]
Giambene, G. Queuing Theory and Telecommunications; Springer: Berlin/Heidelberg, Germany, 2014. [Google Scholar]
Bishop, C.M. Pattern Recognition and Machine Learning; Springer: Berlin/Heidelberg, Germany, 2006; Volume 2, pp. 1122–1128. [Google Scholar]
Zhang, Z. Microsoft kinect sensor and its effect. IEEE Multimed. 2012, 19, 4–10. [Google Scholar] [CrossRef]
Feng, C.; Arshad, S.; Liu, Y. Mais: Multiple activity identification system using channel state information of wifi signals. In Proceedings of the Wireless Algorithms, Systems, and Applications: 12th International Conference, WASA 2017, Guilin, China, 19–21 June 2017; Proceedings 12. pp. 419–432. [Google Scholar]
Depatla, S.; Mostofi, Y. Crowd counting through walls using WiFi. In Proceedings of the 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom), Athens, Greece, 19–23 March 2018; pp. 1–10. [Google Scholar]

Figure 1. BFI-based sensing principle for OOWs and original data sample.

Figure 2. Overview of the methodology.

Figure 3. Preprocessing of the BFI collected from our real-world ship experiment: raw signal (a), interpolated signal (b), and smoothed signal (c).

Figure 4. BFI spectrograms from controlled experiments on a simulated driving platform: unoccupied environments (a) and occupied environments (b).

Figure 5. BFI signal characteristics corresponding to different driving behaviors (a) and results of activity entropy calculations for different numbers of active types (b).

Figure 6. Associated distribution of BFI to the number of active OOWs (a) and schematic diagram of the on-duty active time (b). Data were derived from extensive real-world ship experiments over several days of actual sailing.

Figure 7. Experimental scene and equipment placement: (a) test platform 1 and (b) test platform 2.

Figure 8. Detection results for the selected test: (a) activity status accuracy, (b) number of active drivers, (c) overall number of OOWs, (d) different number of STAs, (e) different antenna configurations, and (f) baseline test.

Table 1. Basic summary of test participants.

No.	#1	#2	#3	#4	#5	#6	#7	#8	#9
Age	22	25	21	23	22	34	25	26	33
Height (cm)	182	168	172	170	186	176	165	174	178
Weight (kg)	73	61	57	52	75	62	58	61	60
Gender	M	M	M	F	M	M	M	M	M

Table 2. Activity time detection accuracy table with different numbers of people.

Number of People	Active Time Accuracy	Average Delay
1	96.5%	37 ms
2	94.3%	45 ms
3	93.5%	41 ms
4	91.3%	47 ms

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).