Current Issue

The KLT (Kanade-Lucas-Tomasi) algorithm to track the motion of features in an image stream was introduced in 1990s. The algorithm assumes that images taken at near time instants are usually strongly related to each other, then the displacement d(ξ,η) of the point at x=(x,y) in image plane between time instants t and (t+τ) can be chosen to minimize the residue error defined by the following double integral over the given window W.⁹⁾

where I(x)=I(x,y,t) is the point at t, J(x)=I(x,y,t+τ) is that point at (t+τ), and ω is the weighting function depending on the image intensity pattern. The KLT algorithm finds the displacement d by rewriting the derivative of residual error in the form of

where the matrix G can be computed from one frame by estimating gradients and computing their second order moments, and vector e can be computed from the difference between the two frames. Based on KLT algorithm, the procedure of tracking target object by the camera attached in the gimbal is proposed as follows :

■ In the first taken frame, a rectangle of interests (ROI) containing the target object is chosen. In grayscale image, corners, parts of ROI contain complete motion information, are detected using minimum eigenvalue algorithm. The bounding box covers all these points, and center of the box is set as the initial location of the target.

■ Next, KLT algorithm is used to track these corners from frame to frame.

■ The tracking results are validated by using forward-backward error technique. In this technique, two tracking trajectories, forward point tracking and backward tracking validation, are compared and if they differ significantly, the tracking is considered as incorrect.

where T^k* is tracking trajectory (f: forward, b: backward) at k^th sequence image, x_t and  x^t are tracked point and validated point from forward and backward tracking trajectories.

■ In the case of successful tracking, the new location of the object is estimated from its center in previous frame and the displacement obtained by KLT algorithm.

As mentioned, the implementation of KLT algorithm may fail to track adequately in various circumstances of interest. Then, the target bounding box and object location in the image frame need to re-detect. Detecting process is implemented as follows :

■ At first, background rejection and object’s region detection are performed by a combination of image thresholding technique, morphological opening technique, and blob detection. In detail :

Background is rejected by using image thresholding techniques.

The image is then converted into a binary image. Morphological opening is performed for removing small objects from an image while preserving the shape and size of larger objects in the image.

Blob detection method is called to detect these regions in the binary image. The object should be in one of these regions, but we cannot conclude where it is exactly.

■ Then, corner features are then extracted in each region, and compared to features in the first frame. If these features are matched together, the object is detected. The bounding box and center are obtained.

■ Finally, KLT algorithm is re-initialized to track the object from these new features.

A flowchart presenting tracking and detecting procedure is shown in Fig. 1. An experimental study is shown in Fig. 2.

Fig. 1 
Flowchart of the proposed tracking procedure

Fig. 2 
Tracking and detecting results

a : Choosing the ROI in the first taken frameb : Corner features in ROI and center locationc, d : Tracking object by KLT algorithme : Tracking failed due to the degraded imagef, g : Background rejection by image thresholding technique (f) and morphological opening performance (g)h : Detecting object by Blob detector and features matchingi, j : Re-tracking by the KLT algorithmk~n : Object detection process when tracking is failed due to the fast-moving targeto : Success rate

In this study, the well-known pinhole camera model is used to obtain the relation between locations of the target object in 3D world coordinate and in 2D image plane. In this model, the optical axis runs through the pinhole perpendicular to the image plane. The location of the object expressed in the coordinate attached with the camera is given by P=[X,Y,Z]^T. The point of the object in the image plane is p=[x,y]^T. The coordinates [x,y] represents the columns and the rows of the image in pixel, respectively. The pinhole camera model yields that :

Fig. 3 
Pinhole camera model

where f is called the focal distance, s_x and s_y are the pixel dimensions. Then automatically :

Consider a two-axis gimbal carrying the camera attached at the center. The gimbal consists of an outer channel actuating pan motion and an inner channel operating tilt motion. In general, the main goal of tracking control applying for gimbal system is to keep the target object in a pre-defined frame of view around the center of the image frame.

If Ψ_di(i=1,2) is the angular position of the object in the reference coordinate having its origin at the inner gimbal, then the additional angle that the gimbal needs to steer in order to point to the object is Ψ_ai=Ψ_di-Ψ_i where Ψ_i is the current position of the gimbal, including the tilt and the pan positions.

In order to obtain steering angles needed, let us consider the two-axis gimbal as an universal joint connecting two channels whose axis are inclined to each other by the tilt angle. Fig. 4 shows planes of rotation of the pan axis and the camera’s axis. Target point A in horizontal axis in image plane and its corresponding location in the camera plane are illustrated. To track the target, the camera need to steer an angle of Ψ_ai in camera plane as.

Fig. 4 
Relationship between target location in image plane and gimbal’s steering angles

Then, from the kinematics of the universal joint, the steering angle about pan axis

Although point A initially lies on the horizontal axis, control the pan motion alone can not bring it to the center of the image, as shown in Fig. 4. Consequently, correction in tilt motion is needed.

In general, if the point of the target in the image plane is p=[x,y]^T, the steering angles that the gimbal needs to perform are :

Assume that the focal length f is a constant during tracking, then the steering angles can be computed from the captured image without knowledge of the distance to the object.

A standard image-based visual tracking problem uses the additional angles calculated from the tracker as the control error as in Fig. 5.

Fig. 5 
Configuration of an standard image-based visual tracking control system

The control law can be easily designed based on the Moore-Penrose pseudo-inverse matrix of camera interaction matrix.^3,4)

Moreover, from the viewpoint of control system, camera and tracker add transportation delay into the system. The delay time consists of time of taking snapshot, transmitting data, and sampling time of tracking algorithm. Therefore, the steering angle obtained from the tracker is actually Ψ_aie^-sT, T is the delay time. Then, the closed-loop system with a standard image-based controller can be described as :

As seen in Eq. (11), e^-sT appears in both numerator and denominator of the transfer function proving that time delay not only reduces system performance but also influent to system stability. Suppose that we can express :

where N(s) and D(s) are real polynomials of degree p and q respectively. Then the characteristic equation of the closed-loop system is :

If q≥p and bp/aq<1, where a_q,b_p are the leading coefficients of D(s) and N(s) respectively, the Nyquist criterion is applicable to check the stability of the system.¹⁶⁾ If not, the system is unstable. The complete set S_R of adjustable parameters of controller K_t(s) such that the system stability is preserved for any delay time T∈[0,T₀] is obtained as :

where SI=S0╲SN,S0, is the complete set of controller adjustable parameters that stabilize the delay-free plant G_i(s),S_N is the set such that the delay-free open-loop expression K_t(s)G_i(s) is an improper transfer function.

And S_L is the set of controller parameters such that K_t(s)G_i(s)e^-sT=-1 for a specific delay time T∈[0,T₀]. It is easily seen that S_R⊆S_I⊆S₀, means that the stability of the delay-time system is preserved in a stricter condition comparing to the delay-free one. The controller parameters make the system achieves good results may not in the subset S_R, so the control performance can be reduced. The longer delay time, the more limited controller parameters and the more sluggish system response.¹²⁾ Based on Smith predictor, a new control structure is proposed as Fig. 6. A Smith Predictor is implemented as Ψ_i-Ψ_ie^-sT. When the amount of delay is known, the position of the target computed from image tracker can be obtained by Eq. (15) such that the system is represented as Eq. (16), respectively.

Fig. 6 
Configuration of the proposed control system

The delay is brought out of the characteristic equation, so the system stability is preserved in the same condition with the delay-free one. The set of controller adjustable parameters is now S₀.

However, time delay still gives an upper bound on the achievable bandwidth of the system. It is recommended that the bandwidth of a system with delay does not exceed π/2T or 2/T.^12,15)

Now, let us generally describe the plant of tracking motion system as follows :

The plant contains the closed-loop of gimbal stabilization control, where there are the plant of the gimbal mechanism and a stabilization controller. In Eq. (17), a_i,b_i and c_i are the nominal parameters of the system model corresponding to system dynamics, and u_i is the control signal of proposed tracking controller.

A PID controller is implemented in the form as :

where controller gains are tuned so that system bandwidth is satisfied.