Referring to Bénasséni (2018) or Enguix-González et al. (2005) and letting x¯I=(1/r)∑i∈Ixi, we know that, when a subset of observations indexed by I is deleted, the covariance matrix S is transformed to S~ which can be expressed as: 1S~=S+(rn−r)[S−1r∑i∈I(xi−x¯)(xi−x¯)t]+(rn−r)2[−(x¯I−x¯)(x¯I−x¯)t]

We then have a perturbation of the form S~=S+ϵM+ϵ2N with ϵ=rn−r, M=S−(1/r)∑i∈I(xi−x¯)(xi−x¯)t and N=−(x¯I−x¯)(x¯I−x¯)t. Following matrix perturbation theory detailed in Wilkinson (1988) for example or referring to Sibson (1979), we know that, if ϵ is sufficiently small, for each simple eigenvalue λ of S there is an eigenvalue λ~ of S~ given by a convergent power series: 2λ~=λ+γ1ϵ+γ2ϵ2+…+γmϵm+O(ϵm+1)

with a corresponding eigenvector which can also be expressed under a convergent power series: 3ϕ~=ϕ+ψ1ϵ+ψ2ϵ2+…+ψmϵm+O(ϵm+1)

The parameters γ₁, γ₂, …, γ_m and ψ1,ψ2,…,ψm are derived by equating the coefficients of ϵ, ϵ2,…,ϵm in the equation S~ϕ~=λ~ϕ~. It should also be noted that the perturbation λ~k of λ_k may not necessarily be the kth largest eigenvalue of S~ if the subset of observations indexed by I has initially a strong influence on λ_k for example. In this case, following Critchley (1985), one can simply assume that the eigenvalues λ~k have been reordered in decreasing order and that their corresponding eigenvectors ϕ~k have been relabeled. However, the reader is referred to Masioti et al. (2023) for a more comprehensive discussion on this topic.

For any integer m≥1, retaining only terms of order lower or equal to m in ϵ in (2) and (3) provides the approximations λ~(m) and ϕ~(m) of order m for the eigenvalues and eigenvectors of S~.

This is the general approach suggested in Bénasséni (2018), Enguix-González et al. (2005) and Wang and Liski (1993). Assuming that ϵ=rn−r is sufficiently small to ensure the convergence of the above power series, Enguix-González et al. (2005) provide the following approximations for k=1,…,p: 4λ~k(1)=λk+(rn−r)(λk−1r∑i∈Iαki2) 5λ~k(2)=λ~k(1)+(rn−r)2[−αkI2−∑j≠k1λj−λk(1r∑i∈Iαkiαji)2] 6ϕ~k(1)=ϕk+(rn−r)∑j≠k(1r∑i∈Iαkiαji)ϕjλj−λk

where αki=ϕkt(xi−x¯), for i=1,…,n and αkI=(∑i∈Iαki)/r.

Details on the derivation of the above expressions are given in Bénasséni (2018). The reader is referred to Enguix-González et al. (2005) for the formula of ϕ~k(2) which is fairly long and therefore omitted in this paper. It should also be pointed out that a comprehensive study of approximations for the unbiased matrix (n−r)S~/(n−r−1) is provided in this last reference. In the remaining of the paper, we define also the approximations of order zero as λ~(0)=λ and ϕ~(0)=ϕ for notational convenience.

Finally, when studying the influence of a single observation, note that approximations are simply obtained by taking r = 1, I={i} and ϵ=1n−1 in the previous developments. The reader more specifically interested by this case will refer to a series of papers including Critchley (1985), Hadi and Nyquist (1993), Masioti et al. (2023) and Wang and Nyquist (1991).

Assuming that weights are given to the observations, Bénasséni (1987) studies the effects of modifying these weights on the eigenvalues and eigenvectors of the covariance matrix. Deleting a small subset of observations indexed by I is therefore a particular case of his approach which consists simply of modifying to zero the corresponding weights. As approximations to λ~k for k=1,…,p, this author suggests using the Rayleigh quotient qk(0)=(ϕ~k(0))tS~ϕ~k(0)/(ϕ~k(0))tϕ~k(0) for S~ and the initial normalized eigenvector ϕ~k(0), and the Rayleigh quotient qk(1)=(ϕ~k(1))tS~ϕ~k(1)/(ϕ~k(1))tϕ~k(1) for S~ and the approximation of order one ϕ~k(1) to ϕ~k.

From a computational standpoint, it is of major interest to evaluate the accuracy of approximations without having to recompute the exact eigenelements of S~. In order to do this, Bénasséni (1987) suggests using inequalities provided in Wilkinson (1988). Focusing on λ~k and its corresponding eigenvector ϕ~k and, from now on, assuming without change of notation that ϕ~k(m) has been normalized, let ηk(m)=‖S~ϕ~k(m)−qk(m)ϕ~k(m)‖2 for m=0,1 where ‖.‖2 stands for the two norm. Assume that ck(m)∈R+∗ is a nonzero positive constant such that |λ~j−qk(m)|>ck(m) for j=1,…,p with j≠k. Then the accuracy of qk(m) as approximation to λ~k, and of ϕ~k(m) as approximation to ϕ~k, is analyzed in Bénasséni (1987) using the following inequalities given in (Wilkinson, 1988, pp. 172–176): 7‖ϕ~k−ϕ~k(m)‖22≤(ηk(m)ck(m))2[1+(ηk(m)ck(m))2]

and if ηk(m)/ck(m)<1: 8|λ~k−qk(m)|≤[(ηk(m))2ck(m)]/[1−(ηk(m)ck(m))2]

Using ‖ϕ~k‖2=‖ϕ~k(m)‖2=1, note that (7), can be written with the cosine between ϕ~k and ϕ~k(m) as: 91−[(ηk(m))2/2(ck(m))2][1+(ηk(m)/ck(m))2]≤cos⁡(ϕ~k,ϕ~k(m)).

In practice, it is necessary to give the parameter ck(m) a value in the previous inequalities. When studying the influence of a single observation, it should be noted that (1) can be expressed as 10S~=nn−1[S−1n−1(xi−x¯)(xi−x¯)t]

so that we have a rank one perturbation. In this case, the parameter ck(m) is given a value in Bénasséni (1987) using bounds, derived from the Courant-Fischer theorem, for the eigenvalues λ~j of the symmetric matrix S~, j=1,…,p, j≠k. In the case where several observations are deleted this author suggests a fairly lengthy procedure assuming that these observations are removed one after the other, so that we have a series of rank one perturbations.

In his work, Bénasséni (1987) only considers the approximations of order one λ~k(1) for the eigenvalues of S~ and provides no comparison of λ~k(2) with approximations based on Rayleigh quotients. However, it is easy to derive the following simple relations. First note that qk(0) can be written as: 11qk(0)=λk+(rn−r)(λk−1r∑i∈Iαki2)−(rn−r)2αkI2

using (1). Therefore we see that we have always λ~k(1)≥qk(0). Furthermore, a simple comparison of (11) with (5) shows that: λ~k(2)−qk(0)=−(rn−r)2∑j≠k1λj−λk(1r∑i∈Iαkiαji)2.

In particular, when focusing on the largest eigenvalue which plays a central role in PCA, this difference is non negative so that we have λ~1(2)≥q1(0). In a similar way, when considering the smallest eigenvalue, we get λ~p(2)≤qp(0).

We omit the derivation of qk(1) which is tedious and leads to a formula too complicated to be interpreted. However, it should be noted that this approximation involves terms up to the order 4 in ϵ=rn−r.

In practice, it is of crucial importance to have bounds as close as possible to the true values of |λ~k−qk(m)| and cos⁡(ϕ~k,ϕ~k(m)) in Inequalities (8) and (9). This will allow for an evaluation as precise as possible of the real accuracy of the approximations without having to recompute the perturbed analysis. For this purpose, we introduce below refined inequalities in the study of covariance matrices in order to get an improved error analysis

Indeed, Inequalities (7), (8) and (9) introduced in the Subsection Error Analysis can be improved using error analysis developed in Chatelin (2012, pp. 180–184). More precisely, it is easily derived from Corollary 4.6.4 in this reference that: 12|λ~k−qk(m)|≤(ηk(m))2ck(m)

and 13sin⁡(ϕ~k,ϕ~k(m))≤ηk(m)ck(m)

under the condition: 14ηk(m)<ck(m).

It is obvious that (12) is more accurate than (8). A similar remark holds for (13) wich improves (9). This last point is easily checked by converting (13) into 15cos2(ϕ~k,ϕ~k(m))≥1−(ηk(m)/ck(m))2.

Then letting a=(ηk(m)/ck(m))2, A=1−a and B=[1−(a/2)(1+a)]2, Inequality (9) becomes cos2(ϕ~k,ϕ~k(m))≥B so that we have simply to prove that A≥B. Developing B, we get B=A+(a2/4)(a2+2a−3). Thus we have that A≥B if and only if the polynomial a2+2a−3 is non-positive. This is the case when a belongs to [−3,1]. Therefore the result follows since 0≤a<1 from (14).

Furthermore, when dealing with the eigenvector associated to the largest eigenvalue, Chatelin (2012, p. 204) points out that Inequality (13) can also be refined into the following tangent based inequality: 16tan⁡(ϕ~1,ϕ~1(m))≤η1(m)c1(m)

since this eigenvalue is assumed to be simple. More precisely, letting α denote the angle between the two vectors ϕ~1 and ϕ~1(m), we have α≤arctan⁡(η1(m)/c1(m))≤arcsin⁡(η1(m)/c1(m)) since 0≤arctan⁡x≤arcsin⁡x for all x∈[0,1]. Thus we obtain a better approximation of α when using the function arctan rather than the function arcsin showing that (16) improves (13).

The sharpness of the bounds in Inequalities (12), (13) and (16) depends on the value of the parameter ck(m). The larger ck(m) is, the sharper are these inequalities. In order to get a suitable value for this parameter, some other results of Chatelin (2012, pp. 180–181) or of Wilkinson (1988, pp. 174–176) turn out to be also of specific interest since they often improve significantly the value of the parameter ck(m) obtained through the Courant-Fischer theorem in Bénasséni (1987). More precisely, from these references we know that there is at least one eigenvalue of S~ in each of the intervals defined for j=1,…,p by [bj(m),Bj(m)]=[qj(m)−ηj(m),qj(m)+ηj(m)] which are often referred to as Krylov-Weinstein intervals. When the interval [bk(m),Bk(m)] is isolated from the p−1 other ones, we know that it contains precisely one eigenvalue. Then, assuming that the Rayleigh quotients satisfy q1(m)>q2(m)>…>qp(m) (after having been reordered if necessary), a value for ck(m) can be easily derived as ck(m)=min(bk(m)−Bk+1(m),bk−1(m)−Bk(m)) if k∈{2,…,p−1}, c1(m)=b1(m)−B2(m) if k = 1 and cp(m)=bp−1(m)−Bp(m) if k=p. Finally, note that these Krylov-Weinstein intervals proved to be also useful in studying the effects of some small errors in the data table itself as emphasized by the work of Bénasséni (1988, pp. 303–310)

It should be pointed out that for very close eigenvalues, giving a value to this parameter can be a real issue. However, once a value satisfying (14) is obtained, we know from (12) that, for k=1,…,p, the eigenvalues λ~k lie in the intervals [qk(m)−(ηk(m))2/ck(m),qk(m)+(ηk(m))2/ck(m)] which are sharper than the Krylov-Weinstein intervals as soon as (14) holds. These new intervals can be used to obtain a larger value for the parameter ck(m), thus improving (12), (13) and (16). This process could be iterated, but no significative improvment is generally observed.

Finally, it was noted in (10) that we have a rank one perturbation when deleting a single observation. Several inequalities and relations for this restricted rank perturbation are given in Bénasséni (1987), Hadi and Nyquist (1993), and Wang and Nyquist (1991). The reader is also referred to more recent works by Bénasséni (2011), Cheng et al. (2014), and Ipsen and Nadler (2009), who suggest new bounds for the perturbed eigenvalues. Although Krylov-Weinstein intervals generally provide a satisfying value for the parameter ck(m), these recent works can also be interesting in the determination of the largest possible constant ck(m) in order to make Inequalities (12), (13) and (16) sharper.

Table 1 provides for each subset I={i}, the perturbed eigenvalue λ~1, its order one and two approximations λ~1(1) and λ~1(2), the Rayleigh quotients q1(0) ans q1(1) and the differences between each approximation and the true perturbed eigenvalue λ~1. In the last two columns we find the bounds to |λ~1−q1(0)| and |λ~1−q1(1)| given by Inequality (12).

The first comment regarding the results in this table is that λ~1(1) is by far the less accurate approximation in all cases. In contrast q1(1) always provides extremely sharp approximations since it deviates from λ~1 by 1.76×10−3 in the worst case (when deleting Observation 4) and that the error is only 2.13×10−13 when deleting Observation 16. It should be noted that λ~1(2) also provides fairly satisfying approximations although clearly less accurate than q1(1). The Rayleigh quotient q1(0) is outperformed by λ~1(2) but remains significantly sharper than λ~1(1). Furthermore, it is worth pointing out that λ~1(1) always overestimates the perturbed eigenvalue while the other three estimations slightly underestimate it. Note also that the results agree with inequalities λ~1(1)≥q1(0) and λ~1(2)≥q1(0) given in the Subsection Some Relations Between the Approximations.

Second, Inequality (12) provides bounds sufficiently close to |λ~1−q1(0)| and |λ~1−q1(1)| to evaluate correctly the accuracy of Rayleigh quotients as approximations to λ~1 without having to recompute the perturbed analysis.

Third, it is easily seen from (10) that the maximum value for the perturbed eigenvalue is obtained when xi=x¯ with λ~1=(20λ1)/19=86.64028. We have the highest perturbed eigenvalue when deleting Observations Number 3, 5, 10, 12, 14, 16, 20 which are fairly close to x¯ and in these cases we get the sharper approximations to λ~1.

Focusing now on the second largest eigenvalue, Table 2 provides results similar to those of Table 1. It turns out that λ~2(1) is the less accurate approximation to λ~2. Except when deleting Observation 13, the Rayleigh quotient q2(1) again provides the best approximation with a very good accuracy since in the worst case corresponding to this observation we have λ~2−q2(1)=7.69×10−4. For this observation, λ~2(2) is slightly better but less accurate in all the other cases while performing fairly well in general. The Rayleigh quotient q2(0) performs in a similar way as q1(0) in Table 1.

It should be noted that again λ~2(1) always overestimates the perturbed eigenvalue but in contrast to Table 1, the other three approximations can as well slightly underestimate or overestimate λ~2.

Another difference with Table 1 is that bounds provided to |λ~2−q2(0)| and |λ~2−q2(1)| by Inequality (12) are not so close to these quantities as they were previously. For a part, this can be explained by the fact that we have a smaller value for c2(0) and c2(1) than for c1(0) and c1(1) when considering the largest eigenvalue. Indeed, for m=0,1 the gap |λ~3−q2(m)|  is smaller than the gap |λ~2−q1(m)| . However, if we except the case of Observation 4, we know from this bound that |λ~2−q2(1)| never exceeds 1.02×10−3 and this is sufficient for practical interpretation.

Results for the eigenvectors corresponding to the two largest eigenvalues are given in Table 3 which provides the sines sin⁡(ϕ~k,ϕk) and sin⁡(ϕ~k,ϕ~k(1)) for k=1,2 and their bounds provided by Inequality (13).

First, we consider the eigenvector associated to the largest eigenvalue. It could be noted that the maximum value of the sine bewteen the unperturbed and perturbed eigenvector is obtained when deleting Observation 4. This value corresponds to an angle of 2.85∘. In this case, as in all the other ones, the order one approximation performs fairly well since its sine with the perturbed eigenvector is equal to only 0.00494 which corresponds to an angle of 0.28∘. Furthermore bounds provided by Inequality (13) are always extremely close to the exact value of sin⁡(ϕ~1,ϕ~1(1)).

Focusing now on the eigenvector corresponding to the second largest eigenvalue, the maximum of sin⁡(ϕ~2,ϕ2) is again obtained when deleting Observation 4 with the value of 0.08114. Even in this case, the order one approximation is fairly close to the perturbed eigenvector since sin⁡(ϕ~2,ϕ~2(1))=0.01954. In contrast to the previous eigenvector, it is worth pointing out that bounds provided by Inequality (13) are not always sufficiently close to the true values of the sine to give an exact account of the accuracy for these approximations.

Finally, since the angles between the eigenvectors studied in the table are always very close to zero, we do not provide the tangent of these angles which only deviates from the sine by an extremely small amount.

Now, we study approximations to the perturbed largest eigenvalue ant its corresponding eigenvector when deleting the subsets of two observations considered for the numerical illustration in Wang and Liski (1993). Results similar to those of the previous subsection are provided in Tables 4 and 5.