Original Article

A Comparative Study of Approximations for Perturbation Analysis of Principal Components

Jacques Bénasséni*¹, Alain Mom¹

[1] Université Rennes, CNRS, IRMAR - UMR 6625, F-35000 Rennes, France.

Methodology, 2025, Vol. 21(1), 27–45, https://doi.org/10.5964/meth.15357

Received: 2024-08-20. Accepted: 2025-02-07. Published (VoR): 2025-03-31.

Handling Editor: Tamás Rudas, Eötvös Loránd University, Budapest, Hungary

*Corresponding author at: Université Rennes 2, Place du Recteur Henri Le Moal CS 24307 - 35043 Rennes cedex, France. E-mail: jacques.benasseni@univ-rennes2.fr

This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Principal component analysis is a well known method for dimension reduction based on the covariance matrix associated to a multivariate data table. Therefore, a large amount of work has been devoted to analyzing the sensitivity of the eigenstructure of this matrix to influential observations. In order to evaluate the effect of deleting one or a small subset of observations, several approximations for the perturbed eigenelements have been proposed. This paper provides a theoretical and numerical comparison of the main approximations. A special emphasis is given to those based on Rayleigh quotients since they are under-utilized given their excellent performance. A general approach, using refined inequalities, is proposed in order to get a precise evaluation of their accuracy without having to recompute the exact perturbed eigenvalues and eigenvectors. This approach is of specific interest from a computational standpoint. Theoretical developments are illustrated with a numerical study which emphasizes the accuracy of approximations based on Rayleigh quotients.

Keywords: approximation, eigenvalue and eigenvector, covariance matrix, principal component analysis, perturbation, Rayleigh quotient.

We consider a n × p data matrix X in which the n rows are the observation vectors $x_{i}^{t} \in R^{p}$ , $i = 1, \dots, n$ . Letting $\bar{x} = (1 / n) \sum_{i = 1}^{n} x_{i}$ denote the mean vector of dimension p, the covariance matrix $S = (1 / n) \sum_{i = 1}^{n} (x_{i} - \bar{x}) {(x_{i} - \bar{x})}^{t}$ is involved in a wide range of statistical methods including principal component analysis or multiple regression for example. Since S is known to be highly prone to influential observations, sensitivity aspects for principal component analysis have been discussed in several papers including Critchley (1985), Jolliffe (2002), Pack et al. (1988), Prendergast (2008), Prendergast and Li Wai Suen (2011), Tanaka (1988) among many others. It should be noted that perturbation issues for this method are still the subject of active research as emphasized by the recent work of Masioti et al. (2023) for example. In order to assess the influence of a small subset I of r observations on the eigenstructure of S, a possible approach consists of studying the effect of removing these r observations on the eigenelements of S. If we note a significant modification of the eigenelements, we know that these observations are strongly influential on the results. In this framework, letting $\tilde{S}$ denote the covariance matrix obtained without the subset of observations indexed by I, several authors have studied the relationship between the eigenelements of S and those of $\tilde{S}$ . More specifically, providing approximations to the eigenelements of $\tilde{S}$ allows to detect influential subsets of observations without having to recompute the exact modified eigenvalues and eigenvectors. Hadi and Nyquist (1993), Masioti et al. (2023), and Wang and Nyquist (1991) study the effect of deleting a single observation while Bénasséni (2018), Enguix-González et al. (2005) and Wang and Liski (1993) focus on the general case where I comprises several observations. In these works, approximations to the eigenvalues and eigenvectors are obtained by retaining the first terms in power expansions of these parameters. Independently of these works, Bénasséni (1987) suggests using approximations based on Rayleigh quotients together with inequalities given in Wilkinson (1988). More generally, approximations for the covariance matrix are also the subject of research in a wider framework as emphasized by other works including for example Enguix-González et al. (2015) which considers the moments of the eigenelements or Enguix-González et al. (2012) dealing with the conditional bias of eigenvalues.

The contribution of this work is twofold. First, some new elements of comparison are provided between approximations derived from power expansions and those based on Rayleigh quotients. This comparison is based on some simple theoretical relations together with the numerical study of a data table which is intended to provide some guidance on the approximations to choose in practice. Second, refined inequalities in Chatelin (2012) are introduced in order to evaluate the accuracy of approximations without having to recompute the eigenelements of $\tilde{S}$ . It is proved that these inequalities always provide a sharper evaluation than those used in Bénasséni (1987). This is a definite advantage from a computational standpoint.

In this work, the eigenvalues $λ_{1} > λ_{2} \dots > λ_{p} \geq 0$ of S are assumed simple and associated to the normalized eigenvectors $ϕ_{1}, ϕ_{2}, \dots ϕ_{p}$ . In the same way, the eigenvalues ${\tilde{λ}}_{1} > {\tilde{λ}}_{2} \dots > {\tilde{λ}}_{p} \geq 0$ of the perturbed matrix $\tilde{S}$ are also assumed simple and associated to the eigenvectors ${\tilde{ϕ}}_{1}, {\tilde{ϕ}}_{2}, \dots {\tilde{ϕ}}_{p}$ .

Approximations Based On Power Expansions

Theoretical Background On Matrix Perturbations

Referring to Bénasséni (2018) or Enguix-González et al. (2005) and letting ${\bar{x}}_{I} = (1 / r) \sum_{i \in I} x_{i}$ , we know that, when a subset of observations indexed by I is deleted, the covariance matrix S is transformed to $\tilde{S}$ which can be expressed as:

1

\begin{matrix} \tilde{S} = S + (\frac{r}{n - r}) [S - \frac{1}{r} \sum_{i \in I} (x_{i} - \bar{x}) {(x_{i} - \bar{x})}^{t}] + {(\frac{r}{n - r})}^{2} [- ({\bar{x}}_{I} - \bar{x}) {({\bar{x}}_{I} - \bar{x})}^{t}] \end{matrix}

We then have a perturbation of the form $\tilde{S} = S + ϵ M + ϵ^{2} N$ with $ϵ = \frac{r}{n - r}$ , $M = S - (1 / r) \sum_{i \in I} (x_{i} - \bar{x}) {(x_{i} - \bar{x})}^{t}$ and $N = - ({\bar{x}}_{I} - \bar{x}) {({\bar{x}}_{I} - \bar{x})}^{t}$ . Following matrix perturbation theory detailed in Wilkinson (1988) for example or referring to Sibson (1979), we know that, if ϵ is sufficiently small, for each simple eigenvalue λ of S there is an eigenvalue $\tilde{λ}$ of $\tilde{S}$ given by a convergent power series:

2

\begin{matrix} \tilde{λ} = λ + γ_{1} ϵ + γ_{2} ϵ^{2} + \dots + γ_{m} ϵ^{m} + O (ϵ^{m + 1}) \end{matrix}

with a corresponding eigenvector which can also be expressed under a convergent power series:

3

\begin{matrix} \tilde{ϕ} = ϕ + ψ_{1} ϵ + ψ_{2} ϵ^{2} + \dots + ψ_{m} ϵ^{m} + O (ϵ^{m + 1}) \end{matrix}

The parameters γ₁, γ₂, …, γ_m and $ψ_{1}$ , $ψ_{2}$ ,…, $ψ_{m}$ are derived by equating the coefficients of ϵ, $ϵ^{2}$ ,…, $ϵ^{m}$ in the equation $\tilde{S} \tilde{ϕ} = \tilde{λ} \tilde{ϕ}$ . It should also be noted that the perturbation ${\tilde{λ}}_{k}$ of λ_k may not necessarily be the kth largest eigenvalue of $\tilde{S}$ if the subset of observations indexed by I has initially a strong influence on λ_k for example. In this case, following Critchley (1985), one can simply assume that the eigenvalues ${\tilde{λ}}_{k}$ have been reordered in decreasing order and that their corresponding eigenvectors ${\tilde{ϕ}}_{k}$ have been relabeled. However, the reader is referred to Masioti et al. (2023) for a more comprehensive discussion on this topic.

Formulae for the Approximations Derived From Power Expansions

For any integer $m \geq 1$ , retaining only terms of order lower or equal to m in ϵ in (2) and (3) provides the approximations ${\tilde{λ}}^{(m)}$ and ${\tilde{ϕ}}^{(m)}$ of order m for the eigenvalues and eigenvectors of $\tilde{S}$ .

This is the general approach suggested in Bénasséni (2018), Enguix-González et al. (2005) and Wang and Liski (1993). Assuming that $ϵ = \frac{r}{n - r}$ is sufficiently small to ensure the convergence of the above power series, Enguix-González et al. (2005) provide the following approximations for $k = 1, \dots, p$ :

4

\begin{matrix} {\tilde{λ}}_{k}^{(1)} = λ_{k} + (\frac{r}{n - r}) (λ_{k} - \frac{1}{r} \sum_{i \in I} α_{k i}^{2}) \end{matrix}

5

\begin{matrix} {\tilde{λ}}_{k}^{(2)} = {\tilde{λ}}_{k}^{(1)} + {(\frac{r}{n - r})}^{2} [- α_{k I}^{2} - \sum_{j \neq k} \frac{1}{λ_{j} - λ_{k}} {(\frac{1}{r} \sum_{i \in I} α_{k i} α_{j i})}^{2}] \end{matrix}

6

\begin{matrix} {\tilde{ϕ}}_{k}^{(1)} = ϕ_{k} + (\frac{r}{n - r}) \sum_{j \neq k} (\frac{1}{r} \sum_{i \in I} α_{k i} α_{j i}) \frac{ϕ_{j}}{λ_{j} - λ_{k}} \end{matrix}

where $α_{k i} = ϕ_{k}^{t} (x_{i} - \bar{x})$ , for $i = 1, \dots, n$ and $α_{k I} = (\sum_{i \in I} α_{k i}) / r$ .

Details on the derivation of the above expressions are given in Bénasséni (2018). The reader is referred to Enguix-González et al. (2005) for the formula of ${\tilde{ϕ}}_{k}^{(2)}$ which is fairly long and therefore omitted in this paper. It should also be pointed out that a comprehensive study of approximations for the unbiased matrix $(n - r) \tilde{S} / (n - r - 1)$ is provided in this last reference. In the remaining of the paper, we define also the approximations of order zero as ${\tilde{λ}}^{(0)} = λ$ and ${\tilde{ϕ}}^{(0)} = ϕ$ for notational convenience.

Finally, when studying the influence of a single observation, note that approximations are simply obtained by taking r = 1, $I = {i}$ and $ϵ = \frac{1}{n - 1}$ in the previous developments. The reader more specifically interested by this case will refer to a series of papers including Critchley (1985), Hadi and Nyquist (1993), Masioti et al. (2023) and Wang and Nyquist (1991).

Approximations Based On Rayleigh Quotients

Rayleigh Quotients as Approximations to the Perturbed Eigenvalues

Assuming that weights are given to the observations, Bénasséni (1987) studies the effects of modifying these weights on the eigenvalues and eigenvectors of the covariance matrix. Deleting a small subset of observations indexed by I is therefore a particular case of his approach which consists simply of modifying to zero the corresponding weights. As approximations to ${\tilde{λ}}_{k}$ for $k = 1, \dots, p$ , this author suggests using the Rayleigh quotient $q_{k}^{(0)} = {({\tilde{ϕ}}_{k}^{(0)})}^{t} \tilde{S} {\tilde{ϕ}}_{k}^{(0)} / {({\tilde{ϕ}}_{k}^{(0)})}^{t} {\tilde{ϕ}}_{k}^{(0)}$ for $\tilde{S}$ and the initial normalized eigenvector ${\tilde{ϕ}}_{k}^{(0)}$ , and the Rayleigh quotient $q_{k}^{(1)} = {({\tilde{ϕ}}_{k}^{(1)})}^{t} \tilde{S} {\tilde{ϕ}}_{k}^{(1)} / {({\tilde{ϕ}}_{k}^{(1)})}^{t} {\tilde{ϕ}}_{k}^{(1)}$ for $\tilde{S}$ and the approximation of order one ${\tilde{ϕ}}_{k}^{(1)}$ to ${\tilde{ϕ}}_{k}$ .

Error Analysis

From a computational standpoint, it is of major interest to evaluate the accuracy of approximations without having to recompute the exact eigenelements of $\tilde{S}$ . In order to do this, Bénasséni (1987) suggests using inequalities provided in Wilkinson (1988). Focusing on ${\tilde{λ}}_{k}$ and its corresponding eigenvector ${\tilde{ϕ}}_{k}$ and, from now on, assuming without change of notation that ${\tilde{ϕ}}_{k}^{(m)}$ has been normalized, let $η_{k}^{(m)} = {‖ \tilde{S} {\tilde{ϕ}}_{k}^{(m)} - q_{k}^{(m)} {\tilde{ϕ}}_{k}^{(m)} ‖}_{2}$ for $m = 0, 1$ where ${‖ . ‖}_{2}$ stands for the two norm. Assume that $c_{k}^{(m)} \in R^{+ *}$ is a nonzero positive constant such that $| {\tilde{λ}}_{j} - q_{k}^{(m)} | > c_{k}^{(m)}$ for $j = 1, \dots, p$ with $j \neq k$ . Then the accuracy of $q_{k}^{(m)}$ as approximation to ${\tilde{λ}}_{k}$ , and of ${\tilde{ϕ}}_{k}^{(m)}$ as approximation to ${\tilde{ϕ}}_{k}$ , is analyzed in Bénasséni (1987) using the following inequalities given in (Wilkinson, 1988, pp. 172–176):

7

\begin{matrix} {‖ {\tilde{ϕ}}_{k} - {\tilde{ϕ}}_{k}^{(m)} ‖}_{2}^{2} \leq {(\frac{η_{k}^{(m)}}{c_{k}^{(m)}})}^{2} [1 + {(\frac{η_{k}^{(m)}}{c_{k}^{(m)}})}^{2}] \end{matrix}

and if $η_{k}^{(m)} / c_{k}^{(m)} < 1$ :

8

\begin{matrix} | {\tilde{λ}}_{k} - q_{k}^{(m)} | \leq [\frac{{(η_{k}^{(m)})}^{2}}{c_{k}^{(m)}}] / [1 - {(\frac{η_{k}^{(m)}}{c_{k}^{(m)}})}^{2}] \end{matrix}

Using ${‖ {\tilde{ϕ}}_{k} ‖}_{2} = {‖ {\tilde{ϕ}}_{k}^{(m)} ‖}_{2} = 1$ , note that (7), can be written with the cosine between ${\tilde{ϕ}}_{k}$ and ${\tilde{ϕ}}_{k}^{(m)}$ as:

9

\begin{matrix} 1 - [{(η_{k}^{(m)})}^{2} / 2 {(c_{k}^{(m)})}^{2}] [1 + {(η_{k}^{(m)} / c_{k}^{(m)})}^{2}] \leq \cos ({\tilde{ϕ}}_{k}, {\tilde{ϕ}}_{k}^{(m)}) . \end{matrix}

In practice, it is necessary to give the parameter $c_{k}^{(m)}$ a value in the previous inequalities. When studying the influence of a single observation, it should be noted that (1) can be expressed as

10

\begin{matrix} \tilde{S} = \frac{n}{n - 1} [S - \frac{1}{n - 1} (x_{i} - \bar{x}) {(x_{i} - \bar{x})}^{t}] \end{matrix}

so that we have a rank one perturbation. In this case, the parameter $c_{k}^{(m)}$ is given a value in Bénasséni (1987) using bounds, derived from the Courant-Fischer theorem, for the eigenvalues ${\tilde{λ}}_{j}$ of the symmetric matrix $\tilde{S}$ , $j = 1, \dots, p$ , $j \neq k$ . In the case where several observations are deleted this author suggests a fairly lengthy procedure assuming that these observations are removed one after the other, so that we have a series of rank one perturbations.

New Developments

Some Relations Between the Approximations

In his work, Bénasséni (1987) only considers the approximations of order one ${\tilde{λ}}_{k}^{(1)}$ for the eigenvalues of $\tilde{S}$ and provides no comparison of ${\tilde{λ}}_{k}^{(2)}$ with approximations based on Rayleigh quotients. However, it is easy to derive the following simple relations. First note that $q_{k}^{(0)}$ can be written as:

11

\begin{matrix} q_{k}^{(0)} = λ_{k} + (\frac{r}{n - r}) (λ_{k} - \frac{1}{r} \sum_{i \in I} α_{k i}^{2}) - {(\frac{r}{n - r})}^{2} α_{k I}^{2} \end{matrix}

using (1). Therefore we see that we have always ${\tilde{λ}}_{k}^{(1)} \geq q_{k}^{(0)}$ . Furthermore, a simple comparison of (11) with (5) shows that:

{\tilde{λ}}_{k}^{(2)} - q_{k}^{(0)} = - {(\frac{r}{n - r})}^{2} \sum_{j \neq k} \frac{1}{λ_{j} - λ_{k}} {(\frac{1}{r} \sum_{i \in I} α_{k i} α_{j i})}^{2} .

In particular, when focusing on the largest eigenvalue which plays a central role in PCA, this difference is non negative so that we have ${\tilde{λ}}_{1}^{(2)} \geq q_{1}^{(0)}$ . In a similar way, when considering the smallest eigenvalue, we get ${\tilde{λ}}_{p}^{(2)} \leq q_{p}^{(0)}$ .

We omit the derivation of $q_{k}^{(1)}$ which is tedious and leads to a formula too complicated to be interpreted. However, it should be noted that this approximation involves terms up to the order 4 in $ϵ = \frac{r}{n - r}$ .

Improved Inequalities in Error Analysis

In practice, it is of crucial importance to have bounds as close as possible to the true values of $| {\tilde{λ}}_{k} - q_{k}^{(m)} |$ and $\cos ({\tilde{ϕ}}_{k}, {\tilde{ϕ}}_{k}^{(m)})$ in Inequalities (8) and (9). This will allow for an evaluation as precise as possible of the real accuracy of the approximations without having to recompute the perturbed analysis. For this purpose, we introduce below refined inequalities in the study of covariance matrices in order to get an improved error analysis

Indeed, Inequalities (7), (8) and (9) introduced in the Subsection Error Analysis can be improved using error analysis developed in Chatelin (2012, pp. 180–184). More precisely, it is easily derived from Corollary 4.6.4 in this reference that:

12

\begin{matrix} | {\tilde{λ}}_{k} - q_{k}^{(m)} | \leq \frac{{(η_{k}^{(m)})}^{2}}{c_{k}^{(m)}} \end{matrix}

and

13

\begin{matrix} \sin ({\tilde{ϕ}}_{k}, {\tilde{ϕ}}_{k}^{(m)}) \leq \frac{η_{k}^{(m)}}{c_{k}^{(m)}} \end{matrix}

under the condition:

14

η_{k}^{(m)} < c_{k}^{(m)} .

It is obvious that (12) is more accurate than (8). A similar remark holds for (13) wich improves (9). This last point is easily checked by converting (13) into

15

\begin{matrix} c o s^{2} ({\tilde{ϕ}}_{k}, {\tilde{ϕ}}_{k}^{(m)}) \geq 1 - {(η_{k}^{(m)} / c_{k}^{(m)})}^{2} . \end{matrix}

Then letting $a = {(η_{k}^{(m)} / c_{k}^{(m)})}^{2}$ , $A = 1 - a$ and $B = {[1 - (a / 2) (1 + a)]}^{2}$ , Inequality (9) becomes $c o s^{2} ({\tilde{ϕ}}_{k}, {\tilde{ϕ}}_{k}^{(m)}) \geq B$ so that we have simply to prove that $A \geq B$ . Developing B, we get $B = A + (a^{2} / 4) (a^{2} + 2 a - 3)$ . Thus we have that $A \geq B$ if and only if the polynomial $a^{2} + 2 a - 3$ is non-positive. This is the case when a belongs to $[- 3, 1]$ . Therefore the result follows since $0 \leq a < 1$ from (14).

Furthermore, when dealing with the eigenvector associated to the largest eigenvalue, Chatelin (2012, p. 204) points out that Inequality (13) can also be refined into the following tangent based inequality:

16

\begin{matrix} \tan ({\tilde{ϕ}}_{1}, {\tilde{ϕ}}_{1}^{(m)}) \leq \frac{η_{1}^{(m)}}{c_{1}^{(m)}} \end{matrix}

since this eigenvalue is assumed to be simple. More precisely, letting α denote the angle between the two vectors ${\tilde{ϕ}}_{1}$ and ${\tilde{ϕ}}_{1}^{(m)}$ , we have $α \leq \arctan (η_{1}^{(m)} / c_{1}^{(m)}) \leq \arcsin (η_{1}^{(m)} / c_{1}^{(m)})$ since $0 \leq \arctan x \leq \arcsin x$ for all $x \in [0, 1]$ . Thus we obtain a better approximation of α when using the function arctan rather than the function arcsin showing that (16) improves (13).

Improved Value for the Parameter $c_{k}^{(m)}$

The sharpness of the bounds in Inequalities (12), (13) and (16) depends on the value of the parameter $c_{k}^{(m)}$ . The larger $c_{k}^{(m)}$ is, the sharper are these inequalities. In order to get a suitable value for this parameter, some other results of Chatelin (2012, pp. 180–181) or of Wilkinson (1988, pp. 174–176) turn out to be also of specific interest since they often improve significantly the value of the parameter $c_{k}^{(m)}$ obtained through the Courant-Fischer theorem in Bénasséni (1987). More precisely, from these references we know that there is at least one eigenvalue of $\tilde{S}$ in each of the intervals defined for $j = 1, \dots, p$ by $[b_{j}^{(m)}, B_{j}^{(m)}] = [q_{j}^{(m)} - η_{j}^{(m)}, q_{j}^{(m)} + η_{j}^{(m)}]$ which are often referred to as Krylov-Weinstein intervals. When the interval $[b_{k}^{(m)}, B_{k}^{(m)}]$ is isolated from the $p - 1$ other ones, we know that it contains precisely one eigenvalue. Then, assuming that the Rayleigh quotients satisfy $q_{1}^{(m)} > q_{2}^{(m)} > \dots > q_{p}^{(m)}$ (after having been reordered if necessary), a value for $c_{k}^{(m)}$ can be easily derived as $c_{k}^{(m)} = min (b_{k}^{(m)} - B_{k + 1}^{(m)}, b_{k - 1}^{(m)} - B_{k}^{(m)})$ if $k \in {2, \dots, p - 1}$ , $c_{1}^{(m)} = b_{1}^{(m)} - B_{2}^{(m)}$ if k = 1 and $c_{p}^{(m)} = b_{p - 1}^{(m)} - B_{p}^{(m)}$ if $k = p$ . Finally, note that these Krylov-Weinstein intervals proved to be also useful in studying the effects of some small errors in the data table itself as emphasized by the work of Bénasséni (1988, pp. 303–310)

It should be pointed out that for very close eigenvalues, giving a value to this parameter can be a real issue. However, once a value satisfying (14) is obtained, we know from (12) that, for $k = 1, \dots, p$ , the eigenvalues ${\tilde{λ}}_{k}$ lie in the intervals $[q_{k}^{(m)} - {(η_{k}^{(m)})}^{2} / c_{k}^{(m)}, q_{k}^{(m)} + {(η_{k}^{(m)})}^{2} / c_{k}^{(m)}]$ which are sharper than the Krylov-Weinstein intervals as soon as (14) holds. These new intervals can be used to obtain a larger value for the parameter $c_{k}^{(m)}$ , thus improving (12), (13) and (16). This process could be iterated, but no significative improvment is generally observed.

Finally, it was noted in (10) that we have a rank one perturbation when deleting a single observation. Several inequalities and relations for this restricted rank perturbation are given in Bénasséni (1987), Hadi and Nyquist (1993), and Wang and Nyquist (1991). The reader is also referred to more recent works by Bénasséni (2011), Cheng et al. (2014), and Ipsen and Nadler (2009), who suggest new bounds for the perturbed eigenvalues. Although Krylov-Weinstein intervals generally provide a satisfying value for the parameter $c_{k}^{(m)}$ , these recent works can also be interesting in the determination of the largest possible constant $c_{k}^{(m)}$ in order to make Inequalities (12), (13) and (16) sharper.

Numerical Study

The numerical illustration of the results is based on the soil composition data in Kendall (1975) which have already been used in several works including, among others, Bénasséni (2018), Critchley (1985), Enguix-González et al. (2005), Enguix-González et al. (2012), Tanaka (1988), Wang and Liski (1993), or Wang and Nyquist (1991) for sensitivity study of covariance based principal component analysis. The data table consists of 20 observations measured on 4 variables. We have the following four eigenvalues for the corresponding covariance matrix: $λ_{1} = 82.30827$ , $λ_{2} = 6.73891$ , $λ_{3} = 0.44783$ , $λ_{4} = 0.24552$ . In the first subsection, we study the effect of deleting each of the 20 observations on the two largest eigenvalues (which account for more than 99% of the total variation in principal component analysis) and on their corresponding eigenvectors. In the following subsection, we study the effects of deleting subsets of two observations on the largest eigenvalue. These subsets are those considered for the numerical study in Wang and Liski (1993). Finally, a short illustration of the potential effects of removing three observations is given in the last subsection.

Approximations When Deleting One Observation

Table 1 provides for each subset $I = {i}$ , the perturbed eigenvalue ${\tilde{λ}}_{1}$ , its order one and two approximations ${\tilde{λ}}_{1}^{(1)}$ and ${\tilde{λ}}_{1}^{(2)}$ , the Rayleigh quotients $q_{1}^{(0)}$ ans $q_{1}^{(1)}$ and the differences between each approximation and the true perturbed eigenvalue ${\tilde{λ}}_{1}$ . In the last two columns we find the bounds to $| {\tilde{λ}}_{1} - q_{1}^{(0)} |$ and $| {\tilde{λ}}_{1} - q_{1}^{(1)} |$ given by Inequality (12).

Table 1

Approximations and Error Analysis for the Largest Eigenvalue With r = 1

I	${\tilde{λ}}_{1}$	${\tilde{λ}}_{1}^{(1)}$	${\tilde{λ}}_{1}^{(2)}$	$q_{1}^{(0)}$	$q_{1}^{(1)}$	${\tilde{λ}}_{1} - {\tilde{λ}}_{1}^{(1)}$	${\tilde{λ}}_{1} - {\tilde{λ}}_{1}^{(2)}$	${\tilde{λ}}_{1} - q_{1}^{(0)}$	${\tilde{λ}}_{1} - q_{1}^{(1)}$	$\| {\tilde{λ}}_{1} - q_{1}^{(0)} \| \leq$	$\| {\tilde{λ}}_{1} - q_{1}^{(1)} \| \leq$
1	81.59976	81.8333	81.59769	81.58031	81.59969	-0.23355	0.00207	0.01945	$6.87 \times 10^{- 5}$	0.01988	$6.91 \times 10^{- 5}$
2	77.18681	77.64932	77.18511	77.17611	77.18667	-0.46251	0.00170	0.01070	$1.42 \times 10^{- 4}$	0.01115	$1.46 \times 10^{- 4}$
3	86.52309	86.52844	86.52306	86.52255	86.52309	-0.00535	$2.53 \times 10^{- 5}$	$5.38 \times 10^{- 4}$	$5.20 \times 10^{- 9}$	$5.40 \times 10^{- 4}$	$5.22 \times 10^{- 9}$
4	77.34975	77.64477	77.32399	77.17133	77.34799	-0.29503	0.02576	0.17842	$1.76 \times 10^{- 3}$	0.18922	$1.78 \times 10^{- 3}$
5	86.57878	86.58112	86.57875	86.57801	86.57878	-0.00234	$2.99 \times 10^{- 5}$	$7.71 \times 10^{- 4}$	$1.06 \times 10^{- 7}$	$7.74 \times 10^{- 4}$	$1.07 \times 10^{- 7}$
6	76.85039	77.33877	76.8502	76.84922	76.85037	-0.48838	$1.86 \times 10^{- 4}$	$1.17 \times 10^{- 3}$	$1.53 \times 10^{- 5}$	$1.28 \times 10^{- 3}$	$1.66 \times 10^{- 5}$
7	79.95562	80.27544	79.95368	79.94044	79.95552	-0.31981	0.00194	0.01518	$1.02 \times 10^{- 4}$	0.01544	$1.02 \times 10^{- 4}$
8	74.71505	75.29919	74.71264	74.70229	74.71477	-0.58414	0.00241	0.01276	$2.72 \times 10^{- 4}$	0.01326	$2.79 \times 10^{- 4}$
9	74.45165	75.05167	74.4498	74.44174	74.45144	-0.60002	0.00185	$9.91 \times 10^{- 3}$	$2.04 \times 10^{- 4}$	0.01073	$2.19 \times 10^{- 4}$
10	86.02449	86.05404	86.02442	86.02318	86.02449	-0.02955	$7.21 \times 10^{- 5}$	$1.30 \times 10^{- 3}$	$3.99 \times 10^{- 8}$	$1.33 \times 10^{- 3}$	$4.04 \times 10^{- 8}$
11	85.32593	85.38314	85.3254	85.31697	85.32593	-0.05721	$5.31 \times 10^{- 4}$	0.001782881	$8.62 \times 10^{- 7}$	$9.06 \times 10^{- 3}$	$8.63 \times 10^{- 7}$
12	86.58382	86.58643	86.58381	86.58359	86.58382	-0.00261	$1.06 \times 10^{- 5}$	$2.26 \times 10^{- 4}$	$2.48 \times 10^{- 9}$	$2.32 \times 10^{- 4}$	$2.53 \times 10^{- 9}$
13	84.13095	84.20755	84.12781	84.07951	84.13094	-0.07660	0.00314	0.05143	$6.97 \times 10^{- 6}$	0.05293	$6.99 \times 10^{- 6}$
14	86.52446	86.52855	86.52439	86.52267	86.52446	-0.00410	$6.56 \times 10^{- 5}$	$1.78 \times 10^{- 3}$	$3.46 \times 10^{- 7}$	$1.82 \times 10^{- 3}$	$3.50 \times 10^{- 7}$
15	82.72790	82.91768	82.72732	82.72175	82.72788	-0.18978	$5.84 \times 10^{- 4}$	$6.15 \times 10^{- 3}$	$1.37 \times 10^{- 5}$	$6.26 \times 10^{- 3}$	$1.39 \times 10^{- 5}$
16	86.63917	86.63923	86.63917	86.63917	86.63917	$- 5.48 \times 10^{- 5}$	$3.36 \times 10^{- 8}$	$6.79 \times 10^{- 7}$	$2.13 \times 10^{- 13}$	$7.21 \times 10^{- 7}$	$2.59 \times 10^{- 13}$
17	83.27902	83.42004	83.27668	83.25055	83.27899	-0.14101	0.00234	0.02847	$3.27 \times 10^{- 5}$	0.02925	$3.29 \times 10^{- 5}$
18	80.80479	81.08976	80.80396	80.79763	80.80476	-0.28497	$8.28 \times 10^{- 4}$	$7.16 \times 10^{- 3}$	$3.42 \times 10^{- 5}$	$7.57 \times 10^{- 3}$	$3.59 \times 10^{- 5}$
19	78.33933	78.75217	78.33900	78.33701	78.33931	-0.41283	$3.31 \times 10^{- 4}$	$2.33 \times 10^{- 3}$	$2.18 \times 10^{- 5}$	$2.53 \times 10^{- 3}$	$2.37 \times 10^{- 5}$
20	86.37150	86.38477	86.37149	86.37133	86.37150	-0.01328	$9.05 \times 10^{- 6}$	$1.73 \times 10^{- 4}$	$1.09 \times 10^{- 9}$	$1.86 \times 10^{- 4}$	$1.17 \times 10^{- 9}$

The first comment regarding the results in this table is that ${\tilde{λ}}_{1}^{(1)}$ is by far the less accurate approximation in all cases. In contrast $q_{1}^{(1)}$ always provides extremely sharp approximations since it deviates from ${\tilde{λ}}_{1}$ by $1.76 \times 10^{- 3}$ in the worst case (when deleting Observation 4) and that the error is only $2.13 \times 10^{- 13}$ when deleting Observation 16. It should be noted that ${\tilde{λ}}_{1}^{(2)}$ also provides fairly satisfying approximations although clearly less accurate than $q_{1}^{(1)}$ . The Rayleigh quotient $q_{1}^{(0)}$ is outperformed by ${\tilde{λ}}_{1}^{(2)}$ but remains significantly sharper than ${\tilde{λ}}_{1}^{(1)}$ . Furthermore, it is worth pointing out that ${\tilde{λ}}_{1}^{(1)}$ always overestimates the perturbed eigenvalue while the other three estimations slightly underestimate it. Note also that the results agree with inequalities ${\tilde{λ}}_{1}^{(1)} \geq q_{1}^{(0)}$ and ${\tilde{λ}}_{1}^{(2)} \geq q_{1}^{(0)}$ given in the Subsection Some Relations Between the Approximations.

Second, Inequality (12) provides bounds sufficiently close to $| {\tilde{λ}}_{1} - q_{1}^{(0)} |$ and $| {\tilde{λ}}_{1} - q_{1}^{(1)} |$ to evaluate correctly the accuracy of Rayleigh quotients as approximations to ${\tilde{λ}}_{1}$ without having to recompute the perturbed analysis.

Third, it is easily seen from (10) that the maximum value for the perturbed eigenvalue is obtained when $x_{i} = \bar{x}$ with ${\tilde{λ}}_{1} = (20 λ_{1}) / 19 = 86.64028$ . We have the highest perturbed eigenvalue when deleting Observations Number 3, 5, 10, 12, 14, 16, 20 which are fairly close to $\bar{x}$ and in these cases we get the sharper approximations to ${\tilde{λ}}_{1}$ .

Focusing now on the second largest eigenvalue, Table 2 provides results similar to those of Table 1. It turns out that ${\tilde{λ}}_{2}^{(1)}$ is the less accurate approximation to ${\tilde{λ}}_{2}$ . Except when deleting Observation 13, the Rayleigh quotient $q_{2}^{(1)}$ again provides the best approximation with a very good accuracy since in the worst case corresponding to this observation we have ${\tilde{λ}}_{2} - q_{2}^{(1)} = 7.69 \times 10^{- 4}$ . For this observation, ${\tilde{λ}}_{2}^{(2)}$ is slightly better but less accurate in all the other cases while performing fairly well in general. The Rayleigh quotient $q_{2}^{(0)}$ performs in a similar way as $q_{1}^{(0)}$ in Table 1.

Table 2

Approximations and Error Analysis for the Second Largest Eigenvalue with r = 1

I	${\tilde{λ}}_{2}$	${\tilde{λ}}_{2}^{(1)}$	${\tilde{λ}}_{2}^{(2)}$	$q_{2}^{(0)}$	$q_{2}^{(1)}$	${\tilde{λ}}_{2} - {\tilde{λ}}_{2}^{(1)}$	${\tilde{λ}}_{2} - {\tilde{λ}}_{2}^{(2)}$	${\tilde{λ}}_{2} - q_{2}^{(0)}$	${\tilde{λ}}_{2} - q_{2}^{(1)}$	$\| {\tilde{λ}}_{2} - q_{2}^{(0)} \| \leq$	$\| {\tilde{λ}}_{2} - q_{2}^{(1)} \| \leq$
1	6.80760	6.83824	6.80935	6.82480	6.80764	-0.03065	-0.00175	-0.01721	$- 4.90 \times 10^{- 5}$	0.22557	$7.06 \times 10^{- 4}$
2	7.03262	7.04231	7.03372	7.03961	7.03271	-0.00969	-0.00110	-0.00699	$- 9.02 \times 10^{- 5}$	0.08423	$1.02 \times 10^{- 3}$
3	6.73724	6.75509	6.73721	6.73727	6.73724	-0.01785	$2.51 \times 10^{- 5}$	$- 3.39 \times 10^{- 5}$	$1.36 \times 10^{- 6}$	$7.24 \times 10^{- 3}$	$1.53 \times 10^{- 6}$
4	5.66955	5.87923	5.68474	5.81532	5.66891	-0.20968	-0.01519	-0.14577	$6.36 \times 10^{- 4}$	2.52388	0.02060
5	6.10058	6.15035	6.10049	6.10071	6.10057	-0.04977	$9.24 \times 10^{- 5}$	$- 1.22 \times 10^{- 4}$	$1.36 \times 10^{- 5}$	0.01169	$1.59 \times 10^{- 5}$
6	7.09264	7.0928	7.09266	7.09276	7.09264	$- 1.56 \times 10^{- 4}$	$- 1.90 \times 10^{- 5}$	$- 1.15 \times 10^{- 4}$	$- 1.71 \times 10^{- 6}$	$1.23 \times 10^{- 3}$	$1.82 \times 10^{- 5}$
7	6.91733	6.94011	6.91920	6.93203	6.91743	-0.02277	-0.00187	-0.01470	$- 9.66 \times 10^{- 5}$	0.17207	$1.12 \times 10^{- 3}$
8	7.03111	7.04295	7.03284	7.04029	7.03131	-0.01184	-0.00173	-0.00917	$- 1.96 \times 10^{- 4}$	0.09986	$2.08 \times 10^{- 3}$
9	7.07835	7.08123	7.07877	7.080583	7.07840	-0.00288	$- 4.22 \times 10^{- 4}$	$- 2.23 \times 10^{- 3}$	$- 4.77 \times 10^{- 5}$	0.02615	$5.16 \times 10^{- 4}$
10	6.95776	6.96488	6.95776	6.95811	6.95776	-0.00712	$- 3.17 \times 10^{- 6}$	$- 3.48 \times 10^{- 4}$	$3.59 \times 10^{- 7}$	0.01375	$3.83 \times 10^{- 7}$
11	6.55595	6.59092	6.55642	6.56447	6.55595	-0.03497	$- 4.69 \times 10^{- 4}$	-0.00851	$2.69 \times 10^{- 6}$	0.11689	$1.33 \times 10^{- 5}$
12	6.85444	6.86361	6.85422	6.85150	6.85444	-0.00917	$2.21 \times 10^{- 4}$	0.00293	$1.79 \times 10^{- 6}$	0.00536	$2.30 \times 10^{- 6}$
13	5.50884	5.62489	5.50826	5.54759	5.50807	-0.11605	$5.83 \times 10^{- 4}$	-0.03875	$7.69 \times 10^{- 4}$	0.84790	$8.23 \times 10^{- 4}$
14	6.04911	6.07404	6.04375	6.02038	6.04857	-0.02493	0.00536	0.02873	$5.42 \times 10^{- 4}$	0.05689	$5.82 \times 10^{- 4}$
15	6.98255	6.99306	6.98304	6.98777	6.98256	-0.01051	$- 4.84 \times 10^{- 4}$	-0.00522	$- 1.04 \times 10^{- 5}$	0.06610	$1.34 \times 10^{- 4}$
16	7.08039	7.08097	7.08038	7.08031	7.08039	$- 5.88 \times 10^{- 4}$	$3.51 \times 10^{- 6}$	$7.55 \times 10^{- 5}$	$1.05 \times 10^{- 9}$	$7.94 \times 10^{- 5}$	$1.15 \times 10^{- 9}$
17	6.48050	6.53032	6.48147	6.50067	6.48043	-0.04981	$- 9.70 \times 10^{- 4}$	-0.02017	$7.06 \times 10^{- 5}$	0.3604578	$3.20 \times 10^{- 4}$
18	7.04966	7.05459	7.04998	7.05253	7.04967	-0.00492	$- 3.18 \times 10^{- 4}$	$- 2.87 \times 10^{- 3}$	$- 1.18 \times 10^{- 5}$	0.03878	$1.56 \times 10^{- 4}$
19	7.09124	7.09159	7.09128	7.09148	7.09125	$- 3.43 \times 10^{- 4}$	$- 3.48 \times 10^{- 5}$	$- 2.38 \times 10^{- 4}$	$- 2.45 \times 10^{- 6}$	$2.68 \times 10^{- 3}$	$2.73 \times 10^{- 5}$
20	7.08664	7.08696	7.08664	7.08662	7.08664	$- 3.23 \times 10^{- 4}$	$1.23 \times 10^{- 6}$	$2.54 \times 10^{- 5}$	$9.25 \times 10^{- 12}$	$3.36 \times 10^{- 4}$	$4.31 \times 10^{- 9}$

It should be noted that again ${\tilde{λ}}_{2}^{(1)}$ always overestimates the perturbed eigenvalue but in contrast to Table 1, the other three approximations can as well slightly underestimate or overestimate ${\tilde{λ}}_{2}$ .

Another difference with Table 1 is that bounds provided to $| {\tilde{λ}}_{2} - q_{2}^{(0)} |$ and $| {\tilde{λ}}_{2} - q_{2}^{(1)} |$ by Inequality (12) are not so close to these quantities as they were previously. For a part, this can be explained by the fact that we have a smaller value for $c_{2}^{(0)}$ and $c_{2}^{(1)}$ than for $c_{1}^{(0)}$ and $c_{1}^{(1)}$ when considering the largest eigenvalue. Indeed, for $m = 0, 1$ the gap $| {\tilde{λ}}_{3} - q_{2}^{(m)} |$ is smaller than the gap $| {\tilde{λ}}_{2} - q_{1}^{(m)} |$ . However, if we except the case of Observation 4, we know from this bound that $| {\tilde{λ}}_{2} - q_{2}^{(1)} |$ never exceeds $1.02 \times 10^{- 3}$ and this is sufficient for practical interpretation.

Results for the eigenvectors corresponding to the two largest eigenvalues are given in Table 3 which provides the sines $\sin ({\tilde{ϕ}}_{k}, ϕ_{k})$ and $\sin ({\tilde{ϕ}}_{k}, {\tilde{ϕ}}_{k}^{(1)})$ for $k = 1, 2$ and their bounds provided by Inequality (13).

Table 3

Approximations and Error Analysis for the Eigenvectors Associated to the Two Largest Eigenvalues with r = 1

I	$\sin ({\tilde{ϕ}}_{1}, ϕ_{1})$	$\sin ({\tilde{ϕ}}_{1}, {\tilde{ϕ}}_{1}^{(1)})$	$\sin ({\tilde{ϕ}}_{1}, ϕ_{1}) \leq$	$\sin ({\tilde{ϕ}}_{1}, {\tilde{ϕ}}_{1}^{(1)}) \leq$	$\sin ({\tilde{ϕ}}_{2}, ϕ_{2})$	$\sin ({\tilde{ϕ}}_{2}, {\tilde{ϕ}}_{2}^{(1)})$	$\sin ({\tilde{ϕ}}_{2}, ϕ_{2}) \leq$	$\sin ({\tilde{ϕ}}_{2}, {\tilde{ϕ}}_{2}^{(1)}) \leq$
1	0.01609	$9.56 \times 10^{- 4}$	0.01644	$9.62 \times 10^{- 4}$	0.01988	0.00153	0.19303	0.01056
2	0.01218	$1.40 \times 10^{- 3}$	0.01267	$1.44 \times 10^{- 3}$	0.01213	0.00142	0.11768	0.01250
3	0.00259	$8.07 \times 10^{- 6}$	0.00261	$8.09 \times 10^{- 6}$	0.00912	$4.62 \times 10^{- 4}$	0.03408	$4.94 \times 10^{- 4}$
4	0.04974	$4.94 \times 10^{- 3}$	0.05280	$4.99 \times 10^{- 3}$	0.08114	0.01954	0.72248	0.06298
5	0.00309	$3.64 \times 10^{- 5}$	0.00311	$3.64 \times 10^{- 5}$	0.01097	0.00154	0.04563	0.00168
6	0.00393	$4.50 \times 10^{- 4}$	0.00428	$4.88 \times - 10^{- 3}$	0.01491	0.00125	0.16533	0.01317
8	0.01356	$1.98 \times 10^{- 3}$	0.01408	$2.03 \times 10^{- 3}$	0.01310	0.00194	0.12507	0.01782
9	0.01169	$1.68 \times 10^{- 3}$	0.01266	$1.80 \times 10^{- 3}$	0.00725	0.00107	0.06553	0.00883
10	0.00403	$2.23 \times 10^{- 5}$	0.00411	$2.26 \times 10^{- 5}$	0.01092	$2.32 \times 10^{- 4}$	0.04628	$2.43 \times 10^{- 4}$
11	0.01066	$1.05 \times 10^{- 4}$	0.01078	$1.05 \times 10^{- 4}$	0.01321	$7.59 \times 10^{- 4}$	0.13947	0.00148
12	0.00167	$5.52 \times 10^{- 6}$	0.00171	$5.63 \times 10^{- 6}$	0.02195	$5.32 \times 10^{- 4}$	0.02915	$5.96 \times 10^{- 4}$
13	0.02555	$2.98 \times 10^{- 4}$	0.02630	$2.98 \times 10^{- 4}$	0.05398	0.01230	0.42319	0.01285
14	0.00468	$6.52 \times 10^{- 5}$	0.00477	$6.60 \times 10^{- 5}$	0.07252	0.00970	0.10386	0.01011
15	0.00897	$4.24 \times 10^{- 4}$	0.00913	$4.28 \times 10^{- 4}$	0.01053	$5.52 \times 10^{- 4}$	0.10247	0.00454
16	$8.97 \times 10^{- 5}$	$5.16 \times 10^{- 8}$	$9.52 \times 10^{- 5}$	$5.70 \times 10^{- 8}$	0.00335	$1.25 \times 10^{- 5}$	0.00347	$1.32 \times 10^{- 5}$
17	0.01918	$6.50 \times 10^{- 4}$	0.01971	$6.55 \times 10^{- 4}$	0.03624	0.00383	0.25215	0.00729
18	0.00962	$6.66 \times 10^{- 4}$	0.01017	$6.97 \times 10^{- 4}$	0.00999	$6.86 \times 10^{- 4}$	0.07882	0.00486
19	0.00549	$5.31 \times 10^{- 4}$	0.00597	$5.76 \times 10^{- 4}$	0.00213	$2.16 \times 10^{- 4}$	0.02021	0.00203
20	0.00142	$3.58 \times 10^{- 6}$	0.00153	$3.84 \times 10^{- 6}$	0.00274	$7.34 \times 10^{- 6}$	0.00715	$2.55 \times 10^{- 5}$

First, we consider the eigenvector associated to the largest eigenvalue. It could be noted that the maximum value of the sine bewteen the unperturbed and perturbed eigenvector is obtained when deleting Observation 4. This value corresponds to an angle of ${2.85}^{\circ}$ . In this case, as in all the other ones, the order one approximation performs fairly well since its sine with the perturbed eigenvector is equal to only 0.00494 which corresponds to an angle of ${0.28}^{\circ}$ . Furthermore bounds provided by Inequality (13) are always extremely close to the exact value of $\sin ({\tilde{ϕ}}_{1}, {\tilde{ϕ}}_{1}^{(1)})$ .

Focusing now on the eigenvector corresponding to the second largest eigenvalue, the maximum of $\sin ({\tilde{ϕ}}_{2}, ϕ_{2})$ is again obtained when deleting Observation 4 with the value of 0.08114. Even in this case, the order one approximation is fairly close to the perturbed eigenvector since $\sin ({\tilde{ϕ}}_{2}, {\tilde{ϕ}}_{2}^{(1)}) = 0.01954$ . In contrast to the previous eigenvector, it is worth pointing out that bounds provided by Inequality (13) are not always sufficiently close to the true values of the sine to give an exact account of the accuracy for these approximations.

Finally, since the angles between the eigenvectors studied in the table are always very close to zero, we do not provide the tangent of these angles which only deviates from the sine by an extremely small amount.

Approximations When Deleting Subsets of Two Observations

Now, we study approximations to the perturbed largest eigenvalue ant its corresponding eigenvector when deleting the subsets of two observations considered for the numerical illustration in Wang and Liski (1993). Results similar to those of the previous subsection are provided in Tables 4 and 5.

Table 4

Approximations and Error Analysis for the Largest Eigenvalue with r = 2

I	${\tilde{λ}}_{1}$	${\tilde{λ}}_{1}^{(1)}$	${\tilde{λ}}_{1}^{(2)}$	$q_{1}^{(0)}$	$q_{1}^{(1)}$	${\tilde{λ}}_{1} - {\tilde{λ}}_{1}^{(1)}$	${\tilde{λ}}_{1} - {\tilde{λ}}_{1}^{(2)}$	${\tilde{λ}}_{1} - q_{1}^{(0)}$	${\tilde{λ}}_{1} - q_{1}^{(1)}$	$\| {\tilde{λ}}_{1} - q_{1}^{(0)} \| \leq$	$\| {\tilde{λ}}_{1} - q_{1}^{(1)} \| \leq$
${8, 9}$	64.59310	67.25005	64.58126	64.56084	64.59026	-2.65695	0.01184	0.03226	0.00284	0.03616	0.00316
${4, 9}$	67.51718	69.72595	67.4481	67.32138	67.50241	-2.20877	0.06908	0.19580	0.01476	0.20842	0.01501
${4, 8}$	67.98219	69.98722	67.85527	67.61002	67.95618	-2.00502	0.12692	0.37218	0.02602	0.40664	0.02660
${6, 9}$	69.40391	69.40294	69.40242	69.39558	69.40366	$9.72 \times 10^{- 4}$	0.00149	0.00834	$2.52 \times 10^{- 4}$	0.00892	$2.66 \times 10^{- 4}$
${2, 9}$	69.74526	69.73074	69.74115	69.72109	69.74455	0.01451	0.00410	0.02417	$7.04 \times 10^{- 4}$	0.02512	$7.19 \times 10^{- 4}$
${2, 8}$	69.98911	69.99202	69.98832	69.98402	69.98899	-0.00291	$7.86 \times 10^{- 4}$	0.00509	$1.21 \times 10^{- 4}$	0.00565	$1.33 \times 10^{- 4}$
${2, 6}$	70.01929	72.14491	70.01294	69.99965	70.01808	-2.12561	0.00636	0.01964	0.00121	0.02072	0.00126
${3, 8}$	78.83587	79.36442	78.83242	78.82489	78.83542	-0.52854	0.00345	0.01099	$4.54 \times 10^{- 4}$	0.01144	$4.59 \times 10^{- 4}$
${2, 4}$	72.60356	72.46791	72.58774	72.46791	72.60172	0.13565	0.01582	0.13565	0.00184	0.14414	0.00188
${4, 7}$	75.53263	75.23992	75.50596	75.22661	75.53053	0.29271	0.02667	0.30602	0.00210	0.33028	0.00212

Table 5

Approximations and Error Analysis for the Eigenvector Associated to the Largest Eigenvalue with r = 2

I	$\sin ({\tilde{ϕ}}_{1}, ϕ_{1})$	$\sin ({\tilde{ϕ}}_{1}, {\tilde{ϕ}}_{1}^{(1)})$	$\sin ({\tilde{ϕ}}_{1}, ϕ_{1}) \leq$	$\sin ({\tilde{ϕ}}_{1}, {\tilde{ϕ}}_{1}^{(1)}) \leq$	I	$\sin ({\tilde{ϕ}}_{1}, ϕ_{1})$	$\sin ({\tilde{ϕ}}_{1}, {\tilde{ϕ}}_{1}^{(1)})$	$\sin ({\tilde{ϕ}}_{1}, ϕ_{1}) \leq$	$\sin ({\tilde{ϕ}}_{1}, {\tilde{ϕ}}_{1}^{(1)}) \leq$
${8, 9}$	0.02254	0.00667	0.02525	0.00745	${2, 8}$	0.00856	0.00132	0.00951	0.00146
${4, 9}$	0.05637	0.01548	0.06010	0.01574	${2, 6}$	0.01740	0.00432	0.01833	0.00449
${4, 8}$	0.07724	0.02042	0.08464	0.02089	${3, 8}$	0.01220	0.00251	0.01269	0.00253
${6, 9}$	0.01128	0.00197	0.01205	0.00207	${2, 4}$	0.04477	0.00522	0.04759	0.00532
${2, 9}$	0.01950	0.00333	0.02026	0.00340	${4, 7}$	0.06609	0.00548	0.07148	0.00553

We note that deleting subsets of two observations can result in larger variations of the eigenvalue of interest than when deleting single observations. Indeed, the perturbed eigenvalue is lower than 70 in the first six lines of Table 4.

Furthermore for all the subsets I studied in this table we have a decrease of the eigenvalue, while we note that this eigenvalue is increased in several cases in Table 1. Despite these significant variations of the eigenvalue, we see that the Rayleigh quotient $q_{1}^{(1)}$ always provides a very accurate approximation to ${\tilde{λ}}_{1}$ since the maximum gap $| {\tilde{λ}}_{1} - q_{1}^{(1)} | = 0.02602$ observed for $I = {4, 8}$ remains fairly moderate. It should also be noted that $q_{1}^{(1)}$ always performs better than ${\tilde{λ}}_{1}^{(2)}$ . This point is fairly well illustrated considering again the case of $I = {4, 8}$ for which we have ${\tilde{λ}}_{1} - {\tilde{λ}}_{1}^{(2)} = 0.12692$ . The Rayleigh quotient $q_{1}^{(0)}$ provides less accurate approximations than ${\tilde{λ}}_{1}^{(2)}$ but should generally be preferred to ${\tilde{λ}}_{1}^{(1)}$ if we except some cases. Finally, it is worth pointing out that bounds to $| {\tilde{λ}}_{1} - q_{1}^{(0)} |$ and $| {\tilde{λ}}_{1} - q_{1}^{(1)} |$ provided by Inequality (12) are always very close to the true values of these two differences thus avoiding to recompute the perturbed analysis.

Turning now to the sine values in Table 5 , we note the largest variations of the eigenvector when deleting the subsets $I = {4, 7}$ , $I = {4, 8}$ and $I = {4, 9}$ . However, the order one approximation ${\tilde{ϕ}}_{1}^{(1)}$ remains fairly satisfying in all the cases since the maximum value of $\sin ({\tilde{ϕ}}_{1}, {\tilde{ϕ}}_{1}^{(1)})$ obtained when deleting the subset $I = {4, 8}$ does not exceed 0.02042 which corresponds to an angle of only ${1.17}^{\circ}$ .

Approximations When Deleting Subsets of Three Observations

While the accuracy of $q_{1}^{(1)}$ is fairly good for the removal of two observations despite the small sample size, one might think that results would probably be much worse when removing three observations and that their performance would deteriorate very quickly as $ϵ = \frac{r}{n - r}$ increases. In order to investigate this point, we decided to perform a very brief study of the performance of the approximations for the largest eigenvalue when removing some sets of three observations. Note that in this case we have: $ϵ = \frac{3}{17} = 0.1765$ instead of $ϵ = \frac{2}{18} = 0.1111$ when deleting only two observations.

First, when removing the three Observations 3, 4 and 7, the largest eigenvalue initially equal to $λ_{1} = 82.30827$ is moderately decreased to ${\tilde{λ}}_{1} = 79.83362$ . The approximations to ${\tilde{λ}}_{1}$ are then the following: ${\tilde{λ}}_{1}^{(1)} = 79.5408$ , ${\tilde{λ}}_{1}^{(2)} = 79.82602$ , $q_{1}^{(0)} = 79.53947$ and $q_{1}^{(1)} = 79.83346$ . We note the excellent accuracy of $q_{1}^{(1)}$ since the error ${\tilde{λ}}_{1} - q_{1}^{(1)} = 0.00016$ is negligible.

Second, when removing the three Observations 2, 4 and 6, the largest eigenvalue is now significantly decreased to ${\tilde{λ}}_{1} = 65.86825$ . The approximations to ${\tilde{λ}}_{1}$ are then the following: ${\tilde{λ}}_{1}^{(1)} = 66.33492$ , ${\tilde{λ}}_{1}^{(2)} = 65.85841$ , $q_{1}^{(0)} = 65.7237$ and $q_{1}^{(1)} = 65.86529$ . We still have a satisfying accuracy of $q_{1}^{(1)}$ since the error ${\tilde{λ}}_{1} - q_{1}^{(1)}$ is equal to 0.00295.

Finally, when removing the very influential set ${4, 8, 9}$ , the largest eigenvalue is drastically decreased to ${\tilde{λ}}_{1} = 55.36930$ and we get the following approximations: ${\tilde{λ}}_{1}^{(1)} = 61, 15213$ , ${\tilde{λ}}_{1}^{(2)} = 55.10147$ , $q_{1}^{(0)} = 54.87525$ and $q_{1}^{(1)} = 55.26709$ . Therefore while the largest eigenvalue is decreased by 26.939 from its initial value, the error ${\tilde{λ}}_{1} - q_{1}^{(1)} = 0.10221$ remains fairly moderate while obviously larger than when considering removal of only two observations.

However, it would be necessary to perform additional studies with other data tables to get more reliable information on the relation between the value of ϵ and the accuracy of the approximations.

Concluding Remarks

The previous numerical study provides some indications on the sharpness of the various approximations in the paper when considering a particular data set. As a result, it may be useful to provide practitioners with some guidance on the choice of approximations for perturbed covariance matrices.

First, when focusing on eigenvalues, it should be noted that, in this example, approximations provided by ${\tilde{λ}}_{k}^{(1)}$ should be avoided as well as the Rayleigh quotients $q_{k}^{(0)}$ which are not sufficiently accurate.

In contrast, Rayleigh quotients $q_{k}^{(1)}$ (based on the perturbed matrix $\tilde{S}$ and the approximations of order one ${\tilde{ϕ}}_{k}^{(1)}$ ) seem to always provide reliable approximations to ${\tilde{λ}}_{k}$ . They must be preferred to the approximation of order two ${\tilde{λ}}_{k}^{(2)}$ in the study of this data table. Furthermore, their accuracy can be evaluated in a precise way by Inequality (12) if the eigenvalue of interest is not too close to the other eigenvalues, as emphasized by results in Table 1 when considering the largest eigenvalue. This is a definite advantage over other approximations.

It should also be noted that these Rayleigh quotients perform fairly well even for values of $ϵ = \frac{r}{n - r}$ which are not necessarily very close to zero. This point is made clear in the first lines of Table 4 where $ϵ = \frac{2}{18} = 0.11$ and the eigenvalue ${\tilde{λ}}_{1}$ is significantly decreased through the perturbation. The Subsection Approximations When Deleting Subsets of Three Observations, briefly dealing with the removal of three observations, tends to indicate that these Rayleigh quotients should still perform in a satisfying way even for higher values of ϵ. However, more in-depth studies would be necessary to get reliable conclusions on this point.

Second, when considering eigenvectors, we note a satisfying accuracy of approximations of order one ${\tilde{ϕ}}_{k}^{(1)}$ . Again, when the eigenvalue corresponding to the eigenvector of interest is sufficiently distant from the other ones, we have a correct evaluation of this accuracy by Inequality (13).

Funding

The authors have no funding to report.

Acknowledgments

The authors are grateful to the reviewers for their careful reading of the paper and their comments.

Competing Interests

The authors have declared that no competing interests exist.

References

Bénasséni, J. (1987). Perturbation des poids des unités statistiques et approximation en analyse en composantes principales. RAIRO—Recherche opérationnelle/Operations Research, 21(2), 175-198.
Bénasséni, J. (1988). Sensitivity of principal component to data perturbation (pp. 303–310). In E. Diday (Ed.), Data analysis and informatics, V. Elsevier.
Bénasséni, J. (2011). Lower bounds for the largest eigenvalue of a symmetric matrix under perturbations of rank one. Linear and Multilinear Algebra, 59(5), 565-569. https://doi.org/10.1080/03081081003709827
Bénasséni, J. (2018). A correction of approximations used in sensitivity study of principal component analysis. Computational Statistics, 33, 1939-1955. https://doi.org/10.1007/s00180-017-0790-7
Chatelin, F. (2012). Eigenvalues of matrices. Society for Industrial and Applied Mathematics.
Cheng, G., Song, Z., Yang, J., & Si, J. (2014). The bounds of the eigenvalues for rank-one modification of hermitian matrix. Numerical Linear Algebra with Applications, 21(1), 98-107. https://doi.org/10.1002/nla.1867
Critchley, F. (1985). Influence in principal component analysis. Biometrika, 72(3), 627-636. https://doi.org/10.1093/biomet/72.3.627
Enguix-González, A., Moreno-Rebollo, J. L., & Muñoz Pichardo, J. M. (2015). A better approximation of moments of the eigenvalues and eigenvectors of the sample covariance matrix. Journal of Multivariate Analysis, 142, 133-143. https://doi.org/10.1016/j.jmva.2015.08.002
Enguix-González, A., Muñoz Pichardo, J. M., Moreno-Rebollo, J. L., & Barranco-Chamorro, I. (2012). Using conditional bias in principal component analysis for the evaluation of joint influence on the eigenvalues of the covariance matrix. Applied Mathematics and Computation, 218(17), 8937-8950. https://doi.org/10.1016/j.amc.2012.02.054
Enguix-González, A., Muñoz Pichardo, J. M., Moreno-Rebollo, J. L., & Pino-Mejías, R. (2005). Influence analysis in principal component analysis through power-series expansions. Communications in Statistics-Theory and Methods, 34(9–10), 2025-2046. https://doi.org/10.1080/03610920500203505
Hadi, A., & Nyquist, H. (1993). Further theoretical results and a comparison between two methods for approximating eigenvalues of perturbed covariance matrices. Statistics and Computing, 3, 113-123. https://doi.org/10.1007/BF00147774
Ipsen, I. C. F., & Nadler, B. (2009). Refined perturbation bounds for eigenvalues of hermitian and non-hermitian matrices. SIAM Journal on Matrix Analysis and Applications, 31(1), 40-53. https://doi.org/10.1137/070682745
Jolliffe, I. T. (2002). Principal component analysis (2^nd ed.). Springer-Verlag.
Kendall, M. G. (1975). Multivariate analysis. Griffin.
Masioti, M., Li-Wai-Suen, C., Prendergast, L. A., & Shaker, A. (2023). A note on switching eigenvalues under small perturbations. Communications in Statistics—Theory and Methods, 53(20), 7311-7325. https://doi.org/10.1080/03610926.2023.2263114
Pack, P., Jolliffe, I. T., & Morgan, B. J. T. (1988). Influential observations in principal component analysis: A case-study. Journal of Applied Statistics, 15(1), 39-52. https://doi.org/10.1080/02664768800000004
Prendergast, L. A. (2008). A note on sensitivity of principal component subspaces and the efficient detection of influential observations in high dimensions. Electronic Journal of Statistics, 2, 454-467. https://doi.org/10.1214/08-EJS201
Prendergast, L. A., & Li-Wai-Suen, C. (2011). A new and practical influence measure for subsets of covariance matrix sample principal components with applications to high dimensional datasets. Computational Statistics and Data Analysis, 55(1), 752-764. https://doi.org/10.1016/j.csda.2010.06.022
Sibson, R. (1979). Studies in robustness of multidimensional scaling: perturbational analysis of classical scaling. Journal of the Royal Statistical Society: Series B (Methodological), 41(2), 217-229. https://doi.org/10.1111/j.2517-6161.1979.tb01076.x
Tanaka, Y. (1988). Sensitivity analysis in principal component analysis: Influence on the subspace spanned by principal components. Communications in Statistics-Theory and Methods, 17(9), 3157-3175. https://doi.org/10.1080/03610928808829796
Wang, S.-G., & Liski, E. P. (1993). Effects of observations on the eigensystem of a sample covariance matrix. Journal of Statistical Planning and Inference, 36, 215-226.
Wang, S.-G., & Nyquist, H. (1991). Effects on the eigenstructure of a data matrix when deleting an observation. Computational Statistics and Data Analysis, 11(2), 179-188. https://doi.org/10.1016/0167-9473(91)90068-D
Wilkinson, J. H. (1988). The algebraic eigenvalue problem. Oxford University Press.