This article was downloaded by: [Memorial University of Newfoundland] On: 25 January 2015, At: 12:23 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Multivariate Behavioral Research Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hmbr20
Convergence Properties of Certain Minres Algorithms Jos M.F. ten Berge & Frits E. Zegers Published online: 10 Jun 2010.
To cite this article: Jos M.F. ten Berge & Frits E. Zegers (1990) Convergence Properties of Certain Minres Algorithms, Multivariate Behavioral Research, 25:4, 421-425, DOI: 10.1207/s15327906mbr2504_1 To link to this article: http://dx.doi.org/10.1207/s15327906mbr2504_1
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.
Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
bfultivariate Behavioral Research, 25 (4), 421-425 Copyright O 1990, Lawrence Erlbaum Associates, Inc.
Convergence Properties of Certain Minres .Algorithms Jos M.F. ten Berge and Frits E. Zegers Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
University of Groningen
Levin (1988) has challenged the convergence properties of the Harrnan and Jones (1966) method of Minres factor analysis. Levin claimed that convergence of the Harman and Jones method is not guaranteed and that a modified version of this method, with proven convergence, is to be preferred. In the present article it is shown that both claims are invali~d. Monotone convergence of the Harman and Jones method is guaranteed whereas the modified version, proposed by Levin, may converge to an incorrect solution. Levin has also claimed that the rankone version of the Harman and Jones method, as implemented in a method by Z,egers and ten Berge (1983) lacks a valid convergence proof, and that a method suggested by Comrey and Ahumada (1964, 1965) should be used instead. It is shown that these claims, too, should be reversed.
Harman and Jones (1966) have developed the Minres method in order to find, for a given n )< n correlation matrix R, an n x r matrix A (r < n) such that the sum of squares of the off-diagonal elements of (R - AA') is minimized. Specifically, Minres seeks to minimize the function (1)
f(A) = IF - I - AA' + Diag(AA')1I2
for a specified rank r of A. The Minres method proceeds by starting with arbitrary initial values for A, and replacing the rows of A iteratively, in cycles of n steps, until a full cycle fails to decreasef(A) according to some elonvergence criterion. The replacement of any row i of A (i = 1 ,..., n) is based on the observation that f(A) can be partitioned in a part that depends on row i plus a constant term that does not vary with row i. Specifically, let a'*be the current ith row of A, let ribe the ith column of (R - I), and let Acdenote the current A, with the elements of row i replaced by zeroes. Then we can write
where ci is independent of a, and gi(a,) contains those terms off(A)~that do vary with aL. Clearly, gi(al) is minimized by taking
OCTOBER 1990
Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
J. ten Betge and F. Zegers
i = 1, ..., n. At this point it is easy to explain the difference between Minres as suggested by Harman and Jones (1966) and a modified method, as advocated by Levin (1988). Harman and Jones suggest evaluating Equation 3 for a value of i, and updating row i ofA at once, before turning to row i + 1(single updating). Levin, on the other hand, suggests evaluating Equation 3 for i = 1, ..., n, from the same A, and updating the n rows of A afterwards, before turning to a next cycle of n steps (simultaneous updating). Levin (1988) has claimed that single updating is incorrect, and that simultaneous updating is necessary in order to have a convergence proof. Similarly, Levin has claimed that single updating in the r = 1case is incorrect, thus challenging a procedure suggested by Zegers and ten Berge (1983), and that simultaneous updating, as implemented in a procedure by Comrey and Ahumada (1964, 1965) is the correct method. The purpose of the present article is to show that, contrary to Levin's (1988) claims, single updating is guaranteed to converge monotonely in terms off(A), whereas simultaneous updating is problematic. That is, simultaneous updating can also be shown to converge, which is the major accomplishment of Levin's article, but not necessarily to a minimum of KA). Convergence of single updating Levin (1988, p. 414) attempted to prove non-convergence for single updating in the following way. When a,, say, is updated by Equation 3, we minimize g,(al). Next, if a, is updated, we minimize g2(a2),at the cost of possibly increasing gl(a,). It is thus seen that there is no monotony for every term g,(az), i = 1, ..., n. From this, Levin (1988) infers that f(A) is not necessarily monotonely decreasing with direct updating. This inference, however, is fully unwarranted. Each row-update minimizesflA) as a function of a subset of the parameters in A. If this subset is updated at once, before turning to a next subset, then the monotone reduction of fTA) is guaranteed by the elementary principles of alternating least squares. Therefore, Levin's claim of non-convergence of direct updating is to be ignored. Levin has only shown that a convergence proof based on monotony of the gl(ai) functions is doomed to fail. This, however, does not imply that all other convergence proofs must fail as well. Convergence for direct updating is guaranteed because of the monotony of the function f(A). The Minres method of Harman and Jones is a monotonely convergent alternating least squares method, in spite of Levin's claim.
422
MULTIVARIATE BEHAVIORAL RESEARCH
J. ten Berge anld F. Zegers
Convergence properties of simultaneous updating
Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
After rejecting single updating on account of its alledged non-monotony, Levin (1988, p. 415) suggested simultaneous updating as a convergent alternative. The formal properties of simultaneous updating can most elegantly 1bedescribed in terms of Levin's Equation 6, that is, in terms of the function (4>
h(A,B) = IIR - I - AB' + Diag(AB')1I2.
For a fixed A, updating its rows simultaneously according to Equation 3 is tantamount to minimizing h(A,B) as a function of B. Conversely, wh~enB is kept fixed and its rows are updated simultaneously by Equation 3, then h(A,B) is minimized as a function ofA. It follows that simultaneous updating; minimizes h(A,B) by alternating least squares, thus monotonely reducing hcl,B). In this way Levin showed that his method is bound to converge. It is of vital importance, however, to note that h(A,B) is not the function to be minimized in minimum residual factor analysis. That is, minimizing h(A,B) of Equation 4 is not equivalent to minimizing f(A) of Equation 1, because the constraintA = B has not been imposed on h(A,B). Nevertheless, simultaneous updating could be salvaged as a Minres method if it could be shown to converge to a limiting pair A,B for which AB' is Gramian (not just symmetric). Levin (1988) has shown that, if [R - I+ Diag(AB1)]is Gramian at the global minimum of h(A,B), then so is AB'. In fact, this result can be shown to hold for every stationary point of h(A,B). The major problem with Levin's method is, that there is no guarantee whatsoever that AB' will be Gramian after convergence. A simple example can demonstrate this. Consider the 4 x 4 correla~tionmatrix
with eigenvalues 2.625, -625, .625, and .125. For r = 2 and
and
OCTOBER 1990
Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
J. ten Berge and F. Zegers
we have a symmetric matrixAB1for which h(A7B)attains the global minimum of zero, yet AB' is not Gramian, with eigenvalues 2,0,O and -.5. Levin (p. 416, footnote) attempts to ignore the negative eigenvalue of -.5 as a case of overfactoring, but this is an adhoc definition of overfactoring rather than an explanation. Instead, the present authors interpret the negative eigenvaiue as a predictable consequence of minimizing the wrong function 4 instead of 1. It can be concluded that simultaneous updating as suggested by Levin is problematic because it converges to a solution for an irrelevant function, from which a solution for the relevant function cannot always be generated. Although Levin's method may work in certain practical applications, it is safer to rely on the Harman and Jones (1966) method, which works in every case. Direct and simultaneous updating in column-wise approaches Comrey and Ahumada (1964,1965) and Zegers and ten Berge (1983) have developed Minres procedures based on column-wise updating of A. The Comrey and Ahumada procedure is based on simultaneous updating as in Levin's (1988) method, whereas the Zegers and ten Berge method is based on direct updating as in the Harman and Jones (1966) method. Levin (1988, p. 416) concluded, in line with his previous claims, that the Zegers and ten Berge method is incorrect whereas the Comrey and Ahumada method is correct. It is clear from the previous sections that these conclusions have to be reversed. That is, the Zegers and ten Berge method monotonely reduces the function f(A) of Equation 1, whereas the Comrey and Ahumada method minimizes an irrelevant function. Again, a small example is highly instructive. Consider the 3 x 3 correlation matrix.
for which the function RA) cannot have a zero value if r = 1. However, the Comrey and Ahumada method minimizes h(A,B) and for r = 1it attains a global minimum of zero for
424
MULTIVARIATE BEHAVIORAL RESEARCH
J. ten Berge and F. Zegers
Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
and b = -a, thus yielding a matrix ab' that is non-Gramian. Ignoring this result as a case of overfactoring would not make sense. Instead, the non-Gramian solution is to be interpreted as a penalty for minimizing the wrong function. Discussion So far the superiority of direct updating over simultaneous updating has been discussed without taking Heywood cases into consideration. Levin (1988,~.416) refers to Harman (1976) and Mulaik (1972) for the handling of Heywood cases. However, these methods, also discussed in Zegers and ten Berge (1983), apply to direct updating. It is not at all clear how He:ywood cases are to be treated with simultaneous updating methods as Levin's. This is another reason for preferring direct updating (Harman &Jones; Zegers & ten Berge) over simultaneous updating (Levin; Comrey & Ahumada). References Comrey, A. L., & Ahumada, A. (1964). An improved procedure and program for minimum residual analysis. Psychological Reports, 15, 91-96. Comrey, A. L., & Ahumada, A. (1965). Note and Fortran IV program for m~inimumresidual factor analysis. Psychological Reports, 17, 446. Harman, H. H. (1976). Modern factor analysis (3rd ed., rev.). Chicago: University of Chicago Press. Harman, H. H., & Jones, W. H. (1966). Factor analysis by minimizing residuals (Minres). Psychometrika, 31,351-369. Levin, J. (1988). Note on convergence of Minres. Multivariate BehavioralResearch, 23,413417. Mulaik, S. A. (1972). The foundations of factor analysis. New-Yorlr: McGriaw-Hill. Zegers, F. E., & ten Berge, J. M. F. (1983). A fast and simple computational method of minimum residual factor analysis. Multivariate Behavioral Research, ICI,331-340.
OCTOBER 1990
Multivariate Behavioral Research Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hmbr20
Convergence Properties of Certain Minres Algorithms Jos M.F. ten Berge & Frits E. Zegers Published online: 10 Jun 2010.
To cite this article: Jos M.F. ten Berge & Frits E. Zegers (1990) Convergence Properties of Certain Minres Algorithms, Multivariate Behavioral Research, 25:4, 421-425, DOI: 10.1207/s15327906mbr2504_1 To link to this article: http://dx.doi.org/10.1207/s15327906mbr2504_1
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.
Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
bfultivariate Behavioral Research, 25 (4), 421-425 Copyright O 1990, Lawrence Erlbaum Associates, Inc.
Convergence Properties of Certain Minres .Algorithms Jos M.F. ten Berge and Frits E. Zegers Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
University of Groningen
Levin (1988) has challenged the convergence properties of the Harrnan and Jones (1966) method of Minres factor analysis. Levin claimed that convergence of the Harman and Jones method is not guaranteed and that a modified version of this method, with proven convergence, is to be preferred. In the present article it is shown that both claims are invali~d. Monotone convergence of the Harman and Jones method is guaranteed whereas the modified version, proposed by Levin, may converge to an incorrect solution. Levin has also claimed that the rankone version of the Harman and Jones method, as implemented in a method by Z,egers and ten Berge (1983) lacks a valid convergence proof, and that a method suggested by Comrey and Ahumada (1964, 1965) should be used instead. It is shown that these claims, too, should be reversed.
Harman and Jones (1966) have developed the Minres method in order to find, for a given n )< n correlation matrix R, an n x r matrix A (r < n) such that the sum of squares of the off-diagonal elements of (R - AA') is minimized. Specifically, Minres seeks to minimize the function (1)
f(A) = IF - I - AA' + Diag(AA')1I2
for a specified rank r of A. The Minres method proceeds by starting with arbitrary initial values for A, and replacing the rows of A iteratively, in cycles of n steps, until a full cycle fails to decreasef(A) according to some elonvergence criterion. The replacement of any row i of A (i = 1 ,..., n) is based on the observation that f(A) can be partitioned in a part that depends on row i plus a constant term that does not vary with row i. Specifically, let a'*be the current ith row of A, let ribe the ith column of (R - I), and let Acdenote the current A, with the elements of row i replaced by zeroes. Then we can write
where ci is independent of a, and gi(a,) contains those terms off(A)~that do vary with aL. Clearly, gi(al) is minimized by taking
OCTOBER 1990
Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
J. ten Betge and F. Zegers
i = 1, ..., n. At this point it is easy to explain the difference between Minres as suggested by Harman and Jones (1966) and a modified method, as advocated by Levin (1988). Harman and Jones suggest evaluating Equation 3 for a value of i, and updating row i ofA at once, before turning to row i + 1(single updating). Levin, on the other hand, suggests evaluating Equation 3 for i = 1, ..., n, from the same A, and updating the n rows of A afterwards, before turning to a next cycle of n steps (simultaneous updating). Levin (1988) has claimed that single updating is incorrect, and that simultaneous updating is necessary in order to have a convergence proof. Similarly, Levin has claimed that single updating in the r = 1case is incorrect, thus challenging a procedure suggested by Zegers and ten Berge (1983), and that simultaneous updating, as implemented in a procedure by Comrey and Ahumada (1964, 1965) is the correct method. The purpose of the present article is to show that, contrary to Levin's (1988) claims, single updating is guaranteed to converge monotonely in terms off(A), whereas simultaneous updating is problematic. That is, simultaneous updating can also be shown to converge, which is the major accomplishment of Levin's article, but not necessarily to a minimum of KA). Convergence of single updating Levin (1988, p. 414) attempted to prove non-convergence for single updating in the following way. When a,, say, is updated by Equation 3, we minimize g,(al). Next, if a, is updated, we minimize g2(a2),at the cost of possibly increasing gl(a,). It is thus seen that there is no monotony for every term g,(az), i = 1, ..., n. From this, Levin (1988) infers that f(A) is not necessarily monotonely decreasing with direct updating. This inference, however, is fully unwarranted. Each row-update minimizesflA) as a function of a subset of the parameters in A. If this subset is updated at once, before turning to a next subset, then the monotone reduction of fTA) is guaranteed by the elementary principles of alternating least squares. Therefore, Levin's claim of non-convergence of direct updating is to be ignored. Levin has only shown that a convergence proof based on monotony of the gl(ai) functions is doomed to fail. This, however, does not imply that all other convergence proofs must fail as well. Convergence for direct updating is guaranteed because of the monotony of the function f(A). The Minres method of Harman and Jones is a monotonely convergent alternating least squares method, in spite of Levin's claim.
422
MULTIVARIATE BEHAVIORAL RESEARCH
J. ten Berge anld F. Zegers
Convergence properties of simultaneous updating
Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
After rejecting single updating on account of its alledged non-monotony, Levin (1988, p. 415) suggested simultaneous updating as a convergent alternative. The formal properties of simultaneous updating can most elegantly 1bedescribed in terms of Levin's Equation 6, that is, in terms of the function (4>
h(A,B) = IIR - I - AB' + Diag(AB')1I2.
For a fixed A, updating its rows simultaneously according to Equation 3 is tantamount to minimizing h(A,B) as a function of B. Conversely, wh~enB is kept fixed and its rows are updated simultaneously by Equation 3, then h(A,B) is minimized as a function ofA. It follows that simultaneous updating; minimizes h(A,B) by alternating least squares, thus monotonely reducing hcl,B). In this way Levin showed that his method is bound to converge. It is of vital importance, however, to note that h(A,B) is not the function to be minimized in minimum residual factor analysis. That is, minimizing h(A,B) of Equation 4 is not equivalent to minimizing f(A) of Equation 1, because the constraintA = B has not been imposed on h(A,B). Nevertheless, simultaneous updating could be salvaged as a Minres method if it could be shown to converge to a limiting pair A,B for which AB' is Gramian (not just symmetric). Levin (1988) has shown that, if [R - I+ Diag(AB1)]is Gramian at the global minimum of h(A,B), then so is AB'. In fact, this result can be shown to hold for every stationary point of h(A,B). The major problem with Levin's method is, that there is no guarantee whatsoever that AB' will be Gramian after convergence. A simple example can demonstrate this. Consider the 4 x 4 correla~tionmatrix
with eigenvalues 2.625, -625, .625, and .125. For r = 2 and
and
OCTOBER 1990
Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
J. ten Berge and F. Zegers
we have a symmetric matrixAB1for which h(A7B)attains the global minimum of zero, yet AB' is not Gramian, with eigenvalues 2,0,O and -.5. Levin (p. 416, footnote) attempts to ignore the negative eigenvalue of -.5 as a case of overfactoring, but this is an adhoc definition of overfactoring rather than an explanation. Instead, the present authors interpret the negative eigenvaiue as a predictable consequence of minimizing the wrong function 4 instead of 1. It can be concluded that simultaneous updating as suggested by Levin is problematic because it converges to a solution for an irrelevant function, from which a solution for the relevant function cannot always be generated. Although Levin's method may work in certain practical applications, it is safer to rely on the Harman and Jones (1966) method, which works in every case. Direct and simultaneous updating in column-wise approaches Comrey and Ahumada (1964,1965) and Zegers and ten Berge (1983) have developed Minres procedures based on column-wise updating of A. The Comrey and Ahumada procedure is based on simultaneous updating as in Levin's (1988) method, whereas the Zegers and ten Berge method is based on direct updating as in the Harman and Jones (1966) method. Levin (1988, p. 416) concluded, in line with his previous claims, that the Zegers and ten Berge method is incorrect whereas the Comrey and Ahumada method is correct. It is clear from the previous sections that these conclusions have to be reversed. That is, the Zegers and ten Berge method monotonely reduces the function f(A) of Equation 1, whereas the Comrey and Ahumada method minimizes an irrelevant function. Again, a small example is highly instructive. Consider the 3 x 3 correlation matrix.
for which the function RA) cannot have a zero value if r = 1. However, the Comrey and Ahumada method minimizes h(A,B) and for r = 1it attains a global minimum of zero for
424
MULTIVARIATE BEHAVIORAL RESEARCH
J. ten Berge and F. Zegers
Downloaded by [Memorial University of Newfoundland] at 12:23 25 January 2015
and b = -a, thus yielding a matrix ab' that is non-Gramian. Ignoring this result as a case of overfactoring would not make sense. Instead, the non-Gramian solution is to be interpreted as a penalty for minimizing the wrong function. Discussion So far the superiority of direct updating over simultaneous updating has been discussed without taking Heywood cases into consideration. Levin (1988,~.416) refers to Harman (1976) and Mulaik (1972) for the handling of Heywood cases. However, these methods, also discussed in Zegers and ten Berge (1983), apply to direct updating. It is not at all clear how He:ywood cases are to be treated with simultaneous updating methods as Levin's. This is another reason for preferring direct updating (Harman &Jones; Zegers & ten Berge) over simultaneous updating (Levin; Comrey & Ahumada). References Comrey, A. L., & Ahumada, A. (1964). An improved procedure and program for minimum residual analysis. Psychological Reports, 15, 91-96. Comrey, A. L., & Ahumada, A. (1965). Note and Fortran IV program for m~inimumresidual factor analysis. Psychological Reports, 17, 446. Harman, H. H. (1976). Modern factor analysis (3rd ed., rev.). Chicago: University of Chicago Press. Harman, H. H., & Jones, W. H. (1966). Factor analysis by minimizing residuals (Minres). Psychometrika, 31,351-369. Levin, J. (1988). Note on convergence of Minres. Multivariate BehavioralResearch, 23,413417. Mulaik, S. A. (1972). The foundations of factor analysis. New-Yorlr: McGriaw-Hill. Zegers, F. E., & ten Berge, J. M. F. (1983). A fast and simple computational method of minimum residual factor analysis. Multivariate Behavioral Research, ICI,331-340.
OCTOBER 1990
