Available online at www.sciencedirect.com

Nurs Outlook 63 (2015) 318e330

www.nursingoutlook.org

Validity and model-based reliability of the Work Organisation Assessment Questionnaire among nurses Leila Karimi, PhDa,*, Denny Meyer, DBLb a

School of Health Sciences, Swinburne University of Technology and School of Public Health and Human Biosciences, La Trobe University, Melbourne, Victoria, Australia b The Brain and Psych Sciences Centre, Swinburne University of Technology, Hawthorn, Australia

article info

abstract

Article history: Received 26 May 2014 Revised 17 September 2014 Accepted 22 September 2014 Available online 28 September 2014

Background: The Work Organisation Assessment Questionnaire (WOAQ) was developed as part of a risk assessment tool for the manufacturing setting (Griffiths, Cox, Karanika, Khan, Toma´s, 2006). The WOAQ was used in this study to assess its validity and model-based reliability among Australian nurses using a conventional second-order and a bifactor confirmatory factor analysis (CFA). Methods: This article reports a study of Australian participants (N ¼ 312) to validate a multidimensional measure of the WOAQ. Data were collected from a group of Australian community nurses. CFA was performed comparing a conventional second-order model and a bifactor model of one WOAQ summative factor and its five subfactors, which together described the most common issues of work design and management in organizations. A group of close fit indices that include relatively greater penalties for model complexity (i.e., the root mean square error of approximation, Non-normed Fit Index, and the Akaike Information Criterion) were chosen to interpret the results. Results: The results of reliability and validity analysis revealed some evidence for the summative measure of the WOAQ. Results indicate that the questionnaire is a valid and reliable scale, but some subfactors are more plausible than others within the Australian nursing community. Conclusion: The study provides future researchers some guidance for bifactor modeling in organizational studies of this type. The WOAQ provides flexibility to measure risk assessment and management procedures.

Keywords: Bifactor modeling Model-based reliability Nursing Risk assessment Second-order modeling Validity Work Organisation Assessment Questionnaire

Cite this article: Karimi, L., & Meyer, D. (2015, JUNE). Validity and model-based reliability of the Work Organisation Assessment Questionnaire among nurses. Nursing Outlook, 63(3), 318-330. http://dx.doi.org/ 10.1016/j.outlook.2014.09.003.

One of the greatest challenges for society is sustaining an individual’s health and the quality of life in the workplace (Cox, 1997). There is a broad body of research revealing damages to health and well-being in workplaces. Increasing awareness of the possible effects of work-related factors on health has led to the enforcement of regulations and the introduction of

legislation in many developed countries to ensure that organizations make the health of their employees a high priority (Faragher, Cooper, & Cartwright, 2004). As a result, management has also been encouraged to conduct risk assessments for psychosocial hazards with a view to protecting employees’ health and safety in the workplace (Rick & Briner, 2000).

* Corresponding author: Leila Karimi, School of Public Health and Human Biosciences, La Trobe University, Melbourne, Victoria 3086, Australia. E-mail address: [email protected] (L. Karimi). 0029-6554/$ - see front matter Ó 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.outlook.2014.09.003

Nurs Outlook 63 (2015) 318e330

There are several driving forces that contribute to making the workplace a less convivial place for employees to work in. For instance, the growing competitiveness of the marketplace, the constant need to improve organization efficiency and profitability, and radical changes in employment conditions are among the major driving forces responsible for increasing stress in the workplace (Faragher et al., 2004); but, in particular, an inability to incorporate proper work design in the workplace leads to a negative effect on both the employees and the organization (Griffiths, Cox, Karanika, Khan, & Tomas, 2006). Much of the attention in the occupational health and safety (OHS) literature has focused on linking this inability to incorporate a suitable work design with the right assessment tools and decreasing negative workrelated outcomes for individuals and organizations (Griffiths et al., 2006). However, the efficacy of an OHS tool in assessing the risk factors in the workplace environment depends greatly on how well it is designed, implemented, and adapted to the nature of the work (LaMontagne, 2004). Adapting such approaches provides a benchmark to identify the main organizational hazards and to progressively improve OHS by improving safe work design and practices. The main challenge is to use a suitable instrument to improve the capture of OHS indicators. Based on recommendations from previous studies, a good risk assessment process can only be achieved by using multiple methods of assessment. A welldesigned assessment should recognize the risks in the workplace and also identify the employees at risk (The Health and Safety Executive Guidelines, 2010). The organizational risk assessment is obtained using questionnaire/survey scales. To evaluate risk and stress effectively, this questionnaire must meet some important criteria such as being reliable and valid; being easy to complete; measuring the possible risks, their predictability of outcomes related to the employees’ health, their size, and impact on the target population; and applying to both organizations as a whole and at different work levels. To be able to meet such criteria, the questionnaires are usually quite lengthy. As a result, the large amount of time it takes to complete a questionnaire leads to a low response rate (Faragher et al., 2004). A short yet comprehensive risk assessment questionnaire is desirable. One such instrument called the Work Organisation Assessment Questionnaire (WOAQ) developed by Griffiths et al. (2006) may be applicable as a substitute to overcome problems identified in previously validated measures because of its short length and comprehensive content. The methodology developed in WOAQ was based on identifying and collecting employees’ opinions on their work, health, and workplace design and management (Griffiths et al., 2006). It was designed to measure risk factors pertaining to the work design and management that may influence employee health and health-related behaviors in a

319

manufacturing setting (Griffiths et al., 2006; WynneJones et al., 2009). The overall score on WOAQ indicates the extent to which the respondents believe that these dimensions of work are problematic and can be used as predictors of well-being, subjective health, and job satisfaction. A high score on WOAQ indicates that the respondents perceive dimensions of work as good, and a low score on WOAQ indicates that the respondents perceive dimensions of work as problematic (Griffiths et al., 2006). WOAQ was initially developed for a manufacturing setting and implemented in a private sector; however, this comprehensive approach to risk assessment may be extended to other settings including a nonmanufacturing or health setting. In this regard, it is important to make sure WOAQ can be implemented in other work settings or professions (Wynne-Jones et al., 2009). Only a few studies evaluated the utility of WOAQ in other workplaces. For example, Wynne-Jones et al. (2009), in their study of two large public sector organizations in South Wales, evaluated the validity and reliability of WOAQ in a public sector. Using a second-order confirmatory factor analysis (CFA), the researchers only found a marginal fit for the original five subfactors of WOAQ. In the end, they identified a two-factor structure of the WOAQ assessing management and work design and work culture. Therefore, one of the aims of this study was to find out if the general and five subfactors of WOAQ can be implemented in a nonmanufacturing, health setting in Australia. Also, in addition to the conventional second-order model of CFA used heavily by other scholars in the field (including Wynne-Jones’s study), a more practical bifactor modeling was used to assess the general factor of WOAQ and the plausibility of its five subfactors in an Australian community nursing setting.

Bifactor Versus Second-order Modeling Constructs are often operationalized as multidimensional units (Diamantopoulos, 2010; Edwards & Bagozzi, 2000). When there are a number of dimensions or related attributes required to form a latent factor, it is considered multidimensional. In a multidimensional construct, dimensions can be constructed under an overall concept or a second-order (higherorder) construct (Law, Wong, & Mobley, 1998). In second-order constructs, two levels of constructs exist: the first-order level with indicators and the secondorder level with first-order constructs (Jarvis, MacKenzie, & Podsakoff, 2003). Such models are known as hierarchical (higher-order or second-order) models. By default, the majority of the researchers in behavioral sciences use higher-order modeling to evaluate multidimensionality. However, this seems not to be the only procedure and not always the best

320

Nurs Outlook 63 (2015) 318e330

way to evaluate a multidimensional model. The use of other approaches, such as bifactor (direct hierarchical order) modeling, are not as common in the literature (Gignac, 2007; Reise, Moore, & Haviland, 2010). In a bifactor model, latent variables are constructed as firstorder constructs with these first-order factors nested within the general factors (Gignac, 2007; Gustafsson & Balke, 1993; Holzinger & Swineford, 1937). Perhaps the early roots of the development of the bifactor (nested factors) model can be related mainly to the work of Holzinger and Swineford (1937) (for a full history of Structural Equation Modeling [SEM] development, see Karimi & Meyer, 2014). However, the bifactor approach is not well appreciated in the literature although there are some advantages in using this procedure as an alternative to the conventional higherorder modeling (Gignac, 2007, 2013). As discussed by Gignac (2007), “the advantages are that bifactor (nested factors) models: (a) tend to be associated with nonnegligibly higher level of model fit; (b) allow for statistical significance testing for all parameter estimates, and (c) allow for less ambiguous interpretations of the factor loadings and the narrow factors ‘nested’ within the higher-order factors”. As asserted by Gignac, one can achieve a better model fit using bifactor modeling. Imposing fewer restrictions on parameter estimates (as opposed to the conventional procedure in CFA) improves the quality and validity of the reported results in bifactor models. In addition, the bifactor model provides some evidence on the plausibility of the subfactors and the extent of their contribution practically. Clearly, the procedure is not without disadvantages. One of the main limitations of using the nested factor model is that it is associated with fewer degrees of freedom. This might lead to model identification problems (Gignac, 2007). However, this problem can be managed simply by constraining some of the parameters in the model (Gignac, 2007, 2013). In this study, all the latent variable variances were constrained to 1.0 in order to achieve the identified model. It is evident that, if the aim for the proposed model is to present both multidimensionality and a general single factor at the same time, then a bifactor model is the best way to present the model (Reise et al., 2010). Using a bifactor model not only shows the contribution of the items to a general factor (broad construct) but also provides information on the item’s contribution to subdimensions (narrow constructs; Reise et al., 2010). In the context of risk assessment in organizations, a tool like WOAQ that presents the overall work condition as a general single factor and at the same time highlights the different subsections of work organization characteristics would be helpful. Having a broad or macrolevel assessment (using WOAQ general factor) would help to get an overall picture of the organization, whereas assessing the organization at a narrow or microlevel (using multidimensions of WOAQ) would have practical implications for the improvement of the specific problematic areas. Thus, evaluating the

plausibility of subfactors is very important in such contexts, suggesting that a direct hierarchical model for WOAQ is a good choice. Therefore, one of the aims of this study was to compare a bifactor (nested factor model) with a conventional second-order (higher order) model of WOAQ in a health setting. This might open up some empirical and methodological venues for further developments in this area. A higher-order model (or full-mediation model) and a direct hierarchical model (partial-mediation model) of WOAQ can be distinguished statistically (Gignac, 2007, 2008, 2013; Yung, Thissen, & McLeod, 1999). To the researchers’ knowledge, no study has been conducted in a nonmanufacturing, health setting in Australia using the proposed procedure. Therefore, the present study used data collected from a group of Australian nurses to assess the validity and modelbased reliability of WOAQ, a well-designed instrument for assessing work and organizational factors as potential risks to employee health. The study aimed to achieve two main goals: first, to assess the validity and model-based reliability of WOAQ in an Australian health setting and, second, to compare a bifactor model (nested factor models) with a conventional second-order model of WOAQ using CFA.

Model-based Reliability One of the commonly used estimates for reliability is coefficient alpha, which was proposed originally by Cronbach in 1951. Coefficient alpha was developed for one-dimensional scales, and its limitations, especially for multidimensional constructs, have been discussed by scholars (Sijtsma, 2009; Zinbarg, Revelle, Yovel, & Li, 2005). When it comes to multidimensional scales, coefficient alpha would not be an accurate representative of the reliability and might lead to overestimation of the reliability (Cortina, 1993). A statistically justified reliability assessment for multiple constructs was introduced many years ago by Bentler (1968) and Heise and Bohrnstedt (1970) for factor analytic types of models and by Bentler (2007, 2009), in a generalized form for any structural equation model with additive errors. Although reliability for a general SEM model is rationalized based on the model’s multidimensional structure, it should be noted that such a coefficient, which we will call r or rho, also can be interpreted as a unidimensional measure that quantifies the proportion of variance explained by the most reliable single dimension in a multidimensional space (Bentler, 2007). However, despite the importance of multidimensional model-based reliability measurements, there are a few empirical studies that report factor-based coefficients (e.g., Gignac & Watkins, 2013; Reise, Bonifay, & Haviland, 2012; Zinbarg et al., 2005). The reliability coefficient rho can be calculated as easily for one-dimensional as for multidimensional

Nurs Outlook 63 (2015) 318e330

models in EQS (Multivariate Software Inc, Los Angeles). Therefore, in this study, if the CFA model of WOAQ describes the data well among Australian nursing, then the multidimensional coefficient rho will be used for evaluating WOAQ reliability.

Predictive (Criterion) Validity of WOAQ Another type of validity that needs to be assessed after establishing the other types of validity and reliability refers to predictive or criterion validity. By assessing criterion validity, one can determine whether the scale or its subscales can predict other related constructs (Bagozzi, 1980). This level of validity can be evaluated by entering some known outcome or antecedent variable into the model and assessing their relationship with the scale or subfactors. The workplace climate has been found to relate to job satisfaction, stress, withdrawing behaviors, and even higher team performance (Grandey, 2000; Laschinger, 2012). In client-focused, high-demand settings like nursing, work characteristics may help employees cope with the emotional challenges of service jobs. In this study, the predictive validity of WOAQ will be assessed using some well-known outcomes (i.e., job stress, job stresserelated presenteeism, well-being factors, and job satisfaction) of workplace characteristics.

Methodology Sample and Procedure Data were collected from Australian nurses. The study design was cross-sectional. A self-report questionnaire was used to capture demographic work characteristics and WOAQ. A questionnaire package that included a cover letter, information sheet, consent form, questionnaires, and reply paid envelopes was forwarded to all potential participants. Three weeks after the mail out, a letter was forwarded to the employees to thank them for their participation or to ask if they could complete and return the questionnaire if they had not done so. A total of 334 surveys were returned. Some of the returned surveys were incomplete with a high percentage of missing data; therefore, a decision was made to remove the incomplete surveys. After data cleaning and removing the incomplete data, 312 surveys were included in the final data analysis.

Measures WOAQ The study measures included a 28-item WOAQ (Griffiths et al., 2006) pertinent to aspects of the

321

respondents’ work organization. Respondents were asked to rate how problematic or good each of the items were for them in the last 6 months on a 5-point Likert scale (from “very good” [5] to “major problem” [1]). WOAQ was measured as a general 28-item summative factor with a five-factor structure. The five-factor structure of the scale includes workload issues, reward and recognition, quality of relationships with management, relationships with colleagues, and physical environment.

Job Stress The Self-Report Emotional Intelligence Test (SREIT) measurement uses 13 items to assess job stress (Parker & Decotiis, 1983). The responses were rated on a Likert scale (e.g., “I have too much work and too little time to do it in”). A high score indicated a high level of job stress.

General Well-being Questionnaire The General Well-Being Questionnaire (GWBQ) of Cox, Thirlaway, Gotts, and Cox (1983) was used to evaluate employee health. GWBQ is a 24-item instrument used to capture suboptimal health, consisting of selfreported symptoms of general malaise. It presents a multifactored set of general symptoms of ill health and consists of 2 subscales of suboptimum health each comprised of 12 items: (a) worn out/exhausted and (b) tense/nervous. For both these subscales, respondents reported on their experience of 12 symptoms in the last 6 months. Their responses ranged from 0 (never) to 4 (all the time). For the final analysis, the scores were reversed to make them consistent with the other scales, which meant that a high score indicated high well-being.

Job Stresserelated Presenteeism Presenteeism occurs when employees are physically present but mentally absent. A self-report scale created by Gilbreath and Frew (2008) is used to measure employee job stresserelated presenteeism. The scale consisted of six items with responses ranging from “all the time” (5) to “never” (1). A higher score on this scale represents a higher level of presenteeism.

Intrinsic Job Satisfaction Intrinsic job satisfaction is referred to as “the degree to which a person reports satisfaction with intrinsic features of the job” (Warr, Cook, & Wall, 1979, p. 133). This variable was assessed using the intrinsic job satisfaction scale of Warr et al. (1979). It is comprised of seven items. For each item the respondents were asked to assess their degree of satisfaction/dissatisfaction with their work on a seven-point Likert response with 1 being “extremely dissatisfied” and 7 “extremely satisfied.” The sample questions were as follows: “How satisfied or dissatisfied are you with: the freedom to choose your own method of working, the recognition you get for good work, the amount of responsibility you are given, the opportunity to use your abilities, your chance of promotion, the attention paid to suggestions

322

Nurs Outlook 63 (2015) 318e330

you make, and the amount of variety in your job?” A higher score on this scale means a higher level of satisfaction.

Ethical Consideration Human Research Ethics Committee approval was obtained from both the lead university and the participating organization. Written consents were obtained from the participants.

Overview of Analyses At the first step of the validation process, the model fit for an independence model was compared with that of a second-order or higher-order model (a higher-order construct that explains its five first order constructs), and a bifactor model (a nested construct in which each item loads on a general construct and the given nested construct). The chosen goodness of fit measures were suited to the study goals. An important factor that was considered in choosing the suitable fit indices was the degree of penalty it includes for model complexity. Based on the suggestion by scholars (e.g., Gignac, 2013) for evaluating bifactor models, it is better to choose those groups of indices that include relatively greater penalties for model complexity (i.e., the root mean square error of approximation [RMSEA], the Nonnormed Fit Index [NNFI]/the Tucker-Lewis Index [TLI], and the Akaike Information Criterion [AIC]). The fit indices reported in this study are summarized as follows: (a) the RMSEA, (b) the TLI or NNFI, and (c) the AIC. RMSEA values of less than .08 and .05 (MacCallum, Browne, & Sugawara, 1996) and NNFI values of greater than 0.90 and 0.95 (Hu & Bentler, 1999) were considered as marginal and good fit respectively. The model comparisons were performed based on a practical improvement in NNFI. NNFI differences of at least .010 show significant model improvement (Vandenberg & Lance, 2000). The AIC is another comparative measure of fit that was used for this study. AIC is meaningful only when two different models are compared. The smaller value of AIC with a drop (DAIC) of more than 10 indicate a superior model fit (Akaike, 1973; Raftery, 1995; Schwarz, 1978). Based on the recommendation by Burnham and Anderson (2004), AIC has more practical and theoretical advantages over Bayesian Information Criterion (BIC) or the chi-square test. The chi-square goodness of fit test was also reported as a conventional, commonly reported measure of fit in the literature. Traditionally, a chi-square statistic is used for assessing if the proposed model describes the data well. Because the chi-square is highly dependent on sample size, the relative chi-square (CMIN/DF) is used as a measure of model fit. A value of 1 to 2 reflects good fit, less than 3 represents acceptable fit (Kline, 1998), and less than 5 represents adequate fit

(Schumacker & Lomax, 2004). As acknowledged by Hu and Bentler (1999), the chi-square statistic is highly dependent on sample size and is not appropriate for complex or non-normal data. The other commonly reported fit indices are the standardized root mean square residual and the comparative fit index; however, none of these fit indices were considered in this study because they do not penalize for model complexity adequately (Gignac, 2013; Marsh, Hau, & Grayson, 2005). The model-based reliability coefficient, rho, was used for testing the reliability of WOAQ. EQS (v6.2) was used to run the CFA analysis using maximum likelihood estimation with robust standard errors (Satorra & Bentler, 1994). Any fit indices reported in this article are “robust” indices. For assessing predictive validity, a number of relationships between the questionnaire factors and measures of job stress, well-being, satisfaction, and job stresserelated presenteeism were examined using regressions.

Results Table 1 presents the frequencies, means and standard deviations for the demographic variables. The majority of the participants were female (94.5%) with an average age of 45.19 years. The majority had more than 4 years of experience working in a nursing setting (97.1%). About 40% of the participants were working full-time, with the remaining 60% part-time employees. Table 1 e Descriptive Statistics of the Demographic Variables* Frequency (%) Sex Male Female Contacts with clients/h/d 8 Years of experience/y 6 Employment status Part-time Full-time Age

31 (5.5) 290 (94.5) 22 (7.1) 30 (9.7) 122 (39.4) 134 (43.2) 2 (0.6) 2 (0.7) 7 (2.3) 16 (5.2) 282 (91.9) 183 (60) 123 (40) Mean (SD) 45.19 (9.54)

SD, standard deviation. * n varies between 306 and 312 because of some missing responses.

Nurs Outlook 63 (2015) 318e330

The 28 WOAQ items were used to assess the validity of the scale (Appendix 1). The dimensionality of the general score of WOAQ and the five subfactors were assessed using CFA. Both the second-order model (higher-order model; Figure 1, model 1) and the bifactor model were compared (Figure 1, model 2) were tested

323

for fit adequacy. The results for modification indices suggested a correlation between three sets of the within construct measurement errors specifically for the environmental factor (safety at work with exposure to physical danger) and the workload factor (impact of family/social life on work with impact of your work on

Figure 1 e The proposed model of WOAQ.

324

Nurs Outlook 63 (2015) 318e330

family/social life; pace of work with workload). As suggested by Kenny (2011), if some items have similar content and are theoretically meaningful, one may correlate the errors for these items. In this context, it made sense theoretically and statistically to correlate those measurement errors. In SEM, by default, maximum likelihood is used for data estimation.

Model Fit Evaluation The results indicated that the second-order model provides a marginally fitting model for a general factor of WOAQ and its five subfactors (RMSEA ¼ 0.06, NNFI ¼ 0.89). The factor loadings for all subfactors on the general factor of WOAQ were strong and significant suggesting well-defined subfactors. The standardised path coefficients were 0.71 for quality of physical environment, 0.57 for quality of relationship with colleagues, 0.93 for quality of relationship with management, 0.99 for reward and recognition, and 0.76 for workload issues. For a meaningful comparison of the second-order loadings with the bifactor model, the Schmid-Leiman (S-L) transformation was conducted. Table 2 provides

the S-L transformed factor loadings for the secondorder model. As suggested by Gignac (2007), the S-L transformation for first-order loadings was calculated by multiplying the first-order factor loadings with their respective higher-order general factor loadings. The S-L was calculated for the subfactors loadings using their respective unique regression weights. The results of the bifactor model presented an acceptable fit (RMSEA ¼ 0.04, NNFI ¼ 0.93). Table 3 presents the results of the CFA evaluation. Based on the results, the bifactor model of WOAQ provides a superior fit and smaller AIC value (AIC ¼ 89.66) compared with the conventional second-order model (AIC ¼ 50.44). DNNFI greater than 0.04 and DAIC of 140.10 demonstrated the superiority of the bifactor model over the second-order model. Important differences also were found in the factor loadings of the bifactor model compared with the second-order model. The most important differences were found for the “quality of relationship with management” and “rewards and recognition” subfactors. The S-L solutions of the second-order model showed similar positive factor loadings for these factors, whereas the bifactor model detected differentially directed loadings for these subfactors.

Table 2 e Completely Standardized Maximum Likelihood Solutions of the Second-order Model (SchmidLeiman [S-L] Transformed) and the Bifactor Model Item S-L Bifactor Number Second Order

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

G

QPE

0.47 0.54 0.57 0.29 0.74 0.52 0.71 0.52 0.42 0.37 0.79 0.00 0.00 0.00 0.43 0.73 0.72 0.34 0.48 0.54 0.57 0.78 0.74 0.60 0.69 0.73 0.72 0.45

0.55 0.62

QRC

QRM

RR

WI

0.23 0.34 0.30 0.46 0.28 0.46 0.49 0.55 0.31 0.02 0.02 0.02 0.38 0.29 0.28 0.39 0.42 0.62 0.02 0.31 0.02 0.02 0.02 0.29 0.28 0.67

G

QPE

.32 .35 .65 .24 .72 .47 .73 .48 .44 .35 .78 .74 .64 .48 .37 .72 .72 .31 .44 .49 .56 .76 .69 .55 .66 .80 .79 .45

.60 .78

QRC

QRM

RR

WI

.18 .29 .42 .37 .12 .37 .33 .65 .33 .11 .06 .43 .58 .29 .18 .28 .52 .52 .06 .40 .30 .09 .44 .05 .11 .55

G, general factor; QPE, quality of physical environment; QRC, quality of relationship with colleagues; QRM, quality of relationship with management; RR, reward and recognition; S-L, Schmid-Leiman; WI, workload issues.

325

Nurs Outlook 63 (2015) 318e330

Table 3 e Summary of Model Fit Statistics of the CFA Models of the Work Organisation Assessment Questionnaire Model SB c2/df RMSEA (95% CI) NNFI AIC DNNFI* DAIC* 0. Independent model 1. Second-order model 2. Bifactor model

11.48 2.14 1.71

0.06 (.05e0.06) 0.04 (0.04e0.05)

0.89 0.93

50.44 89.66

140.10

0.04

AIC, Akaike Information Criterion; CFA, confirmatory factor analysis; NNFI, Non-normed Fit Index; RMSEA, root mean square error of approximation. * Differences between model 2 versus model 1.

Further analysis was performed to assess reliability of the models. The results of the model-based reliability evaluation of the multidimensional WOAQ using the reliability coefficient rho indicated excellent reliability for this scale (rho ¼ 0.94).

Predictive Validity To determine if the dimensions of WOAQ were differentially related to the outcome variables, stepwise linear regression analyses were performed on the dependent variables (i.e., job stress, job satisfaction, stress-related presenteeism, and well-being). The results of the intercorrelation analysis on all study variables are presented in Table 4. As presented in Table 4, the correlations between the WOAQ factors and the outcome variables varied from small to high (r ¼ 0.13e0.91, p  .01). All WOAQ subfactors were negatively correlated with stress-related presenteeism and job stress (r ¼ .19 to .56, p  .01). Also, the five subfactors of WOAQ were positively correlated with job satisfaction and the general factor of well-being and its subfactors (tense/nervous and worn out/ exhausted) (r ¼ 0.13e.66, p  .01), indicating that a higher quality of work organization is associated with a higher level of employee well-being and satisfaction. Job stress was differentially predicted by the subfactors of WOAQ (Table 5). The strongest relationship was found with workload issues (b ¼ 0.56, p  .01). Interestingly, the same result was found for stressrelated presenteeism; again workload issues was the

strongest predictor of this outcome variable (b ¼ 0.47, p  .01). Job satisfaction was predicted by all factors of the WOAQ and the quality of relationship with management (b ¼ 0.64, p  .01), and issues of reward and recognition (b ¼ 0.66, p  .01) were the strongest predictors. The general factor of well-being and its two subfactors of worn out/exhausted and tense/nervous were predicted by all the subfactors, but the R2 values, although statistically significant, were small (0.01e0.06). As shown in Table 6, the general factor of the WOAQ significantly predicted job stress, stress-related presenteeism, job satisfaction, and the general factor of well-being and its subfactors of worn out/exhausted and tense/nervous. The highest predictability was observed with job satisfaction (b ¼ 0.64, p  .01).

Discussion The most common problems detected in the literature of full risk assessment were the facts that questionnaires are either very long and detailed or unable to detect the hazardous nature of the identified problem in a work setting. In response to the evidenced need for a short, valid stress risk assessment, WOAQ was developed. WOAQ (Griffiths et al., 2006) seems to overcome these problems with its short length yet comprehensive contents. The methodology developed in WOAQ was based

Table 4 e Correlation Matrix for the Main Variables with Subfactors of the Work Organisation Assessment Questionnaire (WOAQ) Variables 1 2 3 4 5 6 7 8 9 10 11 12 1.WOAQ general factors a. Quality of relationships with management b. Reward and recognition c. Workload issues d. Quality of relationships with colleagues e. Quality of physical environment 2. Job stresserelated presenteeism 3. Job stress 4. Job satisfaction 5. Well-being: general factor a. Well-being: worn out/exhausted b. Well-being: tense/nervous Note. Significant correlations ( p< .01).

.91 .87 .72 .48 .71 .39 .47 .64 .24 .19 .25

.81 .50 .41 .46 .31 .37 .64 .16 .13 .16

.54 .43 .45 .31 .40 .66 .22 .18 .22

.20 .59 .47 .56 .40 .25 .22 .23

.25 .19 .22 .30 .21 .15 .23

.28 .35 .32 .19 .14 .20

.67 .30 .49 .49 .38

.38 .45 .42 .40

.23 .20 .22

.91 .89

.63

326

Nurs Outlook 63 (2015) 318e330

Table 5 e Regression Analyses with the Outcome Variables of the Five WOAQ Subfactors Variables Job stress Quality of relationships with management Reward and recognition Workload issues Quality of relationships with colleagues Quality of physical environment Well-being Quality of relationships with management Reward and recognition Workload issues Quality of relationships with colleagues Quality of physical environment Worn out/exhausted Quality of relationships with management Reward and recognition Workload issues Quality of relationships with colleagues Quality of physical environment Tense/nervous Quality of relationships with management Reward and recognition Workload issues Quality of relationships with colleagues Quality of physical environment Job satisfaction Quality of relationships with management Reward and recognition Workload issues Quality of relationships with colleagues Quality of physical environment Stress-related presenteeism Quality of relationships with management Reward and recognition Workload issues Quality of relationships with colleagues Quality of physical environment

R2

Adjusted R2

F-Value

b

.13 .16 .31 .05 .12

.13 .15 .31 .04 .12

49.05 59.89 145.45 16.41 43.47

.37z .40z .56z .22z .35z

.02 .05 .06 .04 .03

.02 .04 .06 .04 .03

8.61 17.03* 21.43 14.82 11.77

.16y .22z .25z .21z .19z

.01 .03 .04 .02 .02

.01 .03 .04 .02 .01

5.67 11.04 16.28 7.61 6.48

.13* .18y .22z .15* .14*

.02 .05 .05 .05 .04

.02 .04 .05 .05 .03

8.64 17.19 18.52 18.03 13.64

.16y .22z .23z .23z .20z

.41 .43 .16 .09 .10

.41 .43 .15 .09 .10

220.65 239.88 59.47 32.17 37.70

.64z .66z .40z .30z .32z

.10 .09 .22 .03 .07

.09 .09 .22 .03 .07

35.06 34.40 92.26 11.57 26.58

.31z .31y .47z .19y .28z

*p  .05, yp < .01., zp < .001.

on the identification and collection of employee opinions on their work and health (Griffiths et al., 2006). The present research examines the validity and model-based reliability of WOAQ on a group of Australian employees using the conventional secondorder model and a bifactor model of WOAQ. The scale is relatively short (28 items). The WOAQ second-order model included a general measure of WOAQ at a higher order and five subfactors at the first order, each representing different dimensions of work organization risk assessment. The five subfactors are “quality of relationships with management,” “reward and recognition,” “workload issues,” “quality of relationships with colleagues,” and “quality of physical environment.” The bifactor model of WOAQ included the general measure of WOAQ and its five subfactors as a direct hierarchical model. The results of CFA revealed the superiority of the bifactor model over the conventional second-order model. In addition, very important differences were found between the second-order model and the bifactor model. The most important difference was detected

when the conventional second-order model failed to recognize the low and differentially directed loadings of the “quality of relationship with management.” However, the results of the bifactor model detected that the subfactor of “quality of relationship with management” was poorly defined in this context independently of the general measure of WOAQ. The subfactor of “reward and recognition” was identified as implausible by both models. These results have great practical implications. It shows that in the context of community nursing, although the general measure of WOAQ is a valid and reliable measure of the organizational risk assessment, the most important plausible subscales are “quality of physical environment,” “quality of relationship with the colleagues,” and “the workload issues.” Based on the findings, the two subscales of “the quality of relationship with management” and “reward and recognition” are not that critical in such contexts. Unfortunately, the lack of previous studies makes it difficult to compare the findings with the results of previous studies because the majority of the previous

327

Nurs Outlook 63 (2015) 318e330

Table 6 e Regression Analyses for the Outcome Variables using the General Work Organisation Assessment Questionniare (WOAQ) factor as a predictor Variables R2 Adjusted R2 F-Value b Job stress Stress-related presenteeism Job satisfaction Well-being Worn out/exhausted Tense/nervous

.47 .15 .41 .06 .03 .06

.22 .15 .41 .05 .03 .06

88.54 55.98 216.72 20.17 12.52 21.05

.47z .39* .64* .24* .19* .25*

Note: *p < .05., zp < .001.

studies of WOAQ have been conducted in a manufacturing setting, using the conventional second-order procedure (Griffiths et al., 2006; WynneJones, 2009). However, close evaluation of the reported work setting indicates that these findings should not be a surprise. These findings can perhaps be explained by the nature of the community nursing work environment because, although the nurses belong to a larger organization, they work in different, small branches with their own immediate managers/supervisors. In such an environment, there is a more informal relationship between the nurse and manager/supervisor. The relationships in community nursing settings are more colleaguecolleague relationships rather than nurse-manager. Also, the “reward and recognition” factor is strongly tied to the management relationship, and only items representing a variety of tasks, an opportunity for learning, and using the new skills appeared as important indicators of this subscale. Thus, for risk management improvement in nursing organisations it seems that the most important subfactors are the work physical environment, the relationship with colleagues, and managing the workloads. The results also indicated acceptable reliability and predicative validity for the WOAQ and its subfactors. The general summative score of WOAQ and each of the subscales were differentially associated with the outcome variables of work stressors, confirming the predictive validity of the WOAQ. Overall, WOAQ, especially the single general measure, appears to be a superior instrument for assessing risk factors associated with employee health and health-related behavior because of its satisfactory psychometric properties and short length. More importantly, based on the results from the bifactor analysis, it was shown that some of the subscales are more important than others in a nursing setting. This indicated that concerns relating to the identification of the hazardous nature of the identified problem in the work setting can be solved by applying WOAQ to assess risk factors in the workplace. Ultimately, this would assist management in identifying problem areas, which may cause harm to their employees and the organization, and allow proper action to be accomplished in order to prevent future occurrences of similar events.

Strengths and Limitations The strength of the study is the context of the research and the methodology used. To the researchers’ knowledge, this study is one of the first that is comparing a conventional second-order model with a bifactor model of WOAQ. The methodology used has theoretical and practical implications in organizational studies like this. Based on the nature of work (e.g., manufacturing vs. nonmanufacturing) and occupation types, organizations will have significant differences with each other. A risk assessment tool like WOAQ would be a very useful tool in assessing organizational risk factors, but in practice not all of its subfactors might be as plausible or important in all organizations. In the risk assessment reported in this study, using bifactor modeling, some evidence was found that supports the general measure of WOAQ, but not all the subfactors were equally important. As discussed, the results relate well to real-life expectations. Second, an important factor in choosing the most suitable fit indices was based on the degree of penalty each index includes for model complexity. Those groups of goodness of fit indices that include relatively greater penalties for model complexity (i.e., RMSEA, NNFI/TLI, and AIC) were used due to the complexity of the WOAQ model. Third, the study has used model-based reliability estimation using rho. This targeted the multidimensional nature of WOAQ (i.e., both the general and the five-factor model of WOAQ) to assess the reliability coefficient rho. Using a model-based reliability rather than the conventional coefficient alpha is recommended for multidimensional models such as WOAQ. Fourth, this is one of the first studies that has been conducted on a group of Australian employees in a nursing (nonmanufacturing) setting. The original WOAQ was developed for a manufacturing setting. No previous studies have been completed in a health setting using a comprehensive, short scale risk assessment tool such as the WOAQ. The study initiates a venue for more research in such settings using WOAQ. Finally, WOAQ is a useful tool in practice for its ability to measure organizational risk assessment only using 28 items. This meets most workplace requirements in terms of cost, time, and resources. Using

328

Nurs Outlook 63 (2015) 318e330

bifactor modeling, the most plausible subfactors that are helpful in decision making for improving the organizational risk environment in a nursing settng were identified. One of the limitations in this study is that the study focus was only on health professionals. Further studies are needed to expand the concept to other nonmanufacturing or blue-collar occupations. The lack of background literature in the Australian context using bifactor modeling make it difficult to evaluate or compare the results with other studies. Further studies are needed to expand this area of research.

Summary and Conclusion In this study, attempts were made to assess the validity and reliability of WOAQ in an Australian health setting. Based on the literature, several robust methodological procedures were adopted for assessing the validity of WOAQ, including evaluating the conventional second order model of WOAQ, the bifactor model of WOAQ, and model-based reliability. In general, results showed that WOAQ appears to be a superior instrument for assessing risk factors associated with employee health and health-related behavior because of its satisfactory psychometric properties and short length. Some evidence of multidimensionality was found, and some subfactors appeared to play more critical roles for risk assessment in a nursing setting.

Acknowledgments We would like to acknowledge the community nurses who took part in the study.

references

Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B. N. Petrov & B. F. Csaki (Eds.), Second international symposium on information theory (pp. 267e281). Budapest: Academiai Kiado. Bagozzi, R. P. (1980). Causal modeling in marketing. New York, NY: Wiley & Sons. Bentler, P. M. (1968). Alpha-maximized factor analysis (Alphamax): Its relation to alpha and canonical factor analysis. Psychometrika, 33, 335e345. Bentler, P. M. (2007). Covariance structure models for maximal reliability of unit-weighted composites. In S. Y. Lee (Ed.), Handbook of latent variable and related models (pp. 1e19). Amsterdam: North-Holland. Bentler, P. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74, 137e143. Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference understanding AIC and BIC in model selection. Sociological Methods & Research, 33(2), 261e304.

Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Psychological Bulletin, 78, 98e104. Cox, T. (1997). Workplace health promotion. Work & Stress, 11, 1e5. Cox, T., Thirlaway, M., Gotts, G., & Cox, S. (1983). The nature and assessment of general wellbeing. Journal of Psychosomatic Research, 27, 353e359. Diamantopoulos, A. (2010). Reflective and formative metrics of relationship value: Response to Baxter’s commentary essay. Journal of Business Research, 63(1), 91e93. Edwards, J. R., & Bagozzi, R. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 95(2), 155e174. Faragher, E. B., Cooper, C. L., & Cartwright, S. (2004). A shortened stress evaluation tool (ASSET). Stress and Health, 20, 189e201. Gignac, G. E. (2013). Modeling the balanced inventory of desirable responding: Evidence in favour of a revised model of socially desirable responding. Journal of Personal Assessment, 95, 645e656. Gignac, G. E. (2008). Higher-order models versus direct hierarchical modes: G as superordinate or breadth factor? Psychology Science, 50, 21e43. Gignac, G. E. (2007). Multifactor modeling in individual differences research: Some recommendations and suggestions. Personality and Individual Differences, 42, 37e48. Gignac, G. E., & Watkins, M. W. (2013). Bifactor modeling and the estimation of model-based reliability in the WAIS-IV. Multivariate Behavioral Research, 48, 639e662. Gilbreath, B., & Frew, E. J. (2008). The stress-related presenteeism scale. Pueblo, CO: Colorado State University - Pueblo, Hasan School of Business, Colorado State University e Pueblo. Grandey, A. A. (2000). Emotion regulation in the workplace: A new way to conceptualize emotional labor. Journal of Occupational Health Psychology, 5, 95e110. Griffiths, A., Cox, T., Karanika, M., Khan, S., & Tomas, J. M. (2006). Work design and management in the manufacturing sector: development and validation of the work organisation assessment questionnaire. Occupational and Environmental Medicine, 63, 669e675. Gustafsson, J. E., & Balke, G. (1993). General and specific abilities as predictors of school achievement. Multivariate Behavioral Research, 28, 407e434. Health and Safety Executive. (2010). Guidance. Retrieved from http://www.hse.gov.uk/guidance/. Heise, D. R., & Bohrnstedt, G. W. (1970). Validity, invalidity, and reliability. In E. F. Borgatta (Ed.), Sociological methodology 1970 (pp. 104e129). San Francisco: Jossey-Bass. Holzinger, K. J., & Swineford, F. (1937). The bi-factor method. Psychometrika, 2, 41e54. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Model, 6, 1e55. Jarvis, C. B., MacKenzie, S. B., & Podsakoff, P. M. (2003). A critical review of construct indicators and measurement model misspecification in marketing and consumer research. Journal of Consumer Research, 30, 199e218. Karimi, L., & Meyer, D. (2014). Structural equation modeling in psychology: The history, development and current challenges. International Journal of Psychological Studies, 6, 1-11. Kline, R. B. (1998). Principles and practice of structural equation modeling. New York: The Guilford Press. LaMontagne, A. (2004). Improving OHS policy through intervention research. Journal of Occupational Health and Safety, 20(2), 107e113. Laschinger, H. K. S. (2012). Job and career satisfaction and turnover intentions of newly graduated nurses. Journal of Nursing Management, 20, 472e484.

Nurs Outlook 63 (2015) 318e330

Law, K. S., Wong, C. S., & Mobley, W. H. (1998). Towards a taxonomy of multidimensional constructs. Academy of Management Review, 23(4), 741e755. MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130e149. Marsh, H. W., Hau, K. T., & Grayson, D. (2005). Goodness of fit in structural equation models. In A. Maydeu-Olivares & J. J. McArdle (Eds.), Contemporary psychometrics: A festschrift for Roderick P. McDonald (pp. 225e340). Mahwah, NJ: Erlbaum. Raftery, Adrian E. (1995). Bayesian model selection in social research. Sociological Methodology, 25, 111e195. Parker, D. F., & Decotiis, T. A. (1983). Organizational determinants of job stress. Organizational Behavior and Human Performance, 32, 160e177. Reise, P., Moore, T. M., & Haviland, M. G. (2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544e559. Reise, S. P., Bonifay, W. E., & Haviland, M. G. (2012). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95, 129e140. Rick, J., & Briner, R. B. (2000). Psychosocial risk assessment: Problems and prospects. Occupational Medicine, 50(5), 310e314. Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. von

329

Eye & C. C. Clogg (Eds.), Latent variables analysis: Applications for developmental research (pp. 399e419). Thousands Oaks, CA: Sage. Schumacker, R. E., & Lomax, R. G. (2004). A beginner’s guide to structural equation modeling (2nd ed.) Mahwah, NJ: Lawrence Erlbaum Associates. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461e464. Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107e120. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4e70. Warr, P., Cook, J., & Wall, T. (1979). Scales for the measurement of some work attitudes and aspects of psychological well-being. Journal of Occupational Psychology, 52(2), 129e148. Wynne-Jones, G., Varnaya, A., Buck, R., Karanika-Murray, M., Griffiths, A., Phillips, C., . Main, C. J. (2009). Examination of the work organisation assessment questionnaire in public sector workers. Journal of Occupational & Environmental Medicine, 51(5), 586e593. Yung, Y. F., Thissen, D., & McLeod, L. D. (1999). On the relationship between the higher-order factor model and the hierarchical factor model. Psychometrika, 64, 113e128. Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach’s a, Revelle’s b, and McDonald’s uh: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, 123e133.

330

Nurs Outlook 63 (2015) 318e330

Appendix 1 e The WOAQ and Its Subfactor Items Item Number/Factor Quality of physical environment 1eFacilities for taking breaks 2eWork surroundings 4eExposure to physical danger 9eSafety at work 18eThe equipment/information technology that you use 20eWorkstation/work space Quality of relationship with colleagues 10eYour relationship with your coworkers (socially) 28eHow well you work with your coworkers (as a team) Quality of relationship with management 3eClear roles and responsibilities 5eSupport from line manager/supervisor 7eFeedback on your performance 11eAppreciation of efforts from line managers/ supervisors 16eSenior management attitudes 17eClear reporting line(s) 22eCommunication with line manager/supervisor 26eStatus/recognition in the workplace 27eClear workplace objectives, values, procedures Reward and recognition 12eConsultation about changes in your job 13eAdequate training for your current job 14eVariety of different tasks 21eOpportunities for promotion 23eOpportunities for learning new skills 24eFlexibility of working hours 25eOpportunities to use your skills Workload issues 6ePace of work 8eYour workload 15eImpact of family/social life on work 19eImpact of your work on family/social life