Abstract
Background: The South African Career Interest Inventory – Short (SACII-Short) is used in research settings to measure Holland’s six interest factors. Conclusions reached in studies using the instrument are subject to the measurement properties of the SACII-Short items and scales.
Objective: This study aimed to investigate the psychometric properties of the SACII-Short items and the fit of circumplex structure to the SACII-Short item and scale scores.
Method: Secondary data from South African university students and working adults (n = 673) were used. The graded response model was used to investigate the measurement properties of the items. Factor analysis was used to investigate the circumplex structure of the item and scale scores.
Results: Most of the SACII-Short items showed satisfactory measurement properties. Some concerns were observed with the item locations of the realistic and social items. The item and scale scores showed satisfactory fit to circumplex structure. The wording of some of the items could be reconsidered to increase the applicability of the content to the South African work context.
Conclusion: The SACII-Short demonstrates satisfactory psychometric properties for use in research settings. These psychometric properties support the validity of results obtained from studies that have used the SACII-Short scale scores as a proxy for vocational interests.
Keywords: measurement; reliability; validity; SACII; circumplex; vocational interests.
Introduction
Vocational interests – defined as preferences for activities or environments (Hansen & Wiernik, 2018) – have a long history of use in organisational and counselling settings (e.g. Moore, 1921; Parsons, 1909). At the start of the 20th century, organisational psychologists enthusiastically used interests to understand workplace behaviours (Hansen & Wiernik, 2018). However, by the mid to late 20th century, this enthusiasm had waned, with the measurement and interpretation of vocational interests becoming the primary domain of career counsellors (Dawis, 1991; Hansen & Wiernik, 2018). In recent years, organisational psychologists have encouraged the reintroduction of interests in understanding workplace behaviours and outcomes (e.g. Nye, Su, Rounds, & Drasgow, 2012; Wiernik, 2016) because of promising findings linking interest–environment fit with, amongst others, job performance, satisfaction and well-being (Hoff, Perlus, & Rounds, 2019; Van Iddekinge, Roth, Putka, & Lanivich, 2011; Wiernik, 2016). In this regard, Wiernik (2016) argued that ‘the best decisions can be made (in organisations) when interests are combined with measures of personality traits and ability’ (p. 156).
Researchers investigating the utility of interests in organisational settings have typically used sample groups and occupations from the United States of America (USA). Interests and their interaction with the environment are contextually bound (Einarsdóttir, Rounds, & Su, 2010; Tang, 2009), potentially limiting the generalisation of existing results outside the USA. The utility of interests in organisational settings in South Africa has received little research to date. This lack of research makes it difficult to determine if interest measurement and interpretation can be used to understand workplace behaviours in organisations in South Africa. Recent research using South African data has shown that interest–environment fit is related to burnout (Pillay, 2020) and job-hopping motives (Hall, Morgan, & Redelinghuys, 2022). These studies suggest that results from the USA might generalise to organisations in South Africa and that interests are a potentially useful predictor of workplace behaviours in this context.
Problem statement
Vocational interests are a latent construct that exists as a theoretical abstraction or pragmatic fiction (see Yarkoni, 2022). Their latent status means that an indirect measure is required to obtain observable scores that can serve as a proxy for the unobservable latent scores. These proxy scores and their validity are directly tied to the psychometric properties of the measure used to obtain them (Nunnally & Bernstein, 1994; Yarkoni, 2022). Proxy scores obtained from measures with poor psychometric properties can seriously reduce the trustworthiness of conclusions in research that has used these scores. The two studies mentioned previously obtained interest scores via the short version of the South African Career Interest Inventory (SACII-Short) – a South African measure of John Holland’s six interest factors, which are labelled as realistic, investigative, artistic, social, enterprising and conventional (RIASEC). Several studies have supported the psychometric properties of the original 142-item version of the SACII in South Africa (e.g. Morgan, De Bruin, & De Bruin, 2015; Rabie & Naidoo, 2019; Van Vuuren, 2022). However, little research is available on the psychometric properties of the SACII-Short.
Research question
Studies that have investigated the psychometric properties of the SACII-Short have focused almost exclusively on the validity of the RIASEC scale scores (Hall et al., 2022; Morgan & De Bruin, 2019). This exclusive focus is a problem because it is necessary to investigate the measurement properties of the items in addition to the scale scores to obtain a complete understanding of the psychometric properties of the SACII-Short. Against this background, I set out to investigate (1) the measurement properties of the SACII-Short items using the graded response model (GRM) and (2) the structural validity of the SACII-Short items and scale scores. This investigation holds important implications for the validity of results from studies that have used the SACII-Short to operationalise interests and for continued use of the SACII-Short in research.
Holland’s RIASEC interest factors
The SACII-Short measures John Holland’s six RIASEC vocational interest factors. Holland (1958, 1959, 1997) believed that the RIASEC interest factors are a mixture of personal adjustments, values, attitudes and motivation and thus represent a broad set of adjustive orientations towards the environment (Holland, 1958, 1997). The broad nature of these six factors means that they capture general similarities across more fine-grained interest factors and environments (see Wiernik, 2016). It is comparable to using the five personality factors instead of facets of these factors. I do not want to create the impression here that the RIASEC interest factors are the best way to conceptualise the interest space. There are advantages and disadvantages when operating in low-dimensional versus high-dimensional latent space (see Wiernik, 2016). The RIASEC factors are simply one way that can be used to measure broad tendencies – dimensions that capture shared variance of many similar basic interests – that may or may not be helpful depending on the needs of the researcher, counsellor or client (see Morgan et al., 2019).
Circumplex structure of the RIASEC interest factors
The RIASEC interest factors and their indicators have a circumplex structure (see Glosenberg, Tracey, Behrend, Blustein, & Foster, 2019; Wiernik, 2016), which Holland called a hexagon. Circumplex structure means that the interrelationships between the RIASEC correlation coefficients of the scale scores show a pattern of rising and falling (Guttman, 1954) as one moves away from and then towards the RIASEC correlation matrix diagonal. This pattern implies that the interest factors are located on the circumference of a circle in a two-dimensional space (Fabrigar, Visser, & Browne, 1997). Most of Holland’s model and theoretical predictions depend on this circumplex structure and the correct RIASEC ordering holding. For example, congruence (i.e. interest–environment fit) is based on the distance between a person’s RIASEC interest scores and the environment RIASEC interest scores in the interest space (Holland, 1997). Differentiation and consistency, two derived markers of interest clarity, also depend on circumplex structure and the correct RIASEC ordering (Holland, 1997).
Development of the South African Career Interest Inventory
Morgan and De Bruin (2019) used data collected by Morgan (2014) for the 169-item version of the SACII to develop the SACII-Short. A random sample1 of 1000 participants from the 1543 participants in the data set was selected to investigate the SACII item functioning. The authors applied the Rasch partial credit model and principal components analysis. The Rasch partial credit model was used to investigate the category functioning, item fit and item difficulties or locations of the items within each scale (i.e. the item measurement properties). Hereafter, principal components analysis was used to reduce the item correlation matrix to three orthogonal linear components representing the general factor and the two circumplex factors (see Tracey, 2000). The component loadings on the second and third linear components were transformed from rectangular to polar coordinates, giving each item’s angular location and vector length (Morgan & De Bruin, 2019). The vector length is the square root of the communality coefficient on the circumplex factors and thus represents the distance from the origin to the circle’s circumference. Polar coordinates are used to select items that match the theoretical RIASEC ordering and have the largest vector length (see Tracey, 2000).
The Rasch partial credit model showed that 17 items were too easy or difficult to endorse and that 34 items did not fit the Rasch model. No items had angular locations that deviated markedly from the other items in their respective scales, and 29 items had vector lengths < 0.30 (approximately 9% variance). Morgan and De Bruin (2019) used the information above and theoretical considerations to select five items for each RIASEC scale. Reliability coefficients ranging from 0.80 to 0.89 were found for the six scale scores. Morgan and De Bruin then obtained a new sample group of 183 participants to cross-validate the fit of circumplex structure to the SACII-Short scale scores. Circumplex structure and RIASEC ordering also held in the second sample group, although the reliability coefficients of the scale scores were lower, ranging from 0.72 to 0.83. The Rasch model was not applied to the items in this second sample group. Therefore, the measurement properties of the items within each SACII-Short scale remain relatively unknown.
Emoji response format
The SACII-Short initially used a Likert-type response format ranging from ‘strongly disagree’ to ‘strongly agree’, where a participant indicated the extent to which ‘I am interested in and would like to do the following activity in my job’ (Morgan & De Bruin, 2019). Phan, Amrhein, Rounds and Lewis (2019) suggested that interest items use an emoji response format instead because of the inherent affective or emotional aspect involved when rating interest activities (e.g. Silvia, 2001; Strong, 1943). The response format of the SACII-Short was changed to a dislike–like rating scale. In this scale, a participant indicates their preference for each of the activities in the items below according to the emoji response categories (see Naidu, 2020 for more details). The emoji faces are presented alongside their corresponding interpretation (i.e. strongly dislike, dislike, unsure, like and strongly like).
Naidu (2020) found slightly better reliability of the emoji item scores and a somewhat better fit of circumplex structure to the RIASEC scale scores than the Likert-type response format. Pillay (2020) found support for the circumplex structure in a different sample group who completed the Likert-type response format. Naidu (2020) found reliability coefficients ranging from 0.76 to 0.90, and Pillay (2020) found reliability coefficients ranging from 0.82 to 0.94. Juliana and Gunawan (2021) translated the SACII-Short to Indonesian. They investigated the convergent validity of the scale scores with the O*Net Interest Profiler, a measure of the RIASEC interest factors developed in the USA. The authors reported reliability coefficients ranging from 0.76 to 0.90 and correlation coefficients of the RIASEC scale scores with the O*Net Interest Profiler scale scores ranging from 0.73 to 0.79.
Summary
The SACII-Short is used in research to measure John Holland’s RIASEC interest factors. The validity of these RIASEC interest scores depends on the measurement or psychometric properties of the SACII-Short. This study therefore set out to investigate (1) the measurement properties of the SACII-Short items and (2) the structural validity of the SACII-Short item and scale scores.
Method
Sample
I used secondary data of participants who completed the emoji response format of the SACII-Short. Makhura (2022), Naidu (2020) and Strydom (2022) collected the primary data.2 These three studies used the emoji response format of the SACII-Short. The secondary data consists of 673 South African participants. The participants’ ages ranged from 18 to 61 years, with a mean and standard deviation of 24.76 and 7.16 and skewness of 2.15. The sample consisted of slightly more self-identified women (n = 342, 59%) than self-identified men (n = 233, 41%). Most of the participants identified as black African (n = 504, 76%), followed by Indian and Asian (n = 65, 10%), white (n = 50, 8%), mixed-race (n = 38, 6%) and Middle Eastern (n = 2, 0%). Approximately one-third of the participants indicated that they had full-time employment (n = 203, 30%), with the remainder either having part-time employment, contract employment, self-employment or no employment (n = 465, 70%).
Procedure
The three previously mentioned studies used different procedures. Naidu (2020) collected data from university students and working adults using physical copies of the SACII-Short. Makhura (2022) and Strydom (2022) used digital copies of the SACII-Short hosted on Google Forms. For the former, permission was obtained from the relevant higher education institution to advertise participation in the study on the institutional student learning platform. A Google Forms link was included in the advertisement. The latter research obtained participants via the Prolific platform. These participants were paid approximately R30 to participate.
Instrument
The SACII-Short was previously discussed and is therefore not presented here. Items are available in the online supplement (https://osf.io/hbkmt/?view_only=cb84a73eb7c74c45b3d805347b40a562).
Analysis
The measurement properties of the SACII-Short items were investigated using the GRM (Samejima, 1969). This model is a polytomous generalisation of the two-parameter logistic model for ordered item responses. I first investigated the slope or discrimination parameters (a) and the threshold parameters (bij). To assist in interpreting these threshold parameters, I also included the item response function generalised item difficulty as a measure of the central location for each item (Ali, Chang, & Anderson, 2015). I then inspected the item and test information to determine the reliable range of the latent trait measured by the items. This inspection used the area under the curve for the information function bounded between –4.00 and 4.00 units. The fit of the model to each item was investigated using the plausible-value imputations of the Q1 statistic (Chalmers & Ng, 2017) and the signed χ2 test (Orlando & Thissen, 2000) and local dependence was investigated using Yen’s (1984) Q3 statistic. Bias-corrected and accelerated bootstrapped confidence intervals (BCA) using 5000 random samples were used to determine the statistical significance of the Q3 statistic. Recommended adjustments to the expected value of the Q3 statistic were made.
The fit of circumplex structure to the item responses was investigated indirectly by fitting a full-information factor analysis (i.e. multidimensional GRM) with three factors extracted. The unrotated factor loadings were then rotated to an orthogonal target matrix with targets on the first factor, given as 0.71, and targets on the second and third factors, given as the sine and cosine rectangular coordinates to the hexagonal model. The fit of the model to the items was only investigated using the signed χ2 test because the plausible-value imputations of the Q1 statistic are not available for multidimensional models. The rectangular coordinates on the second and third factors were converted into polar coordinates to obtain the item angular locations and the communality coefficients on these two factors. The fit of a circumplex model to the RIASEC factors was investigated using Browne’s (1992) stochastic circular modelling approach (CCSM). Expected a posteriori factor scores from the GRM model applied to each RIASEC scale were used in the analysis. I used one free beta parameter in the Fourier series function (i.e. m = 1) and allowed scaling parameters, angular locations and unique latent RIASEC variances to be freely estimated. Maximum likelihood estimation with robust standard errors (MLR) was used to obtain the parameter estimates.
All analyses were conducted using R version 4.1.3 (R Core Team, 2021). The GRMs were fit in the mirt package version 1.35.1 (Chalmers, 2012) using EM estimation. The CCSM was fit in the lavaan package version 0.6-10 (Rosseel, 2012). BCA were estimated using the coxed package version 0.3.3 (Kropko & Harden, 2020).
Ethics
Makhura (2022), Naidu (2020), and Strydom (2022) received ethical clearance from the College of Business and Economics or the Department of Industrial Psychology and People Management Research Ethics Committee to collect data for their studies. This study obtained ethical clearance from the Department of Industrial Psychology and People Management Research Ethics Committee to use the secondary data. The ethical clearance codes were CBEREC18JBS11, IPPM-2020-420(M), IPPM-2021-515(M) and IPPM-2021-477. More information on the ethical considerations can be obtained from the three previously mentioned studies. Participants were provided with a detailed participant information sheet explaining the nature and purpose of the study and the conditions of participation. Those who agreed to participate were also required to complete a consent form. No personally identifiable information was collected so that all respondents could remain anonymous.
Results
Table 1 presents the model fit statistics and reliability coefficients for the five SACII-Short scales. The model fit statistics were mostly satisfactory for the realistic, artistic and enterprising scales and somewhat less satisfactory for the investigative, artistic and conventional scales.3 The marginal reliability coefficients ranged from 0.80 to 0.88, with a mean marginal reliability of 0.84. Similar results were obtained when calculating the coefficient omega total on the raw item responses, with these coefficients ranging from 0.77 to 0.88 and having a mean coefficient omega total of 0.84. Table 2 presents the GRM parameters and item fit statistics for the six SACII-Short scales.
| TABLE 1: RIASEC model fit statistics and reliability coefficients. |
| TABLE 2: Graded response model item parameters and item fit. |
Realistic scale
The item slope parameters of the realistic scale ranged from 2.08 (item R1) to 3.29 (item R4), with a mean slope parameter of 2.91. The easiest and most difficult items to endorse were item R1, with a generalised difficulty of 0.53, and item R4, with a generalised difficulty of 0.89. The mean generalised difficulty was 0.72. Items R1 (p = 0.020) and R5 (p = 0.033) showed statistically significant misfit.4 The item wordings for these two items are ‘do routine maintenance of machines’ and ‘weld metal parts together’. Inspection of item and response characteristic curves showed that participants who scored low on the realistic trait tended to score higher than expected on item R1, and that misfit was evident in the first three response categories. The item characteristic curve showed little observable misfit for item R5. However, the category characteristic curves showed some misfit for the fourth and fifth response categories. The realistic items showed the most test information between approximately –1.00 and 3.00 units on the latent trait, with item R1 having the lowest item information with an area under the item information curve of 5.47 units. Yen’s Q3 statistic suggested that there was minor local dependence between items R1 and R2 (Q3 = 0.16), R1 and R3 (Q3 = 0.20), R2 and R4 (Q3 = –0.14) and R3 and R5 (Q3 = –0.16).
Investigative
The investigative item slope parameters ranged from 1.62 (item I3) to 3.00 (item I2), with a mean slope parameter of 2.33. The easiest and most difficult items to endorse were item I1, with a generalised difficulty of –0.10, and item I4, with a generalised difficulty of 0.69. The mean generalised difficulty was 0.21. Item I4 (p = 0.015) showed statistically significant misfit. Inspection of the item and response characteristic curve showed that participants who scored near the mean on the investigative trait scored lower than expected on I4 and that misfit was evident for the fourth and fifth response categories. Item I3 did not show a statistically significant misfit (p = 0.058). However, visual inspection of the response category curves showed a misfit for the first three response categories. The item wordings for these two items are ‘study insects in a laboratory’ and ‘research ancient monuments’. The investigative items showed the most test information between approximately –2.00 and 2.00 units on the latent trait. Items I3 and I4 had the lowest item information, with areas under the item information curve of 3.63 and 4.69 units. Yen’s Q3 statistic suggested that there was minor local dependence between items I1 and I2 (Q3 = 0.24), I1 and I4 (Q3 = –0.17), I1 and I5 (Q3 = –0.13) and I2 and I3 (Q3 = –0.12) and large local dependence between items I2 and I5 (Q3 = –0.24), I3 and I4 (Q3 = 0.29), I3 and I5 (Q3 = 0.29) and I4 and I5 (Q3 = 0.29).
Artistic
The slope parameters for the artistic items ranged from 1.14 (A1) to 2.23 (A5), with a mean slope parameter of 1.74. The easiest item to endorse was item A3, with a generalised difficulty of –0.89, and the most difficult item to endorse was item A5, with a generalised difficulty of –0.18. The mean generalised difficulty was –0.45. None of the artistic items showed a statistically significant misfit. However, an inspection of item characteristic curves showed that for item A5, participants who scored lower on the trait scored lower than expected on the item and those who scored higher on the trait scored higher than expected on the item. The wording for this item is ‘write poetry’. The artistic scale items showed the most test information between approximately –2.50 and 2.00 on the latent trait. Item A1 had the lowest item information, with an area under the curve of 2.19. Yen’s Q3 statistic suggested that there was little to no local dependence.
Social
The item slope parameters for the social items ranged from 1.34 (S2) to 3.03 (S5), with a mean slope parameter of 1.81. The easiest and most difficult items to endorse were item S2, with a generalised difficulty of –1.53, and item S4, with a generalised difficulty of –0.43. The mean generalised difficulty was –0.99. No items showed a statistically significant misfit. However, an inspection of item characteristic curves showed that for item S5, participants who scored lower on the trait tended to score lower than expected on the item and that participants who scored high on the trait scored higher than expected on the item. Inspection of the response characteristic curves for this item showed a misfit in the first and fourth response categories. The wording for this item is ‘teach people life skills’. The social items showed the most test information on the latent trait between approximately –3.00 and 2.00 units. Item S2 had the lowest item information, with an area under the item information curve of 2.91 units. Yen’s Q3 statistic showed that there was little to no local dependence.
Enterprising
The enterprising item slope parameters ranged from 1.68 (E5) to 3.24 (E2), with a mean slope parameter of 2.45. The easiest and most difficult items to endorse were item E1, with a generalised difficulty of –1.39, and item E5, with a generalised difficulty of –0.54. The mean generalised difficulty was –1.01. Item E2 (p = 0.036) showed statistically significant misfit. Inspection of the item and category characteristic curves showed little observable misfit for this item but that there was a slight misfit for the first response category. The wording for this item is ‘increase efficacy of a business’. The enterprising items showed the most test information between approximately –3.00 and 1.50 units on the latent trait. Item E5 had the lowest item information, with an area under the item information curve of 4.03 units. Yen’s Q3 statistic suggested that there was minor local dependence between items E2 and E4 (Q3 = –0.18) and large local dependence between items E4 and E5 (Q3 = 0.35).
Conventional
The conventional item slope parameters ranged from 2.11 (C1) to 3.20 (C4), with a mean slope parameter of 2.57. The easiest and most difficult items to endorse were item C2, with a generalised difficulty of –0.26 and item C1, with a generalised difficulty of –0.05. The mean generalised difficulty was –0.13. None of the items showed a statistically significant misfit. Inspection of category characteristic curves suggested some misfit for the first response category across all the conventional items. The conventional items showed the most test information between approximately –2.00 and 2.00 units on the latent trait. Item C1 had the lowest item information, with an area under the item information curve of 5.56 units. Yen’s Q3 statistic suggested that there was minor local dependence between items C2 and C5 (Q3 = –0.19), C3 and C4 (Q3 = –0.23), C3 and C5 (Q3 = –0.13) and large local dependence between items C2 and C3 (Q3 = 0.51), C2 and C4 (Q3 = –0.29) and C4 and C5 (Q3 = 0.32).
Item locations and vector lengths
The three-factor model produced an M2 statistic of 3313.03 on 348 degrees of freedom and an RMSEA with 90% confidence intervals of 0.11 [0.11, 0.12]. The standardised root mean residual (SRMR), Tucker–Lewis index (TLI) and comparative fit index (CFI) were 0.07, 0.81 and 0.84, respectively. Table 3 presents each item’s target rotated factor loadings, polar coordinates, multidimensional discrimination indices, item RMSEAs and signed χ2 fit statistic p-values. The items showed a circular structure with the correct RIASEC ordering. The communality coefficients (i.e. h2) of the items on the two circumplex factors ranged from 0.08 (item S3) to 0.55 (item R5), with a mean communality coefficient of 0.25. The multidimensional discrimination index ranged from 0.82 (item S3) to 3.33 (item R5), with a mean multidimensional discrimination index of 1.82. None of the items showed a statistically significant misfit.
| TABLE 3: Target-rotated full information item factor analysis of the 30 SACII-Short items. |
RIASEC circumplex structure
Figure 1 presents the estimated RIASEC angular locations and communality coefficients (i.e. h2). Inspection of this figure shows that the circumplex model had a good fit to the data and that the correct RIASEC ordering emerged (fit statistics are reported in the figure note). The communality coefficients ranged from 0.28 (realistic) to 0.72 (investigative), with a mean communality coefficient of 0.52. As seen in Figure 1, the communality coefficient of the realistic and social factors was relatively low compared with the other four factors. This suggests that the fit of a circumplex model to the realistic and social factors was somewhat wanting in this sample group and that these two factors had more unique variance than circumplex variance. The general factor accounted for 55% of the common variance, and the circumplex factors accounted for 45% of the common variance.
 |
FIGURE 1: RIASEC angular locations and communality coefficients. |
|
Discussion
This study set out to investigate (1) the measurement properties and (2) the fit of circumplex structure to the SACII-Short item and scale scores. In the following sections, I discuss the results for the SACII-Short as a whole and then discuss the results for the SACII-Short scales and items.
Overall results
The overall results provide evidence in support of the psychometric properties of the SACII-Short. The SACII-Short items showed mostly satisfactory reliability, given that each scale consists of only five items (e.g. Nunnally & Bernstein, 1994). The GRM generally fit the items well and inspection of item and category characteristic curves did not show any serious concerns. The circumplex model fit the RIASEC factor scores well, and the correct RIASEC ordering of these factor scores was observed. The realistic and social factors did show relatively low communality coefficients. Morgan and De Bruin (2019) found low communality coefficients for the realistic scale in their first sample but not in the second sample and low communality coefficients for the social scale in their second sample but not in their first sample. It is therefore unclear if this is a problem with the realistic and social scales or if it is a sample-specific finding. The general factor accounted for just over half of the common variance in the circumplex model in this study. Similar results were obtained by Morgan and De Bruin (2019). The relative dominance of the general factor over the circumplex factors is not unexpected, as vocational interests are known to have a large interpretable general factor (e.g. Tracey, 2012; Wiernik, 2016).
Results for the scales and items
Most SACII-Short items appeared to have satisfactory measurement properties, supporting the overall reliability and validity of the SACII-Short scale scores. There were, however, a few concerns that should be considered. The first concern is that coverage of the latent trait for some scales could be improved as indicated by the information functions. The SACII-Short items should ideally be reliable (i.e. provide information) between approximately –2.00 and 2.00 units on the latent trait, because this range would cover most trait scores in the population (about 95%, assuming a normal distribution of the latent scores). Items in the investigative, artistic and conventional scales showed satisfactory coverage of the latent trait in this range. In contrast, the realistic items provided little information at the lower end of the trait and the social and enterprising items showed little information at the upper ends of the trait. The mean generalised difficulties showed that the realistic items were generally too difficult to endorse and that the social and enterprising items were generally too easy to endorse. These generalised difficulties are one explanation for the limited trait coverage over the –2.00 to 2.00 range. Similar results have been obtained for the SACII (e.g. Morgan, 2014; Morgan et al., 2021). It is therefore possible that this is a problem inherited from the SACII. Increased trait coverage of these scales in the SACII-Short could be addressed by including easier or more difficult-to-endorse items.
The second concern is that approximately seven of the SACII-Short items were potentially problematic. Three of these items were from the social scale. These three items generally had lower slope parameters, meaning that they did not adequately discriminate between those who scored low and high on the latent trait or lower communality coefficients on the circumplex factors, meaning that there was a relatively large proportion of unmodelled variance not attributed to the three-factor model in these items. Morgan and De Bruin (2019) did not find problematic communality coefficients for these items in their first sample group. However, they used principal components (all of the variance) analysis rather than factor analysis (the common variance), which means that the communality coefficients cannot be directly compared. Items I3 (‘research ancient monuments’) and A1 (‘act in a play’) consistently emerged as potentially problematic items. The term ‘ancient monuments’ is intended to serve as a proxy for archaeology activities (Morgan, 2014). Alternative terms that could be considered are fossils or bones, origins of human life and cultures. The term ‘play’ in item A1 is intended to refer to theatre productions (Morgan, 2014). However, it has a possible dual meaning, as it could refer to play as an activity, playing games or play as in a stage performance. It might be preferable to use theatre, or musical productions or performing arts.
A third concern is that the enterprising and conventional items and factors cluster relatively close together on the circle. The same clustering was found by Morgan and De Bruin (2019) and in other studies that have used the SACII (e.g. Morgan, 2014; Morgan et al., 2021). The problem with this clustering is that it makes it difficult to distinguish between these two interest factors in a person’s responses (e.g. multicollinearity). For example, a person who scores high on the conventional interest might score this way because of an overlap with a high score on the enterprising interest rather than because of a true conventional interest. It also has implications for calculating congruence coefficients that rely on an approximate hexagon or regression-based statistical techniques that obtain profile patterns. That said, it is unclear if this clustering represents the RIASEC structure in South Africa, if it is sample related or if it is because of the enterprising and conventional items in the SACII-Short and SACII. Each of these options has different implications and solutions.
Implications
The results from this study support the use of the SACII-Short in its current form for research purposes. This in turn supports the trustworthiness of the RIASEC scale scores obtained from the SACII-Short in the past research (e.g. Hall et al., 2022; Naidu, 2020; Pillay, 2020). However, the results also show that the SACII-Short could be improved in at least three ways. Firstly, easier or more difficult items should be added to the realistic, social and enterprising scales to increase coverage of the relevant trait. As it currently stands, the mean of the raw social and enterprising scale scores will almost always be higher than the mean for the realistic scale scores. It is unclear whether these differences represent real trait differences or are because of differences in the difficulties of the items. Regardless, it would be advisable to standardise (centre and scale) the scale scores, depending on the statistical techniques used and the purpose of the data, before using them as-is. This standardisation is especially important when using the scale scores to calculate congruence coefficients. Secondly, there is evidence that some items should be removed or reworked, as they are somewhat detrimental to the psychometric properties of the SACII-Short. Thirdly, clarity should be obtained on potential reasons for the enterprising–conventional clustering so that corrections can be made to the SACII-Short if necessary. However, this is more of a theoretical investigation of the RIASEC structure in South Africa than a SACII-Short investigation.
Limitations
The data collected on the SACII-Short thus far have used sample groups exclusively from the adult population. This means that there is little to no evidence for the psychometric properties of the SACII-Short in adolescent sample groups. Therefore, researchers should be hesitant to use the SACII-Short with adolescents in research settings without obtaining preliminary evidence for its psychometric properties and fit of a circumplex model to the RIASEC item and scale scores. The sample group used in this study, as with most other studies conducted on the SACII, tended to over-represent occupations in the social–enterprising and enterprising–conventional spaces. The problem with this over-representation is that the realistic, investigative and artistic scale scores are subject to potential range restriction. This range restriction in turn can have implications for the statistical techniques used in this study, such as reducing the amount of statistical information available when analysing specific model parameters.
Conclusion
This study set out to investigate (1) the measurement properties of and (2) the fit of circumplex structure to the SACII-Short item and scale scores. Results support usage of the SACII-Short scale scores for research purposes when using adult sample groups.
Acknowledgements
The author would like to thank Ms Naidu, Mr Makhura and Ms Strydom for access to their data and Prof. Wiernik for his assistance with analysing circumplex models.
Competing interests
I am an editorial member of the African Journal of Career Development.
Author’s contributions
I declare that I am the sole author of this research article.
Ethical considerations
Ethical clearance to conduct this study was obtained from the University of Johannesburg, Department of Industrial Psychology and People Management Research Ethics Committee. (ref. nos. CBEREC18JBS11, IPPM-2020-420(M), IPPM-2021-515(M) and IPPM-2021-477).
Funding information
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Data availability
Data used in this study can be obtained from the corresponding author.
Disclaimer
The views and opinions expressed in this article are those of the author and do not necessarily reflect the official policy or position of any affiliated agency of the author.
References
Ali, U.S., Chang, H.-H., &Anderson, C.J. (2015). Location indices for ordinal polytomous items based on item response theory. ETS Research Report Series, 2015(2), 1–13. https://doi.org/10.1002/ets2.12065
Browne, M.W. (1992). Circumplex models for correlation matrices. Psychometrika, 57(4), 469–497. https://doi.org/10.1007/BF02294416
Chalmers, R.P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
Chalmers, R.P., & Ng, V. (2017). Plausible-value imputation statistics for detecting misfit. Applied Psychological Measurement, 41(5), 372–387. https://doi.org/10.1177/0146621617692079
Dawis, R.V. (1991). Vocational interests, values, and preferences. In M.D. Dunnette & L.M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 2, pp. 833–871). Thousands Oaks, CA: Consulting Psychologists Press.
Einarsdóttir, S., Rounds, J., & Su, R. (2010). Holland in Iceland revisited: An emic approach to evaluating U.S. vocational interest models. Journal of Counseling Psychology, 57(3), 361–367. https://doi.org/10.1037/a0019685
Fabrigar, L.R., Visser, P.S., & Browne, M.W. (1997). Conceptual and methodological issues in testing the circumplex structure of data in personality and social psychology. Personality and Social Psychology Review, 1(3), 184–203. https://doi.org/10.1207/s15327957pspr0103_1
Glosenberg, A., Tracey, T.J.G., Behrend, T.S., Blustein, D.L., & Foster, L.L. (2019). Person-vocation fit across the world of work: Evaluating the generalizability of the circular model of vocational interests and social cognitive career theory across 74 countries. Journal of Vocational Behavior, 112, 92–108. https://doi.org/10.1016/j.jvb.2019.01.002
Guttman, L. (1954). A new approach to factor analysis: The Radex. In P.F. Lazarsfeld (Ed.), Mathematical thinking in the social sciences (pp. 258–348). New York, NY: Free Press.
Hall, A.A., Morgan, B., & Redelinghuys, K. (2022). The relationship between job-hopping motives and congruence. SA Journal of Industrial Psychology/SA Tydskrif vir Bedryfsielkunde, 48(0), a1938. https://doi.org/10.4102/sajip.v48i0.1938
Hansen, J.-I.C., & Wiernik, B.M. (2018). Work preferences: Vocational interests and values. In D.S. Ones, N. Anderson, C. Viswesvaran, & H.K. Sinangil (Eds.), The SAGE handbook of industrial, work & organizational psychology: Personnel psychology and employee performance (pp. 408–448). Thousands Oaks, CA: Sage Reference.
Hoff, K.A., Perlus, J.G., & Rounds, J. (2019). Vocational interests: Revisiting assumptions about their development and what they predict. In J.A. Athanasou & H.N. Perera (Eds.), International handbook of career guidance (2nd ed., pp. 673–692). Germany: Springer.
Holland, J.L. (1958). A personality inventory employing occupational titles. Journal of Applied Psychology, 42(5), 336–342. https://doi.org/10.1037/h0047330
Holland, J.L. (1959). A theory of vocational choice. Journal of Counseling Psychology, 6(1), 35–45. https://doi.org/10.1037/h0040767
Holland, J.L. (1997). Making vocational choices: A theory of vocational personalities and work environments (3rd ed.). Odessa, FL: Psychological Assessment Resources.
Juliana, N., & Gunawan, W. (2021). Uji validitas dan reliabilitas South African career interest inventory short. Jurnal Psikologi, 17(2), 191–204. https://doi.org/10.24014/jp.v17i2.12676
Kropko, J., & Harden, J.J. (2020). coxed: Duration-based quantities of interest for the Cox proportional hazards model. Retrieved from https://CRAN.R-project.org/package=coxed
Makhura, R.F. (2022). The relationship between vocational interests and career indecision in university students. Master’s dissertation in preparation. Johannesburg: University of Johannesburg.
Moore, B.V. (1921). Personnel selection of graduate engineers: The differentiation of apprentice engineers for training as salesmen, designers, and executives of production. Psychological Monographs, 30(5), i–85. https://doi.org/10.1037/h0093191
Morgan, B. (2014). Toward the development and evaluation of the South African Career Interest Inventory. Unpublished doctoral dissertation. Johannesburg: University of Johannesburg.
Morgan, B., & De Bruin, G.P. (2019). Development and evaluation of a short RIASEC interest inventory. International Journal of Educational and Vocational Guidance, 19, 437–454. https://doi.org/10.1007/s10775-019-09387-2
Morgan, B., De Bruin, G.P., & De Bruin, K. (2015). Constructing Holland’s hexagon in South Africa: Development and initial validation of the South African career interest inventory. Journal of Career Assessment, 23(3), 493–511. https://doi.org/10.1177/1069072714547615
Morgan, B., Naidoo, A. V., Henn, C., & Rabie, S. (2019). Using interests and Holland’s model of vocational personality types in career counselling: Some thoughts and research findings. In J. G. Maree (Ed.), Handbook of innovative career counselling (pp. 485–503). Switzerland: Springer.
Morgan, B., Teye-Kwadjo, E., Asumeng, M., Rabie, S., Naidoo, A.V., & De Bruin, G.P. (2021). Psychometric properties of the African career interest inventory in Ghana. International Journal of Educational and Vocational Guidance. https://doi.org/10.1007/s10775-021-09499-8
Orlando, M., & Thissen, D. (2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24(1), 50–64. https://doi.org/10.1177/01466216000241003
Naidu, M. (2020). Emoji’s as response anchors in the South African Career Interest Inventory. Unpublished master’s dissertation. Johannesburg: University of Johannesburg.
Nye, C.D., Su, R., Rounds, J., & Drasgow, F. (2012). Vocational interests and performance: A quantitative summary of over 60 years of research. Perspectives on Psychological Science, 7(4), 384–403. https://doi.org/10.1177/1745691612449021
Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
Parsons, F. (1909). Choosing a vocation. Boston, MA: Houghton Mifflin Company.
Phan, W.M.J., Amrhein, R., Rounds, J., & Lewis, P. (2019). Contextualizing interest scales with emojis: Implications for measurement and validity. Journal of Career Assessment, 27(1), 114–133. https://doi.org/10.1177/1069072717748647
Pillay, A. (2020). The relationship between person-environment fit and burnout: A vocational interest perspective. Unpublished master’s dissertation. Johannesburg: University of Johannesburg.
Rabie, S., & Naidoo, A.V. (2019). Validating the adaptation of the first career measures in isiXhosa: The South African career interest inventory – isiXhosa version. South African Journal of Psychology, 49(1), 109–121. https://doi.org/10.1177/0081246318772419
R Core Team. (2021). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/
Rosseel, Y. (2012). lavaan: An R package for structural equation modelling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34, 1–97. https://doi.org/10.1007/BF03372160
Silvia, P.J. (2001). Interest and interests: The psychology of constructive capriciousness. Review of General Psychology, 5(3), 270–290. https://doi.org/10.1037/1089-2680.5.3.270
Strong, E.K., Jr. (1943). Vocational interests of men and women. Redwood City, CA: Stanford University Press.
Strydom, N. (2022). The moderating effect of job crafting on the relationship between interest-environment fit and burnout. Master’s dissertation in preparation. Johannesburg: University of Johannesburg.
Tang, M. (2009). Examining the application of Holland’s theory to vocational interests and choices of Chinese college students. Journal of Career Assessment, 17(1), 86–98. https://doi.org/10.1177/1069072708325743
Tracey, T.J.G. (2000). Analysis of circumplex models. In H.E.A. Tinsley & S.D. Brown (Eds.), Handbook of applied multivariate statistics and mathematic modelling (pp. 641–664). San Diego: Academic Press.
Tracey, T.J.G. (2012). Problems with single interest scales: Implications of the general interest factor. Journal of Vocational Behavior, 81(3), 378–384. https://doi.org/10.1016/j.jvb.2012.10.001
Van Iddekinge, C.H., Roth, P.L., Putka, D.J., & Lanivich, S.E. (2011). Are you interested? A meta-analysis of relations between vocational interests and employee performance and turnover. Journal of Applied Psychology, 96(6), 1167–1194. https://doi.org/10.1037/a0024343
Van Vuuren, N. (2022). A psychometric investigation of the South African career interest inventory – isiXhosa version among secondary school learners using the Rasch measurement model. Unpublished master’s dissertation. Stellenbosch University.
Wiernik, B.M. (2016). The nomological network of classic and contemporary career preferences. Unpublished doctoral dissertation. Minnesota: University of Minnesota.
Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, e1. https://doi.org/10.1017/S0140525X20001685
Yen, W.M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125–145. https://doi.org/10.1177/014662168400800201
Footnotes
1. Random numbers were generated in Excel.
2. Makhura was investigating the relationship between interests and career indecision of university students, and Strydom was investigating the relationship between interests, burnout and job crafting. Strydom’s data included participants from four countries. I only used the responses from South African participants.
3. Fit statistics should not be over-interpreted, as there are no universal cut-off criteria for what constitutes acceptable fit.
4. I only inspected the statistical significance of the plausible-value imputations of the Q1 statistic across all the unidimensional models.
|
Hide
Show all
Email this article (Login required)
