About The ARDA | Tools | FAQs | Contact Us
Post-Hoc Scales

It is often possible, while engaged in secondary analysis of existing data, to create multi-item scales that measure concepts more reliably than any one item can do. They measure somewhat general concepts that may include subscales and other more limited concepts. It is generally best to do this on the basis of theory, although at times a more empirical approach may be appropriate. Our example will be the 1963 Survey of Northern California Church Bodies, conducted by Charles Y. Glock and Rodney Stark. Sixteen years after the data were collected, Stark returned to the data in collaboration with William Bainbridge, and one goal was to refine conceptions of sectarian tension, the extent to which religious groups distance themselves from secular norms. Several items relevant to this issue had been included in the original questionnaire, but without the theoretical refinements of the intervening sixteen years that suggested a way of organizing them.

By the end of the 1970s, sectarian tension was being conceptualized as a socio-cultural dissonance between a religious group and the wider society, perhaps described as subcultural deviance but with the understanding that all religious groups exhibit some tension, some more than others. Thus, sectarian tension is a dimension of variation, with mainstream denominations at the low-tension end, and extreme sects at the high-tension end.

It is general practice in those social sciences that think in terms of scales and dimensions to give them names and clear definitions. But it is also common for such scales to have internal structure, often called subscales, that can even be described as separate dimensions when analysis is focusing on fine details. Dimension, of course, is a metaphor from the physical world, that is especially apt when we graph data – a graph represents the data as if it were a physical shape. In talking about physical objects, we often speak of the volume of the object, as if volume were one dimension, and yet objects are three-dimensional. Similarly, a one-dimensional scale can have multiple subscales within it, that can be thought of as dimensions in their own right, stretching the physical metaphor but not violating it.

Sectarian tension was conceptualized as a general dimension with three sub-aspects: difference, antagonism, and separation. The more the beliefs and practices of a religious groups differ from those of the majority in society, it exists in higher tension. The more negatively members of the group feel about outsiders, and the more negatively outsiders feel about members, the greater the hostility and thus the tension. Finally, separation refers to social distance between members and non-members, and social encapsulation of the members – for example having fellow members as their friends.

A very large number of items in the questionnaire related to the three aspect of sectarian tension, so many that it proved useful to divide them into groups. For example, members of high-tension religions differed from the average person in terms of norms, behavior and beliefs, so each of these could be handled as a sub-sub-scale. The five items that reflect belief differences provide good illustrations of some issues one faces when constructing a scale from items in an existing questionnaire, that were not originally deigned to be a scale, and thus were not all in the same format. Here are the five items:

97) The Bible tells of many miracles, some credited to Christ and some to other prophets and apostles. Generally speaking, which of the following statements comes closest to what you believe about Biblical miracles? (BLVMRCL)

100) Would you please think about each of the religious beliefs listed below and then indicate how certain you are that it is true. — Devil actually exists (DVLEXIST)

103) Would you please think about each of the religious beliefs listed below and then indicate how certain you are that it is true. — Jesus walked on water. (WLKONWTR)

130) Do you believe Jesus will actually return to the earth some day? (LORDRTRN)

435) Do you tend to agree or disagree with Darwin’s Theory of Evolution, which maintains that human beings evolved from lower forms of animal life over many millions of years? (EVOLUTN)

As their wording suggests, items 100 and 103 were part of the same battery, with identical response choices:
1) Completely true
2) Probably true
3) Probably not true
4) Definitely not true

Thus, it would be easy to combine the data for these two items, for example simply adding together the scores for all those respondents who answered both questions. If there were many items with the same responses, and many people failed to answered one or two, the data could be combined by taking an average of the scores for each respondent. This approach using averages would produce a more accurate measure of an individual’s beliefs, the more of the questions the respondent answered, but could give a better picture of the whole population if it allowed a larger number of respondent to be included in the analysis. So, there is always a trade-off in procedural decisions like this.

The three other belief items, unfortunately, had different response formats. For example, item 130, about the Second Coming, had these responses:
1) Definitely
2) Probably
3) Possibly
4) Probably not
5) Definitely not

This set of 5 responses is not very different from the sets of 4 responses for the two other items, but it is unlikely to be appropriate simply to add the score.

One has many choices in how to handle situations like this. Two principles can guide researchers: 1) Do your best, given the scientific purposes of your study, to pick the method least likely to introduce errors into the particular analysis you are doing. 2) In reporting results, be careful to tell the reader exactly what you did.

A statistically sophisticated approach is to transform each variable into a common measure of how far above or below the average each respondent is. Especially popular are standard scores (z-scores) based on standard deviations. However, technically this method tends to assume that the data have a normal distribution, and that the steps on the original scales are equal steps apart. This is not the case for the variables described here.

In fact, the three items just described have very skewed (unbalanced) distributions. Fully 44.6 percent of respondents said it is “completely true” the Devil exists, and an actual majority of 56.3 percent said it is “completely true” Jesus walked on water. The item with a 5-step scale also showed unbalanced responses, 46.2 percent “definitely” believing in the Second Coming. This suggests a very different approach for combining these items into a scale. Recoding the less extreme (and less popular) responses all to “0” would make all items dichotomous – just scores of 1 and 0, and adding together would be an accurate count of how many intensely religious responses the person gave.

Both methods are defensible, but have different strengths. Standard scores make use of all the variation in the data, but the recoding-and-adding method provides results that are easier to interpret rigorously. One could use both methods, and see if they gave comparable results.

Whatever choices one makes, it is important to keep in mind the wider context of the original data collection. The respondents were not a random sample of the general publication, but members of religious denominations that agreed to participate in the study. Furthermore, the number of respondents in each denomination do not mirror exactly the percentages among church members at large. Thus, the original sectarian analysis publication chose not to try to combine the items statistically into comprehensive scales, but to report the distribution of responses across the surveyed denominations for each item separated, but grouping into each of several tables the items that related to the same concept. This is a third analytical approach. A good practice for a scientific paper would be to do this first, presenting results of the items separately in one or more sections of the paper, and then combine scores into scale only for the concluding section of the paper.

Glock, Charles Y., and Rodney Stark. 1966. Christian Beliefs and Anti-Semitism. New York: Harper and Row.

Bainbridge, William Sims, and Rodney Stark. 1980. “Sectarian Tension,” Review of Religious Research 22: 105-124.

QuickSearch The Knowledge-Base

To search the knowledge-base, enter a term below:

Select a Theory below to learn more:

Select a Concept below to learn more:

TCM Contributors

Would you like to be considered for a position on the Theories, Concepts, and Measures contribution team? If so, click on the link below and complete the TCM Online Request Form.

TCM Online Request Form

If you are already a contributor for Theories, Concepts, and Measures and need assistance in managing the content, please click on the link below for instructions.

Site Administration Instructions