TCAT Research
One of the most important tests of the value of a survey is the “sniff test.” We all learned this test from our mothers. Mothers have a highly developed sense of smell and could tell with one whiff when we hadn’t taken a bath, been around someone who had been smoking, and even know what we had been eating.
The first step of the research was a quantitative research study of Protestant churches. Among the first seven thousand pastors who completed the quantitative survey, their churches were from 123 different denominations as indicated from our sample. Within the survey we had a shorter list to choose from which showed that 6 percent of participants were non-denominational (or had no connection with a denomination-like affinity group, such as Independent Fundamentalist networks). The goal was to establish benchmarks of key metrics, but more importantly to identify the best churches. These churches would be the source of additional insights in Phase 2.
Because of the volatility in church growth numbers among small churches (e.g., one family coming or going has a large impact on percentage growth), the church must have a minimum of fifty in current worship attendance.
As you can tell, we were looking to change the scorecard. We were not saying that things like attendance and conversions don’t matter – they do! But they are not the only things that matter. For too long we have focused only on nickels, noses, and numbers. We wanted to look at lives changed, churches growing, and communities being changed and served. Hence, we started with some expanded metrics.
Phase 2: Qualitative Research to Obtain In-depth Insights
We then contacted the 298 churches with the top 10 percent responses based on the Phase 1 criteria who also permitted us to contact them. We requested to interview the senior pastor, in person if at all possible. With much persistence by LifeWay church consultants, more than 250 interviews were completed, as they traveled to meet pastors and church leaders from Maine to California, from Washington to Florida.
Principles that were common among the top 10 percent churches were used to develop questions for the church assessment tool. Thom Rainer and Ed Stetzer and our LifeWay Research Team (particularly Scott McConnell and Lizette Beard) looked at them extensively. The questions were reviewed by a panel of experts that included:
An extensive list of questions was chosen initially– twenty questions each for the seven elements of the Transformational Loop – 140 questions in all. The lengthy survey was used so that the paring down of the survey would be based on objective criteria rather than the opinions of a few.
Phase 4: Church Assessment at the Congregation Level
Additional randomly selected Protestant churches were recruited to test the validity of the final assessment tool. Of the 280 Protestant churches that were randomly selected for this test, 94 obtained enough responses from their attendees to provide results representative of their church with a confidence interval within +/-10 percent. This constituted a 34 percent success rate.
Conclusion
In the end, we are confident that the principles were discovered among effective churches. These Transformational Church elements were assembled in a reliable survey that will consistently measure these principles in different churches. The Transformational Church Assessment Tool (TCAT) questions have been and continue to be rigorously tested. The assessment tool produced by this study is both internally and externally valid. And, the actionable Transformational Church principles predict desirable church outcomes such as attendance growth, new commitments to Jesus Christ, and service.
In the same way research must pass the sniff test. Both the results and how the research was conducted need to make sense and not hint at any bias when the reader gets a whiff.
Here is the process we used to develop and test the Transformational Church research and assessment tool.
Phase 1: Identify Churches for Deeper Learning
A calling list was randomly drawn from a list of all Protestant churches in the United States of America. Up to six calls were made to reach a sampled phone number. Each interview was conducted with the senior pastor, minister, or priest of the church called. The calling was done in waves of one thousand completed interviews working through a portion of the sample in each wave systematically with quotas for church size (sizes of the churches were proportional to the normal size of U.S. churches). The calling was done by a professional call center with experienced supervision.
After analyzing survey responses, we used the following criteria to determine the most effective churches among the 4,006 churches that were interviewed in the first four waves of Phase 1. To aid in recruiting churches for Phase 3, a total of seven thousand pastors ultimately completed Phase 1 questions.
The basis of the interview was a pair of topic guides that the interviewer used to encourage each pastor to describe the activities of the church, why it is working, and how they know it is effective. Because of the depth required in the interviews, the nine general subject areas were split between the two topic guides. Most interviews used only one, but a few generous pastors were interviewed using both.
Each pastor was given the opportunity to provide additional facets of ministry they considered vital to an effective ministry. The additional information provided learning beyond the general areas of inquiry that included worship, evangelism, discipleship, small groups, prayer, leadership, how new people are connected, the quality of life within a congregation, and level of engaging a lost and hurting world.
The interviewers employed by LifeWay in the process were church consultants that had extensive ministry experience. To prepare each consultant and to ensure that each interview was carried out in a consistent manner, two training sessions were provided for conducting interviews.
Each interview was recorded and transcribed. Each church consultant completed a summary immediately following the interview highlighting the most important things they heard. We were able to enhance interview quality by sharing the early transcripts among church consultants for them to hear other interviewers and to observe when useful information was obtained and when it was missed.
Interviewer observations and analysis of the transcripts yielded principles that led to the framework for the Transformational Loop and provided many practical examples that Ed Stetzer and Thom Rainer included in the Transformational Church book.
Phase 3: Church Assessment on the Layperson Level
A random set of Protestant churches was then invited to have their pastor and adult members use the Transformational Church assessment tool.
Analysis was done on 4,077 members who completed the initial version of the survey. The analysis validated the scales (sets of questions in each Transformational Church element that have a similar underlying pattern of response). In each of the elements, outliers constituted 1 percent or less of the values.
Factor analysis was run using the questions for each Transformational Church element in the survey. Cronbach’s Alpha for each factor proved very reliable with the lowest being .826. This internal consistency establishes the reliability of the questions in the assessment tool. Individual questions that did not have at least 10 percent of values on each half of the Likert scale were eliminated as were questions that did not load with a factor in their section. The final set of 82 questions comprised the assessment tool.
The 82 questions were reassessed for validity and reliability to doubly insure the accuracy of the survey method. With a total of more than 9,000 church-attendee responses to compare to the 4,077 in Phase 3, frequency distributions were found to possess negligible differences. Validity of the final method was supported by this consistency between Phase 3 and Phase 4.
Internal consistency estimates were run to test the reliability of the survey method. Cronbach’s alphas for Phase 4 were all between .87 and .94, and Spearman-Brown coefficients for the survey were all between .85 and .95. These results support strong internal reliability for the survey method. This effectively supports the reliability and validity of the final survey form.
The resultant Transformational Church element sums were all highly correlated to each other (above .69). This confirms statistically the convergence of the Transformational Church elements mentioned in chapter 2 of the Transformational Church book.
Separate multiple regression analyses for each sum revealed that one or more of the Transformational Church elements are predictive of each of the following church outcomes:
