Summary of the
Program Evaluation Results

Program Objective

Excellence Achieved?

PO 1: Faith and Worldview. Graduates demonstrate a lively Christian faith and the ability to integrate their Christian faith and Reformed worldview with their counseling. Our benchmark for excellence in achieving PO 1 is achieving excellence on each of the measures below. No
1. Clinical performance

Our benchmark for excellence on this item is 90% of students achieving a summative evaluation score of 3 or better on the Student Counselor Outcome Report question “Graciously and appropriately addresses spiritual issues in treatment conceptualization and implementation” at the end of their third internship semester.

Result: 100% (n=32)

Yes
2. Capstone Interview performance

Our benchmark for excellence on this item is 90% of student responses score of 3 or more on a 4 point scale on the Capstone Topic Presentation Evaluation.

Result: 94% (n=32)

Yes
3. Internship/Externship site administrative assessments

Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the Internship Information Survey question 5, “Please rate…the most recent RTS students you have had as interns (on): Handling of spiritual issues.”

Result: 4.3 (n=3)

No
4. Employer Assessments

Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the Employer Information Survey question 5, “Please rate…RTS graduates that you have recently hired (on): Handling of spiritual issues.”

Result: Mean score of 5.4 (n=4)

Yes
5. Alumni Assessments

Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the following two questions of the Alumni Information Survey:

Area 8: Training satisfaction

Question 1: How satisfied were you with…

Item 3: Relating theology to therapy?

Result: Mean score of 5.3 (n=35)

Area 10: RTS Experience

Question 1: How satisfied were you with…

Item 4: (The) impact (of your training) on your spiritual life?

Result: Mean score of 5.3 (n=35)

Yes
PO 2: Professional Competence (Eight Core areas, Clinical Mental Health Specialty, Dispositions). Graduates have the necessary knowledge, skills, and dispositions to be professionally competent conceptually and clinically in the areas of (1) professional counseling orientation and ethical practice, (2) social and cultural diversity, (3) human growth and development, (4) career development, (5) counseling and helping relationships, (6) group counseling and group work, (7) assessment and testing, (8) research and program evaluation, and (9) clinical mental health counseling.  Our benchmark for excellence on PO 2 is achieving excellence on each of the measures below. Yes
1. Employer Evaluation

Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the following two questions on the Employer Information Survey:

Area 2: RTS Graduate Performance. Rate RTS graduates’…

Item 2: General knowledge

Result: Mean score of 5.0 (n=4)

Item 3: Clinical skills

Result: Mean score of 5.4 (n=4)

Yes
2. Alumni Assessments

Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the following three questions of the Alumni Information Survey:

Area 8: Training Satisfaction.

Question 1: How satisfied were you with…

Item 1: Training in techniques and clinical skills

Result:  Mean score of 5.5 (n=35)

Item 2: Theoretical perspective of the program

Result:  Mean score of 5.6 (n=35)

Area 9: Preparation for Professional Practice

Question 1: How satisfied were you with…

Item 4: Preparation compared with other Master’s level counselors you have encountered

Result: Mean score of 5.6 (n=35)

Yes
3. Key Performance Indicators of knowledge and skills

The benchmark for program excellence is that annually students meet expectations in 90% of the 18 Key Performance Indicators (KPI) of student achievement in the core and specialty areas required of professional counselors.

Result: 97.9% of KPIs measured achieved this benchmark (n=524)

Yes
4. Key Professional Disposition Indicators

Our benchmark for excellence on this item is 90% of students achieving a summative evaluation score of 3 or better on Student Counselor Outcome Report questions 1,2,3,5,11,12,22 and 23 and a mean score of 5 or more on Client Feedback Form questions 1-5 in their third internship semester.

Result: 100% of students achieved this benchmark (n=32)

Yes
PO 3: Diverse populations. Graduates have the necessary knowledge, skills, and dispositions to render service to churched and unchurched populations in our multicultural and pluralistic society. Taken together, the four items below contribute to evaluation of program effectiveness in achieving PO3. Our benchmark for excellence on PO 3 is achieving excellence on each of the measures below. No
1. Supervisor evaluation

Our benchmark for excellence on this item is 90% of students achieving a summative evaluation score of 3 or better on SCOR questions 11 and 12 at the end of their third internship semester.

Result: 100% (n=32)

Yes
2. Client evaluation

Our benchmark for excellence on this item is 90% of students achieving a mean score of 5 or better on Client Feedback Form Questions 1, 2, 3 during the first half their third internship semester.

Result: 94.4% (n=805)

Yes
3. Internship site administrator evaluation

Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the following two questions on the Internship Information Survey:

Area 2: Intern performance.  Rate RTS interns’…

First item: Overall performance

Result: 4.33 (n=3)

Fifth item: Handling of Spiritual matters

Result: 4.5 (n=3)

No
4. Employer evaluation

Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the following two questions on the Employer Information Survey:

Area 2: RTS Graduate Performance. Rate RTS graduates’…

Overall performance

Result: Mean score of 5.4 (n=4)

Fifth item: Handling of spiritual matters

Result: Mean score of 5.4 (n=4)

Yes
PO4: Licensure. Graduates have the necessary knowledge, skills, and dispositions successfully to be licensed as professional counselors.  Our benchmark for effectiveness in achieving PO 4 is achieving excellence on both measures below. Yes
1. National Counselor Exam Pass Rate (NCE or NCMHCE)

Our benchmark for excellence on this item is a pass rate of 90% or more as reported by alumni on the Alumni Information Survey on the question: Did you pass the licensure exam?

Result: 100% (n=13)

Yes
2. Favorable Employer Review

Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the Employer Information Survey on the question: Rate RTS graduates’ Overall performance.

Result: Mean score of 5.4 (n=4)

Yes

Internship Information Survey Results

The program reviewed the results of the Internship Information Survey which were below our benchmarks for overall performance and handling of spiritual matters.  We note the small sample size for this survey (n=3) due to the small number of internship and externship sites we currently have. The program will review our benchmarks for this survey given this small sample size and may make adjustments for future years.

Program Modifications Since Last Report

The Master in Arts in Counseling program at Reformed Theological Seminary made a number of changes to the program in the 2019-2020 academic year.  First, in response to feedback received from students on the difficulty of taking Research in the Winter term, as well as feedback indicating that the statistical concepts covered in the Research course would be helpful in preparation for the Assessment and Testing course, the program decided to alter its curriculum and swap when the Research course is offered.  The Research class is now scheduled during the cohort’s first year, and the Assessment and Testing is offered during the cohort’s second year.

Significant temporary changes in the program were necessary to respond to the COVID-19 pandemic.  Classes in the Spring of 2020 term were switch to a synchronous (live) online format. This format was continued during the Summer term of 2020.  The clinical hours required to complete Practicum and Internship were reduced from 400 direct hours (spread over Practicum and Internship) to 280 direct hours (the minimum required by CACREP).