Summary of the Program Evaluation Results
Program Objective |
Excellence Achieved? |
| PO 1: Faith and Worldview. Graduates demonstrate a lively Christian faith and the ability to integrate their Christian faith and Reformed worldview with their counseling. Our benchmark for excellence in achieving PO 1 is achieving excellence on each of the measures below. | No |
| 1. Clinical performance
Our benchmark for excellence on this item is 90% of students achieving a summative evaluation score of 3 or better on the Student Counselor Outcome Report question “Graciously and appropriately addresses spiritual issues in treatment conceptualization and implementation” at the end of their third internship semester. Result: 100% (n=32) |
Yes |
| 2. Capstone Interview performance
Our benchmark for excellence on this item is 90% of student responses score of 3 or more on a 4 point scale on the Capstone Topic Presentation Evaluation. Result: 94% (n=32) |
Yes |
| 3. Internship/Externship site administrative assessments
Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the Internship Information Survey question 5, “Please rate…the most recent RTS students you have had as interns (on): Handling of spiritual issues.” Result: 4.3 (n=3) |
No |
| 4. Employer Assessments
Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the Employer Information Survey question 5, “Please rate…RTS graduates that you have recently hired (on): Handling of spiritual issues.” Result: Mean score of 5.4 (n=4) |
Yes |
| 5. Alumni Assessments
Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the following two questions of the Alumni Information Survey: Area 8: Training satisfaction Question 1: How satisfied were you with… Item 3: Relating theology to therapy? Result: Mean score of 5.3 (n=35) Area 10: RTS Experience Question 1: How satisfied were you with… Item 4: (The) impact (of your training) on your spiritual life? Result: Mean score of 5.3 (n=35) |
Yes |
| PO 2: Professional Competence (Eight Core areas, Clinical Mental Health Specialty, Dispositions). Graduates have the necessary knowledge, skills, and dispositions to be professionally competent conceptually and clinically in the areas of (1) professional counseling orientation and ethical practice, (2) social and cultural diversity, (3) human growth and development, (4) career development, (5) counseling and helping relationships, (6) group counseling and group work, (7) assessment and testing, (8) research and program evaluation, and (9) clinical mental health counseling. Our benchmark for excellence on PO 2 is achieving excellence on each of the measures below. | Yes |
| 1. Employer Evaluation
Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the following two questions on the Employer Information Survey: Area 2: RTS Graduate Performance. Rate RTS graduates’… Item 2: General knowledge Result: Mean score of 5.0 (n=4) Item 3: Clinical skills Result: Mean score of 5.4 (n=4) |
Yes |
| 2. Alumni Assessments
Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the following three questions of the Alumni Information Survey: Area 8: Training Satisfaction. Question 1: How satisfied were you with… Item 1: Training in techniques and clinical skills Result: Mean score of 5.5 (n=35) Item 2: Theoretical perspective of the program Result: Mean score of 5.6 (n=35) Area 9: Preparation for Professional Practice Question 1: How satisfied were you with… Item 4: Preparation compared with other Master’s level counselors you have encountered Result: Mean score of 5.6 (n=35) |
Yes |
| 3. Key Performance Indicators of knowledge and skills
The benchmark for program excellence is that annually students meet expectations in 90% of the 18 Key Performance Indicators (KPI) of student achievement in the core and specialty areas required of professional counselors. Result: 97.9% of KPIs measured achieved this benchmark (n=524) |
Yes |
| 4. Key Professional Disposition Indicators
Our benchmark for excellence on this item is 90% of students achieving a summative evaluation score of 3 or better on Student Counselor Outcome Report questions 1,2,3,5,11,12,22 and 23 and a mean score of 5 or more on Client Feedback Form questions 1-5 in their third internship semester. Result: 100% of students achieved this benchmark (n=32) |
Yes |
| PO 3: Diverse populations. Graduates have the necessary knowledge, skills, and dispositions to render service to churched and unchurched populations in our multicultural and pluralistic society. Taken together, the four items below contribute to evaluation of program effectiveness in achieving PO3. Our benchmark for excellence on PO 3 is achieving excellence on each of the measures below. | No |
| 1. Supervisor evaluation
Our benchmark for excellence on this item is 90% of students achieving a summative evaluation score of 3 or better on SCOR questions 11 and 12 at the end of their third internship semester. Result: 100% (n=32) |
Yes |
| 2. Client evaluation
Our benchmark for excellence on this item is 90% of students achieving a mean score of 5 or better on Client Feedback Form Questions 1, 2, 3 during the first half their third internship semester. Result: 94.4% (n=805) |
Yes |
| 3. Internship site administrator evaluation
Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the following two questions on the Internship Information Survey: Area 2: Intern performance. Rate RTS interns’… First item: Overall performance Result: 4.33 (n=3) Fifth item: Handling of Spiritual matters Result: 4.5 (n=3) |
No |
| 4. Employer evaluation
Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the following two questions on the Employer Information Survey: Area 2: RTS Graduate Performance. Rate RTS graduates’… Overall performance Result: Mean score of 5.4 (n=4) Fifth item: Handling of spiritual matters Result: Mean score of 5.4 (n=4) |
Yes |
| PO4: Licensure. Graduates have the necessary knowledge, skills, and dispositions successfully to be licensed as professional counselors. Our benchmark for effectiveness in achieving PO 4 is achieving excellence on both measures below. | Yes |
| 1. National Counselor Exam Pass Rate (NCE or NCMHCE)
Our benchmark for excellence on this item is a pass rate of 90% or more as reported by alumni on the Alumni Information Survey on the question: Did you pass the licensure exam? Result: 100% (n=13) |
Yes |
| 2. Favorable Employer Review
Our benchmark for excellence on this item is a mean respondent score of 5 or more on a 6 point scale on the Employer Information Survey on the question: Rate RTS graduates’ Overall performance. Result: Mean score of 5.4 (n=4) |
Yes |
Internship Information Survey Results
The program reviewed the results of the Internship Information Survey which were below our benchmarks for overall performance and handling of spiritual matters. We note the small sample size for this survey (n=3) due to the small number of internship and externship sites we currently have. The program will review our benchmarks for this survey given this small sample size and may make adjustments for future years.
Program Modifications Since Last Report
The Master in Arts in Counseling program at Reformed Theological Seminary made a number of changes to the program in the 2019-2020 academic year. First, in response to feedback received from students on the difficulty of taking Research in the Winter term, as well as feedback indicating that the statistical concepts covered in the Research course would be helpful in preparation for the Assessment and Testing course, the program decided to alter its curriculum and swap when the Research course is offered. The Research class is now scheduled during the cohort’s first year, and the Assessment and Testing is offered during the cohort’s second year.
Significant temporary changes in the program were necessary to respond to the COVID-19 pandemic. Classes in the Spring of 2020 term were switch to a synchronous (live) online format. This format was continued during the Summer term of 2020. The clinical hours required to complete Practicum and Internship were reduced from 400 direct hours (spread over Practicum and Internship) to 280 direct hours (the minimum required by CACREP).