The Impact of an Assessment Certificate on Faculty Perceptions and Knowledge

Abstract Administrators are struggling to understand how to best promote and implement a culture of evidence-based decision making to stakeholders. The research study presented explored best practices on creating meaningful professional development experiences using both direct and indirect evidence of learning -this article will describe the effectiveness of a hybrid certificate program designed to educate faculty about assessment and its impact on faculty learning gains, perceptions, and self-efficacy. The study used a pre-/post-test design to measure participant knowledge using quizzes for each of the four modules of the certificate and participant perceptions using a survey. The modules covered writing student learning and program outcomes, curriculum mapping, developing assessment methods, creating assessment instruments, collecting data, analyzing and reporting results, and using results for improvement. Certificate completers demonstrated increased knowledge of assessment terminology, procedures, and best practices, as well as improved assessment-related self-efficacy. However, their perception regarding assessment did not change. Data gathered through this study can help inform decisions on needed assessment-related faculty professional development activities.


Introduction
Over time, the role of faculty has expanded from teaching to include research and service (Boyer, 1990). Institutions of higher education are placing greater demands on faculty with higher teaching loads and increasing expectations of scholarly productivity (e.g., research publications, funding). Engaging in the scholarship of teaching and learning, which includes the assessment of student learning, is a central focus for institutes of higher education because it is required to maintain regional accreditation. In addition, the assessment of student learning is critical to ensure students are prepared to enter the workforce with the necessary knowledge, skills, and abilities to engage in continued learning beyond graduation (Boud & Falchikov, 2007). However, as Boud and Falchikov (2007) argue, the assessment discourse is "…commonly dominated by the needs of certification" (p. 4) and fails to truly capture its impact on student learning. Faculty are subject-matter experts rather than pedagogical experts; therefore, they often lack the knowledge and skills necessary to transmit concepts essential to the discipline in ways that optimize student learning (Boyer, 1990;Saroyan & Amundsen, 2004). This knowledge gap between content and pedagogy in the classroom hinders faculty efficacy in the assessment of student learning and in the development and implementation of strategies to improve student learning. For this reason, higher education institutions need to support faculty not only in their research and service appointments, but also in their pedagogical development (Hott & Tietjen-Smith, 2018), which encompasses assessment of student learning for the purpose of continuous academic improvement. In this study, we examine the effectiveness of a faculty professional development certificate program created to (1) instill and enhance knowledge of assessment best practices in student learning, (2) improve attitudes and beliefs about assessment, and (3) improve participants' self-efficacy regarding assessment.
Review of the Literature Content and pedagogical knowledge are essential to developing and implementing meaningful assessment practices that lead to academic improvement. However, this can only occur when faculty are knowledgeable of effective teaching, learning, and assessment strategies; therefore, faculty development is needed in these areas. The following sections will explore relevant literature related to content and pedagogical knowledge, the current state of assessment practices in higher education, and the need for faculty development in pedagogy.

Content Knowledge and Pedagogical Content Knowledge
Content knowledge is the deep understanding of discipline-specific concepts and principles necessary to make pedagogical and curricular judgments (Shulman, 1986). To ensure learning, faculty must manage numerous components in the classroom (e.g., learner differences, context, background knowledge) while effectively transmitting content knowledge to the students in a meaningful context. This ability to transmit content knowledge to students was coined by Shulman (1986) as pedagogical content knowledge. Shulman (1986) differentiates it from content knowledge by arguing that pedagogical content knowledge is the "…amalgam of content and pedagogy (Shulman, 1987, p. 5)" or "…the dimension for subject matter knowledge for teaching (Shulman, 1986, p. 9)," which involves understanding, organizing, and adapting content for all learners. Pedagogical content knowledge is necessary for faculty to employ strategies that optimize student learning, which is determined through the assessment of important competencies to be attained by the students. Then, faculty are informed by these assessment data to make data-driven improvements in their classroom, which further expands their pedagogical content knowledge.

Assessment in Higher Education
There has been little change in assessment practices within higher education. According to Ronald Barnett, assessment is not utilized to its maximum potential (Boud and Falchikov, Chapter 3, 2007) and continues to be mostly a bureaucratic process for faculty. David Boud calls for a transformation of the current discourse in higher education into one that reframes the purpose of assessment, how we talk about it, and how we describe it (Boud and Falchikov, Chapter 2, 2007). This transformation can only take place if we empower stakeholders (e.g., faculty, administration) with the knowledge and understanding of assessment best practices. In order to accomplish this, active collaboration (Banta, Jones & Black, 2009) is needed to "…find ways of thinking about assessment that have positive consequential influence on learning (Boud and Falchikov, 2007, p. 19)." Aligning scholarship and teaching for the improvement of assessment practices, beginning with faculty development, is necessary to transform the assessment discourse.

Faculty Development
Higher education institutions hire faculty as subject-matter experts and expect them to effectively convey knowledge to their students; yet, many faculty lack pedagogical training. De Golia et al. (2019) surveyed psychiatry faculty to assess faculty development needs and teaching skills workshops including teaching methods, assessment skills, and pedagogy was identified as an unmet need. Faculty teaching in an online education doctorate program identified the need for more professional development in the area of pedagogy in a qualitative study conducted by Berry (2019). A study conducted by Behar-Horenstein, Garvan, Catalanotto, Su, and Feng (2016) assessed faculty development needs amongst dental faculty; their findings suggest the need for faculty development that enhances teaching. More importantly, Rutz, Condon, Iverson, Manduca, and Willett's (2012) study looked at the relationship between faculty development, pedagogy, and student achievement and identified a direct relationship between faculty development and improved pedagogy. In summary, the findings of these studies demonstrate the need, and faculty desire, for professional development opportunities in the areas of pedagogy and assessment of student learning.

Problem Statement
Current literature (Allan & Driscoll, 2014;Behar-Horenstein et al., 2016;Berry, 2019;De Golia et al., 2019;Hott & Smith, 2018;Pawlyshyn & Hitch, 2016;Rutz et al., 2012) notes the importance of faculty development and its positive impact on student learning. With increasing faculty expectations and growing accountability measures from regional accrediting agencies, higher education needs to expand faculty development in the area of assessment. As Boud and Falchikov (2007) state, "…focus on assessment practices is needed, not simply on labelled methods considered independently of their consequences (p. 12)." In response to this identified need, our intervention -an assessment certificate program -was provided to faculty to impart knowledge related to best practices in the assessment of student learning and program efficiency for the purpose of continuous academic improvement.

Research Questions
Grounded on literature that stresses the importance of faculty development (Allan & Driscoll, 2014;Boyer, 1990;Hott & Smith, 2018;Pawlyshyn & Hitch, 2016;Saroyan & Amundsen, 2004) and the need for meaningful assessment practices (Allan & Driscoll, 2014;Boud & Falchikov, 2007), this study examined the effectiveness of an institutionwide assessment certificate created for faculty to learn assessment best practices in student learning and program efficiency. The following research questions served as a guide for the design of this intervention: 1. Did knowledge about assessment terminology, procedures, and best practices improve? 2. Does participation in a certificate program improve participants' perception of the effectiveness of support systems? 3. Did participation in the certificate program improve participants' self-efficacy?
The current study took place in the fall 2018 term at Florida International University (FIU). FIU is a large urban public research university with over 58,000 students and 2,300 faculty members. A pre-test post-test experimental design was used to analyze learning gains and participant perception data gathered in this study.

Methodology
The Institutional Effectiveness team within the Office of Academic Planning and Accountability at Florida International University developed a hybrid (delivered partially online and face-to-face) certificate program designed to educate faculty about assessment and its impact on faculty learning gains, perceptions, and self-efficacy. The certificate program consisted of four modules delivered over the 12 weeks. The modules covered (1) writing student learning and program outcomes, (2) curriculum mapping, (3) developing assessment methods, creating assessment instruments, collecting data, analyzing and reporting results, and (4) using results for improvement.
Each on-line module consisted of interactive activities to facilitate learning of assessment best practices. The information included in each module was guided by the work of Banta and Blaich (2010), Kuh et. al. (2014, 2015, and Suskie (2009). Participants completed assigned readings, participated in discussion boards, and developed a comprehensive assessment plan. Participants also attended two in-person workshops during which the institutional effectiveness team provided detailed feedback on the assessment plans developed by the faculty. Furthermore, the in-person sessions were designed to reinforce course content by reviewing learning outcomes for each module and providing active learning opportunities for participants (e.g., discussions to where faculty were asked to apply competencies to their programs and courses, activities to create outcomes). This study used a pre-/post-test design to measure participant knowledge using quizzes for each of the four modules and participant perceptions using a survey.

Participants
The participants in this study consisted of faculty, instructors, and staff who enrolled in a semester-long, hybrid assessment certificate program. Recruitment for participants was initiated as an e-mail to a convenience sample of faculty who work on assessment reports for academic programs, certificate programs, and general education courses. Recruitment was focused on faculty from the College of Arts, Sciences and Education (CASE) and the Steven J. Green School of International & Public Affairs (SIPA). The researchers targeted these two colleges as participants in a pilot study since they represented the bulk of the general education courses and programs at the institution. Out of the 81 people who demonstrated interest, 50 were selected based on their availability and willingness to complete all requirements of the certificate program. By the end of the semester, only 45 participants completed the fourth and final post-test of the program. The analysis of the data reflects the drop-out of two participants from Module 1 through Module 3 and the drop-out of three remaining participants in Module 4. Demographic data for the 48 participants that completed the first three modules are presented in Table 1.

Instruments
Two instruments were used to answer the three research questions. The first instrument given to the participants was a perception survey. The survey consisted of 59 items and was designed by the researchers to measure faculty perceptions of assessment practices, knowledge, and utility.
A Likert scale was used to quantify the rating in the perception survey. For items related to beliefs about assessment and willingness to do assessment tasks, a 4-point Likert scale was used where 1 represented "strongly disagree" and 4 represented "strongly agree." For items related to self-efficacy or utilization of assessment best practices, a 4-point Likert scale was used where a score of 1 indicated "not at all" and 4 indicated "to a very great extent." Finally, items related to assessment support and culture in their department, college, and institutional effectiveness office were rated using a 3-point scale where 1 indicated "no", 2 indicated "sometimes", and 3 indicated "yes." The survey questions were broken down into the following categories: 1. Effectiveness & utility of assessment (10 items) 2. Willingness to participate in assessment activities (6 items) 3. Self-efficacy (10 items) 4. Perception of assessment -Department Level (9 items) 5. Perception of assessment -College Level (9 items) 6. Perception of assessment -Institutional Effectiveness Office Level (9 items) 7. Extent to which assessment results are perceived to be useful (6 items) The same survey was distributed twice, the first iteration before the certificate program began and second iteration after participants completed all modules and assignments of the certificate program (including the in-person session and mid-term and final projects). Both validity and reliability tests of the instrument were conducted. The survey validation method selected was construct validity, which is "the instrument's ability to relate to other variables (Burton, & Mazerolle, 2011)." Thus, Exploratory Factor Analysis was the appropriate statistical technique to identify instrument constructs (Turocy, 2002). To conduct this analysis, the seven categories were grouped according to their possible answer choices (refer to Table 2).
For the categories Effectiveness and Utility and Willingness, the Kaiser-Meyer-Olkin (KMO) and Bartlett's value (chi-square=761.591, p<.05, sig=.000) exceeds the heuristic of .60, indicating adequate correlations to continue with factor analysis (Burton, & Mazerolle, 2011); refer to Table 3. Communalities range from .476 to .894 (refer to Table 4). Since they all exceed the 0.4 minimum, factor analysis was conducted using all items.
And as shown on Table 5, using the Kaiser rule with eigenvalues greater than 1, two factors emerged (eigenvalues 9.119 and 2.773). In total, 16 components were extracted, accounting for 100% of the variance. The first factor accounts for 56.99% of the variance and the second factor accounts for 17.33% of the variance. The total amount of variance accounted for by the first two principal components solution is 74.32%.
For the categories of Self Efficacy and Results Utility, KMO and Bartlett's value (chi-square=728.430, p<.05, sig=.000) exceeds the heuristic of .60, indicating adequate correlations to continue with factor analysis (Burton, & Mazerolle, 2011); refer to Table 7. Communalities range from .524 to .898 (refer to Table 8). Since they all exceed the 0.4 minimum, factor analysis was conducted using all items.
As shown on Table 9, using the Kaiser rule with eigenvalues greater than 1, three factors emerged (eigenvalues 7.770, 3.402, and 1.164). In total, 16 components were extracted, accounting for 100% of the variance. The first factor accounts for 48.56% of the variance, the second factor accounts for 21.26% of the variance, and the third factor accounts for 7.28% of the variance. The total amount of variance accounted for by the first three principal components solution is 77%.
The Rotated Component Matrix indicates (boxed in black) the items belonging to each of the three components (refer to Table 10). Factor cross loading at 0.5 or above occurred for two items (boxed in red). As Burton and Mazerolle (2010) suggest, these items should be removed in future administrations.
For the categories of Perception of Department, College, and Institutional Effectiveness, KMO and Bartlett's value (chi-square=1190.405, p<.05, sig=.000) exceeds the heuristic of .60, indicating adequate correlations to continue with factor analysis (Burton, & Mazerolle, 2011); refer to Table 11. The question asking participants to rate whether assessment is valued by the Institutional Effectiveness team was removed from this analysis, as this variable had zero variance (i.e., all responses were "Yes"). Communalities range from .627 to .921 (refer to Table 12). Since they all exceed the 0.4 minimum, factor analysis was conducted using all items.
As shown on Table 13, using the Kaiser rule with eigenvalues greater than 1, five factors emerged (eigenvalues 11.050, 3.127, 2.357, 2.058, and 1.289). In total, 26 components were extracted, accounting for 100% of the variance. The first factor accounts for 42.5% of the variance, the second factor accounts for 12.03%, the third factor accounts for 9.07%, the fourth accounts for 7.92%, and the fifth accounts for 4.99%. The total amount of variance accounted for by the first two principal components solution is 76.51%. These results indicate further refinement of these categories is also necessary, as participants' responses were grouped and analyzed based on three constructs.
The Rotated Component Matrix indicates (boxed in black) the items belonging to each of the five components (refer to Table 14). Factor cross loading at 0.5 or above occurred for three items (boxed in red). As Burton and Mazerolle (2011) suggest, these items should be removed in future administrations. In addition to Exploratory Factor Analysis, a Cronbach's Alpha was done to test the reliability of the items. Reliability indicated high internal consistency (α > 0.70) per category, as shown on Table 15.
The second instrument consisted of pre-test and post-test quizzes for each of the four modules of the certificates to measure participant knowledge about assessment terminology, procedures, and best practices. The pre-tests and post-tests used the same items. Table 16 describes the total number of questions per module and a breakdown of the learning areas the questions focused on.
Participants completed the pre-test before each module and did not see the answers to the questions. The same questions were then presented at the end of the module as a posttest and they had one opportunity to respond correctly. Items were scored dichotomously (1=correct, 0=incorrect). A higher score indicated greater knowledge.

Data Analysis
Descriptive statistics and paired sample t-tests were used to answer the first research question: Did learning about assessment terminology, procedures, and best practices improve? Descriptive statistics were used to calculate averages of the pre-test and posttest and were used as a direct measure for assessing learning gains for each of the four modules. A secondary analysis using paired sample t-tests was conducted to calculate significance of the differences in means. The last two research questions related to changes in attitudes/beliefs, self-efficacy, and effectiveness of support services were answered using independent sample t-tests since the survey was anonymous and we were not able to pair pre-test scores with post-test scores. Alpha levels for all tests were set at the .05 level.

Results
Results for the first research question indicate a significant increase in mean assessment knowledge from pre-to post-test for each of the four modules (Refer to Table 17).
Results for the second research question, change in perceptions regarding assessment after completion of the certificate program, did not indicate significant differences between pre-and post-test scores. However, results for the third research question indicate a posttest significant increase in participants' perceived self-efficacy on most of the areas surveyed (Refer to Table 18). It is important to note pre-and post-survey data were not paired. Since the pre-survey was completed anonymously, multiple submissions resulted in a larger sample size (n=60); this was rectified prior to completion of the post-survey, which yielded an accurate sample size (n=47). This is reflected by the 105 degrees of freedom of the two-tailed t-test (refer to Table 3).

Discussion and Future Directions
Results of this study suggest that structured professional development activities are effective in teaching faculty assessment best practices. Though average increases and significant differences could be attributed to test-retest validity since the same questions were used for pre-and post-test quizzes and survey, the need remains for higher education institutions to invest in assessment-related professional development activities for faculty, as they (for the most part) are only subject-matter experts (Allan & Driscoll, 2014;Boyer, 1990;Hott & Smith, 2018;Pawlyshyn & Hitch, 2016;Saroyan & Amundsen, 2004). However, as Banta (2009) states, this must be done with the support of senior-level administrators and with faculty who are committed to the process. Hence, collaboration amongst senior-level administrators, institutional effectiveness teams, and faculty is pivotal in creating and sustaining a culture of meaningful assessment practices within higher education institutions.
Beyond the formal instruction provided through the certificate program, this intervention served as a springboard to initiate a culture of faculty-driven assessment practices throughout our institution. A longitudinal follow-up study to this research is forthcoming to examine whether or not the learning achieved by the participants affects the quality of assessment reports as measured by a standardized rubric. Improving assessment practices among faculty could facilitate more impactful improvement strategies that lead to enhanced student learning; thereby perpetuating a positive cycle of continuous improvement in both teaching and learning.
It is also important to note that the impact of learning experiences such as this certificate on self-efficacy should be further explored. The results indicating that self-efficacy was significantly increased can be meaningful since research shows that self-efficacy is correlated with motivation as it relates to learning and applying/transferring learned concepts from the training program (Chiaburu & Lindsay, 2008). Sorrenti, Filippello, Buzzai, Butto & Costa (2017) found that self-efficacy was positively correlated with traits of conscientiousness, extraversion, openness to experience, and agreeableness and negatively correlated with learned helplessness. These studies suggest that self-efficacy needs to be taken into consideration when creating learning environments for learners such as the faculty and staff in this study. Self-efficacy may be an important factor in influencing perceptions of assessment and motivation to improve and apply assessment best practices. Follow-up studies should include an investigation of whether self-efficacy is correlated with competency mastery and application of learned skills in program/course assessment practices.
Future research should further explore the effectiveness of assessment-related professional development activities, as well as perhaps identify additional assessmentrelated competencies faculty should master to ensure student success. A phenomenological study would also be beneficial to better understand the challenges institutions and faculty face when developing and implementing assessment practices; survey responses cannot capture the depth, intricacies, and differences amongst institutions. Another untapped area is the long-term effect of assessment-related faculty development activities and its impact on student learning. Beyond faculty preparedness, the preparation of administrative staff (e.g., registrar, student affairs) to assess the quality and effectiveness of their processes, initiatives, and areas of oversight should also be explored, as they too are tasked with supporting the institution's mission. Finally, further refinement of the survey instrument used in this study should be explored, as it will provide researchers with a valid and reliable instrument to assess categories discussed.    Table 3.    Table 7.