Assessing the Effectiveness of a Writing Intensive Course in Business

Many business professionals and academics claim that students’ ability to communicate in writing has deteriorated over the past few decades. In response to this decline, colleges and universities addressed the problem by changing existing courses to include more writing intensive (WI) assignments as well as introducing new WI courses with an emphasis on writing in the disciplines. The researchers analyzed the students’ rubric scores on their assigned cases as they progressed through a WI international financial management course which was fully online. During the current Covid-19 pandemic, increasingly colleges and universities conducted online courses. To assess learning, increases in the rubric scores during the semester were tested for statistical significance. Significant overall improvement in student writing occurred, but students’ grammar and spelling remained weak.


Introduction
Many colleges are now offering writing intensive (WI) courses "in the disciplines" as part of their curriculum to enhance writing skills used in the student's specific major. In response, a writing intensive version of an international financial management course was developed. The researchers assessed the effectiveness of teaching a WI course in the business discipline by testing the statistical significance of five specific rubric categories.

Literature Review
Many criticisms regarding the inability of undergraduate students to write clearly and logically have been noted by researchers in recent years. According to Kelly & Gaytan (2020), "The demand for writing skills is becoming increasingly prevalent within the U.S. job market" (p. 96). Fischer & Meyers (2017) state that, "Writing skills are one of the most important skills college graduates need to possess; however, college graduates struggle to complete written communications proficiently in the workforce" (p. 69). According to Washington (2014), "Effective writing skills are becoming more essential to workplace success and thus a central focus in business programs across the country" (p. 285). Middleton (2011) states, "While M.B.A. students' quantitative skills are prized by employers, their writing and presentation skills have been a perennial complaint" (p. 2). Articles reflecting efforts to incorporate additional writing into economics/business courses and to assess the results were published in prior years by Cohen and Spencer (1993), Davidson and Gumnior (1993), Hansen (1993), McElroy (1997), Biktimirov andKlassen (2008), andMcGoldrick (2008). The clear consensus of these researchers is that incorporating WI courses in economics/business college curriculums is essential for the future long-term success of students in their careers. According to a study by The National Commission on Writing for America's Families, Schools, and Colleges (2004), "Writing is a 'threshold skill' for both employment and promotion," and "People who cannot write and communicate clearly will not be hired and are unlikely to last long enough to be considered for promotion." The overwhelming consensus in the literature is that writing intensive college courses are needed to improve students writing, but assessment of the effectiveness of such courses needs further research and publication of the results.
A preliminary study was conducted by the authors with a small sample of 14 students (one class), and the results were presented at the 2015 Northeast Business and Economics Association Annual Meeting (Pasmantier & Di Liberto, 2015). Based on the findings of this preliminary study, modifications were made to the writing assignments and the rubric prior to the data collection of this larger multi-semester study.

Research Methods
The sample used in this study included data from 63 students collected over four semesters from multiple sections that were all taught by the same instructor. However, for convenience, we refer to our sample of 63 students as our "class". The writing assignments were persuasive essays of 1,000-1,500 words based on four cases with each student working on the same case/topic. There were, in fact, five writing assignments in total since Case 1 had two submissions, a draft and final revised submission. The resubmission of Case 1 was assigned to adhere to the requirement of our institution's WI program that, "requires students to complete a writing task in multiple drafts for at least one assignment".
It should be noted that the students who addressed their writing problems in Case 1, which were noted by the instructor's comments, did in fact score higher on the resubmission of Case 1. This may have introduced an upward bias in the Case 1 resubmission score, and thus, the resubmission score may not be a good measure of the students' ability for Case 1. This is particularly true for grammar and spelling since the average score for every skill level dropped in Case 2 from the resubmission of Case 1. Since only one submission for Cases 2, 3, and 4 was required, these assignments do reflect the students' writing abilities as does the initial submission of Case 1 draft. Had resubmissions been allowed for the latter cases, it is quite probable that those scores would have also been higher. This would have been especially true for the last assignment, Case 4, since many of the remaining errors at this point in the course were in grammar/spelling and citations, which could easily have been corrected in a revised submission.

Rubric Used to Assess Students' Writing Assignments
The course instructor developed a rubric specifically to evaluate the persuasive essay assignments in the International Financial Management course. To ensure consistency in the data, the same individual, a single instructor, used this rubric to grade all the writing assignments across all sections for all semesters of the study. The course syllabus contained a copy of this rubric which had the additional advantage of helping students understand what was expected of them, and it used five criteria for grading each case. (Table 1.) The rubric used by the instructor to grade each persuasive essay contained five criteria. They were: 1) opening/attention grabber (opening), 2) support for the position/evidence and examples (evidence), 3) closing paragraph (closing), 4) documentation and citations (citations), 5) grammar and spelling (grammar/spelling).
Each WI assignment was evaluated for the five rubric criteria, and each criterion received a score of 4 points for having met the requirements above standards, 3 points for having met the standards, 2 points for having approached the standards, and 1 point for having been below standards. With values of 1-4 points for each of the five criteria, the maximum grade for each WI assignment was 20, and the assignment grades ranged from a low of 5 points to a high of 20 points. The introductory paragraph has a hook or attention grabber, but it is weak, rambling, or inappropriate for the audience.
The author has an interesting introductory paragraph but the connection to the topic is not clear.
The introductory paragraph is NOT interesting and is not relevant to the topic.

Support for Position/ Evidence and Examples
Includes 3

Closing Paragraph
The conclusion is strong and leaves the reader solidly understanding the writer's position. Effective restatement of the position statement begins the closing paragraph.
The conclusion is recognizable. The author's position is restated within the first two sentences of the closing paragraph.
The author's position is restated within the closing paragraph, but not near the beginning.
There is no conclusion -the paper just ends.

& Citations
All citations used for quotes, statistics and facts are credible and cited correctly (parenthetical citations and bibliography).
All citations used for quotes, statistics and facts are credible and most are cited correctly.
Most citations used for quotes, statistics and facts are credible and cited correctly.
Many citations are suspect (not credible) and/or are not cited correctly.

Grammar & Spelling
Author makes no errors in grammar or spelling that distract the reader from the content.
Author makes 1-2 errors in grammar or spelling that distract the reader from the content.
Author makes 3-4 errors in grammar or spelling that distract the reader from the content.
Author makes more than 4 errors in grammar or spelling that distract the reader from the content.

Statistical Testing for Improvement in Students' Rubric Scores Results
Rubric scores of the participants were evaluated by the researchers to examine the level of writing improvement. To assess the data further, the sample was divided into three groups of students with weak, modest, and strong writing skills based on the writing skill level when the class began. The groups were examined to assess and compare the writing improvement across these three skill levels. Finally, the data for each of the five rubric criteria were analyzed to see how the class, as well as the students at the three skill levels advanced in each skill. The skill level data were also reviewed for patterns in the progress of the five criteria.

Class Rubric-Score Means of the Five Writing Assignments
The mean rubric-scores of the five assignments for the entire class were reported in Table  2. The class means improved throughout the semester, albeit very gradually, following the large increase in the scores after the revised final Case 1 (C1f) submission.

Statistical Tests for Improvement in the Class's Rubric Scores After Completing the WI Course
To test for a statistically significant improvement in the students' rubric scores after completing the WI course, a t test of the differences between matched sample pairs was used. First, for every student, the difference between a student's final Case 4 (C4) score and the initial Case 1 draft (C1d) score was estimated. This difference (C4-C1d) was positive if a student's writing improved, zero if there was no change, and negative if a student's writing actually deteriorated while taking the WI course.
Second, the class mean of the (C4-C1d) score differences was calculated. Third, the null hypothesis of no improvement in the mean score (H0: μD < 0) was tested against an alternative hypothesis of improvement for the class (H1: μD > 0) using an upper-tail t test of the differences from a sample of matched pairs. The results of the paired difference t tests are reported in Table 3. With a potential total score of 20, the class's mean increased 3.81 pts. (27.8%) from an initial C1d score of 13.68 pts. to a final C4 score of 17.49 pts. The upper-tail paired difference test of the 3.81 pts. mean difference has a t(62) value of 9.41 which was highly significant at a .001 level. The rejection of the null hypothesis indicates that, on average, the difference (C4-C1d) was significantly positive confirming that the WI course accomplished its goal of improving the writing skills of our business students.

Tests for Improvement in the Class's Rubric Scores on Subsequent Assignments
Although the cumulative impact of the course generated a significant improvement in writing, the progress between each assignment, while consistently positive, was not always statistically significant (Table 3).
The largest advance in the class's mean scores occurred between C1d and C1f, when a 20.2% improvement of 2.76 pts. was highly significant at a .001 level with a t(62) value of 9.08. In submitting C1f, students were able to correct their errors and improve their C1d in response to the instructor's specific comments. Clearly, this process of draft revision is a very instructive exercise.
Following Case 1, the class maintained their new higher level of writing as evidenced by their total score, which only improved and never slipped back to their initial level. Subsequent changes in the class's means between assignments, while smaller, were consistently positive. A 1.6% increase of 0.27 pts. from C1f to Case 2 (C2) with a t(62) value of 1.02 and a 0.5% increase of 0.08 pts. from C2 to Case 3 (C3) with a t(62) value of 0.27 were not significant. However, the 4.2% increase of 0.70 pts. from C3 to C4 with a t(62) value of 3.19 was very significant at the .01 level. To summarize, the students' overall writing skills continually improved throughout the course.

Grouping Students According to their Initial Writing Skills at the Start of the Course
A wide variation in students' writing skills was evident. In order to assess where students were weak and how they progressed through the course, the class was sorted into three groups based on students' initial performance in their first writing assignment (C1d).
Unfortunately, the sample of 63 students did not divide evenly into three equal groups because of the uneven and slightly skewed distribution of scores. (See Figure 1.)

Frequency Distribution of Rubric Scores for the Draft of Case 1
Consequently, 19 students with scores between 5 and 12 were classified as low scorers with weak initial writing skills. The mid scorers, who initially displayed modest writing skills, had the highest frequency with 26 students despite having the smallest range with scores between 13 and 15. The high scorers, who already had developed strong writing skills when the class began, included 18 students with scores of 16 and above. Figure 2 illustrates the mean rubric scores of the three skill level groups throughout the course and the data are reported in Table 2. At the start of the course, there are large differences between the C1d means of the low, mid, and high scorers (9.79, 13.88, and 17.50 pts., respectively).

Mean Group Scores for the Four Cases Sorted by the Score of the Case 1 Draft
Using an upper-tail separate-variance t test, the group mean rubric scores were compared. (Table 4.) The difference between the high and low scorers' C1d means is 7.71 pts. with a t(28) of 14.70; the difference between the high and mid scorers' C1d means is 3.62 pts. with a t(30) of 11.69; while the difference between the mid and low scorers' means is 4.10 pts. with a t(22) of 8.42. These differences between the C1d means were all highly significant at the .001 level. Therefore, at the start of the course, the scores of the low, mid, and high scorers were significantly different from one another, showing highly varied student writing skill levels.

Statistical Tests for Improvement in the Group Rubric Scores after Completing the WI Course
To see if each group benefited from the WI course, an upper-tail t test of the matched pair (C4-C1d) differences was again used (Table 3).
The low scorers with initially weak writing skills achieved a final C4 score of 17.00 pts., representing an enormous 73.7% improvement of 7.21 pts. over their C1d mean of 9.79 pts. that is highly significant at the .001 level with a t(18) of 11.83. The mid scorers with modest skills finished C4 with a score of 17.27 pts., increasing their scores by a very large 24.4% (3.38 pts.) from their initial mean of 13.88 pts. that is also highly significant at a .001 level with a t(25) of 9.41. The high scorers, who demonstrated strong writing skills with a high C1d mean score of 17.50 pts., completed C4 with a score of 18.33 pts. showing only a 4.8% improvement of 0.83 pts. in their rubric mean scores over the course. However, even the high scorers' small improvement is statistically significant at the .05 level with a t(17) of 2.19.

Differences Between the Three Skill Level Groups at the End of the Course
By the time the low, mid, and high scorers wrote their last assignment (C4) with scores of 17.00, 17.27, and 18.33 pts., respectively, the dramatically wide 7.71 pts. spread between the high and low scorers' C1d means had shrunk to only 1.33 points (Table 2).
Although the gap between the high and low scorers closed considerably to 1.33 pts. over the semester, a one-tail separate variance t test with a t(34) of 3.16 confirms that the highscoring group's C4 mean remained significantly higher than the low-scoring group's mean at a .01 level ( Table 4). The difference between the high and mid scorers' C4 means (1.06 pts.) was also significant at a .01 level with a t(40) of 2.54. However, by the end of the course, the small difference between the mid and low scorers' C4 means (0.27 pts.) was no longer significant with a t(41) of 0.63. The results indicated that all students significantly improved their scores and that a WI course in business was valuable for students with weak, modest, or strong writing skills.
The course was particularly important for those students who initially displayed weak or modest skills since they improved to proficiency levels just below their high-scoring peers and virtually closed the gaps between the groups. The low-scoring students benefited the most as their scores became statistically indistinguishable from the mid-scoring students by the end of the course.

The Opening, Evidence, and Closing Criteria
The class's mean scores for all five rubric criteria are illustrated in Figure 3. Note that the opening, evidence, and closing criteria curves in Figure 3 were very closely aligned with the students displaying similar levels of initial proficiency as well as development throughout the course. The class means of the three criteria never varied by more than 0.19 pts. on any assignment.

The Class's Mean Rubric Scores of the Five Criteria for the Four Cases
When the class began, the students' skills were already quite strong in the opening, evidence, and closing criteria with C1d class means of 3.30, 3.25, and 3.11 pts., respectively. All three means began above 3 out of a possible score of 4 pts., and therefore, the students were meeting the rubric standards already. By the end of the course, the C4 class means for the opening, evidence, and closing criteria were nearly perfect scores of 3.95, 3.90, and 3.94 pts., respectively (Table 5). Despite the relatively small increases in the class's three criteria means, the upper-tail t test of the matched pair (C4-C1d) differences for the opening, evidence, and closing criteria were highly significant at the .001 level with t(62) scores of 5.98, 5.52 and 6.39, respectively (Table 6).

The Citations and Grammar/Spelling Criteria
The students' writing skills in citations and grammar/spelling were considerably weaker than for the opening, evidence, and closing criteria as shown in Figure 3. On the initial C1d assignment, the class's mean rubric scores were only 2.29 pts. (approaching standards) for citations and 1.70 pts. (below standards) for grammar/spelling. At the end of the course on the C4 assignment, the students averaged 3.44 (meeting standards) in citations and 2.21 pts. (approaching standards) for grammar/spelling (Table 5).
While these scores remain below the near-perfect C4 scores for the opening, evidence, and closing criteria, the respective improvements over the term of 1.16 and 0.51 pts. in citations and grammar/spelling were highly significant at a .001 level with their matched pair (C4-C1d) differences with t(62) scores of 9.48 and 3.92, respectively (Table 6).
Of the five criteria, grammar/spelling was the class's lowest mean score on the initial assignment. The 1.70 pts. C1d mean indicates that, on average, students were below standards and making more than four errors per assignment. As noted above, the class's small 0.51 pts. improvement in grammar/spelling was highly significant, however, the resulting C4 mean of 2.21 pts. indicated that the class was still only "approaching standards" by making between three and four errors per assignment at the end of the course.

Analysis of the Progress in the Five Writing-Skill Criteria by Groups Sorted by their Initial Writing Assignment Scores
To further analyze the impact of the course on students' writing, the progress made in each of the five criteria by the students with strong, modest, and weak writing skills at the start of the course was examined. The differences and similarities in the progress between these three student groups are apparent in Table 5 where the group means of the five criteria for all assignments are presented for the high, mid, and low scorers.

The Opening, Evidence, and Closing Criteria
As noted above, the patterns of the class's improvement for the opening, evidence, and closing criteria in Figure 3 were virtually identical. However, not only were the class's three criteria means similar across assignments, each skill-level group had a distinctive improvement pattern that was essentially the same across the three criteria ( Figure 4).

Figure 4 The Group Mean Scores for the Five Criteria When Students Are Sorted According to the Case 1 Draft Score
While the class's high average scores in these three criteria implied well-developed writing skills at the start of the course, the first three panels of Figure 4, however, reveal this was not the case for everyone in the class. The low scorers were struggling with the C1d opening, evidence, and closing and only approaching rubric standards with means of 2.32, 2.53, and 2.05 pts., respectively, while the high scorers were meeting standards by displaying robust skills with means of 3.89, 3.78, and 3.94 pts., respectively. The mid scorers were slightly below their high-scoring classmates and meeting standards with means of 3.62, 3.42, and 3.31 pts. for the opening, evidence, and closing criteria, respectively.
By the end of the course, all groups achieved strong scores and met the rubric standards. The high-scoring students had honed their skills and displayed near-perfect scores with C4 means of 4.00, 3.89, and 3.94 for the opening, evidence, and closing criteria, respectively. Despite the wide gap between the high-scoring students and the rest of the class in those three criteria at the start of the course, the low and mid scorers were able to improve their skills so that the gaps between themselves and their high-scoring classmates virtually closed. The near-perfect opening, evidence, and closing C4 means are 3.89, 3.84, and 3.84 pts. for the low-scoring students, and 3.96, 3.96, and 4.00 pts. for the mid-scoring students, respectively.

The Citations Criterion
The Citations panel of Figure 4 indicates that citations were more challenging for students at all writing levels than the opening, evidence, or closing criteria. The low, mid, and high scorers started with considerably lower C1d means of 1.42, 2.23, and 3.28 pts. for citations indicating that students were below, approaching, and meeting rubric standards, respectively.
The low scorers improved after the lessons learned from completing C1d and C1f, and from C2 on, their performance closely matched the mid scorers. For C2 and C4, the weak students with means of 3.11 and 3.32 pts, actually overtook the mid scorers, who had slightly lower mean scores of 3.08 and 3.31 pts., respectively. Thus, by C4, both groups met standards.
The high scorers did their best citation work in C1f (3.89 pts.) following the professor's comments on C1d. Although the high scorers maintained some of the improvement over their C1d score in the subsequent assignments, these students never achieved the level that they displayed in C1f and finished C4 meeting standards with a strong, but somewhat lower, mean of 3.78 pts.

The Grammar/Spelling Criterion
The students' performance in grammar/spelling is noticeably weaker than it is in the other criteria as shown in Figure 4. The C1d grammar/spelling means for the low, mid, and high scorers were 1.42, 1.31, and 2.56 pts., respectively. The scores indicated that both the low and mid scorers began the course below standards averaging more than four errors per assignment, while the high scorers approached standards averaging between three and four errors. The C1f revision had higher means of 1.95, 1.85, and 3.06 pts for the low, mid, and high scorers, respectively.
Although the low and mid scorers retreated slightly from their C1f grammar/spelling scores in C2, they continued to improve and finished the course with C4 scores of 2.11 and 1.96 pts., respectively. Thus by C4, the low scorers now approached standards averaging three to four errors per assignment, while their mid-scoring peers came very close to this level. (It is interesting to note that the "low scorers" who initially had the weakest overall scores had higher grammar/spelling means than the mid scorers did on every writing assignment.) Unfortunately, the high scorers never improved upon their C1f performance as both the low and mid scorers did. Their final C4 score was only 2.67 pts., barely above the initial C1d mean of 2.56 pts., indicating that they were still approaching standards making three to four errors per assignment. This low grammar/spelling mean contrasts with the other four criteria means of C4 that were at or above 3.78 pts. Grammar/spelling remained a challenge even for otherwise excellent students.

The Overall Progress of Low, Mid, and High Scorers
Figures 5, 6, and 7 illustrate the differences between the initial writing skills and general patterns of progress in the five criteria of the low, mid, and high scorers, respectively.
Low scorers. Initially, the low scorers gave weak performances in all five criteria with three (opening, evidence, closing) approaching standards and citations and grammar/spelling below standards ( Figure 5). However, these students made strong progress during the course. The upper-tail t test of matched pair (C4-C1d) differences showed a highly significant improvement at a .001 level for the opening, evidence, closing, and citations criteria with t(18) values of 8. 22, 5.18, 9.12, and 12.55, respectively, while their improvement in grammar/spelling was highly significant a .01 level with a t(18) value of 2.82 pts. After the WI course, the low scorers significantly improved all five writing skill criteria with four of them meeting standards and grammar/spelling approaching standards.

Figure 5
The

Rubric Score Averages of the Low Scorers When Sorted by the Case 1 Draft Score
Mid scorers. The mid scorers started the term with meeting standards and strong scores in the opening, evidence, and closing criteria, as shown in Figure 6. These mid scorers improved their writing during the course, and by C4, many achieved a perfect score of 4 pts. in these three criteria. The upper-tail t test of matched pair (C4-C1d) differences showed a highly significant improvement for the evidence and closing criteria at a .001 level with t(25) values of 4.24 and 3.99, respectively, while the opening improvement was very significant at the .01 level with a t(25) value of 3.14.

Figure 6
The Rubric Score Averages of the Mid Scorers When Sorted by the Case 1 Draft Score The mid scorers had a weak start in citations and were only approaching standards. By C4, the mid scorers were meeting standards, and the upper-tail t test of matched pair (C4-C1d) differences indicated a highly significant improvement in citations at a 0.001 level with a t(25) value of 5.62.
Unfortunately, the mid scorers performed below standards in grammar/spelling at the start of the term, with a C1d mean (1.31 pts.) only slightly above the lowest possible score of 1 pt. However, by the end of the course, the mid scorers almost reached approaching standards in grammar/spelling, making very significant progress at a .01 level using the upper-tail t test of matched pair (C4-C1d) differences with a t(25) value of 3.05.
High scorers. The high scorers' criteria means were high when the course started, and thus, there was little room for improvement. The opening, evidence, and closing criteria means for the high scorers are virtually horizontal lines near the highest possible score of 4 pts., clearly meeting standards (Figure 7). Given the very slight changes during the course, the upper-tail t test of matched pair (C4-C1d) differences for the opening, evidence, and closing criteria were not statistically significant with t(17) values 1.46, 0.70, and 0.00, respectively.

Figure 7
The

Rubric Score Averages of the High Scorers When Sorted by the Case 1 Draft Score
For citations, the high scorers started the term meeting standards with a strong C1d mean score. They did their best work in C1f responding to the professor's comments, as shown in Figure 7. Although the high scorers never again achieved this quality of work, the students ended the term with a very high score for C4. The upper-tail t test of matched pair (C4-C1d) differences with a t(17) value of 3.00 showed very significant improvement in citations at the 0.01 level.
The high scorers were able to achieve near-perfect scores in four of the five rubric criteria by the end of the WI course. However, their grammar/spelling mean score remained weak. As with the citations, C1f was their best work. Unfortunately, the C2 score retreated showing no improvement over the initial C1d. Thereafter, the high scorers made only slight improvements in C3 and C4. The upper-tail t test of matched pair (C4-C1d) differences with a t(17) value of 0.57 indicated that the high scorers made no significant improvement in the grammar/spelling criterion, and they remained only approaching standards.

Summary
The assessment of the WI course in International Financial Management indicated statistically significant results confirming that students' writing skills dramatically improved from taking the course. The analysis of the rubric scores from the sample of 63 students revealed a great disparity in the initial writing skills and showed how students progressed during the course.
The low scorers with weak writing skills and the mid scorers with modest skills exhibited large improvements in their writing during the course. Although the two groups started the course far apart in the opening, evidence, closing, and citations criteria, they performed at virtually identical levels of writing proficiency by the end of the WI course. Almost all students in those two groups achieved perfect scores of 4 pts. in those four rubric criteria just as their high-scoring peers did. The performances of the low and mid scorers were virtually identical in grammar/spelling throughout the course. However, while those two groups made significant progress in all their writing-skill criteria, those students, on average, were still making 3-4 errors in grammar and spelling per assignment.
The high scorers with strong writing skills demonstrated a very high proficiency at the start of the course in the opening, the evidence, and the closing, as well as in the citations after the "refresher" lessons in C1d and C1f. The grammar/spelling of the high scorers, while consistently better than their lower-scoring peers, only improved marginally during the course and remained an area where these otherwise excellent students could improve.
Because of the individual analysis of the criteria, the researchers now recognize that grammar and spelling are areas in which our students struggle and that a greater focus on these skills needs to be included in the course. Additionally, more attention needs to be given to these areas in first-year developmental writing courses. As a result of this assessment, students in this WI class now are required to check their writing with a spelling/grammar check program prior to their case submissions. This requirement is now clearly stated in the syllabus. In fact, the researchers are currently collecting data to assess whether the use of such spelling/grammar tools will address the students' writing weakness in this area.
As a result of this study, we found that WI courses improve writing skills and that honing these skills is necessary for students' written communication to become effective. Teaching students how to improve their writing and communication skills is of extreme importance to schools and to the businesses that will employ them after graduation.