Cognitive Complexity and Degree Planning: Student Perceptions of and Needs From Technology Tools

Participant Recruitment and Study Sample

This paper draws on a robust dataset drawn from two broad-access, four-year public institutions. Institutions were selected because (a) they were actively exploring ways to improve advising and advising technologies on their campuses; (b) they were diverse in location and student demographics; (c) they were public and broad-access, thereby generally representative of the types of institutions in which new-majority students enroll; and (d) they were willing to partner with the researchers to support data collection activities.

One institution was located in the Midwest, and enrolls approximately 11,000 students, 24% of whom identify as racially-minoritized, and 41% of whom are Pell-eligible. The other institution was located in a northeast urban center and is part of a multi-institution system. It enrolls approximately 15,000 students, 85% of whom identify as racially-minoritized, and 52% of whom are Pell-eligible.

Both institutions assign students to advisors, but the extent to which students were required to meet with their advisors varied, based on students’ majors and class year. Advisor caseloads varied across and within the institutions, as well, with some advisors serving upwards of 600 students and others serving many fewer. Appendix A provides additional information about the two institutions.

This study was approved by the IRB at both institutions. For all data collection procedures, we were careful to ensure participant confidentiality and voluntary participation. We were particularly cognizant of the fact that we (a) were asking students to discuss potentially uncomfortable questions around race and class and (b) for advisors and staff, we were asking them about their places of employment. We were also aware of our positionality as researchers; two of the three researchers engaged in data collection for this study are white, and all of us have advanced degrees. Our data collection methods acknowledged this by making all demographic disclosures (as well as participation) voluntary and by spending time at the beginning of each interview and focus group building rapport.

Our primary data are drawn from virtual focus groups conducted with students at both institutions during the spring of 2022. Representatives from each college helped us identify and recruit students who identified as BLI/LI. These representatives also sent recruitment information to the identified students, providing them with details about the study, their rights as potential participants, and instructions on how to participate. Students were offered a $50 gift card to participate in the focus group. In order to maintain confidentiality, participants registered for the study with the research team—not the college personnel who aided in recruitment. We spoke with 24 students across the two institutions. To contextualize and triangulate our student data, we conducted interviews with college personnel who work with advising technologies or serve as advisors at each institution. We spoke with 14 advisors and 13 additional personnel.

As part of the data collection, we asked students to provide self-identifying information, as they were comfortable. We prompted them to provide information about their race, gender/gender identity, language status, full/part-time enrollment status, financial aid status, and “anything else that may help us understand your experiences in college” as part of the process mapping described below. We also offered the opportunity for students to not self-disclose any demographic or descriptive information.

This approach to collecting information about the sample means we necessarily have imperfect descriptors. However, it was in line with our theoretical framework, which emphasizes the importance of elevating what individuals believe to be important, the intersectional nature of identity, and ensuring that non-majority identities are not framed as others or the default. We did not want to presume which identities might matter to students as they navigate course planning, nor did we want to create an environment where they felt compelled to disclose identities they were not comfortable sharing with outsiders.

As a result, students provided demographic information that was most salient to them, rather than predetermined categories. In many ways, we have a richer understanding of our sample than if we had used more traditional methods; students opted to disclose things we would not necessarily have asked about, such as their statuses as DACA recipients, members of the LGTBQIA+ community, or former foster youth.

Seventeen students self-identified as Black or Latine; 15 self-identified as low-income. Twelve students identified as female and eight identified as male. Nine students indicated they are first-generation college goers; eight work full- or part-time; at least one student was undocumented and ineligible for financial aid; and one student was a runaway/emancipated minor. Given that students were not required to disclose demographic information, these numbers may underrepresent BLI/LI representation in the dataset.

Focus Groups and Process Mapping

Focus groups are a tool used for the collection of qualitative data and are described by Nastasi and Hitchcock (2015) as, “interviews (typically semistructured) conducted with a small group of informants for the purpose of exploring specific topics, domains, constructs, beliefs, values or experiences from the perspectives of cultural members” (p. 75). We chose focus groups because they are well suited for working with new-majority students as they “allow for the opportunity to validate their experiences of subjugation and their individual and collective survival and resistance strategies” (Mertens, 2015, p. 383). For this study, focus groups offer two additional benefits. First, focus groups serve as an efficient means of data collection from multiple people (Nastasi & Hitchcock, 2015). A second benefit is that the interactions between group members provide additional insight beyond that collected in individual interviews (Mertens, 2015).

All focus groups in this study were conducted virtually, using video technology. Although virtual focus groups can carry some data collection risks (i.e., difficulty building rapport, less opportunity to observe nonverbal cues), we opted for virtual engagement for a number of reasons. First, meeting with students virtually allowed us more scheduling flexibility, enabling us to conduct focus groups over weeks rather than over a few condensed days of a site visit, as well as varying the time of day. Given the many obligations faced by students in this study, scheduling flexibility allowed us to maximize participation while respecting students’ time.

Second, data collection occurred in 2021 and while the COVID-19 pandemic was technically over, we were mindful of heightened concerns regarding health and safety. Meeting virtually allowed us to maintain safe distance and minimize participants’ exposure to others who may be ill. These heighted safety precautions were particularly important because so many of our participants were from communities disproportionately impacted by the pandemic.

The focus group itself had two parts, leading to two different types of data. During the first section, we guided our participants through a process mapping exercise. Process mapping is widely used in business to visually represent the steps individuals take while completing a task or series of steps. It is typically used to identify breakdowns or inefficiencies in a business process (Dietz et al., 2008). Process mapping allows for an in-depth examination of a single process, in this case, planning and registering for courses. Prior research in similar contexts of community colleges has recommended the use of process mapping to explore advising processes and degree completion (Karp et al., 2021; Wesley & Parnell, 2020).

We used virtual process mapping to collect visual representations of students’ experiences and practices when planning and registering for classes in order to capture detailed, step-by-step roadmaps taken by each student while using academic planning software to make gains towards degree completion. Using Google Slides, we guided focus group participants through the process by asking them to identify what they think about, what they do, and what resources they rely upon to plan their schedules and remain on track to graduation. We then asked them to array those steps into a visual that represents their planning and registration process. We also asked them to “annotate” their maps, identifying the aspects that were easiest and most challenging. Appendix B provides a sample of these maps.

During the second half of the focus group, we used a semistructured interview protocol to ask the students to reflect upon the mapping experience, their planning processes, and the tools available to them. We audio recorded and took detailed notes during these discussions, which were used to supplement, clarify, and extend the visuals students provided via their maps.

As noted, these data were collected towards the end of the COVID-19 pandemic. As a result, our response rates were lower and our no-show rates were higher than we had hoped for—a phenomenon experienced by many researchers during the same time period (see, for example, Rothbaum & Bee, 2022; LeBlanc et al., 2023). Readers should note that our sample may not include students most negatively impacted by the pandemic or who have the greatest caregiving and work responsibilities, as well as those with inconsistent access to technology. As such, our findings may actually underestimate the extent to which non-academic considerations play into students’ planning and registration processes.

Interviews

To capture back-end user perspectives and experiences we conducted virtual interviews with college personnel familiar with advising and degree planning technologies, including academic advisors, IT and IR staff, and registrars. Similar to focus groups, interviews are a tool for qualitative data collection that allow for in-depth exploration of a single individual’s perspective and experience on a given topic. Our college contacts helped us identify these personnel, but all interviews were scheduled and conducted by external members of the research team in order to preserve confidentiality.

These interviews used a semistructured protocol, and we took notes during the discussion. Advisor interviews focused on their use of planning and scheduling tools, perception of the tools and their usefulness for BLI/LI students, and suggested improvements of the tools. These interviews also provided context for the broader advising structure within which the tools are used. IT, IR, and registrar interviews focused on the constraints faced by broad-access institutions when implementing and deploying advising products, as well as suggestions for improvement.

Analysis

Student focus groups resulted in two types of data for analysis: the maps themselves, and the debrief conversation. We manually coded the process maps into the primary categories of the illustration: when and how students used advising and planning tools; the things they thought about, did, and used while creating course plans; their perceptions of advising and planning tools; and the challenges they faced when using those tools. We also thematically coded pain points and challenges, “trouble words,” and any written annotations. Once the initial coding was complete, we added in codes for students’ self-identified BLI/LI and first-generation status in order to identify any themes that were salient for subsets of new-majority students.

We took detailed notes during the debrief conversation and used those as our primary analytic source; however, we also voice-recorded the debrief and used the recording to fill in missing aspects of our notes, confirm direct student quotes, and parse contextual meaning as necessary. We used the notes to triangulate on the initial findings from our process map analysis, engaging in close reading to further understand the themes emerging from the process maps. For example, once we identified the cognitive complexity of program planning in the maps, we read the notes to understand how students describe that complexity, how they feel about it, and the implications it has for their decision-making processes and degree completion paths.

We also took detailed notes during the college personnel interviews. We analyzed the interview notes thematically, using them to triangulate on what we heard from students and to supplement our understanding of institutional constraints that could influence technology purchases, adoption, and implementation processes and practices.

Trustworthiness and interpretive reliability were achieved in a number of ways. First, researchers read the notes and identified potential coding schemes individually. During team meetings, we compared our potential codes and themes and identified those that resonated with the full group. Areas of disagreement were discussed until we could reach consensus on what appropriate codes and themes would be, given our respective reading of the data. These consensus codes and themes became the structure for our analysis.

Once we began coding, we again worked individually at first. During team meetings, we discussed tricky codes and contradictory or confusing thematic findings. Again, through discussion and close collective reading of the data, we came to consensus regarding how to resolve these analytic challenges. Finally, we ensured external validity by sharing early findings with key contacts at each college for member-checking and feedback, as well as during the report authorship phase when a fourth researcher joined the team and reread findings with fresh eyes.

Finding 1: Students Experience Course Planning as a Cognitively Complex Process

The students’ process maps and their semistructured reflections provided rich insight into how BLI/LI students approach course planning—and indicate that course planning, scheduling, and registration is a remarkably complex cognitive process. It is dynamic and multifaceted, reflecting the nuances of new-majority students’ day-to-day lives and priorities. Jesus, a Latino male who speaks Spanish at home and is an adult student, summed up this complexity by saying that when he logs into his planning tool, the first thing he thinks is, “Where do I even start?”

In analyzing the process maps, we found that, on average, students need to think about 4.25 things just to create a course plan for a single semester.2 Some students listed as many as nine. Students thought about obvious planning and scheduling concerns, such as meeting course requirements. But they also thought about things seemingly unrelated to degree progression including the quality of teaching in a given section, their work and family obligations, unexpected fees and costs, and whether or not they find a class interesting or relevant to their interests.

In fact, 21 of 24 of the students thought about graduation requirements. But 14 students also thought about their personal or work schedules; 11 thought about course timing, usually in relation to other obligations; and 10 thought about their course load. Overall, there were 28 mentions of graduation requirement-related “thinkings,” but 60 mentions of thinkings not directly related to graduation requirements. The cognitive complexity of program planning is further reflected in the number of steps students take (3.7, on average) and tools and information sources they access (3.8, with some students relying on as many as 7).

The complexity of the program planning and scheduling is evident in both students’ maps and their reflections on the mapping processes. Al, the student we met at the beginning of this paper, described his map by saying,

I made mine messy as an artistic statement. I find the process very confusing and frustrating and a lot of going back-and-forth between different documents . . . I was surprised at how much thinking goes into registering that we don’t think about . . .

If we look at his map (see Appendix B), we can see how there are multiple paths to registering for courses, and the number of considerations that have nothing to do with graduation requirements.

Valeria, a Chicana student who is undocumented and therefore ineligible for financial aid, reiterated the point that each step in the planning process requires cognitive complexity. (Her map is also shared in Appendix B). She said, “My process is kind of a step-by-step process but each step requires a lot of thinking and going over multiple times so I can choose the best options.” She noted that the available tools were useful since they showed what courses had been completed and “what to focus on next,” but also implied that there were many considerations outside of what was provided by those tools. In her map, she notes having to drop a class due to unexpected and unstated course fees, reaching out to professors to get additional information about courses, and using ratemyprofessors.com and speaking to others who have taken classes. These aspects of planning indicate that she needs information about the classes themselves—not just the requirements listed in the planning tool—to build an academic plan and schedule that will work for her.

That students consider many things when registering for classes is not a new insight. However, the quantification of this complexity via program maps is important—and as we will see, has implications for the efficacy of course planning software. The specific types of things students think about are the types of things they need software to address. Understanding how students themselves approach the cognitive task of course planning and registration is an important first step in building tools that meet their needs.

Finding 2: The Need to Supplement Program Planning Tools

Based on our analyses of the maps and semistructured interview data, students generally engage with advising and planning tools when it is time to register for classes or check their graduation status. By and large, they do not use the tools to create long-term academic plans or communicate with advisors about their academic progress, although those functionalities are available. Instead, students log in to check the classes they need, when they are offered, and whether or not a given course meets their requirements. In other words, students engage with planning tools in ways that are task-oriented.

Students find that the tools are adequate for these basic scheduling purposes. They felt like the tools did what they were supposed to do—give them insight into the classes they had taken and what they needed to take, and provide an avenue for registering for the next semester. When explicitly asked how they felt about the planning tools at their disposal, 11 students made comments that were explicitly positive, while only five gave negative comments (the remainder had no opinion).

That said, further analysis indicates that technology-based products do not fully address the needs faced by BLI/LI students during the course planning and registration process. Recall that students’ maps indicated that on average, students in our study used 3.8 different tools and information sources to plan and register for their courses, with some students relying on up to 7. Students noted that they need to supplement campus-based advising tools with external tools, notably the website ratemyprofessors.com (8 students) and pencil and paper (7 students).

Notably, most of these tools were not program planning tools, indicating that the tools as they currently exist do not provide functionality to meet all of the cognitive and logistical demands faced by students during the course planning process. In other words, because program planning is such a complex task, students need a complexity of information—but that information is not currently housed in available technology tools, thus forcing them to turn elsewhere and introducing even more complexity into their course planning process.

Finding 3: BLI/LI Students Experience Challenges Using Existing Degree Planning Tools

The data collected during the reflective portion of the focus groups provides further evidence that advising and planning tools are not helping BLI/LI students build course plans that help them progress to graduation. When probed, students indicated that the tools have room for improvement and that the available technology, while adequate, can create pain points or challenges. These challenges were also identified by advisors and back-end staff, confirming student perceptions that existing tools are insufficient to accelerate student success.

First, students in our study viewed some of the technology tools not as simple to use, easy to interpret, or as intuitive as they would like. They found that often, information was hidden or opaque, or the visual appearance of the tool was confusing. Eight students indicated that this was a problem for them. Teddy, a Black male student, described his frustration by saying that the tool “is a pie chart. A lot was green so I thought I was done. But if you click something else, there are major requirements. And that was a lot of red.” As a result of misinterpretation like Teddy’s, a number of students ended up making scheduling and registration mistakes that cost them time or money; others indicate that the confusion leads them to use other information sources or resources.

Back-end users confirmed this perception, adding that usability was particularly challenging for students who relied on mobile interfaces, as is common among low-income students. Miranda, who leads a student-facing service unit, explained that:

Some of the systems are not intuitive or not mobile friendly. There are a lot of pop-up windows that do not work on students’ phones. Most if not all students are using their phones and workflows are not designed for mobile. It frustrates them and they need help to complete the tasks on a computer.

Second, students and staff alike find the available advising and planning technology tools “clunky.” Students noted that the tools often take a long time to navigate, do not connect to other tools or information sources, or take a long time to access and use. These complexities go beyond the presentation of information to the functionality of a product itself. Octavia, a Black, first-generation college student who works full time as part of a single-parent household, described using a course scheduling tool by saying that the “whole website bugs out . . . The page won’t work, it sends you back two pages, go back to the class. You have to make sure you press to modify the search or it will redo the whole search.”

College staff confirmed these challenges, often noting that the reliance on multiple systems with slow data transfers led to the “clunkiness” felt by students and, ultimately, inaccurate information. For example, overnight or longer lead times to transfer data between two systems can affect the student experience with registration or create confusion for the student. Julie detailed one example of this, explaining,

The data between the student information system and the degree audit is not always refreshed overnight. It can take 3 or 4 days even . . . They declare it in the SIS system but the degree audit is not updated. It is showing students as undeclared and their courses are not populated properly in the degree audit system.

A third challenge posed by degree planning tools for students in our study related to the language used. New-majority students found that much of the terminology was overly complicated or unfamiliar. As part of the process mapping exercise, we asked students to list words they encountered during planning, registration, and scheduling that were confusing or made it difficult to make good choices. Students listed 25 different “trouble” words, many of which are used in the tools themselves but are not explained to or defined for students. The most common of these were: corequisite, prerequisite, asynchronous, and Writing Intensive. A full list of these words can be found in Appendix D.

Finally, and perhaps most importantly, students found that the information they needed most was not included in the tools, limiting their efficacy. As discussed in the previous section, students take a robust set of considerations into account when planning and scheduling their courses. Much of this information is not available in the institution-provided tools. Students frequently noted that they had trouble finding out if courses were online or in person; accessing information about additional fees; if course sections have already been filled; or specifics regarding the instructor and course expectations. Some students, like Nika, a Black student who works, noted that available information is often outdated.

College personnel concurred with this assessment, though often contextualized this challenge within the constraints faced by institutional technology and data infrastructures. For example, a student success staff member named Hakim noted that at his institution, it can take up to three days for a new course section to be added into a degree planning system. Though such delays may be understandable given the resource constraints faced by broad-access institutions, they still created challenges for students and ultimately reduced the ability of degree planning tools to meet those students’ needs and support their academic progression.

Finding 4: Low-Income Students are Particularly Disadvantaged by Existing Planning Tools

Because we were keen to center the experiences of BLI/LI students, we also analyzed the data with an eye towards differences across self-reported racial and income groups. Students report engaging with technology similarly. Most also indicated that they believe the tools are race-neutral, in that their experiences do not vary based on their race or ethnicity (both in terms of what they report and with regards to answers when explicitly asked if race influences their technology engagement).

However, students did indicate that income status substantially influences their engagement with advising and planning technology tools. Students report that the reliance on technology disadvantages low-income students, particularly because these students are less likely to have consistent internet access, which in turn creates barriers to timely registration. This might seem like a small challenge, but when contextualized within the constraints discussed earlier, it is compounding and therefore substantial. Al, whom we have already met, explained that he has trouble leveraging his priority registration status to get the classes he needs because his internet goes down. His graduation was delayed as a result. He said, [we all] “register at the same time and people with consistent internet are prioritized just by the fact that it’s first-come, first-served.”

The challenges identified in finding 3, in particular, had real-world consequences for the students in our study. As our findings make clear, building a course schedule is, for many BLI/LI students, a complicated puzzle in which they need to fit together academic requirements, work demands, family obligations, and financial realities. Any delay in access to a course schedule or registration opportunity, or any missing piece of information—the very pain points identified by students in finding 3—can create a cascading impact that results in excess stress, delayed graduation, or additional costs.

Students are careful to plan course schedules that enable them to balance work and school; if they are unable to register on time, and the class sections they need are filled, it is often impossible for them to select an alternative. Instead, they have to wait until a later semester to take the classes they need. Sireana, a low-income, first-generation college student who works full time to support her family of four, explained that she tries to build a draft schedule as soon as courses are released. “I figure out my schedule months ahead of time, as soon as the mock schedules come out, I have to start planning for what my semesters will look like so I can work around it.” Her map is included in Appendix B.

Valeria, whom we met earlier, explained that she enrolled in a course only to find out that the materials cost extra, and she had to drop it as a result. Jennifer, a Latina first-generation student who took a break from college and has now returned, explained how tools that were difficult to interpret created undue stress and almost waylaid her graduation. She relies heavily on a degree planning and audit tool but did not see a section of the tool that indicated she needed additional courses. She did not find out that she was missing required courses until she applied for graduation—at which point, she had to scramble to find an “alternative credit option” that would enable her to stay on track. With great emotion, she described how, even though she ultimately worked it out, the situation was “stressful.” She emphasized that using the tools is not difficult but “what’s difficult is to know what I do not know. . . . I didn’t know [what was missing or what alternative credits were.] I’m first gen. I don’t have anyone to tell me it’s all right.”

As these examples illustrate, technology pain points were particularly salient for low-income and first-generation3 college students. The students in our study were clear that the technology tools they were expected to use assumed a base level of knowledge that they did not have. Even supposedly simple things, like knowing when to register or what higher education terminology meant, were not simple for those who did not have family or friends to guide them. Mariaumu summed up this point by saying, “You have to know what you’re looking at. [The technology] assumes you know things.” In other words, like with much of higher education, assumptions undergird the resources and technological tools designed to support student success.