Assessing Student Learning in Academic Advising Using Social Cognitive Theory Free

For this study, we empirically examined the possible changes in students' self-efficacy and self-regulated learning-strategy levels in academic planning during academic advising processes. Using social cognitive theory constructs of self-efficacy and self-regulated learning, we addressed the call for measuring student learning outcomes within the context of academic advising at a California community college. The research directly applies to academic advisor instructional methods used to develop self-efficacy and self-regulated learning and to the assessment of the outcomes. Specifically, it can provide answers to questions regarding the specific instructional methods used to increase self-efficacy and learning. As Schunk (2008) stated,

With this purpose in mind, we created the following three research hypotheses:

H1 helps determine whether intervention processes affect the desired learning outcome of self-efficacy in academic planning.

H2 shows whether the intervention processes exerted any effects on the desired learning outcome of self-regulated learning in academic planning.

H3 shows whether a positive reciprocal relationship existed between student-rated self-efficacy beliefs and academic advisor–rated self-regulated learning-strategy levels. Such a relationship would mean that these two variables build upon each other, mutually enhancing their effects.

We used three instruments (Appendices A, B, and C) for assessing self-efficacy and self-regulated learning in academic planning: the Counselor Rubric for Gauging Student Understanding of Academic Planning (Erlich, 2008) (hereafter the rubric) (Appendix A), Micro-analytic Assessment Questions for Self-regulated Learning Phases and Academic Planning Strategies (Erlich, 2009a) (hereafter micro-analytic questions) (Appendix B), and the Student Survey for Understanding Academic Planning (Erlich, 2009b) (hereafter the survey) (Appendix C). A detailed analysis of these instruments' validity and reliability were previously published (Erlich & Russ-Eft, 2012).

The academic advisor rated students' pre- and post-intervention self-regulated learning-strategy levels for academic planning using students' answers to micro-analytic questions and the rubric definitions. Four strategy levels for academic planning were used in the rubric (value): no recognition (1), recognizes (2), chooses (3), and creates (4). We assessed five content areas in academic planning: associate degree general education, associate degree major, earned associate degree, transfer general education, and transfer major.

The advisor did not ask students to rate their self-regulated learning-strategy levels for two reasons. First, we wanted to obtain two different perspectives about the academic advising session: from the advisor's perspective of a student's demonstrated self-regulated learning-strategy levels and from the student's perspective as per his or her rated self-efficacy in academic planning. Second, as an expert in the field, the advisor could judge a student's strategy levels within a broader context of academic planning than students could apply to themselves.

Students were asked specific advising questions, which fell within a broad range of knowledge and strategy content. Using the rubric, the advisor placed each student's responses within this broader context when evaluating self-regulated learning-strategy levels.

Following academic advisor interventions, students rated their post-intervention self-efficacy for academic planning and then their pre-intervention self-efficacy. Using the survey, students rated their own self-efficacy on a 0-to-10 scale on three different challenge levels of difficulty—recognizes, chooses, and creates—allowing us to assess their self-efficacy on the same five academic-planning content areas that their academic advisor evaluated with the rubric.

Although this study's quasi-experimental design did not address causality, it allowed us to examine the relationship influences on learned outcomes stemming from intervention processes. Statistically significant differences in learning between pre- and post-intervention, as administered within a 30-minute advising session, raises the question: “What learning principles constituted the intervention processes being delivered?”

The core learning principles used by the academic advisor in structuring the academic advising interventions were based on social cognitive theory (Bandura, 1986, 1997). We hypothesized that by cycling students through the self-regulated learning phases of forethought, performance, and self-reflection (Zimmerman, 2000; Zimmerman & Cleary, 2006) we would see increases in self-efficacy and self-regulated learning for academic planning. These interventions included using a) micro-analytic questions as prompts to facilitate students' cycling through the self-regulated learning phases, b) observational learning, c) guided mastery, and d) cognitive and mastery modeling with instructional aids. Furthermore, this guidance incorporated the four sources for building self-efficacy: practicing the newly learned skills, comparing skills to the ones modeled by the academic advisor, receiving supportive encouragement and feedback from the advisor, and reducing anxiety. The student focused on graduated process goals while the academic advisor guided the direction of the session toward meeting the student's outcome goals. Such guided mastery followed Bandura's (1997) recommendations for this process. To meet the session goal, the academic advisor primarily helped students become more self-confident and self-directed in their academic planning.

We used the retrospective pre- and post-test design to study the students receiving specific advising interventions, and we imposed a pre- and post-test design for determining academic advisor ratings. The advisor implemented the following sequence of steps in the advising session:

1. Student arrived for a prescheduled appointment. If the student wanted academic advising services, then the counselor asked micro-analytic questions. Offered as the forethought phase, these questions were designed to assess the student's educational goals and strategic plans for reaching them. Specifically, micro-analytic Question No. 2 served as the pre-test measure for H2 (regarding pre-intervention self-regulated learning-strategy levels in academic planning). The advisor used the rubric to rate a student's demonstrated self-regulated learning-strategy level in academic planning.

2. As part of the intervention phase, the advisor used cognitive modeling with instructional aids to teach students a new, more complex academic-planning strategy level. Via this process, the advisor answered the student's questions, specifically addressing reasons for seeking academic-advising services.

3. In the performance phase, students practiced (emulated) the new academic- planning strategy modeled by the academic advisor and received constructive feedback for improved performance.

4. The self-reflection phase was characterized by a student's demonstrated comprehension of the new academic-planning strategy. First, the advisee explained it, reasons for using it, and the benefits of its use. Student answers to micro-analytic Question No. 3 constituted the self-regulated learning-strategy-level outcome, the post-test measure for H2, which the advisor assessed at post-intervention using the rubric.

5. Each student completed the survey to retrospectively rate pre-intervention self-efficacy beliefs at the beginning of the session as well as current (or post) self-efficacy beliefs in academic planning. This process allowed for the testing of H1.

6. At the end of the session, the advisor obtained student's consent to include her or his self-efficacy answers from the survey as well as the academic advisor's ratings of the student's self-regulated learning in academic planning as determined from the rubric and micro-analytic questions.

To minimize Type I and II errors, the α value was set at .05, or a willingness to make either type of error 5 times out of 100 possibilities. To maximize the power (1−β) of detecting a significant difference, β was set at .90, or in other words, 9 times out of 10 a significant difference would be detected. Based upon these calculations, the necessary sample size needed was 44 students (Lenth, 2006-2009).

In the current study, N = 120, we conducted statistical analyses for two types of content in the advising session as determined by advisee questions regarding associate degrees and transfer: Seventy students asked about and received advising interventions about both associate degree and transfer content; 15 of the 120 students asked only about associate degrees; 35 of the 120 were concerned only with transfer. Because relatively few students expressed interest only in the associate degree or only in transfer, we combined their data, as follows, to provide valid and reliable information: Seventy students who received all interventions plus the 15 tagged as interested only in associate degrees comprised the associate degree group, and the transfer group included the same 70 participants plus the 35 solely interested in transfer.

To determine whether significant mean differences existed between the associate degree and transfer groups in post self-regulated learning-strategy levels, we conducted a one-way analysis of variance (ANOVA). Results showed significant post–intervention mean differences between these two groups (Appendix D). These data justified our division of the sample into two different groups for data analysis comparisons.

The associate degree group consisted of 43 women and 42 men, and the transfer group included 51 women, 51 men, and 3 who did not identify gender. The women ranged in age from 18 to 51 years, and the men were between 18 and 60 years old; the mean age was 23 years and 70% of the sample ranged in age from 18 to 24 years.

We conducted a test of normality on the distribution of responses generated from both the rubric and the survey using the Shapiro-Wilk (W) statistic. Results on the rubric and survey both showed significant pre- and post-intervention differences: W = .81 to .85, p < .001, on the premeasure compared to .59 to .72, p < .001, on the postmeasure. The null hypothesis was rejected, which means that the data were not normally distributed. Therefore, both parametric and nonparametric statistics were used for conducting mean difference and correlation tests on the three hypotheses. The two sets of tests led to the same qualitative conclusions.

The Oregon State University Institutional Review Board approved the procedures of this study. To protect students' rights, we gave a consent form to each student that detailed the nature of the study and the right to refuse inclusion of answers in the study.

We examined three hypotheses to determine a) any increased levels of student self-efficacy between the retrospective pre and post scores following academic advising interventions, b) increased levels in self-regulated learning-strategy levels between pre and post scores as rated by the academic advisor, and c) a positive reciprocal relationship between self-efficacy and self-regulated learning in academic planning.

H1 stated that students engaged in the academic advising process experience increased levels of self-efficacy in academic planning strategies as a learned outcome. We conducted a paired-samples t test and a related samples Wilcoxon signed-ranks test to evaluate this hypothesis. The survey showed students' post-intervention self-efficacy ratings in academic planning as well as retrospective pre-intervention ratings. The paired-samples t test and related samples Wilcoxon signed-ranks test measure differences, if any, in the scores of one condition (post) and another condition (pre) (Neil Willits, Senior Statistician Consultant at the University of California, Davis, personal correspondence, June 22, 2010).

The paired-samples t tests showed significant differences (p < .001) on all pre and post self-efficacy means across all associate degree and transfer content as well as across all challenge levels of difficulty (recognizes, chooses, and creates) (Appendix E). Pre-intervention means ranged from 4.91 to 6.78. Post-intervention means ranged from 9.30 to 9.82 on a 10-point scale. Students rated their self-efficacy in academic planning significantly higher on the post condition than they did when retrospectively rating their pre-intervention self-efficacy. The related samples Wilcoxon signed-ranks test showed the same significant differences on the pre and post self-efficacy means as did the t tests (p < .001). Results support H1: Students engaged in the academic advising process experienced increased levels of self-efficacy in academic planning as a learned outcome.

H2 stated that students engaged in the academic advising process demonstrate increased self-regulated learning-strategy levels in academic planning as a learned outcome. Using micro-analytic Question Nos. 2 and 3, we applied the rubric to rate students' demonstrated pre and post self-regulated learning of academic-planning strategy levels. A paired-samples t test and a related samples Wilcoxon signed-rank test were employed to test this hypothesis. Using the rubric definitions, the academic advisor rated a student's demonstrated self-regulated learning strategy levels both at pre-intervention and post-intervention periods using the student's answers to the two micro-analytic prompts. See Appendix F.

Using paired-samples t tests on all five content areas, pre and post self-regulated learning-strategy level means showed significant differences (p < .001). Signed-rank tests showed the same results.

Pre-intervention strategy-level means for associate degree content ranged from 1.87 to 1.88, which contrasted with post-intervention strategy-level means that ranged from 3.10 to 3.11. Pre-intervention strategy-level means for transfer content ranged from 2.58 to 2.64, which contrasted with post-intervention strategy-level means that ranged from 3.48 to 3.50. The related samples Wilcoxon signed-ranks test also showed significant mean differences on all five content pairings (p < .001). Judging from the mean changes within each condition as evaluated by the advisor, students demonstrated increased self-regulated learning-strategy levels following interventions. H2 was supported.

However, we add a caveat to this finding. We cannot attribute pre- and post-measure increases in strategy levels completely to the academic advisor interventions. A significant (p < . 01) and strong correlation between pre and post self-regulated learning-strategy levels existed on all five content areas: r = .70 for the associate degree and r = .89 for the transfer topics (Appendix G). In other words, the advisor chose interventions based upon the student's demonstrated pre-intervention strategy levels. Students came to the session with a foundation of knowledge, which the advisor measured by the pre-intervention assessment of the student's self-regulated learning-strategy levels. Thus, students' increased self-regulated learning-strategy levels demonstrated at post-intervention were influenced by both the interventions within the session and their pre-intervention strategy-level foundations.

A linear regression analysis between pre- and post-intervention strategy levels on each of the five content areas confirmed the significance (p < .001) and strength of this predictive relationship. Pre-intervention strategy levels on associate degree content proved to be moderate predictors of strategy-level learned outcomes (R² = .49). Pre-intervention strategy levels on transfer content proved to be strong predictors of strategy-level learned outcomes (R² = .80).

H3 stated a positive reciprocal relationship exists between self-regulated learning-strategy levels (as rated by the advisor) and self-efficacy levels in academic planning (as rated by the student after the intervention as well as retrospectively). If a positive reciprocal relationship exists between self-efficacy beliefs and self-regulated learning in academic planning, then as the value of one variable increases, the value of the other variable should also increase.

We used a one-way ANOVA with post hoc Tukey comparison to test whether between-group differences existed in pre-intervention self-efficacy (as rated by the student retrospectively after the intervention) and on pre-intervention strategy levels in academic planning (as rated by the advisor). This analysis divided students into four groups based on the advisor's ratings of students' pre-intervention strategy levels on a scale of 1 to 4: no recognition = 1, recognizes = 2, chooses = 3, creates = 4. Through the analysis, we compared the four groups using students' pre-intervention self-efficacy ratings. Students had rated their pre-intervention self-efficacy retrospectively on associate degree and transfer content covering three challenge levels: recognizes, chooses, and creates. Because they were never exposed to advanced tasks during the evaluation, those in the no-recognition strategy-level group were not asked to answer creates self-efficacy questions; their self-efficacy ratings on this level would be irrelevant.

With the one-way ANOVA, we found significant mean differences in retrospective pre-intervention self-efficacy beliefs between the four strategy-level groups. Results on the associate degree group (general education, major, earned degree) and challenge levels (recognizes and chooses) showed significant mean self-efficacy differences (p < .001) (Appendix H).

Post hoc Tukey comparisons showed that significant mean differences existed in the associate degree group for students' retrospective pre-intervention self-efficacy ratings. Specifically, the retrospective pre-intervention self-efficacy ratings for students categorized by the advisor at the no-recognition strategy level were significantly lower than those categorized at the recognizes, chooses, and creates strategy levels (Appendix I).

Self-efficacy means showed increases in conjunction with higher, more complex strategy levels. Students who had higher retrospective pre-intervention self-efficacy ratings also functioned at higher pre-intervention strategy levels as rated by the advisor. See Figure 1.

We conducted a one-way ANOVA with post hoc Tukey comparison on data from the transfer group. Results showed significant mean differences (p < .001) for students across all strategy levels (no recognition, recognizes, chooses, and creates), as rated by the advisor, in retrospective pre-intervention self-efficacy (Appendix J). The post hoc Tukey comparisons showed significant mean differences in retrospective pre-intervention student self-efficacy based on advisor strategy-level ratings of no recognition, recognizes, chooses, and creates strategy levels (Appendices K1 & K2), except between chooses and creates, which showed no significant differences.

Just as for the associate degree group, retrospective pre-intervention self-efficacy means for the transfer group continuously rose in conjunction with higher, more complex strategy levels. Students who had higher pre-intervention self-efficacy ratings also functioned at higher pre-intervention strategy levels. See Figure 2.

These results lend support to H3 that a positive, reciprocal relationship existed between pre-intervention self-efficacy (as rated retrospectively by students) and pre-intervention strategy levels in academic planning (as rated by advisor). As one variable increased, so did the other.

How strong was this relationship between students' retrospective pre-intervention self-efficacy ratings with advisors' pre-intervention strategy-level ratings on all five content areas of academic planning? Pearson and Spearman correlations were significant (p < . 001) on all five content areas of academic planning. Associate degree group Pearson correlations ranged from .39 to .51, and on transfer group data, Pearson correlations ranged from .51 to .67 (Appendix L).

These correlations showed that students and the academic advisor expressed moderately strong positive agreement on their independent ratings of self-efficacy and self-regulated learning. For example, if the advisor rated a student's pre-intervention strategy level at no recognition, students in this category tended to rate their pre-intervention self-efficacy lowest (Figures 1 and 2). If the advisor rated a student's pre-intervention strategy level at creates, students in this category tended to rate their pre-intervention self-efficacy high (Figures 1 and 2). Because the ratings were independent from each other—that is, students conducted their own rating of self-efficacy and the advisor rated student self-regulated learning separately—this finding is significant in supporting H3.

Conclusions drawn from these results confirmed that when rated retrospectively, increased self-efficacy rose in association with increasingly complex self-regulated learning-strategy levels. H3 was posed to determine whether a positive reciprocal relationship existed between self-efficacy beliefs in academic planning, as rated by the student, with self-regulated learning-strategy levels, as rated by the advisor. A positive reciprocal relationship was found between students' retrospective pre-intervention self-efficacy with pre-intervention strategy levels as rated by advisor. The increase in one variable was associated with an increase in the other variable.

In this quasi-experimental, post- and retrospective pre-test design, two internal validity factors were uncontrolled per Campbell and Stanley (1963): testing and instrumentation. Testing refers to the effects of completing the first test on the scores of the same test taken a second time. The post ratings of students and the academic advisor may have been influenced by their pre-ratings.

In addition, biases have been identified from retrospective pre- and post-test designs (Taylor, Russ-Eft, & Taylor, 2009). Retrospective self-reports tend to inflate changes and effect sizes. However, Taylor et al. argued for the validity and efficacy of retrospective ratings in certain settings.

Instrumentation refers to use of unique measures with different scorers. Students rated their own self-efficacy using the same assessment instrument. However, these 120 students each used their own standards to judge their self-efficacy, so agreement across student ratings was probably low. In addition, any advisor diversion from the rubric could have introduced bias into the pre and post ratings.

To help mitigate these biases, we used both student and advisor ratings for comparison. Additionally, to eliminate guesswork about learning levels, the advisor rated behaviorally demonstrated students' self-regulated learning-strategy levels with the rubric.

Those in the field have been looking for ways to view academic advising from a learning perspective as well as assess student learning outcomes. In this study, we applied a highly sophisticated learning and motivation theory to the issue with hopes of facilitating advisors' use and measurement of self-efficacy and self-regulated learning within their practice. We attempted to show how social cognitive theory (Bandura, 1986, 1997), and specifically the constructs of self-efficacy and self-regulated learning (Zimmerman, 2000), could be productively applied to assess student learning processes, determine student learning levels within the academic advising setting, and improve advising processes and interventions.

We described the core academic advising intervention principles, which involved a) cycling students through the three self-regulated learning phases, b) choosing an appropriate intervention level that matched student's current regulatory skills, c) using observational learning, d) cognitive and mastery modeling with instructional aids, e) incorporating four sources for building self-efficacy, and f) maintaining student focus on graduated process goals while the advisor guided the session toward meeting student outcome goals. We encourage advisors to conduct studies using these intervention principles and assessment instruments in a variety of settings and share results.

Those undertaking research in self-regulated learning have investigated whether teachers could adapt their teaching methods to increase their students' self-regulated learning (Perry, Vandekamp, Mercer, & Nordby, 2002; Zimmerman, 2008). The results from this study showed that academic-advising teaching methods could be adapted to increase students' self-regulated learning strategies. The academic advisor in the study successfully identified the appropriate intervention levels needed to move students to the next higher level in self-regulated learning-strategy levels.

These results foreshadow the future development of a practical education-accountability system that uses, within the learning context, the expertise of the educator who structures goal-directed learning interventions while students take responsibility for their own learning. Within this interaction, process, and feedback loop, assessed short-term student-learning outcomes could eventually lead to realization of long-term program-learning outcomes.

With this study, we demonstrate the theoretical usefulness of applying social cognitive theory to the field of academic advising for assessing student learning outcomes. Social cognitive theory explains human agency through the interdependence of three main determinants using a three-point model called “triadic reciprocal causation” (Bandura, 1986, 1997). The model visually resembles a triangle with the following points interacting and mutually influencing each other: personal factors (P), which include cognitive, affective, and biological events; environment (E); and behavior (B). By using the constructs of self-efficacy and self-regulated learning (P) in academic planning, we identified increases in self-efficacy in academic planning (B) following interventions (E), increases in self-regulated learning-strategy levels (B) following interventions (E), and a positive, reciprocal relationship between retrospective pre-intervention self-efficacy (as rated by students) and pre-intervention self-regulated learning-strategy levels in academic planning (as rated by the academic advisor).

H1 was supported, showing students rated their post-intervention self-efficacy higher than their retrospective pre-intervention self-efficacy rating. Students asked the advisor questions they wanted answered. Therefore, within the specific context of the studied academic-advising session, students showed post-intervention understanding of the answers to their questions and ways to arrive at these answers. This perception of goal progress substantiates their increased self-efficacy in performing academic planning as taught during the advising session.

H2 was supported, showing students' self-regulated learning-strategy levels (as rated by the academic advisor) were significantly higher at post-intervention than at pre-intervention. Additionally, pre-intervention strategy levels were strong predictors of post-intervention strategy-level learned outcomes. In other words, both advisor interventions and student's pre-intervention strategy-level knowledge influenced the noted increase in self-regulated learning-strategy levels. The students' foundations of knowledge in academic planning played a major part in choices of the intervention administered.

H3 was supported, showing that increasing pre-intervention self-efficacy levels (retrospectively rated by students) were associated with increasing pre-intervention self-regulated learning-strategy levels (rated by the academic advisor). A complementary relationship existed between these two variables such that as self-efficacy rose (as rated by students), so did strategy level accomplishments (as rated by the academic advisor), and as self-regulated learning-strategy levels rose, so did self-efficacy.

These findings comport with Zimmerman's (2000) self-regulated learning model and bidirectional influences on changing behavior as espoused in social cognitive theory. Zimmerman viewed self-regulated learning as self-generated thoughts, feelings, and actions that adapt through a three-step cycle. This three-step cycle includes forethought, performance, and self-reflection. Using Zimmerman's model, in the self-reflection phase, students compared their current self-efficacy beliefs to those held during the forethought phase (personal factors). Based upon their learning experiences throughout the academic advising interventions (environment), their demonstrated academic-planning strategy levels changed (behavior). This in turn influenced their judged pre- and post-intervention self-efficacy beliefs. “Within the model of triadic reciprocality, action, cognition, and environmental factors act together to produce changes” (Bandura, 1986, p. 521).

Note. ASSIST (n.d.); GE is general education; CSU is California State University; IGETC is Intersegmental General Education Transfer Curriculum. Copyright © Richard J. Erlich, Counselor, Sacramento City College, May 18, 2008. Used with permission. See the Authors' Notes for contact information on the rubric and scoring.

Note. Copyright © Richard J. Erlich, Counselor, Sacramento City College, September 18, 2009. Used with permission (Erlich, 2009a). *Erlich (2008, 2009b). See the Authors' Notes for contact information on the rubric and scoring.

Note. ASSIST (n.d.); CSU is California State University; IGETC is Intersegmental General Education Transfer Curriculum. Copyright © Richard J. Erlich, Counselor, Sacramento City College, August 9, 2009. Used with permission.