At the beginning of my LDT 506: Evaluation of Learning System course, I was tasked with completing an Evaluator Self-Assessment based on the AEA Evaluator Competencies. My results did not surprise me since I am new to the realm of evaluation. If I were to rate myself on a scale of 1-6, I would firmly place myself at a 2. My background in teaching created transferable skills in collaboration and teamwork, which translate well to people-oriented domains such as Interpersonal Competence and Planning/Management. However, my experience in assessment rather than evaluation has led to weaknesses in the more technical domains, such as methodology. Most importantly, this assessment shifted my mindset of the focus of evaluation itself. While effective methodology is vital to evaluation, it’s not the only piece to the puzzle.
My two main strengths are specific competencies in the Interpersonal Competence and Planning/Management domains. These two competencies are AEA 4.7 (teams with others when appropriate) and AEA 5.2 (listens to understand and engage different perspectives). I exhibited both of these strengths during a district meeting where my fellow science teachers and I were responsible for selecting a new science curriculum. After hearing various presentations about our curriculum choices, I, along with many other teachers, were drawn to an inquiry-based program, which reflected our state standards closely. However, there was pushback from veteran teachers from my school’s science team initially. Upon first glance, it appeared they simply did not want change to the current curriculum. After further conversation, my colleague and I uncovered the true reason for the resistance to change: the amount of time required to overhaul the years of extensive work these teachers had put into their current slideshows, assignments, and assessments. The veteran teachers understood the merit of the new curriculum, but they were worried about the time commitment needed to essentially start from scratch. I understood their frustration rather than brushing off their concerns, and I took the time to listen to their perspectives, which aligns closely with AEA 5.2.
After listening to my colleagues, I teamed with them to create an effective solution to mitigate their concerns. Rather than each individual teacher take on the responsibility of creating their own unique set of materials, we would work together to create a specific set of resources to share with the team. Two teachers would be responsible for developing the slide decks, two teachers would create the assignments, and one teacher would be responsible for crafting the assessments. Working together as a group ensured the success of each individual teacher, which follows competency AEA 4.7. Before completing my self-assessment, I thought these types of skills were not necessary for evaluation practice. However, Stevahn et al. (2005) found that people-oriented competencies such as interpersonal competence, reflective practice, and situational analysis are crucial to becoming a successful evaluator. The AEA Evaluator Competencies also place a heavy emphasis on these skills in their Interpersonal and Context domains. Evaluation is not only a numbers game, but a people game too.
This experience shone a light on my growth areas too. The two main competencies I would like to focus on are AEA 1.3 (selects evaluation approaches and theories appropriately) and AEA 2.14 (uses evidence and interpretations to draw conclusions, making judgments and recommendations when appropriate). Relating back to my previous example, our old curriculum had not been evaluated to determine its success rate. It would have been extremely helpful if our district office had evaluated the old curriculum and provided us with data showing why it may have been ineffective. While the new curriculum appeared to align with our state standards because it was inquiry-based, I did not have evidence to back my conclusions regarding how this would impact student success. Similarly, our veteran teachers did not have evidence to back their conclusions regarding student success with the old curriculum. With this data, we would have been able to conduct a more data-driven conversation. After reflecting on this experience, I can see how effective evaluation can make it easier to make recommendations, especially when they involve change.
Moreover, I found a few surprises when reviewing the evaluator competencies. Initially, I believed evaluation only focused on methodology such as collecting and analyzing data, but the importance of cultural competence can be found across all of the AEA domains. While the success of the school, business, and organization we are serving is crucial, we must also focus on the success of our learners. By understanding the background of our learners and their own unique cultural contexts, we can ensure the judgments and recommendations we provide are impactful.
Ultimately, I realized I do not need to be in the role of an evaluator to possess an evaluator mindset. I am currently piloting an AI tool in the ASU Online Enrollment Center, which is assisting enrollment coaches while they are on calls with students. The AI tool recognizes questions students ask, and it will pull up the appropriate resource for enrollment coaches to reference in the conversation. I would like to collect some of my own data regarding its effectiveness (how quickly it pulls up resources and if the information is correct) to make evaluative judgments I can share during our weekly debrief meetings. Within Module 2, I would like to focus on AEA 1.3 when completing the Evaluation Proposal Analysis assignment. I plan to carefully look over the evaluation procedures to determine if the correct approach has been selected. If not, I will be able to explain why the current approach is insufficient, and I will be able to offer actionable suggestions for improvement. Currently, this skillset is not within my wheelhouse, so I am looking forward to using this week’s resources to learn and grow!
References
American Evaluation Association. (2018). AEA evaluator competencies. https://www.eval.org
Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26(1), 43–59.
