Chapter 9. Evaluation

Measuring progress

Evaluation is the primary responsibility of the College’s resident experts in engineering education evaluation at the UM Center for Research on Learning and Teaching (CRLT), who routinely provide assistance and expertise to conceptualize, develop, and carry out evaluations of educational innovations.

Partners designed a comprehensive evaluation of the following areas:

Improved student motivation to pursue STEM study and/or careers and improved student achievement in STEM coursework, grades, and test results.

Increased STEM knowledge of students. In collaboration with the Michigan Intermediate School Districts, student-tracking systems will assess progress (test scores, course grades, graduation, and college enrollment).

Successful partnerships among graduate student Teaching Fellows and Teachers. Evaluation will focus on the process and outcomes of collaborations.

Improved teacher STEM content knowledge and skills. Evaluation assesses the success of teacher training and professional development, especially in inquiry-based learning.

Increased parental involvement in encouraging student achievement. Students’ parents provide data about their own participation in the “Parent Learning Community” and information to evaluate program influences that is parallel to that elicited from students.

The plan developed for this partnership employs short-term and long-term surveys, anecdotal questions, participant and student tracking, and performance measurement. In addition, joint assessment strategies focus on evaluating the experiences of all participating individuals as well as the operational effectiveness of the program. For the first year, partners evaluated YPSD secondary students’, Faculty Affiliates’, and Teaching Fellows’ experiences in learning, teaching, and mentoring.

The Process

Over time, the annual evaluation process will document student motivation to learn and greater achievement in coursework as well as guide program development. Baseline statistical data on participating District students has been collected at the beginning of each of the two pilot year terms, and students will be tracked in areas partners intend to impact (e.g. application to and enrollment in college, grade point average, etc.). Obtaining data at multiple time points will enable the application of growth curve analysis to statistically model how members of the target groups, particularly secondary school students, changed as a function of program participation (e.g., in their intentions to pursue engineering and/or STEM careers). Evaluations methodology will include focus groups and interviews when appropriate.

Survey data collected on paper forms from ~600 secondary students per term is entered by hand, and free-response answers are typed into spreadsheets. Teacher surveys are administered via Zoomerang, an online survey tool that allows users to create and send surveys and analyze results on-demand. Teaching Fellow surveys were administered on paper forms for the first term 2007-8, but will be administered via Zoomerang or a similar tool for the second term. Parent Learning Community surveys are administered via paper forms during meetings.

We collected survey data and hired a Ph.D. candidate, Allison Bell, from the Center for the Study of Higher and Postsecondary Education at the University of Michigan, who constructed the initial assessment. Ms. Bell ran tests during the second term FY 2008 to ensure that classes without Teaching Fellows are not statistically different from classes with Teaching Fellows, and to determine whether classes are different at the beginning and at the end. In addition, Bell ran tests from post tests between classes to determine whether the classes with Teaching Fellows have different answers than those classes without Teaching Fellows.

Quantitative results and summative evaluation from both first and second terms, were presented at the ASEE National Conference in June, 2008.

Who is surveyed?

Secondary Students. The Partnership agreement includes College access over time to relevant District student academic progress (aggregate test scores, course grades, graduation, and college enrollment). The evaluation of student motivation includes program impact on STEM-related efficacy, value (interest, utility, importance), identity, epistemic beliefs, and intentions to pursue STEM careers. Other assessed variables include perceived parent support for STEM, student participation, and perceived teacher and graduate student influences on STEM (e.g., whether and how they have impacted students). Students also report their level of engagement in and the value of program activities. Comparison data from students in classrooms without Fellows is available from the District.

Parents. Students’ parents provide data about their own participation in the Parent Learning Community. The parents in the pilot group were selected on the basis of their children's interest in science, technology, engineering careers, but after the pilot year, many more parents have been invited to be part of the PLC program. Parents are surveyed each time they attend the Parent Learning Community meetings. Comments from the survey show that all parents believe they are receiving necessary information about student summer enrichment programs and college expectations and opportunities, and ways to apply for them. Parents say the PLC-format discussion groups are very helpful, and give parents another way to share information among each other. Information is presented in such a way that parents do learn the material. Parents have suggested that more parents should be invited to join the group, and have already invited their high school age children who are attending YPSD to join the group.

Graduate Student Teaching Fellows. Evaluation of Fellows focuses on the process and outcomes of their collaboration with teachers and faculty affiliates. Thematic analyses of their (at least weekly) journaling in semi-structured (private) blogs include critical incidents, reflections on the collaboration, and ways to improve the process and the program. Structured assessment will focus on Fellows’ efficacy with regard to general pedagogical skills, pedagogical content knowledge, coaching and mentoring skills, STEM-related identity, and perceived changes as a function of program participation. Fellows’ public and online discussions are examined for related content.

Faculty Affiliates (Teachers). Evaluation design includes assessment of the success of teacher training and professional development, especially in inquiry-based learning. Teachers report on self-efficacy levels in areas of professional development focus, on the collaborative process with fellows, how their students have been impacted as a function of program participation, and parent involvement.

Participants also evaluate specific program components, such as workshops, visits to labs, and materials. Data is collected and maintained in accordance with applicable institutional IRB rules and regulations.

Results, First Year (Academic Year 2007-8)

Click on each of the following to download copies of the pre- and post-term surveys:

Secondary Students Pre Semester Survey

Secondary Students Post Semester Survey

Teachers Survey

Teaching Fellows Survey

To further provide information on evaluation, especially as the process relates to National Science Foundation funded projects, we provided to teachers and students Useful Links for Professional Evaluation.

Useful Links for Professional Evaluation.

Looking Forward

The College and the District have committed to building and sustaining this partnership and have placed no time limit on the program. The College’s approach to building trust among District administrators, teachers and parents begins with a promise to remain as an active, perpetual partner. Following the pilot year, components of the partnership will be replicated at neighboring Districts, specifically Detroit’s International Academy and Douglass Academy, Detroit University Preparatory Math+Science Middle School. Within three years, the program is expected to impact over 1,800 students and more than 20 teachers in the partnering urban school districts.

The majority of the students who will participate in the project are considered economically disadvantaged and participate in the federal lunch program. For comparison, Michigan State Averages are 26% underrepresented minorities and 36% eligible for free or reduced-price lunch programs. YPSD participating schools’ demographics reflect YPSD-wide figures: more than 63% are underrepresented minority students, and at least 56% are eligible for free or reduced-price lunch programs. Urban Detroit’s UPA / UPSM demographics are 99% underrepresented minority students, and at least 66% eligible for free or reduced-price lunch programs. Detroit Public Schools’ Douglass Academy (all males) demographics include 99% African American population with 88% eligible for free or reduced-price lunch program, and Detroit International Academy (all females) is 97% African American population with 74% on the free or reduced-price lunch program.