Temple University, College of Public Health
|
|
The purpose of this project was to provide a systematic evaluation of the Association for Prevention and Teaching Research Online Learning Modules (http://www.aptrweb.org/?PH_LearningModules) to determine the impact on knowledge and understanding of major public health issues, as well as the ways in which various health-related disciplines can work together to address current and emerging issues in population health. During AY 2015-2016, Temple University’s College of Public Health decided to use the Online Learning Modules in all of the College’s graduate programs as the foundation for a required course, HRPR 5001 (Current and Emerging Issues in Public Health). The purpose of HRPR 5001 is to ensure all graduate students graduating are grounded in the five core areas of public health through Healthy People 2020, as well as an introduction to the value of inter-professional education. Students must complete 6 required modules and then select an additional 6 modules from the remaining list of 12 modules. In addition, there were many graduate instructors who decided to embed at least one module into the work required for a course other than HRPR 5001, thereby increasing the students’ exposure to the learning modules.
After the first year of implementing this course, the majority of respondents believe that the modules “definitely” or “probably” help them to: describe how health, healthcare, and human service professionals work together to address individual and population health issues; define health and wellness and the way various professions consider promoting health and preventing morbidity and mortality; explain the role of Epidemiology as the basic science for Public Health; recognize local, state, and federal healthcare and public health systems and the way they interact; describe environmental health hazards and their potential effects on human health; describe and interpret the use of biostatistics in public health from published public health data; compare various theoretical constructions and models of healthcare including their social, cultural, and behavioral causes; synthesize the impact of major policy, ethical, economic, professional and legal issues facing healthcare providers, insurers, policymakers and consumers; and differentiate among availability, acceptability, and accessibility of health care across diverse populations.
During the second year of implementation, we have continued to assess student satisfaction and learning. We are also taking feedback from students and faculty to identify where there are gaps in the modules and working to develop new modules that are responsive to current and emerging issues in public health.
After the first year of implementing this course, the majority of respondents believe that the modules “definitely” or “probably” help them to: describe how health, healthcare, and human service professionals work together to address individual and population health issues; define health and wellness and the way various professions consider promoting health and preventing morbidity and mortality; explain the role of Epidemiology as the basic science for Public Health; recognize local, state, and federal healthcare and public health systems and the way they interact; describe environmental health hazards and their potential effects on human health; describe and interpret the use of biostatistics in public health from published public health data; compare various theoretical constructions and models of healthcare including their social, cultural, and behavioral causes; synthesize the impact of major policy, ethical, economic, professional and legal issues facing healthcare providers, insurers, policymakers and consumers; and differentiate among availability, acceptability, and accessibility of health care across diverse populations.
During the second year of implementation, we have continued to assess student satisfaction and learning. We are also taking feedback from students and faculty to identify where there are gaps in the modules and working to develop new modules that are responsive to current and emerging issues in public health.
impact
|
Established a required course intended to ground all graduate students across the College of Public Health (700+/year) in the core areas of public health through an exploration of Healthy People 2020. |
The majority of respondents report that the modules have helped increase their knowledge of Healthy People 2020 and that the modules demonstrate the value of interprofessional education |
APTR/CDC EVALUATION Mini Grant
During academic year 2015-2016, Temple University College of Public Health made the decision to utilize the Association for Prevention and Teaching Research Online Learning Modules in all of the College’s graduate programs. The College’s Interdisciplinary Education Committee, which is composed of representatives from each department in the College, reviewed the modules and felt that Healthy People 2020 could serve as a common foundation for all disciplines to learn together. The modules were used to create a zero credit course, HRPR 5001 (Current and Emerging Issues in Public Health) that is required of all graduate students in the College. The purpose of HRPR 5001 is to ensure all students graduating from the College of Public Health are grounded in the five core areas of public health (per CEPH accreditation requirements) through Healthy People 2020, as well as an introduction to the value of inter-professional education. Since the modules were designed to be applicable to a variety of audiences we felt that implementing the modules into the curriculum for the College of Public Health would provide our students with an excellent value added component to their program of study. Additionally, it was viewed as an opportunity to encourage our faculty to promote the importance of inter-professional education by incorporating the modules into coursework or program activities and inviting faculty to develop additional modules to be added to the course.
The HRPR 5001 course has been active and accessible since Fall 2015. Currently, we are enrolling graduate students in the course who are matriculating in Fall 2017. During the implementation of the HRPR 5001 course, we have identified a variety of unanticipated issues and challenges associated with the utilization of the modules course. We are continually working to address these issues so students and faculty are getting the best experience they can with the modules and will encourage broader utilization as we believe that the modules have the greatest impact when they are embedded into coursework.
The HRPR 5001 course has been active and accessible since Fall 2015. Currently, we are enrolling graduate students in the course who are matriculating in Fall 2017. During the implementation of the HRPR 5001 course, we have identified a variety of unanticipated issues and challenges associated with the utilization of the modules course. We are continually working to address these issues so students and faculty are getting the best experience they can with the modules and will encourage broader utilization as we believe that the modules have the greatest impact when they are embedded into coursework.
current activity/project description
The College of Public Health initiated a process of restructuring in Spring 2014 to bring together the various disciplines that call the College home. The College also applied to become an accredited School of Public Health and hosted the CEPH site visit team in March 2017. The HRPR 5001 course serves of the foundation for the College’s identity as a leader in interprofessional education. The course is intended to ground all graduate students across the College of Public Health in the core areas of public health through an exploration of Healthy People 2020. The College includes a range of disciplines (Communication Sciences and Disorders, Epidemiology and Biostatistics, Health Services Administration and Policy, Kinesiology, Nursing, Occupational Therapy, Physical Therapy, Therapeutic Recreation, Social Work and Social and Behavioral Sciences), but Healthy People 2020 works as a great common ground from which to work for our inter-professional efforts. HRPR 5001 will continue as a required part of all of our graduate programs and we are committed to learning from the student experience and developing additional learning modules.
EVALUATION PROJECT ADDRESSES:
- Courses
|
PROFESSIONS/DISCIPLINES/SCHOOLS INVOLVED:
The College of Public Health was completely involved, including the following departments:
|
FORMAL AFFILIATION AGREEMENTS WITH THE SCHOOLS MENTIONED ABOVE:
ACADEMIC LEVELS TARGETED:
|
COURSE DESCRIPTION:
- Number of students enrolled in course during the most recent academic year: 1200
- Is this a shared/cross-listed course? Yes
- Title of Course: Current and Emerging Issues in Public Health and Health Professions
- Number of credits offered: 0 credits (equivalent to 3 credits)
EVALUATION METHODS USED FOR THE SUBGRANT PROJECT:
- Quantitative surveys
- Qualitative surveys
- Faculty surveys
Implementation
Following the change of array approval by the Board of Trustees to add the HRPR 5001 course to graduate programs, we needed to migrate the modules into the learning management platform BlackBoard. Working with the College instructional technology team, a BlackBoard course site for each program was developed. During meetings with program directors during the spring, it was determined that graduate program directors would be responsible for tracking the completion of the modules by all graduate students entering their program beginning in Fall 2015. Operating under a tight timeline, the sites were completed in early Fall 2015. Each site contains all 18 modules, six of which are required (Advancing Health People 2020, Social Determinants of Health, Access to Health Services, Health Information Technology, Public Health Infrastructure in the United States, and Environmental Health) and students choose six to complete from the remaining 12. In order to “complete” the module, students must pass a quiz at the end. A passing grade is 80% and students must pass the quiz within three attempts. If necessary, a student can be given the opportunity to take the quiz an additional time in order to pass, but they must get permission from their program director. Once a module is passed, a student receives a “badge” for that module. We are utilizing the same quizzes created as part of the APTR online learning modules program. A syllabus outlining the course requirements and objectives is also included on each site.
Each program is given the autonomy to implement the HRPR 5001 course based upon their program and students’ needs. For example, a program could determine that each of the six required modules would be implemented into the classes students registered for and then students would complete the final six elective modules on their own. Programs are also able to determine when the modules must be completed to meet the graduation requirement. For example, MPH students must have all 12 modules successfully completed before they begin their fieldwork practicum which occurs during the final two semesters in the program. That means that for students who are in the MPH program full-time, they must ensure that all modules are completed within their first year of study. Master of Occupational Therapy (MOT) students are introduced to the modules during their orientation and complete a number of modules as part of their orientation to the program. After Fall 2015, newly matriculated students are enrolled in their program BlackBoard site. It is expected that students are introduced to the HRPR 5001 course during their orientation by their program director and/or advisor so they have the opportunity to be engaged with the modules early on in their academic career.
Early on we identified the need to create a BlackBoard site specifically for non-matriculated students. Frequently, we have prospective students who enroll in introductory courses. A number of these courses may choose to utilize the modules given the alignment of the content. These students are not enrolled in any of our programs so they would not have access to the modules. Upon an instructor’s request to College administrators, the non-matriculated student will be enrolled in the Non-Matriculated Students BlackBoard site so they can access the assigned modules. If the student applies and matriculates into the College degree program, any modules they completed as a non-matriculated student will count toward their required 12 modules.
College administrators and program directors are able to track student completion of modules by exporting the BlackBoard grade center into Excel and utilizing a formula which calculates student progress toward successful completion of a minimum of 12 modules. This formula makes the tracking of student progress very simple and straightforward.
Following the change of array approval by the Board of Trustees to add the HRPR 5001 course to graduate programs, we needed to migrate the modules into the learning management platform BlackBoard. Working with the College instructional technology team, a BlackBoard course site for each program was developed. During meetings with program directors during the spring, it was determined that graduate program directors would be responsible for tracking the completion of the modules by all graduate students entering their program beginning in Fall 2015. Operating under a tight timeline, the sites were completed in early Fall 2015. Each site contains all 18 modules, six of which are required (Advancing Health People 2020, Social Determinants of Health, Access to Health Services, Health Information Technology, Public Health Infrastructure in the United States, and Environmental Health) and students choose six to complete from the remaining 12. In order to “complete” the module, students must pass a quiz at the end. A passing grade is 80% and students must pass the quiz within three attempts. If necessary, a student can be given the opportunity to take the quiz an additional time in order to pass, but they must get permission from their program director. Once a module is passed, a student receives a “badge” for that module. We are utilizing the same quizzes created as part of the APTR online learning modules program. A syllabus outlining the course requirements and objectives is also included on each site.
Each program is given the autonomy to implement the HRPR 5001 course based upon their program and students’ needs. For example, a program could determine that each of the six required modules would be implemented into the classes students registered for and then students would complete the final six elective modules on their own. Programs are also able to determine when the modules must be completed to meet the graduation requirement. For example, MPH students must have all 12 modules successfully completed before they begin their fieldwork practicum which occurs during the final two semesters in the program. That means that for students who are in the MPH program full-time, they must ensure that all modules are completed within their first year of study. Master of Occupational Therapy (MOT) students are introduced to the modules during their orientation and complete a number of modules as part of their orientation to the program. After Fall 2015, newly matriculated students are enrolled in their program BlackBoard site. It is expected that students are introduced to the HRPR 5001 course during their orientation by their program director and/or advisor so they have the opportunity to be engaged with the modules early on in their academic career.
Early on we identified the need to create a BlackBoard site specifically for non-matriculated students. Frequently, we have prospective students who enroll in introductory courses. A number of these courses may choose to utilize the modules given the alignment of the content. These students are not enrolled in any of our programs so they would not have access to the modules. Upon an instructor’s request to College administrators, the non-matriculated student will be enrolled in the Non-Matriculated Students BlackBoard site so they can access the assigned modules. If the student applies and matriculates into the College degree program, any modules they completed as a non-matriculated student will count toward their required 12 modules.
College administrators and program directors are able to track student completion of modules by exporting the BlackBoard grade center into Excel and utilizing a formula which calculates student progress toward successful completion of a minimum of 12 modules. This formula makes the tracking of student progress very simple and straightforward.
LESSONS/OUTCOMES/RESUlTS
Lessons Learned/Unexpected Outcomes:
Administrative Issues
When creating the syllabus for this course, the interdisciplinary committee wanted to ensure programs had the flexibility to implement this course in a manner that made the most sense for their respective programs. The committee also wanted to encourage faculty to utilize the modules in their coursework. The completion of these modules is a graduation requirement so the tracking of the completion of the modules by program administrators is of critical importance which is why the BlackBoard sites are organized according to program. Additionally, only the program director and College administrators were granted instructor rights in the BlackBoard sites as a quality control measure.
We have noted that this structure has created challenges for faculty who are choosing to utilize a module in their class. Faculty who have assigned a module as part of their class, must ask students to print out or email a copy of the earned badge to document the completion of the module. The badge does not provide the student’s actual score on the quiz which the faculty may want to record. If a faculty member would like the score they must ask the program director to retrieve the score or ask the student to report it. This process is not as seamless as we would like it to be. We are working with departments on an individual basis to determine the best structure for adding their faculty while ensuring that data is not compromised, for example not granting all faculty the ability to “override” a student’s grade on a quiz so we can track the number of attempts.
A similar challenge identified is related to students enrolled in one graduate program, but taking a course taught by another department which is utilizing a module in that course. For example, the department of nursing embedded a number of modules into their courses, and at least one of their courses is very popular among students across the College. The graduate program director in nursing added all faculty to utilizing the modules in their courses to the BlackBoard site so faculty could track student completion. When non-nursing students are enrolled in a nursing course utilizing the modules, the faculty member teaching the course cannot track those students’ completion of the modules. This requires the faculty member to be in communication with College administrators in order to access the student’s quiz grade and/or track completion. While this is not a huge hurdle, it can certainly be viewed as a barrier by faculty which may lead them to choose not to embed a module. We are working to identify the best way to address this issue with faculty to assure seamless integration.
Administrative Issues
When creating the syllabus for this course, the interdisciplinary committee wanted to ensure programs had the flexibility to implement this course in a manner that made the most sense for their respective programs. The committee also wanted to encourage faculty to utilize the modules in their coursework. The completion of these modules is a graduation requirement so the tracking of the completion of the modules by program administrators is of critical importance which is why the BlackBoard sites are organized according to program. Additionally, only the program director and College administrators were granted instructor rights in the BlackBoard sites as a quality control measure.
We have noted that this structure has created challenges for faculty who are choosing to utilize a module in their class. Faculty who have assigned a module as part of their class, must ask students to print out or email a copy of the earned badge to document the completion of the module. The badge does not provide the student’s actual score on the quiz which the faculty may want to record. If a faculty member would like the score they must ask the program director to retrieve the score or ask the student to report it. This process is not as seamless as we would like it to be. We are working with departments on an individual basis to determine the best structure for adding their faculty while ensuring that data is not compromised, for example not granting all faculty the ability to “override” a student’s grade on a quiz so we can track the number of attempts.
A similar challenge identified is related to students enrolled in one graduate program, but taking a course taught by another department which is utilizing a module in that course. For example, the department of nursing embedded a number of modules into their courses, and at least one of their courses is very popular among students across the College. The graduate program director in nursing added all faculty to utilizing the modules in their courses to the BlackBoard site so faculty could track student completion. When non-nursing students are enrolled in a nursing course utilizing the modules, the faculty member teaching the course cannot track those students’ completion of the modules. This requires the faculty member to be in communication with College administrators in order to access the student’s quiz grade and/or track completion. While this is not a huge hurdle, it can certainly be viewed as a barrier by faculty which may lead them to choose not to embed a module. We are working to identify the best way to address this issue with faculty to assure seamless integration.
Coursework Issues
As noted previously, a BlackBoard site was created for each program and a syllabus was loaded onto the site in addition to modules. The syllabus outlined the course objectives and explained the purpose of the HRPR 5001 to include that students may take the modules embedded in a course or on their own. Over the course of the academic year, we tracked a small number of students (approximately 10), who required quizzes to be reset because they failed the quiz three times. It was brought to our attention that the language on the syllabus indicated that students could have unlimited attempts to the pass the quizzes. We did note that for these students who failed the quizzes three times, all three attempts were within minutes of each other so the students likely did not go back and review the module content before attempting the quiz again. We want to be sure the students are engaging with the content and not merely trying to pass a quiz. We will work with program directors and faculty in the coming year to encourage students to utilize the modules in a meaningful way. In addition, we identified that one of the quizzes had incorrect answers associated with it. A faculty member who embedded the module into her course brought it to our attention after the majority of her class failed the quiz on the first attempt. We took the quiz offline, have made revisions and are in the process of re-grading the student attempts. When we notified students of the issue, a few students responded with suggestions to review other quizzes that they thought might have incorrect answers. Upon review, we did not find any other quizzes with similar issues.
As noted previously, a BlackBoard site was created for each program and a syllabus was loaded onto the site in addition to modules. The syllabus outlined the course objectives and explained the purpose of the HRPR 5001 to include that students may take the modules embedded in a course or on their own. Over the course of the academic year, we tracked a small number of students (approximately 10), who required quizzes to be reset because they failed the quiz three times. It was brought to our attention that the language on the syllabus indicated that students could have unlimited attempts to the pass the quizzes. We did note that for these students who failed the quizzes three times, all three attempts were within minutes of each other so the students likely did not go back and review the module content before attempting the quiz again. We want to be sure the students are engaging with the content and not merely trying to pass a quiz. We will work with program directors and faculty in the coming year to encourage students to utilize the modules in a meaningful way. In addition, we identified that one of the quizzes had incorrect answers associated with it. A faculty member who embedded the module into her course brought it to our attention after the majority of her class failed the quiz on the first attempt. We took the quiz offline, have made revisions and are in the process of re-grading the student attempts. When we notified students of the issue, a few students responded with suggestions to review other quizzes that they thought might have incorrect answers. Upon review, we did not find any other quizzes with similar issues.
Evaluation Results:
Learning Outcomes Data
The majority of graduate students enrolled in the online learning modules have completed less than 6 modules as they have been in their program for less than one year. According to the survey respondents, the most frequently completed modules were: Advancing Healthy People 2020 Learning and Practice (71%); Social Determinants of Health: A Lens for Public Health (63%); and Emergencies: Prevention, Preparedness, Response and Recovery (53%). The least frequently viewed modules were: Public Health Infrastructure in the United States: An Integrated System (26%); Bringing it all Together: Healthy People in the Classroom and Beyond (24%); Using Policy and Best Practices in Maternal, Infant, and Child Health: Maternity Care Coalition (24%); Oral Health Across the Lifespan (21%); Injury Prevention: Targeting Teen Driving (21%).
Fifty-one percent of respondents reported that the modules they completed were embedded into a course they were enrolled; 24% indicated that the modules they completed were selected based on interest and 24% indicated the modules they completed were a combination of embedded and selected based on choice. This data would indicate that most students are taking the modules as part of a class and have not yet begun to take complete the modules on their own. As previously stated, the modules are a relatively new component of the curriculum, and the completion of the modules as a graduation requirement remains at least one year away for the majority of the students. We expect the number of students taking modules in a self-directed fashion to increase in future semesters. For the majority of respondents (77%), the embedded modules were required viewing outside of class, 36% of students responded that the module content was part of an ungraded assignment and 27% reported that the content was part of a graded assignment. The content was also incorporated into class lectures (18%) and class discussions (23%).
Overall, the module content seems to complement course content across the programs as the majority of respondents responded strongly agree (14%) and somewhat agree (50%) when asked if the module content aligned with course content for embedded modules. This also holds true for the respondents who selected modules to complete on their own – 14% strongly agree and 43% somewhat agree that the module content aligns with courses previously taken or currently taking. In addition, the majority of respondents report that the modules have helped increase their knowledge of Healthy People 2020 (26% strongly agree and 42% somewhat agree) and that the modules demonstrate the value of inter-professional education (23% strongly agree and 45% somewhat agree). Respondents provided positive feedback about the learning platform as well – 32% strongly agree and 42% somewhat agree that the online learning platform through BlackBoard is easy to use. When asked if they are satisfied with the online learning experience, 16% of respondents strongly agree and 35% somewhat agree.
The majority of respondents believe that the modules “definitely” or “probably” help them: describe how health, healthcare and human service professionals work together to address individual and population health issues (22% definitely, 50% probably); define health and wellness and the way various professions consider promoting health and preventing morbidity and mortality at the individual and population levels (26% definitely, 48% probably); explain the role of Epidemiology as the basic science for Public Health by discussing uses and applications of epidemiology in Public Health (17% definitely, 41 probably); recognize local, state, and federal healthcare and public health systems and the way they interact (26% definitely, 33% probably); describe environmental health hazards and their potential effects on human health (20% definitely, 53% probably); describe and interpret the use of biostatistics in public health from published public health data (7% definitely, 47% probably); compare various theoretical constructions and models of healthcare including their social, cultural, and behavioral causes (10% definitely, 55% probably); synthesize the impact of major policy, ethical, economic, professional and legal issues facing healthcare providers, insurers, policymakers and consumers (16% definitely, 42% probably); differentiate among availability, acceptability, and accessibility of health care across diverse populations(13% definitely, 58% probably). This is important feedback, and we must continue to try and capture this information from students as they move through the program. Overall, students are satisfied with the online learning experience, but would not choose to complete more modules than those required.
It will be important to work with program directors, faculty and students to impress upon them the importance of reviewing and retaining the information from the modules. During the first year of implementation, 65% of students taking the modules quizzes took at least one quiz more than once. Of the multiple attempts, 26% were three attempts which is the maximum number of attempts. When reviewing the attempt data, the vast majority of re-takes are occurring within minutes of the first attempt which may mean that students are re-taking the quiz within reviewing the material again.
Conclusion
Given the design of the HRPR 5001 course, we have the opportunity to work with program directors and faculty to better engage students in the module content. In addition, we can work with faculty in the coming year to develop additional module content which may encourage students to complete more modules. In addition, we can further refine the evaluation plan so that it aligns better with student completion of the modules now that we have a better sense of when students complete them and how they complete them. We will also update the assessment to solicit feedback from students on additional modules that would be of interest.
Learning Outcomes Data
The majority of graduate students enrolled in the online learning modules have completed less than 6 modules as they have been in their program for less than one year. According to the survey respondents, the most frequently completed modules were: Advancing Healthy People 2020 Learning and Practice (71%); Social Determinants of Health: A Lens for Public Health (63%); and Emergencies: Prevention, Preparedness, Response and Recovery (53%). The least frequently viewed modules were: Public Health Infrastructure in the United States: An Integrated System (26%); Bringing it all Together: Healthy People in the Classroom and Beyond (24%); Using Policy and Best Practices in Maternal, Infant, and Child Health: Maternity Care Coalition (24%); Oral Health Across the Lifespan (21%); Injury Prevention: Targeting Teen Driving (21%).
Fifty-one percent of respondents reported that the modules they completed were embedded into a course they were enrolled; 24% indicated that the modules they completed were selected based on interest and 24% indicated the modules they completed were a combination of embedded and selected based on choice. This data would indicate that most students are taking the modules as part of a class and have not yet begun to take complete the modules on their own. As previously stated, the modules are a relatively new component of the curriculum, and the completion of the modules as a graduation requirement remains at least one year away for the majority of the students. We expect the number of students taking modules in a self-directed fashion to increase in future semesters. For the majority of respondents (77%), the embedded modules were required viewing outside of class, 36% of students responded that the module content was part of an ungraded assignment and 27% reported that the content was part of a graded assignment. The content was also incorporated into class lectures (18%) and class discussions (23%).
Overall, the module content seems to complement course content across the programs as the majority of respondents responded strongly agree (14%) and somewhat agree (50%) when asked if the module content aligned with course content for embedded modules. This also holds true for the respondents who selected modules to complete on their own – 14% strongly agree and 43% somewhat agree that the module content aligns with courses previously taken or currently taking. In addition, the majority of respondents report that the modules have helped increase their knowledge of Healthy People 2020 (26% strongly agree and 42% somewhat agree) and that the modules demonstrate the value of inter-professional education (23% strongly agree and 45% somewhat agree). Respondents provided positive feedback about the learning platform as well – 32% strongly agree and 42% somewhat agree that the online learning platform through BlackBoard is easy to use. When asked if they are satisfied with the online learning experience, 16% of respondents strongly agree and 35% somewhat agree.
The majority of respondents believe that the modules “definitely” or “probably” help them: describe how health, healthcare and human service professionals work together to address individual and population health issues (22% definitely, 50% probably); define health and wellness and the way various professions consider promoting health and preventing morbidity and mortality at the individual and population levels (26% definitely, 48% probably); explain the role of Epidemiology as the basic science for Public Health by discussing uses and applications of epidemiology in Public Health (17% definitely, 41 probably); recognize local, state, and federal healthcare and public health systems and the way they interact (26% definitely, 33% probably); describe environmental health hazards and their potential effects on human health (20% definitely, 53% probably); describe and interpret the use of biostatistics in public health from published public health data (7% definitely, 47% probably); compare various theoretical constructions and models of healthcare including their social, cultural, and behavioral causes (10% definitely, 55% probably); synthesize the impact of major policy, ethical, economic, professional and legal issues facing healthcare providers, insurers, policymakers and consumers (16% definitely, 42% probably); differentiate among availability, acceptability, and accessibility of health care across diverse populations(13% definitely, 58% probably). This is important feedback, and we must continue to try and capture this information from students as they move through the program. Overall, students are satisfied with the online learning experience, but would not choose to complete more modules than those required.
It will be important to work with program directors, faculty and students to impress upon them the importance of reviewing and retaining the information from the modules. During the first year of implementation, 65% of students taking the modules quizzes took at least one quiz more than once. Of the multiple attempts, 26% were three attempts which is the maximum number of attempts. When reviewing the attempt data, the vast majority of re-takes are occurring within minutes of the first attempt which may mean that students are re-taking the quiz within reviewing the material again.
Conclusion
Given the design of the HRPR 5001 course, we have the opportunity to work with program directors and faculty to better engage students in the module content. In addition, we can work with faculty in the coming year to develop additional module content which may encourage students to complete more modules. In addition, we can further refine the evaluation plan so that it aligns better with student completion of the modules now that we have a better sense of when students complete them and how they complete them. We will also update the assessment to solicit feedback from students on additional modules that would be of interest.
