The Challenge: Prepare Students for U.S. Civics Major Courses

Students come to Duke from a variety of educational backgrounds, and not all students have the same background coming in to their major courses. Duke Learning Innovation wondered whether a short, online course emphasizing foundational knowledge could give students the background they needed to be successful in required courses, especially gateway courses and key major courses. Learning Innovation called this model a “level-up” because it brings all of the students up to more or less the same level of knowledge.

In the Spring 2018 semester, Learning Innovation worked with two professors from Duke’s political science and public policy departments to pilot the level-up idea. The two faculty members, Bruce Jentleson and Nicholas Carnes, had noticed that many students came to the core course in Duke’s public policy major curriculum lacking extensive background knowledge of U.S. civics—topics such as electoral processes or how social movements worked. They wanted to see if a level-up could provide that basic knowledge without taking away from course time needed to cover more advanced topics.

The goals of the level-up pilot project were:

• Provide all students in a class the opportunity to learn the foundational material needed to be successful in a key required course in the public policy major
• Present the material in a format that would not take away from in-class instructional time
• Create a learning experience that could be completed before or at the beginning of the semester, but that students could refer to throughout the semester as needed
• Structure the learning experience to be somewhat self-paced so students with different backgrounds can work at different paces, but time-bounded so all students complete the material by a given point in the campus course

A MOOC-inspired course design

For the format of the level-up, Learning Innovation looked to a familiar model: the MOOC. Learning Innovation is the central MOOC-creation group at Duke, and over the last five years has created around 60 short, self-paced online courses.

The U.S. Civics level-up followed the typical structure of a MOOC: the course material was divided into three modules, each consisting of 5-6 short video lectures followed by quizzes. Several Duke faculty members were recruited to contribute video lectures so the course would not appear to be “owned” by any single course instructor. The course was hosted on the Coursera platform, but was only available to Duke students.

Students in the two courses were required to complete the level-up either before the semester started (during winter break) or within the first 2 weeks of class. There was a 5% penalty from their final grade for failing to do so; only three students out of 122 failed to complete the course as required.

How we evaluated the pilot

To evaluate the effectiveness of the pilot, Learning Innovation conducted a survey with all the students who were enrolled in the two classes about one month before the end of class. The survey asked students to identify the most and least useful topics they learned about in the course, to self-evaluate how much they knew about various topics before and after taking the class, and to provide some data on how they used the level-up course.

The survey was distributed to 89 students in the Political Science class and 33 students in the Public Policy class. The survey completion rate was 92% in political science (82 completes) and 100% in public policy. We asked students to evaluate their knowledge on nine key topics that the course instructors identified as important topics to understand in order to be successful in the public policy major. All of them were topics covered in the video course, but not all were topics that had been covered in the in-person course at the time of the survey.

Rather than ask students to evaluate their knowledge before and after the online course, we elected to conduct a retrospective self-evaluation of knowledge and an evaluation of currently knowledge at the same time towards the end of the semester. This is because past research has shown that people do a poor job of self-evaluating how much they know about a topic when they do not know very much. Essentially, people are unable to accurately judge how much they do or do not know until after they have learned about a given topic. In hindsight, however, they are able to fairly accurately rate how much they knew at a prior point in time.

Pilot Results

Building foundational knowledge

Students reported significant gains in their knowledge on every key concept we asked about on the survey. The nine concepts we asked about included a mix of topics that had already been also covered in the in-person class and topics that had not yet been covered, and the results were similar. The topics we asked about in the survey were the topics that the two course instructors identified as topics that had been challenging in prior semesters for students without prior knowledge of U.S. civics.

For this reason, we conclude that the level-up achieved the goal of teaching students the foundational knowledge they needed to be more prepared to study U.S. civics in an advanced undergraduate course.

Providing flexible course materials

We found notable variation in how students used the course materials. About a third of the students skipped some or all of the videos and only took the required quizzes. Just under a third watched the videos and stopped while watching to take notes, and the rest watched most of the videos without taking notes.

Because the students were only required to complete the quizzes to receive credit for taking the course, the course ended up allowing students to essentially “test out” of topics they already knew while watching videos at whatever pace they wanted to learn about new topics. In the open-ended questions on our survey, students commented on this flexibility.

We also wanted to know if students with prior subject knowledge more likely to skip videos and just take the quizzes. We found that students with high prior knowledge were 5%-10% more likely to skip parts or videos or entire videos and go straight to the quizzes. Students with low prior knowledge were 10%-20% more likely to watch videos in their entirety.

Connecting level-up material to in-class learning

Finally, we found that students appreciated when the instructors made reference throughout the course to where in the level-up students could return to refresh their understanding of different topics. Many students felt that completing the entire level-up at the beginning of the semester was rushed, and they did not retain the information that was not subsequently referenced until the end of the semester. They recommended breaking the content into two mini-courses and assigning them at the beginning and mid-point of the course.