For the last five years now, I have been involved in teaching the bachelor course on Industrial Automation. The course focuses on discrete and logical control. Apart from the lectures, the course contains two projects, one based on low-level PLC programming. The other focuses on modeling high-level discrete control programmed in Java. At the end of the course, the students integrate the Java and PLC parts of the project to have an end-to-end automated system.

Over the last five years, I have been improving and tweaking the Java part to automate most technical administration. I have integrated the checking of assignments into a Continuous Testing framework using GitHub classrooms and Travis to reduce my load of correcting them. This post is intended to document that process.

GitHub Classrooms

The first step was to organize a way to distribute the assignment files to the students. Previously, they were given all the source files as a compressed archive, which they would modify and submit. It would sometimes get messy since the students worked in groups. When they returned the assignment, we had many unwanted and unknown folders generated by the system/IDE they used.

We changed this and started using GitHub to distribute and collect the assignment. This way, we could control the uploaded files and allow them to collaborate in a much better (industrially accepted) manner. Furthermore, it was easy to give comments and feedback on their code regularly.

We setup GitHub Classrooms to handle how the assignments were distributed. It was easy to setup and use; though it wasn’t mature back then, most of the essential features (assigning groups) worked out quite well. GitHub Classrooms has grown over the years. I am really impressed by the direction it has taken.

The next step was to automate the grading.

Student reports

The assignment was a bunch of functions the students need to implement. The specifications for these were defined in natural language, along with unit tests to help the students. When submitted, we would test their code against a set of unit tests to ensure correctness. Additionally, we wanted the students to get a test report when they submit their code.

The solution was to use Travis to run the tests when the students submit the assignments.

The flow is something like this:

1. Students push their code.
2. Travis boots a VM and clones the student repository.
3. Travis deletes the test folder and replaces the test folder containing all the tests (from a private repository).
4. Travis runs the tests.
5. Travis uploads the reports file to the student repository.

The Travis config file is as follows:

language: java
jdk: openjdk8
sudo: false
branches:
  only:
  - master
before_install:
- rm -r src/test
- git clone --depth 1 --branch test2019 git@github.com:Organization/repo.git
- cp -r ssy066/src/test ./src/
- find . -type d | sed -e "s/[^-][^\/]*\//  |/g" -e "s/|\([^ ]\)/|-\1/"
script: mvn surefire-report:report
after_success:
- git checkout master
- mkdir Reports
- cp ./target/site/surefire-report.html ./Reports/
deploy:
  provider: releases
  api_key: $API_KEY
  file: "./Reports/surefire-report.html"
  skip_cleanup: true
  on:
    tags: true

Additionally, Travis requires access to the private repository with the unit tests. We grant access by giving the SSH keys to Travis so it can access them. Travis also requires permissions to upload to the student repository. This is made possible by authorizing Travis and providing an API key generated by GitHub.

Unfortunately, we cannot add the SSH keys and API key from the start. Apparently, the secrets are encrypted on a per-repository basis, which means that we need to manually add these keys when the students create their repositories. Luckily, Travis has a CLI interface, that can be used to automate the task.

Install the CLI from here: https://github.com/travis-ci/travis.rb#installation Once you have the command on your PATH use the following script to upload the keys:

 travis repos --active --owner <organization> --com | grep "java-2020" | xargs -I % travis sshkey --upload id_travis_rsa -r % --description "CI dependencies"

travis repos --active --owner <oranization> --com | grep "java-2020" | xargs -I % travis env set API_KEY <KEY> -r % --pro

The grep part is used to filter out specific repositories.

Automated Grading

At Chalmers we use Canvas as an LMS, previously it was Pingpong. It is here that we need to mark/grade the students. Though the GitHub reports are useful, it takes a lot of clicks to mark the grade in Pingpong/Canvas for each group. Instead here is how I do it:

Clone all repositories from GitHub using the Classroom Assistant.

Then I use the following script to run through all of them and run the tests within them.

find . -maxdepth 2 -mindepth 2 -type d | while read dir; do
	echo "$dir"
	cd "$dir"
	mvn -Dtest=Test1,Test2 test | grep -v INFO | grep "Tests run"
	echo "============================="
	cp -r ../../AllTestsProject/src/test ./
	mvn -Dtest=Test1,Test2 test | grep -v INFO | grep "Tests run"
	cd ../..
	pwd

done

The maxdepth and mindepth need to be modified based on the folder structure and the grep part helps filter out the lines I want to look at.

The output of this will be as follows:

./Assignment/(Par 16)
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.025 sec
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0
=============================
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.027 sec
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0

At the top I have the student name/group to mark in canvas.

Ofcourse, one can further automate this to give you a list and then connect it with canvas (they apparently have an open API).

Some random thoughts

This workflow of mine is quite outdated. GitHub and the tools around it have matured over the years and offer much better ways to do the same. GitHub Classrooms can automatically connect to LMS systems like Canvas. And, GitHub actions make it much easier to setup auto-grading for the assignments using GitHub Actions.

At one point I had Travis send me notification messages on Slack when a student successfully completes the assignment. But for various reasons this did not feel productive since I had to manually search for failing builds and give feedback to students.

GitHub Classroom has been a great tool to reduce the administrative load from the teaching and focus more on the content. It takes some additional time and effort at the start; once setup, it is a breeze to maintain.