Last revised on June 30, 2025
Section 8: Remote Judging
Overview: Remote Judging occurs when either the Engineering Notebook Judging process, Initial Team Interviews, or both, are done remotely ahead of the event.
Remote Judging follows the same rules and general guidelines as in-person judging, but allows Event Partners and Judge Advisors to utilize volunteers who may not be available in person. It also begins the event day with some aspects of the judging process completed, making it easier to complete judging on schedule in the time allotted.
Determining the judging format (in-person or remote) for an event requires a conversation between the Judge Advisor and Event Partner. Ultimately the decision on the judging format falls to the Event Partner, but the Judge Advisor should be comfortable with working in the chosen format. Remote judging can help better utilize volunteer resources available for the event day, but Judge volunteers need to be comfortable with any additional time and/or technology requirements that may be required of them.
It is important to note that remote judging, in general, requires more volunteer hours than in-person judging during an event. Judges during initial interviews do not get an opportunity to inspect robots in person, and may lose information that would have been conferred by body language. Still, this is a way for events to get some of the work of judging done before the event, and as such may be a viable alternative for some events to entirely in-person judging.
Remote judging follows all guidelines of in-person judging. The following sections highlight the key differences in the judging process if some of the judging tasks usually done in person are conducted remotely. Remote judging can occur in the form of remote Digital Engineering Notebook judging, or remote initial Team Interviews, or a combination of both.
<RJ1> Consistency Within an Event
All teams being judged for an event must be judged in the same format to ensure consistency in the judging experience, and to remove the potential of format-based bias from impacting deliberations. For example, if Engineering Notebooks are submitted for evaluation via links to digital notebooks ahead of the event for some teams, then physical notebooks cannot be evaluated in-person the day of the event for other teams. For Team Interviews, either all teams are given an initial remote interview, or all teams are initially interviewed in person. Teams that are not remotely interviewed cannot be initially interviewed in person at the event.
Remote judging should take place as close to the event as possible so the teams and robots that Judges observe in initial interviews are as similar as possible to what is being brought to competition.
<RJ2> Notebook Submission Deadlines
Primary Team Coaches must provide Digital Engineering Notebook links in their RobotEvents.com account before the stated deadline for the event. The Event Partner and the Judge Advisor should set a deadline that gives the Judges adequate time to review the Digital Engineering Notebooks before the event. The Event Partner will share the list of links with the Judge Advisor, who will assign Judges to review each Digital Engineering Notebook according to the Engineering Notebook evaluation process (see Section 5). All Digital Engineering Notebooks should be evaluated under similar conditions and time constraints.
<RJ3> Remote Digital Engineering Notebook Judging
- Digital Engineering Notebooks are judged remotely before the event.
- Teams will upload links to their Engineering Notebook via RobotEvents.com. Teams cannot be asked to submit notebooks using a method other than the RobotEvents link or as specific file type, and additional requirements that do not appear in this guide cannot be imposed.
- Once a Digital Engineering Notebook (DEN) link is uploaded via RobotEvents.com, teams may still update their DEN on an ongoing basis, even on event day. Notebook content is expected to change over time, which is part of the Engineering Design Process. However, it is not expected that Judges will re-evaluate a notebook based on materials submitted after the initial evaluation.
- Note: Teams must submit the Innovate Award Submission Form by the posted DEN submission deadline. Alterations or additions after this period may not be reviewed and/or accepted.
- The Event Partner will provide a list of links to the Judge Advisor.
- It is recommended that the Event Partner and/or Judge Advisor check all links before the submission deadline, and give all teams equal opportunity to replace non-functional links, if possible. There is no obligation to notify teams if they have a non-functioning link. If it is decided to do so, all teams must be contacted in a consistent manner.
- Digital Engineering Notebooks should be freely viewable by the Judges by using the link. Teams should ensure that permissions to view their notebooks are set to allow the judges to view without special permissions or logins.
- The Judge Advisor will organize Judges into groups to review and score notebooks using the Engineering Notebook Rubric.
- Digital Engineering Notebooks should be handled remotely under similar circumstances to ensure consistency.
- Each Digital Engineering Notebook should be evaluated by multiple Judges to establish a ranking of finalist notebooks.
- Events should abide by <EN18>, which stipulates that Judges at the event must have access to all team Engineering Notebooks; this may involve the EP providing equipment such as computers and internet access for use by judges to view Engineering Notebooks at the event.
- Some events may want to conduct a variation on this evaluation format. The overriding principle remains that all notebook submissions are to be evaluated utilizing the same submission format and in the same timeframe, so that no entries have any real or perceived preference or advantage.
<RJ4> Remote Initial Team Interviews
- Initial Team Interviews are done remotely before the event, using the Team Interview Rubric and Initial Award Candidate Ranking Sheet.
- Team participants can log into the meeting from a single location sharing a webcam, or from multiple locations.
- The goal of initial remote Team Interviews is to identify nominees for each award (step 1 of the deliberation process).
- Judge Advisors should set up a way to collate judging notes to assist in final deliberations.
- Follow-up interviews for final award nominees (step 2 in the deliberation process) must be done in person to account for team and robot observations at the event.
- In-person Judges for follow-up interviews should not move teams from one award category to another. Doing so would invalidate the initial deliberations of the Remote Judges and effectively restart the judging process without giving equal treatment to all teams.
Note: Remote judging does not take the place of in-person follow up interviews and deliberations on the day of the event. It is meant to provide flexibility for Event Partners and judging volunteers to perform some judging tasks ahead of the event day. Remote judging can allow a smaller group of Judges to take advantage of the longer time frame by scheduling judging ahead of the event and allows use of volunteers that may not be able to attend an event in person.
Remote Interview Protocols
- All judging principles and guidelines still apply.
- Youth protection must be upheld. While conducting remote interviews, each participating team must have one adult representative (18+ and not a high school student) logged in and visible on camera during the entirety of the interview. This adult representative should join the interview before any students arrive. The adult may be in the same room as the students or logged in separately to the remote call. This adult cannot participate in or contribute to the content of the team interview in any way. Their presence ensures there are multiple adult parties involved in any video meeting.
- A Judge should never be alone in a remote interview with a team, but instead should work as part of a group of two or more Judges. With the inclusion of the team’s adult representative, this puts the minimum number of adults in a remote interview at three.
- Just as in-person interviews do not allow recording, remote interviews should also never be recorded by any party.
Any meeting notes or data spreadsheets that result from Remote Judging should be under the control of the Judge Advisor, and the information contained in them should be destroyed at the conclusion of the event.
<RJ5> Scheduling Remote Initial Team Interviews
Scheduling the Remote Judging Volunteers
Interview scheduling requires coordination between the Event Partner, Judge Advisor, Remote Judges, and teams. It is recommended that the Event Partner first create a schedule of interview times, then ensure that Remote Judges and the Judge Advisor are available for those times. While the Judge Advisor may not need to participate in each interview, it is highly recommended that they be on hand to help manage any issues that may arise. Additionally, if a Remote Judge ends up not being able to attend or has a technology issue, the Judge Advisor can step in and serve as a Remote Judge so teams can be interviewed at their scheduled time.
Scheduling the Teams
Remote initial Team Interview sign-up times should be provided for teams to schedule themselves via a first-come, first-served sign-up system. If teams miss a clearly communicated sign-up deadline, the Judge Advisor may work with them to schedule them into any available remaining interview times. It is recommended that remote interviews be completed a few days ahead of the event in case extra time is needed due to a volunteer or technology issue disrupting the schedule.
If there are enough Remote Judge volunteers to support it, multiple interviews can be conducted in parallel. For example, participants could log into a single remote judging link with a main room for incoming teams and breakout rooms for each team of Remote Judges. Each team would be moved from the main room into a breakout room for their interview. It may be helpful to have two or more adults (the Judge Advisor and other event staff members) greet teams in the main room as they arrive, ensure they have their adult representative visible on camera, and check that it is the correct team for the time slot before moving teams in to see their Remote Judges. Having this “waiting room” also prevents teams from inadvertently interrupting another team’s interview.
Note: Past experience has shown that half-hour interview cycle times work well: 30 minutes allows ample time for teams to enter the remote judging environment; for Remote Judges to conduct a 10–15 minute interview; and for Remote Judges to discuss, score the interview, and fill out the Initial Award Candidate Ranking Sheet before the next team arrives.