ReproducibiliTea is a grassroots initiative, largely led by ECRs. ReproducibiliTea is evolving worldwide, and our steering committee needs to as well. In 2020, we set out plans for rotating the steering committee and posted a call for applications in December 2020. Now, we are excited to introduce you all to our updated steering committee!

Please join us in welcoming our new committee members:

Jan Vornhagen @VornhagenJB

Jan is a cognitive Psychologist and Doctoral Candidate in Computer Science at the AERIS Lab - Aalto University, Finland - where he also runs the local, HCI-focused RTea chapter

Jan Vornhagen

Alexa von Hagen @alexavonhagen

Alexa is currently a postdoc in the field of school psychology at Goethe University Frankfurt in Germany. Alexa first started with ReproT Singapore last year, but am now contributing to Frankfurt ReproT.

Alexa von Hagen

William Ngiam @will_ngiam

William is a postdoc in the Department of Psychology at the University of Chicago, where he started and co-organizes the ReproducibiliTea journal club!

William Ngiam

And, our new committee chair:

Sam Parsons @sam_d_parsons

Sam replaces Amy Orben as the chair of the steering committee. He is a Postdoctoral Research Associate in Experimental Psychology at the University of Oxford. He started ReproducibiliTea with Sophia and Amy, and also founded the Framework for Open and Reproducible Research Training (FORRT) project.

Sam Parsons

And, give a ‘welcome back’ to our returning members:

Matt Jaquiery @MJaquiery

Matt is an Oxford DPhil student on secondment to the UKRN as Coordinator for Open Research Training. He manages the Oxford Experimental Psychology ReproducibiliTea and does a bunch of technical stuff for the core organisation.

Matt Jaquiery

Sophia Crüwell @cruwelli

Sophia is a PhD student in Philosophy of Science at the University of Cambridge, working on conceptual issues of the replication crisis and also working on empirical metaresearch projects with METRIC-B in Berlin. Sophia started the ReproducibiliTea journal clubs at Oxford (with Sam & Amy), Amsterdam (with Angelika Stefan & Florian Wanders), and Berlin (with Corinna Hartling).

Sophia Crüwell

Katie

Katie is a PhD student in metaresearch at the University of Bristol. She is deputy lead at the University of Bristol Reproducibility, and is responsible for the ReproducibiliTea blog.

Reflections

In our call for applications, We asked applicants to submit their CV, a cover letter and 100 words on how they thought ReproducibiliTea could improve in the next two years. We received six applications for steering committee positions, and one for the steering committee chair position.

The selection process looked like this:

  1. We separately read each application and marked whether we felt the candidate demonstrated each of our selection criteria - the applications were not blinded.
  2. We then met as a group and collated these ratings. We discussed any differences until we came to a consensus.
  3. We discussed each applicant’s 100 word statement on how ReproducibiliTea can be improved. We felt it was the most useful as an insight into each applicant’s unique perspective on where ReproducibiliTea can grow and best support our members.
  4. Finally, we collated the scores and notes on each candidate and decided to invite three candidates to fill positions on the steering committee.

Every application was impressive, and we are not merely paying lip service to say that the selection process was difficult. It gave us all cause to reflect on how we can improve the process.

Selecting new committee members is not something we undertook lightly. From developing the selection process and posting the call for applications, we were aware that we had the responsibility to ensure fairness. At the same time, we knew that the process would be imperfect. In the interests of transparency, and to ensure that our selection process improves, we each reflected on the process.

Pros and cons of the 100 word statement on improving ReproducibiliTea

  • Give applicants more detail, for example we could have been clearer about what we wanted from the 100 words statement on improving ReproducibiliTea. We found that the statements we were more positive about were those that included more specific ideas or plans that the candidate wanted to enact. Next time, we can rephrase this item along the lines of “100 words on how you plan to help ReproducibiliTea improve”.
  • It’s also worth mentioning a reflection from one of our applicants on the 100 word summary. “I definitely found this 100 word document the hardest to prepare. On one hand, I had lots of ideas I wanted to share and I felt 100 words was not enough for this. On the other hand, I felt like I needed to know more about the broad next steps you have in mind for ReproT to be able to suggest ideas that could connect to that and build on all your previous work.”
  • We also reflected that we gained more insight from the 100 word statement than other application materials. We found CVs the least useful in our decision making process, which might reflect how tailored CVs are to academic or research positions, rather than the steering committee requirements. We might consider asking several short answer questions like this in the future.

Selection criteria

  • We should use more concrete and direct questions. Steering committee members marked some of the selection criteria differently. A strength of our discussion was that we were able to resolve these and come to an agreement. We should agree how each criteria will be met and judged before advertising positions. We must be clearer in the call for applications how applicants can demonstrate that they fulfil the selection criteria.
  • One of the selection criteria was ‘early career status’. Next time, we might consider specifically asking for graduate student applicants and postdoc applicants separately. Committee members will academically ‘age’ quickly, and we want to encourage applications from as early as possible. This will also help us navigate some of the differences we observed between candidates pre- and post- phd, insofar as experiences with application processes will equip those later in their academic career to write strong applications.
  • We will remove “communication skills” from the criteria. This criteria opens the possibility for the committee to judge applications from e.g. people with english as a second language or neurodiverse individuals.
  • We may need to update selection criteria as the committee roles evolve. For example, positions such as treasurer and communications officer may require different skills and experiences.

Representation

  • We need to think more about diversity and how people who do have different backgrounds from us will be able to meet the selection criteria. We should not expect people to need to self-disclose things such as being neurodiverse, and should strive to create criteria that include everyone. Some of the above reflections speak to this point, however we should keep this firmly in mind.
  • Two of our selection criteria were “Connection to academia in parts of the world underrepresented by ReproducibiliTea” and “Connection to academic disciplines underrepresented by ReproducibiliTea”. On reflection, these statements are quite vague and we will need to consider how to better phrase these and ask applicants to highlight these aspects.
  • We cannot escape having previous experience with applicants. ReproducibiliTea is well connected via social media, and as we pursue a more integrated community and links between journal clubs we should expect more connections between the steering committee and applicants. There is no easy solution to this as it offers some benefits (e.g. a demonstration of more widespread efforts to improve ReproducibiliTea or reach wider groups) as well as drawbacks (e.g. failing to see an applicant’s full potential or understanding their perspective). In any case, we need to continue to reflect on how this subjectivity interacts with our assessments, particularly with respect to diversity of applicants.