Thursday, May 10, 2018

The benefits of academic workshops?



All IB graduate students should have received an email announcing the new batch of programs offered by the university to help graduate students with one of the most difficult academic skills: writing.

"Dear Graduate Faculty, Coordinators, and Graduate students, Our analytics indicate that those students who receive writing instruction and consultation early in their graduate programs complete their degree requirements sooner. Our programs are designed for graduate student writers at all levels and stages. Individual Writing Consultations: (starts January 15)."
The claims made at the start of the email ("Our analytics indicate [...] complete their degree requirements sooner.") are contrary to a recent publication that received some exposure: "Null effects of boot camps and short-format training for PhD students in life sciences" by Feldon et al. 2017. The title already give the results away, but here is the main conclusion:

"Here we show that participation in such programs is not associated with detectable benefits related to skill development, socialization into the academic community, or scholarly productivity for students in our sample."

This study was no slouch, with 295 participating students, followed for several years with performances evaluated using standard rubrics, etc, and thus the pushback has been struggling with the quality of the study. For instance, the Software Carpentry organization, whose reason for existence is offering these workshops, responded to this almost existential threat in quite some detail. While I think I perceived an undertone of defensiveness in their response, I think that the easiest explanation for these null effects are identified by the Software Carpentry people:
"Benefits of a short course are easily lost in a sea of positive outcomes resulting from graduate training, but that has little bearing on the impact such courses may have when they stand alone."
But that is of course also a sort of cop out. Although given that "only" 16% of the participants participated in such a workshop, the lack of detecting the influence of a single workshop in the wide range of personal outcomes is not that surprising. This is one of the cases where a post-hoc power analysis might have been informative.

The second, standard, scientific cop out is that the authors did not measure the correct outcome. The end point identified by the original notification email (time to completion) was very different from the the endpoints measured in the long-term study (productivity, academic skills development, socialization into the academic community). However, in the appendix of the PNAS article, there is a variable "Time to Degree (T2)" that is not significant. It would be interesting to take a closer look at the Guelph data in that respective.

So while the effectiveness of these workshops at a global level is not exactly clear, I think that it is clear that, from a graduate student perspective, you have to start somewhere. You will not immediately become the best writer in your cohort because you took a 2-day writing tune-up workshop, but you will become a better writer compared to the old you.

Monday, January 15, 2018

Significance versus explanation

Next time I have to teach the difference between explanation and significance, this will be my go-to example: "Scientific aptitude better explains poor responses to teaching of evolution than psychological conflicts" by Mead et al. in Nature Ecology & Evolution.

The article has several figures that look like this:
The P-value looks amazing (9 x 10^-15), but the spread is equally large. I would not even want to guess what the R-square of that relationship is. The authors in the figure legend also specify that:
"The regression line is the best-fit line of y predicted by x. However, as assumptions of linear regression are not fully met it is provided for illustrative purposes alone to indicate the trend."
Good point, because without that line it would be impossible to tell what the relationship would be, which is always a dire sign. But those two sentences should be part of a master class in scientific writing: Necessary, to the point, anticipating a reader's needs and confusion, packaged in a short and succinct statement.

This study works perfectly well in a hypothetico-deductive framework with well-laid out hypotheses, logical predictions, and strong and convincing statistical tests. It also leaves the reader (or some of the readers, for instance me) with this "but is it really important?" question. I could not find any mention of this variability issue, and only lots of highly significant p-values.  What is actually the best explanation of these students' evolutionary understanding?

Monday, January 8, 2018

Let's talk about mental health

Today. Tomorrow. Every day. Because it requires attention year round. 

And the start of the semester is an appropriate time to bring up this study by Levecque and co-authors, covered in Science, that focuses on graduate students. It quantifies the problem, but also identified consistent predictors of mental health issues: 

  • work-life conflicts
  • high job demands
  • low job control 
The story I come up with based on these results is that low job controls leads to (a perception of) high job demands, to work-life conflicts, to mental health symptoms. And what could be related to low job control and/or high job demands? Maybe supervisor style, which can either diminish the prevalence of mental health issues when it is perceived as inspirational, or increase when it is perceived as laissez-faire. This again stresses the crucial importance of the supervisor-graduate student relationship, one of the recurrent themes in these news items. 

Another recurrent theme in these news items is the importance of reasonable job expectations, potentially non-academic ones. It is thus maybe not a coincidence that "Positive perception of career outside academia" has a strong positive effect on graduate mental health. If you know that all the hard work will eventually lead to a satisfying job that requires this education, and not necessarily a low probability faculty position after an endless series of postdocs, maybe that makes all the hard work worth it.