The Crime of Parsimony

The Crime of Parsimony: A comment on The Smartphone Trap

This post is related to (but not a part of) the Youth Suicide Rise project.

 

Science is impossible without verifiability and verification is impossible without the sharing of necessary information.

In a NY Times essay by Jonathan Haidt and Jean M. Twenge titled This Is Our Chance to Pull Teenagers Out of the Smartphone Trap, the authors appeal to findings published in The Journal of Adolescence when they argue that smartphones and social media are primary candidates for a culprit behind remarkably global increases of alienation felt by 15-year-old students in schools (per PISA surveys).

The paper Haidt & Twenge refer to is Worldwide increases in adolescent loneliness, which they co-wrote with four additional authors.

I previously complained about some misleading terminology in the JoA paper and now I will complain even more about another issue.

A lot more.

The JoA paper lacks some of the most basic information necessary to evaluate the arguments and utilize the findings presented by the authors. As to attempting verification, good luck.

This is not an issue of insufficient space. The authors provide pages of long tables supplying evidence of their primary findings about international alienation increases in 37 nations, and pages more on this topic in a Supplementary file accessible online.

We are Sorry, your Cell Phone Plan has Zero Data Allowance

 It is when it comes to their secondary focus, mainly the argument that smartphones are the most likely cause of rising student alienation, that information becomes as scarce as if the authors were guarding the secret recipe for a multi-billion dollar soda formula or the algorithms behind Facebook personalized ads.

The authors tell us they obtained data on prevalence of student access to smartphones from the triennial PISA ICT surveys.

They do not share the results!

The authors fail to provide even the international averages at 2012 versus 2018 for the group of the 37 countries under consideration in the paper, or the median change for this group, and so on.

Did access to smartphones increase by 5 percentage points or by 50 percentage points? Apparently the authors do not think it important for us to know.

If you blame a massive increase in harm Y on the increase in exposure X, is it too much to ask that you actually document the supposed increase in exposure X?

The authors are similarly stingy with data for the time students spent online, which is another measure from the PISA ICT surveys the authors appeal to yet do not show.

Please Visit Alternative Timelines for Relevant Data

The authors also provide very little data regarding the four alternative factors – GDP, GINI, Unemployment, Fertility – that they seek to dismiss as rival explanatory variables. This of course makes it all the more difficult to evaluate their arguments.

The lack of info on units of measurement makes it hard to interpret some the beta values provided in the modeling section. With GDP, the authors never even clarify if they mean GDP per capita.

The authors place great emphasis on the statistical significance of the loneliness correlations they calculated with various factors. The validity of this approach aside, a serious drawback is that the authors omit the p values!

To understand the problem, see that in model B (Table 4) the only ‘significant’ correlation is the one with the Internet – which authors indicate by a single * next to its beta value, meaning p < 0.05.

Without giving us the actual p values, however, it could be that the evidence for the association with online time might be only a tiny bit higher than the evidence for some of the other factors: say p = 0.049 for the Internet and p = 0.051 for unemployment.

Similarly, associations with the Internet and unemployment in the ‘additive’ models of Table 3 differ only by one star (** vs *), meaning the actual difference between the p values could be arbitrarily small.

Using such potentially slight differences to make important distinctions between the Internet and other factors would be dubious.

The various models within the paper are given without precise specification, and since these are multilevel models, the actual form of implementation is not trivial to guess.

Shortly after publication, the authors admitted some of the modeling results printed in their paper were given incorrectly. With so little data and model information, it can be nearly impossible for the reader to discern if some strange result indicates an error in the design of the model or if it is just a typing mistake.

This is especially problematic given that neither the apparent model designs, nor the predictions of these models, nor some of the remarks by the authors about these models make much sense to me.

Since I am far from an expert on multilevel models, it is hard for me to be certain, without sufficient information about these models, that their implementation is indeed as flawed as it seems to be.

As to any attempts at replication … well as I said, good luck.

Dear Editors: This is Your Fault!

If I sound irritated, it is because I am, rightfully, irritated. Obtaining the data on my own proved far more problematic than expected – see Notes – and is a needless waste of time. And no, one should not be reduced to repeatedly contacting the authors to obtain all the various information that should have been already in the paper (although I do attempt below the enumeration of the missing information and will provide the list to the authors).

To be fair to the authors, the lack of data and other information is not the worst that I've seen -- in fact some of the papers written by their opponents (it terms of concerns about digital tech) are more deserving of public castigation in this regard. It is this particular paper, however, that combines two topics that I care about greatly -- school climate and teenage mental health -- and no remarks on it would be reasonably complete if they did not include the mention of missing data and model specifications.

Furthermore, as much as I am irked by all this, my indignation is directed primarily not against the authors but against the Journal of Adolescence editors.

Why?

There will always be authors who will fail to supply all the necessary data and information required for a decent research publication. Some authors will be in a hurry, some will be lazy, some will fail to comprehend the need for sharing certain types of information.

It is the job of the editors to be the guardians of proper scientific standards. If editors fail to perform even such a basic job as is the inclusion of fundamental information within papers, then perhaps scientific journals should be obsolete and we can all just communicate via PsyArXiv, at least in psychology.

So ultimately this is primarily the responsibility of editors, and will remain so as long as papers are published in scientific journals.

Notes:

Data Availability: Some of the data is harder to obtain than it might seem at first. For example, the results for the smartphone access question from the PISA questionnaire are not in the data available from the PISA Data Explorer interface.

Strangely, I found no publication that would even give OECD averages for 2012 to 2018. There are code books that include answer percentages for all the students queried, but using these could be highly misleading, since say the UAE has three times the number of sampled students as France in one survey year.

That leaves having to download raw data files, about 1 Gig in size total, and in changing formats from one survey year to another, and then analyzing the data using R or Python. It should be clear by now to the reader how time consuming it could be for one to obtain all the missing data.

Model Specifications: To see a proper and useful model specification, let us turn to the ‘competition’ of the Journal of Adolescence, namely the International Journal of Adolescence and Youth.

The paper Student- and school-level factors related to school belongingness among high school students published in it has a tortured title but its models are nicely specified and their interpretation made clear:

 

 

and

 

 

Together with additional info in the paper, it is fairly easy even for me, who is not highly familiar with hierarchical models, to understand precisely what and how is being modeled.

[See https://www.tandfonline.com/doi/pdf/10.1080/02673843.2020.1730200]

Missing Info List:

Here are the items I think the authors should provide online in order to allow readers access to all the information necessary to properly evaluate the Worldwide increases paper:

1) Country results for each of the 6 questions compromising the ‘loneliness’ (really alienation) index.

2) Country results for the following 6 factors: smartphone access, online time, GDP (per capita), GINI, Unemployment, Fertility.

3) Units for all the beta values in the models.

4) The p values for all the factors in all the models

5) Are the LL/UL values for 95% or for 90% confidence intervals?

6) How exactly were the p values calculated?

7) How exactly were the confidence intervals calculated?

8) How exactly were the ‘std. b’ values calculated?

9) What are the model specifications for the ‘additive’ and ‘interactive’ models as well as the ‘comprehensive’ models A and B? See above for an example of sufficient specification.