The Perils of Improper Terminology

 

The Perils of Improper Terminology: A Comment on The Smartphone Trap

This post is related to (but not a part of) the Youth Suicide Rise project.

In a NY Times essay by Jonathan Haidt and Jean M. Twenge titled This Is Our Chance to Pull Teenagers Out of the Smartphone Trap, the authors state the following:

Teenage loneliness was relatively stable between 2000 and 2012, with fewer than 18 percent reporting high levels of loneliness. But in the six years after 2012, rates increased dramatically. They roughly doubled in Europe, Latin America and the English-speaking countries, and rose by about 50 percent in the East Asian countries.

Very Lonely Includes Not Lonely ?!?

The above assertion by Haidt & Twenge implies that high levels of loneliness in school were affecting roughly one third of all students in many countries of the world by 2018. Indeed one can seemingly find numerous examples of this in Table 2 (Percent high in school loneliness) of the paper Worldwide increases in adolescent loneliness that Haidt & Twenge refer to in their essay: US 37%, Canada 35%, UK 33%, France 31%, Australia 34%, Brazil 35% and Russia 45% being some of them.

The authors based the calculations of these percentages on PISA surveys, which include an item asking 15-year-old students if they agree or disagree, perhaps strongly, with the statement “I feel lonely at school.”

One might naturally expect that, in these countries, at least one third of the students did not disagree they are lonely – indeed one might expect that roughly one third of the students agreed outright strongly that they feel lonely in school.

In reality, students who agreed or strongly agreed were a much smaller fraction than those deemed high in loneliness by Haidt & Twenge:

	HL	SA	ASA
Australia	34	6	20
Brazil	35	7	24
Canada	35	6	20
France	31	4	12
United Kingdom	33	5	16
United States	37	7	24
Russia	45	7	27

Note: Here HL = high loneliness per Haidt & Twenge; SA = strongly agreed; ASA = agreed or strongly agreed (with “I feel lonely at school.”)

As we can see, the definition of high loneliness per H&T includes a large portion of students who disagreed they feel lonely – in France these students make up the majority of those supposedly high in loneliness, in UK one half of them, and over a third in the other countries on the list.

Clearly this is not a reasonable definition of a high level of loneliness.

What went wrong?

We find no hint in the NY Times essay as to what went wrong with the definition of high loneliness but we can discover the reasons in the Worldwide increases paper.

PISA survey has a Sense of Belonging section where it asks students to agree or disagree, perhaps strongly, with each of the following six questions: I feel like an outsider at school, I feel like I belong at school, I feel awkward and out of place in my school, Other students seem to like me, I make friends easily at school, and I feel lonely at school.

The authors constructed a single score out of the responses to these 6 questions.

Which is fine.

And they decided to label this score a measure of loneliness.

Which is a mistake.

Only one of the six questions is directly about loneliness, and three of the questions may be viewed by many students to be about school itself rather then about relationships with peers.

The authors should have called their construct a measure of alienation if they did not wish to use the PISA term sense of belonging. Had they done so, there would have been no danger of confusion as to what is actually measured.

If you write a paper about banana exports, and yet your data is actually about overall fruit exports, it is no excuse that bananas are fruits. What is true of fruit exports need not be true of banana exports, especially if bananas make up only one sixth of the country’s fruit production.

Similarly, when you make statements about trends in loneliness, they better be about loneliness, and not about a more general concept like alienation.

Since the Worldwide increases paper presents no data specifically about loneliness, not even in its Supplementary file, its readers are left without proof that the findings on alienation, mislabeled as loneliness, are also true for loneliness.

Grading Alienation on the Curve

The trouble with terminology does not end with the mislabeling of alienation as loneliness.

The definition of high loneliness, which produces such nonsensical results when confronted with answers to the loneliness question in countries like France and UK, would remain problematic even if the authors had named it high alienation.

The problem is that the cut off point is so low that it can include students who reveal signs of alienation on only two of the six questions – or just on one if they agree strongly they feel alienated.

Even if we label alienated every student who shows a sign of alienation on just one item out of the six, vast majority of them would be highly alienated per the Haidt & Twenge cut off point.

The reason for this unrealistic cut off point is that the authors lost contact with the concrete during abstraction: they defined the cutoff based purely on relative positions of student scores.

To understand how this can lead to nonsense, imagine that no students agreed they feel alienated on any of the six items, but some disagreed strongly while others just disagreed. This could produce a roughly normal curve scale, but the cutoff methodology used by the authors would lead us to label as highly alienated students that actually showed no sign of alienation.

There are arguably some concepts, such as being tall, that are entirely relative. It makes no sense, however, to call a lean child fat just because its peers are starving, or a happy child unhappy because its peers seem even happier. And it is the same with loneliness as well as alienation.

Real Trends in Student Loneliness

Given that the paper Haidt & Twenge refer to in their statements on loneliness does not offer any data specifically about loneliness, let us look at the information available from PISA and directly examine the 2012-2018 trends in students feeling lonely as determined by their responses to the “I feel lonely at school” question (we define lonely = agreeing or agreeing strongly).

What we find is that in all of the 37 OECD countries with data, the fraction of lonely students increased by at least 10% from 2012 to 2018, with average and median changes at over +70% (the average OECD prevalence of loneliness increasing from about 9% in 2012 to about 16% in 2018).

And in the 60 countries (both OECD and not) with data, the fraction of lonely students increased by at least 10% in 54 countries, with average and median changes at over +60%.

The trends in high levels of loneliness, if measured by agreeing strongly with “I feel lonely in school”, are even steeper: the median change in OECD countries was +100%, the average change was +137%.

These are remarkably strong and universal trends, unusually so in the world of international education.

And the U.S. is no exception: the fraction of lonely students in the U.S. increased from 12% in 2012 to 24% in 2018.

Why these trends have been largely ignored by the educational community is inexplicable to me.

Haidt and Twenge deserve praise for bringing attention to this troubling matter.

The Perils of Miscommunication

By mislabeling alienation as loneliness and employing the doubly flawed definition of high loneliness, Haidt & Twenge risk losing the trust of their audience, many of whom may end up feeling misled and misinformed.

That would be unfortunate, because I do not believe that Haidt & Twenge were trying to mislead and because the trends they wrote about in their NY Times essay are truly worrisome and deserve attention.

As we have seen above, international increases in loneliness are indeed very strong and nearly universal. There is therefore no doubt the authors could have easily proven worldwide rises specifically in loneliness – the misleading terminology does not help hide any weaknesses in their primary finding.

As to prevalence rates, Haidt & Twenge could have easily alarmed most of their NY Times readership with the fact that nearly 1 out of 4 U.S. students agreed they felt lonely in 2018.

Principal Snyder would not care, but anyone else tasked with the well-being of American students should care. When nearly one quarter of students report feeling lonely in school, something is not right.

The Importance of Proper Terminology

The lesson here is the importance of proper terminology.

Some may object that the authors of the Worldwide increases paper made it perfectly clear in their text that their ‘loneliness’ scale was a construct based on 6 questions, all of which they listed. The methodology behind the definition of high loneliness is also made abundantly clear in the paper.

The problem is that once you start using misleading terminology, someone will sooner or later end up being needlessly misled.

And even if the authors avoid the misleading terminology in their communication with the public, some reporters as well as colleagues will end up using the misleading terminology in their communication with the public.

We should avoid misleading terminology as much as reasonably possible, and editors and reviewers should demand that needlessly misleading terminology is corrected before publication.

Notes:

The Title: I also object to the Worldwide increases in adolescent loneliness title itself, as it gives no clue it refers to student loneliness in schools. Adding ‘student’ or ‘in school’ to the title would not make it unreasonably long. And my objection here is not just a matter of principle – the lack of any hint of educational context in the title makes it harder for educational researchers to note this important paper when they search for articles on school climate and student attitudes toward school.

The Scale: The authors created the ‘loneliness’ (really alienation) scale by assigning integers 1,2,3,4 to to the responses strongly disagree, disagree, agree, strongly agree (reverse-coded when appropriate) and then averaging for the final ‘loneliness’ score. Note that this construct is partly arbitrary: there can be other reasonable value assignments, such as 1,2,4,5 or 1,3,4,6.

The Graphs: There are jumps from 2003 to 2012 on the x axis of the graphs without any visual clue that the gap between survey years just increased from 3 to 9 years! See below for one example.

This means the jump is easy to miss and the graph can be very misleading. The authors need to either triple the distance from 2003 to 2012 or at least indicate the gap by a visual clue such as zigzags (squiggles) in the middle of the way to 2012.

Sidenote: I do not object to the authors not starting the y axis at 1 (the minimum value of the 1-4 scale) in the loneliness graph – in fact I consider such complaints to be preposterous, especially since none of the country measures, not to mention the international averages, are anywhere near the minimum. Demanding its inclusion is an absurdity akin to proposing that patient temperature charts must include absolute zero.

Furthermore, those who declared that the loneliness (alienation) changes are trivial because they would look small on a graph with the origin added need to realize that science is about facts – not visual impressions! If you are going to belittle some finding on the basis of your feelings please first do the math to check if your feeling is in sync with reality. Those changes on the graph correspond with massive increases in alienation and loneliness around the world – e.g. the doubling of lonely U.S. students from 12% in 2012 to 24% in 2018.

The Authors: The Worldwide increases paper has altogether 6 authors, all of whom deserve credit for helping to bring attention to a disturbing trend in student welfare: Jean M. Twenge, Jonathan Haidt, Andrew B. Blake, Cooper McAllister, Hannah Lemon, and Astrid Le Roy.

Limitations: The PISA data explorer returns only whole numbers for percentages, so besides sampling issues this is another reason to view the loneliness trends calculated by me as approximate.

Links:

Jonathan Haidt and Jean M. Twenge: This Is Our Chance to Pull Teenagers Out of the Smartphone Trap in The New York Times July 31, 2021.

[https://www.nytimes.com/2021/07/31/opinion/smartphone-iphone-social-media-isolation.html]

A large excerpt of the article can also be accessed at [ https://aspenbraininstitute.org/blog-posts/this-is-our-chance-to-pull-teenagers-out-of-the-smartphone-trap ] for free.

Twenge, Jean M., et al. "Worldwide increases in adolescent loneliness." Journal of Adolescence (2021). [https://doi.org/10.1016/j.adolescence.2021.06.006]

PISA data can be explored at [https://pisadataexplorer.oecd.org/ide/idepisa/] but the site is often down. Raw PISA data in ASCII files is available at [https://www.oecd.org/pisa/data/].

I placed loneliness results for the 37 OECD countries with 2012 and 2018 PISA data on my site at [https://theshoresofacademia.blogspot.com/p/student-loneliness-data-from-pisa.html].