Supreme court justices & health services researchers cite bad science in Affirmative Action debate

It doesn't matter what your cause is, you should never cite bad science

VINAY PRASAD

Recently, the Supreme Court ruled on the constitutionality of Affirmative action in two cases. At least two justices wrote dissents, and a number of physicians online commented on the ruling. If you are interested in a full legal analysis, look elsewhere. This post is about what interests me.

In discussing the ruling, both a Supreme court justice and a team of physician scientists cite specific research that claims Black doctors lower the risk of death among Black patients/ people. That research is not only flawed, but, given observed effect sizes, has to be wrong.

My post is not on affirmative action— but something broader. Many academics are happy to conduct, promote and cite flawed research on sociopolitical issues if it supports their policy preference. This is fundamentally unscientific and unsound. Other physicians are unwilling to say this work is flawed b/c of the growing culture of demonization and name-calling. Ultimately, everyone loses when we conduct, cite and praise bad science. It furthers no cause. It undermines science itself.

Let’s turn to KB Jackson’s dissent in the Affirmative action cases. She specifically cite this.

The paper in question is catastrophically flawed. First, consider that it is a bold claim that a white doctor is twice as likely to kill a black baby. The effect size (TWICE as likely!) is massive.

Next, in my podcast from Aug 2020 I discuss why this paper is flawed (full podcast is 91 min. but relevant discussion runs from 1:31:00 to 0:52:00 mark). Those notes are also captured here.

The key issues are:

If white doctors have so much worse outcomes, one would expect they are making different decisions in the care of neonates than Black doctors— but this paper cannot show the mechanism of the difference

The paper assumes doctor-baby pairings are quasi randomized, but that is unfounded assumption. It may not be quasi randomized and well off Blacks may be more likely to have Black doctors

A baby born is seen by a team of doctors— pediatricians, anesthesiologists, obs— which doctor is ascribed the ‘assigned provider’ per baby. What determines this assignment? (the authors do not provide this information)

Since, the paper was published it was revealed that some hospitals put a treating doctor on the form and others put the head of the unit. (massive bias)

A baby born is seen by a team— nurses, staff, doctors, etc— why are the races (and racial concordance) of these people not accounted for.

If a baby gets sick, and goes to NICU and dies, which doctor is ascribed responsibility. If NICU doctors have different racial make up than other doctors could this not bias results?

Broader issues of administrative data/ multiple hypothesis testing detailed in the episode.

In another essay on the decision, some academic authors claim that lives are put at risk.

To justify this claim, they cite these data.

But this study is even worse than the one before. On Sensible Medicine, I break it down.

Quickly put, the limitations are:

Effect size is preposterously large

No proof that the black doctors are caring for black patients (what is the mechanism?)

Race is defined from self reports. But not all racial groups self report equally, and how do I know it doesn’t vary by geography/ region/ county?

Why is the exposure: the ratio. Why not just # of black doctors per capita, or # of black doctors per capita black people? Did the authors preregister this ratio? Did they try other things before settling on this? How many other groups and how many other analytic plans exist? According to this ratio, if I fired 10% of the white doctors, I would improve life expectancy for blacks. Bizarre.

The model is complicated, but my reading is that the mortality changes seen here arose between 0 and 10 years. Given the data is coded in only 3 times (and each 5 years is time 0,1,2,) we expect the ratio in 2019 to reflect the immediate impact on life expectancy, and the ratio in the 2 prior times to work over 5 and 10 years max. How do PCPs impact survival so dramatically and so quickly? And doesn’t the inclusion of 2019 ratio and outcomes imply we are also looking for immediate effects.

Because of the choice of the ratio, the authors have to discard 50% of data b/c they could not find even 1 self reported black PCP in a county. How would the data look, if these points were included and the exposure was # black PCPs/ per capita or per capita black people? Also this to me is the biggest finding in and of itself! No need to gild the lily and offer a dubious association.

Adjusting for poverty rate is not the same as adjusting for SES.

Analytic flexibility is astronomical , “after testing several models for the level 1 residuals (eg, homoscedastic, autoregressive error structure, etc), mixed-effects growth models with an unstructured residual covariance matrix were used (1) to regress life-expectancy, age-adjusted all-cause mortality rates, and a log-transformed measure of mortality rate disparity between Black and White individuals on the log-transformed representativeness ratio within each county”. Analytic plans for things like this should be pre-registered.

Note, the way PCP is coded— isn’t it likely some sub-specialists are miscoded as PCPs? How do I know this doesn’t vary by geography, location and race?

What about counties with more vs. less cross county health care receipt? Counties adjacent to large cities, etc? Does this not vary by geography and time?

How we do we know direction? Do more black physicians as fraction of all physicians standardized for blacks in the population improve black life expectancy? Or people (richer/ healthier) retire and move to regions that culturally suit them, and have doctors they prefer?

What is the lesson here?

Many people have policy preferences, and want to push for them. Good, me too! But science is concerned only with pursuit of the truth. It undermines science to make strong claims based on weak evidence. Moreover, it creates the appearance of bias or motivated reasoning. It results in massive loss of reputation (of those doing it) and broader distrust of science.

To me, the claim: it would be desirable for doctors in America to be as racial diverse as people in America- is a value judgement, but one I share. The best way to achieve this both in medicine and other desirable careers is sweeping reform to early childhood education. Ironically, school closures during COVID was the single most racist policy— and ironically, it was broadly supported by people who claim to support racial equality. Improving opportunities for everyone (not just the few who become doctors) is the deep solution to disparities.

However, the claim that Black doctors lower mortality rates is entirely unsupported and unlikely to be true. Some tiny experimental data show greater adherence to a handful of interventions— but these studies may not be reproduced and moreover do not show impacts on hard outcomes. The claims of these two research papers, are entirely unfounded. If the topic was something that did not have strong sociopolitical importance, the papers would likely have been rejected repeatedly for weak methods.

What does it say about science and scientists that many appear more concerned with politics than truth? It should scare us. COVID has shown that partisanship can be so blinding. A smart doctor can say with a straight face that two year olds should cloth mask except when they nap. The power of motivated reasoning is strong.

Science works only if scientists hold themselves to the standard of impartial pursuit of truth. If they choose not to, then science loses standing as something objective or apart from politics. It risks becoming a subdivision of political parties. That would be the worst possible outcome, but sadly one we keep marching towards. It’s important for scientists to call out bad methods— even on hot topics.

Source: Vinay Prasad's Observations and Thoughts