New York City Department of Health Creates Coalition to Address Prejudice and “Racial Normalization” in Clinical Algorithms


New York City announced a recent effort to end racial adjustments in clinical algorithms used by city hospitals that experts say perpetuate racist assumptions in healthcare and may lead to chronic care. inferior quality for minority populations.

Clinical algorithms are used by healthcare facilities to help with the way medical care is delivered to patients, and many hospitals use a process called racial normalization – also known as racial adjustment – which has been found to have a negative impact on the type of care. Black and Latino residents of the city receive.

In a statement, city experts used several examples of real-world situations that affected city residents, including an “adjustment” factor for black patients, in reference to kidney function, which calculates. that the kidney function levels of blacks are healthier than white patients. for the same measured result. This resulted in delays in the care needed for black patients.

Racial normalization in maternal care has led black and Latin mothers to be more likely to have unnecessary Caesarean sections despite the same age, health status, and childbirth history as white women. These algorithms have more generally led to less favorable outcomes for black and Latin American people giving birth and have exacerbated existing inequalities in maternal health and health, according to the city.

In order to address this issue, the New York City Department of Health announced the formation of the Coalition to End Racism in Clinical Algorithms (CERCA) to End the Inclusion of Race Adjustment. in clinical algorithms.

“The Coalition to End Racism in Clinical Algorithms will help create a healthier and fairer city,” said Health Commissioner Dr Dave Chokshi.

Deputy Commissioner and Chief Medical Officer Dr. Michelle Morse said that raising awareness of the problematic concept and practice of racial adjustment in clinical algorithms has the potential to create more healthcare outcomes and experiences. equitable for those who identify as Black, Indigenous and people of Color.

NYC Health + Hospitals President and CEO Mitchell Katz added that the city’s public health system is eager to collaborate with the health department and other health systems.

“This coalition has the power to further accelerate the work and impact of our system’s existing ‘Medical Eracism’ initiative, which also aims to challenge the status quo of race-based algorithms present in diagnosis and processing which have been widely accepted in our field for decades. “Katz said.

CERCA has 12 members – NYC Health + Hospitals, Maimonides Medical Center, MountSinai Health System, New-York Presbyterian, Northwell Health, NYU Langone Hospitals, One Brooklyn Health, SBH Health System, SUNY Downstate, Wyckoff Heights Medical Center and Cortelyou Medical Associates.

Each said they would end racial adjustment in at least one clinical algorithm and create plans for racial inequality assessment and patient engagement. They also plan to meet bimonthly for two years and will produce an annual report, the first of which will arrive in June 2022.

Black medical experts have long disputed the racial adjustments used in the algorithms hospitals deploy for treatment. Dr Dorothy Roberts, professor of African studies, law and sociology at the University of Pennsylvania, said that racial adjustments in medicine are “not only based on false racist beliefs about human biology, but that it has also been shown to harm black patients “.

Caitlin Seeley George, campaign manager for the technology advocacy group Fight for the Future, said that race adjustments are not only an “extremely deplorable medical practice based on racist beliefs,” but are also a prime example of the how artificial intelligence can exacerbate these racist beliefs.

“The concept of race-based biological differences existed long before AI, but this technology has been militarized to exacerbate these racist practices. There are so many algorithms in place that mostly work in a black box with little or no no transparency about how they make their decisions. In order to demolish the racist structures in our society, we have to open up these algorithms to find what other racist beliefs are part of their calculations, “said Seeley George.

“We need all of our health systems to do this job, we need legislation that prevents the adoption of harmful algorithms, and in New York, it has to go beyond a clinical algorithm (this which is their current reach) .And these efforts must also include real people who have been harmed by these algorithms as well as human rights organizations and experts who can provide critical insight into the damage these algorithms have on real people. . “

Dr Danya Glabau, assistant professor at NYU Tandon School of Engineering and director of the university’s department of science and technology studies, culture and society, said that “algorithms” in the context medicine have a much longer history than the systems informatics that the board seems poised to tackle.

Glabau has written extensively on how mundane technologies can have a significant effect on health outcomes.

Medical algorithms, according to Glabau, are essentially any type of decision tree that physicians use to make treatment decisions, and the use of non-computerized medical algorithms has been the cornerstone of evidence-based medicine for decades.

Automated digital algorithms, however, move the doctor’s judgment more or less away from treatment decisions, as a computer makes the decisions, she explained, adding that when digital algorithms were deployed, it was believed that elimination humans would eliminate humans. racism.

However, since the decisions of automated algorithms are based on data from past human decisions, human biases like racism and classism are still factored into these tools. them alone because the history of medicine is racist, ”Glabau said.

“It’s hard to say exactly how prevalent digital algorithms are and how many of them are interested in race in particular. But it is likely that most providers use more than one on a daily basis and are aware of it or not. We know that digital algorithms are embedded in a wide variety of tools in medicine today, from planning systems and medical imaging tools to diagnostic technology and treatment delivery devices. tend to be proprietary and it is not always clear how they take into account different data or factors. ”

Glabau explained that another issue is that organizations adopting these algorithms may not have people with programming or ethics expertise involved in the review process to appropriately verify the possibility that the algorithms can discriminate. . Even algorithms that disregard race, she added, can explicitly detect it indirectly through things like postcode.

The council needs to investigate how many digital algorithms shape care, how many take into account race, and how many take into account other factors that could correlate with race or serve as a proxy, Glabau told ZDNet.

Researchers like Amy Moran-Thomas have shown that even simple devices like pulse oximeters can have racist results.

“In this case, the designers just didn’t take into account how skin color would affect the readings given by an optical sensor. We also know that tools like planning software can have racist results. although the scheduling did not seem to have anything to do with race. But black patients in particular were booked in duplicate or triplicate as many had difficulty getting to their appointments on time due to factors beyond their control Glabau said.

“These examples show how difficult it can be to anticipate how algorithms and other digital systems will have racist outcomes. In a city like New York City, where COVID has hit BIPOC communities hard and where the zip code correlates with the income and racial segregation, like seemingly mundane technologies can have significant health consequences.

But the only way for the board to be successful is to give it full access to hospital system operations, software and technical documentation from the companies that produce the algorithms, Glabau added.

They should also have the space to develop binding guidelines that can be implemented across the city.

“If this advice doesn’t have teeth, it may find shocking information or make well-meaning recommendations, but it won’t change anything for patients or fulfill its anti-racist mandate,” Glabau said.

“On the tech side, more transparency and regulation of medical algorithms is badly needed. The FDA has revised and strengthened its medical algorithm review procedures since 2017, which I see as a positive first step. Race and racism should be an explicit measure. On the side of hospitals and medical providers, greater internal expertise in ethics and social sciences is needed to understand the possible implications of using new algorithmic systems. Computer and data science experts should also be involved in the procurement process so that healthcare organizations fully understand how the algorithm is designed, how it uses past and new patient data, and how it formulates recommendations. ”


Comments are closed.