After winning TV game Jeopardy !, IBM AI appears ready to revolutionize medicine. Doctors are still waiting for the miracle that IBM AI will bring

In 2014, the multinational computer technology corporation IBM, USA opened a new headquarters in the field of artificial intelligence, this product is called artificial intelligence software - IBM Watson supercomputer. Inside a glass tower in lower Manhattan, IBM manufacturers can take prospective customers and journalists on a tour of the "immersion chamber", like a miniature planetarium. There, in the dark, customers sit on swivel chairs while fancy graphics flashed around the curved screens that covered the walls. It's sometimes the closest you can get into Watson's electronic brain, the makers say.

The massive launch in 2014 of the Watson supercomputer, showed its potential in medical reforms using AI (Artificial Intelligence) - a goal for which Virginia Rometty IBM Group CEO often referred to as the "moon shot" of the company. In the demo, Watson synthesized the patient's strange symptoms and produced a diagnostic list with each comment on Watson's confidence level and links to supportive medical research.

Watson never fails to make an impression: Its memory stores knowledge of every rare disease and its processor is not susceptible to cognitive bias, which suggests yes may not need a doctor. It can solve a difficult case in seconds. If Watson can bring that instant expertise to hospitals and clinics around the world, then it looks like AI can reduce diagnostic errors, optimize treatments and even alleviate lack of Missing doctors, not by replacing doctors, but by helping doctors work faster and better.

Outside the corporate headquarters, however, IBM discovered that the technological prowess did not match the rather messy medical situation of today's healthcare system. And while attempting to apply Watson to cancer treatment - one of the biggest challenges in medicine, IBM encountered a fundamental mismatch between how machines learn to do and how doctors work.

In fact, projects that IBM Corporation claims on the first day of launch do not yield commercial products. Eight years later, IBM announced more high-profile efforts to develop artificial intelligence in medical technology, many of which have failed, and some of them have failed miserably. The company has spent billions on acquisitions to bolster its internal efforts, but insiders say the acquisitions have not contributed much. And products emerging from the IBM Watson Health division are not like the brilliant AI doctor ever envisioned. These products are like AI assistants with only a few routine tasks.

Mr. Robert Wachter - Dean of the Faculty of Medicine at the University of California, San Francisco, is also the author of the book The Digital Doctor: Hope, Hype and Harm at the Dawn of Medicine. "In general, I think they're in a little bit of trouble," says McGraw-Hill, 2015 (McGraw-Hill). In part, he said, IBM Corporation is suffering from its own ambitions: It is the first company to make a big push to bring AI to clinical facilities. But IBM was also blamed for bad intentions and skeptical of bragging about Watson's abilities. "They offer the form of advertising first, followed by the product and make people excited," said Robert Wachter. “After that, Watson was brought to use. This is a series of extremely difficult problems, by first launching, IBM proved it to everyone. "

At the 2017 IT Professionals Conference in Health, IBM Group CEO Rometty told the crowd: “AI is real, it's key, AI is here and AI can change most. everything about medical care, ”she added,“ it could usher in the golden age of Medicine ”. Rometty is not alone in seeing this opportunity: Medical and computer science experts agree that AI has the potential to transform the healthcare industry. So far, however, that potential has mostly been demonstrated in carefully controlled experiments. Only a few AI-based tools have been approved by regulators for use in hospitals and doctor's offices. These pioneering products work primarily in the field of vision, using computer vision to analyze images such as x-rays and retinal scans. (IBM Corporation does not have diagnostic imaging analysis products, although IBM has research projects in that area.)

But looking beyond the realm of imaging diagnostics, today's best AI is actively working to understand complex medical information. So encoding a specialist in software turned out to be a very difficult problem. IBM Corporation has learned these painful lessons in the marketplace as the world has watched. While IBM hasn't abandoned this moon shot technology, the failures show how difficult it is for technologists and doctors to build an "AI doctor".

Winner of the Jeopardy TV game! 2011 showed the outstanding skills of the Watson supercomputer with NPL (NLP - Natural Language Processing - The ability to process natural language). To get into this game, Watson had to analyze puny complex clue syntax, search a large text database for possible answers, and determine the best answer. The Watson supercomputer is not a glorified search engine; it does not document based on keywords. Instead, it uses hundreds of algorithms to map "entities" in a sentence and understand the relationships between them. It used this skill to understand the Jeopardy! by exploiting millions of text sources.

IBM Research Key Medical Scientist at Jeopardy! Mr. Martin Kohn said: “Watson seems to be able to understand the meaning of language, rather than just recognizing word patterns. This is a technology that is more powerful than any that has ever existed before. ”Moreover, Watson developed this ability on his own through machine learning. The IBM researchers trained Watson by giving it thousands of Jeopardy! Clues, and the answer was either true or incorrect by default. In this complex data set, the AI ​​discovered patterns and created a model of how to get data from the input (a clue) to the output (a correct answer).

Long before the Watson supercomputer joined Jeopardy !, IBM Corporation looked at its potential in the medical field. Medicine, with its vast array of patient data, appears to be a clear fit, especially as hospitals and doctors are turning to electronic medical records. Some of those data can be easily processed by machines, such as test results and vital signs measurements, but most of it is “unstructured” information, such as Doctor and hospital notes summary of hospital discharge records. Written narrative accounts for about 80% of the medical record of a typical patient using jargon, stenogram and subjective statements.

Mr. Kohn came to IBM Group with a medical degree from Harvard University and an engineering degree from Massachusetts Institute of Technology (MIT - Massachusetts Institute of Technology), and was very excited to help the Watson supercomputer solve the language of medicine. . "It seems that Watson is capable of overcoming these complications," he said. By turning NLP (Natural Language Processing) (NLP) capabilities into medicine, Watson is able to read patient medical records as well as all medical documents: textbooks , general articles, approved drug lists, ... With access to all of this data, Watson could become a super doctor, able to make a differential diagnosis without any human being. can be found.

“Doctors go to work every day, especially those who have collaborated with IBM Group in their early medical efforts,” said Herbert Chase, professor of medicine and medical bioinformatics at Columbia University. frontline doctors, primary health care doctors, with the understanding that: they cannot know everything they need to know to practice in the best, most effective way ”. But he says, Watson can keep up - and if Watson is turned into a "clinical decision aid" tool, it could also allow doctors to keep up. Instead of providing clues like Jeopardy !, doctors can give Watson a history of a case and ask Watson to diagnose or come up with an optimal treatment plan.

Professor Chase worked with IBM Group researchers on prototypes for a diagnostic tool that overwhelmed visitors in the Watson soaking room. But IBM Corporation decided not to commercialize it and Professor Chase parted ways with IBM Corporation in 2014. He has been frustrated with Waston's slow progress in medicine since then. "I don't believe there is any spectacular race here," he said.

Professor Chase is one of many early Watson enthusiasts who are demoralized. Eliot Siegel - Professor of Diagnostic Imaging and Vice President of Information Systems at the University of Maryland, has also collaborated with IBM Corporation on diagnostic research. Although he thinks AI-powered tools will be indispensable for doctors within a decade, he is not confident that IBM Corporation will build them. "I don't think they're good at AI," Siegel said. The most exciting things are happening at Google, Apple and Amazon ”.

For Mr. Kohn, who left IBM Group in 2014, he said that IBM fell into a common trap: “Just proving that you have powerful technology is not enough. Prove to me that it will really do something useful, it will make my life better and my patients' lives better ”. Kohn said that he is waiting to see review articles about AI in medical journals, to prove that AI can improve patient outcomes and save costs for the medical system. "There have been very few such articles today, and no results on Watson," he said.

AI for the first time ever breaks into healthcare

Doctors are a conservative group - for good reason - and slow to adopt new technologies. But in some areas of healthcare, medical professionals are starting to find the artificial intelligence system reliable and useful. Here are a few first steps towards AI medicine.

When trying to bring AI into medical care, IBM Corporation took on a huge engineering challenge. But lagging behind tech giants like Google and Apple in many other compute areas, IBM needed something big to survive it deserved. In 2014, IBM invested US $ 1 billion in Watson's unit, which is developing technology for many business areas. In 2015, IBM announced the creation of a special Watson Health division, and by mid-2016, Watson Health acquired four medical data companies for a total cost of about $ 4 billion. It seems that IBM Corporation has the technology, resources, and commitment needed to make AI thrive in the healthcare industry.

Currently, the head of IBM Group, talking about the Watson Health effort as a "journey" downhill with many turns. IBM Senior Vice President of Cognitive and Research Solutions - Mr. John E. Kelly III said: “It is a difficult task and a challenge to bring AI into healthcare. But we are doing it. ” Kelly has been guiding Watson's efforts since Jeopardy !, and by the end of 2018, he also took over direct supervision of Watson Health. "We continue to learn, our services will change as we learn," he said.

“Such a diagnostic tool, for example, is not marketed because the market is not there,” said Ajay Royyuru, vice president for health care research and IBM life sciences. Diagnosis is not a business ”. "There are a few things the experts have done quite well," he added. It is a difficult task, and no matter how well you do with AI, AI will not be able to replace specialist doctors. (Not everyone agrees with Mr. Royyuru: A 2015 report on diagnostic errors from the National Academy of Science, Engineering and Medicine states that: improving diagnostics represents a destiny the order "ethics, professionalism and public health").

In an effort to find business cases that bring AI into the health sector, IBM has pursued a large number of projects targeting all different people in the healthcare system such as doctors, practitioners. primary, insurance company and patient. "What ties all of these objects together is an attempt to provide" decision support using AI [analyzing] giant datasets, "said Kelly. IBM's most widely publicized project focuses on Cancer, where they hope to deploy Watson's "cognitive" capabilities, to turn big data into personalized cancer treatments for each. patient.

In many attempted applications, Watson's Natural Language Processing (NLP) capabilities have struggled to understand the meaning of medical text, as well as many other AI systems. Mr. Yoshua Bengio - professor of computer science at the University of Montreal and leading researcher on AI said: “Right now we are working extremely well with NLP compared to 5 years ago, but AI is still bad much more than humans ”. "In the literature, the AI ​​system cannot understand ambiguity and cannot pick out subtle clues that a human doctor might perceive." Mr. Bengio said that current NLP technology can help the healthcare system: “This is not a complete understanding to do something extremely useful. But so far, no AI has been built that can match the human doctor's understanding and insights. "No, we have not been there yet." He said.

The IBM Group's work on Cancer serves as a prime example of the challenges IBM faces. Mr. Mark Kris - a lung cancer specialist at Memorial Sloan Kettering Cancer Center in New York City, who has led his partnership with IBM Watson since 2012, says: “I don't think people with ideas take this long and complicated way ”.

Efforts to improve cancer treatment have two main projects. Kris and other elite physicians at the Sloan Kettering Cancer Research Center trained an AI system to become the Watson for Oncology product in 2015. Across the country, the elite The MD Anderson Cancer Center residency at the University of Texas, Houston, partnered with IBM Corporation to create another tool called the Oncology Expert Advisor (ie the Oncology Expert Advisor tool). The MD Anderson Center tested the tool in the department of leukemia, but it never became a commercial product.

Both of these efforts have received strong criticism. An interesting article on Watson for Oncology alleged that these two efforts provide useless and sometimes dangerous recommendations (IBM Corporation is arguing these allegations). Looking broader, Kris says he often hears criticism that the product is not real AI. And the MD Anderson Center project failed significantly: A 2016 audit by the University of Texas found that the MD Anderson Center spent $ 62 million on the project before canceling it. A closer look at these two projects reveals a fundamental mismatch between the promise of machine learning and the reality of healthcare - between the "real AI" and the requirements of a real. Functional products for doctors today.

Watson for Oncology is said to have acquired extensive medical literature on cancer and real cancer patient medical records. With his superior computing power, it is hoped that Watson will check hundreds of variables in these medical records - including anthropology, tumor characteristics, treatments, treatment outcomes - and Discovers elements invisible to humans. It will also be updated with articles in the journal about cancer treatments published every day. To the oncologists at the Sloan Kettering Center, it sounds like a potential breakthrough in cancer care. To IBM Corporation, it seemed like a great product. "I think people don't know what we're doing," said Mr Kris.

Watson learned quite quickly how to scan articles about clinical research and determine the underlying results. But it couldn't read the articles in the doctor's way. "Information doctors extracted from an article that doctors use to change their care, may not be the main point of the study," Kris said. Watson's mindset is based on statistics, Kris explained, so all Watson can do is collect statistics to come up with results. "However, the doctors don't work that way."

For example, in 2018, the FDA (FDA - Food and Drug Administration - US Food and Drug Administration) approved a new "inviolable" cancer drug, effective against all tumors exhibited a specific gene mutation. The drug is rapidly monitored based on impressive results with only 55 patients, of which 4 have lung cancer. "Right now, we say that every patient with lung cancer should have this gene screened," said Kris. All previous guidelines have been removed, based on 4 patients ”. But Watson will not change conclusions based on just four patients. To get around this problem, experts at the Sloan Kettering Center created "synthetic cases" that Watson can learn from, essentially patients with cancer anthropometric profiles and traits. certain. "I believe in analysis that can find anything," Kris said. However, with Cancer, it was not really successful ”.

Do you agree?

Several studies have compared the cancer treatment recommendations of Watson for Oncology with oncologists at the hospital. The right percentage indicates how well Watson's advice is aligned with the specialist's treatment plans.

The realization that Watson cannot independently extract insights from new news in medical literature is only the first strike. The researchers also found that it was unable to extract information from the patient's electronic medical record as they expected.

At the MD Anderson Center, researchers brought Watson to work with the medical records of leukemia patients - and they quickly discovered how difficult it was for Watson to work with the records. there. Yes, Watson already has extraordinary NLP (NLP - Natural Language Processing) skills. But in these records, data may be missing, vaguely written, or out of chronological order. In a 2018 article published in The Oncologist, the team reported that the Oncology Expert Advisor provided by Watson has had varying success in extracting information from Text documents in medical records. It has a 90-96% accuracy score when it comes to clear concepts like diagnosis, but scores only 63-65% for time-dependent information like therapy.

In a final test for the AI ​​super doctor dream, researchers realized that Watson couldn't compare a new patient with a series of pre-come, cancer patients to uncover hidden factors. hide. Both the Sloan Kettering Center and MD Anderson Center hope that AI will mimic the capabilities of their oncologists who already have experience with their patients, treatments and treatment outcomes. when they come up with strategies for a new patient. A machine that can do the same sort of population analysis - more rigorously, and employing thousands more patients would be very powerful.

But current standards in the healthcare system discourage such practical learning. The MD Anderson Center's Oncology Expert Advisor tool only provides “evidence-based” recommendations of association with official medical regimens and results of studies. published in the literature. If an AI system relies on its advice about patterns it has detected in the medical record, for example this group of patients will respond better to this drug - then their recommendation would not. considered evidence-based, the gold standard in medicine. Without the tight control of scientific research, such a finding would be considered only a correlation, not a causal relationship.

Kohn, previously working for IBM Group and many others think, healthcare standards must change for AI to realize its full potential and to transform medicine. "The gold standard isn't really precious," said Mr Kohn. "AI systems that can look at more factors than ever would represent a clinical trial and can classify patients into different categories to deliver" truly personalized treatment "packages. . Infrastructure must also change: Medical facilities must agree to share proprietary data and control their privacy, so that AI systems can learn from millions of patients tracked across multiple year.

According to anecdotal reports, IBM Corporation has had difficulty finding buyers for its Watson oncology products in the United States. Some oncologists say they believe in their own judgment and don't need Watson to tell them what to do. Others say that Watson only suggests standard treatments they know of. But Mr Kris said, some doctors are finding it useful as a second opinion they can share with anxious patients, saying: “Not perfect, and limited, but It is very useful". The IBM Group sales representative has had more fortunes with markets outside the US, hospitals in India, Korea, Thailand, and more have adopted this technology. Many of these hospitals proudly use the IBM Watson brand in their marketing, and tell their patients that they will be treated for cancer with artificial intelligence power.

Over the past few years, these hospitals have begun publishing studies of their experiences with Watson for Oncology (Watson for Oncology). In India, doctors at the Manipal Comprehensive Cancer Center evaluated Watson on 638 breast cancer cases and found a 73% match in treatment recommendations; Watson's scores were reduced due to poor performance against metastatic breast cancer. Watson also performed poorly at Gachon University's Gil Medical Center, South Korea, where its top recommendations for 656 colon cancer patients, matched specialists 49% of the time. Doctors reported that Watson did poorly with elderly patients, Watson did not recommend certain standard drugs and had an error that caused him to be offered follow-up, rather than cumulative treatment. extreme for some patients with metastatic cancer. 

These studies aim to determine, whether Watson for Oncology is working as expected or not. But there are no studies that show any benefit to patients. Observers of the University of California, San Francisco say, a growing problem with IBM Corporation: “IBM knows, wins Jeopardy! and the partnership with the Sloan Kettering Center will put IBM in medical care. But IBM needs to show up quickly and have impact on difficult results. ” Observers said that IBM had to convince hospitals that the system was well worth the financial investment. “This is really a very important issue for success. Success is that there's an article published in the New England Journal of Medicine that shows that when we use Watson, the patient gets better or we save more money ”, the followers are still waiting to see which such articles appear.

Kris - The Sloan Kettering Center is not discouraged; Technology will only get better, he said: "As a tool, Watson has extraordinary potential." “I hope that those with intelligence and power who love computer technology will stick with Watson. Although it has to go a long way, it is very valuable.

Several success stories are emerging from Watson Health that, in some narrow and controlled applications, Watson appears to be adding value. For example, the Watson for Genomics product, was developed in collaboration with the University of North Carolina, Yale University and other organizations. These products are used by genetic laboratories, they generate reports for practice oncologists: Watson takes in a file listing the patient's genetic mutation, and in just a few minutes, it has Can generate a report describing all drugs and clinical trials. “We enable laboratories to scale up,” said Vanessa Michelini, an IBM Distinguished Engineer who led the development and launch of Watson for Gen (Watson for Genomics) products in 2016. tissue ”.

Watson had a relatively easy time with genetic information presented in structured files and without ambiguity - either there was a mutation there, or not. The tool does not use NLP to mine medical records, instead uses the NPL to search textbooks, scientific articles, drug approvals, and clinical trial announcements, where it searches for very specific statement.

IBM Group Partners at the University of North Carolina published the first paper on the effectiveness of Watson for Genomics (Watson for Genomics) in 2017. Of the 32% of cancer patients participating in that study, Watson There were found important mutations that were not determined by human judgment, making these patients good candidates for a new drug or a new clinical trial. But so far, there is no indication that Watson for Gen (Watson for Genomics) has yielded good results.

The US Secretary of the Veterans Affairs (VA) uses Watson's reports for Gen (Watson for Genomics) at more than 70 hospitals nationwide, said Michael Kelley, VA's national program director, on cancer. The Veterans' Association initially tested the system on lung cancer and now uses it for all solid tumors. "I think it helps to improve patient care," he said. When VA oncologists are deciding on a treatment plan, "that's a source of information they can bring to a discussion." But Kelley said he doesn't think that Watson is a robot doctor: "I tend to think of it as a master medical librarian."

Most doctors would probably be very happy to have an AI librarian by their side, and if that was what IBM Corporation had originally promised the doctors, today doctors would probably not have done so. hope. The Watson Health story is a cautionary tale about arrogance and hype. Everyone likes ambition, loves moon shots, but no one wants to climb on a rocket that isn't working.

So far, little has been achieved

IBM Corporation began its efforts to bring Watson into the healthcare industry in 2011. Since then, IBM has issued nearly 50 announcements of partnerships aimed at developing new AI-enabled tools for medicine. Some collaborate on tools for doctors and organizations; Some work on consumer applications. Although many of these alliances have yet to produce commercial products, IBM says these research efforts are valuable and many relationships are underway. Below are representative samples of projects:

Source: 

Translation summary: Dr. Nguyen Huu Tung et al

Explore more:

1. Jeopardy! : Is a TV knowledge quiz program in the US with quizzes of history, literature, art, pop culture, science, sports, geography, puns, and more . The program has a unique answer-and-question format in which the candidate is given the clue in answer form, and must produce the answer in question form.

2. IBM stands for International Business Machines, is a multinational computer technology corporation headquartered in Armonk, New York, USA. IBM was founded in 1911 in New York City, originally as Computing Tabulating Recording (CTR) and changed to International Business Machines in 1924.

3.Moon shot means breakthrough and potential inventions, but difficult to execute and with high risk of failure

4.