COVID-Net: Final Reflections

by Sheldon Fernandez

Thoughts and thanks as we transition the project to the University of Waterloo

“Sheldon, ping me when you have a moment – I have something you should see”.

I can still recall reading the Slack message above from our Chief Scientist and wincing. Not intentionally of course. Professor Wong, or Alex, as I’d called him since we started DarwinAI, was the most brilliant mind I’d encountered in my professional career (and recently recognized by Stanford University as one of the world’s top 2% of scientists). The problem was, I didn’t have a moment. Or a half-moment. It was March of 2020 and the world was on fire due to Covid-19.

Of course that’s how it felt in the formative days of the pandemic. Rising infection rates, ominous warnings from abroad, and general ignorance of the virus and its societal implications spawned anxieties few of us will forget. For me personally, add to that a board prudently imploring us to take drastic measures to keep our business afloat and a childcare plan that collapsed overnight (daycare, what daycare?) and I was overwhelmed. 

But Alex persisted. A second message followed, then a third. And finally he dropped it on me: in the seven days since the provincial lockdown, he and the team had developed a neural network that could diagnose COVID-19 with uncanny accuracy. 

I smiled. At the absurdity of it all or being humbled by a more powerful mind. I’m not sure, but it was a small moment of levity amidst the global whirlwind: while our COO and I were taking steps to ensure the survival of DarwinAI, Alex and the team were taking steps to ensure the survival of the species.

***

We released COVID-Net on March 22, 2020. And today – 613 days later – we’re transitioning ownership of the project along with maintenance and support of the codebase to the wonderful team at the Vision and Image Processing Lab at the University of Waterloo.

While the story of how we rapidly developed COVID-Net is interesting and highlights key elements of our technology, in this article – our final on the topic – I’d like to reflect on our twenty month journey with the initiative and the contributors and corporate partners that made it possible.

***

The response to our initial announcement around COVID-Net was overwhelming. As I wrote at the time, in determining “how to best deploy our skills in service of the present crisis,” we “collaborated with researchers at the University of Waterloo’s VIP Lab to develop COVID-Net: a convolutional neural network for COVID-19 detection via chest radiography.” Moreover, we decided to fully open source the project “in hopes of developing a robust tool to assist health care professionals in combating the pandemic.”

Academics, researchers and developers were invited to peruse and enhance the codebase and did so in an impressive fashion and we were flooded with suggestions and updates from motivated individuals around the world. Word of the project also spread through substantial press coverage in those initial months, including pieces in MIT Tech, Forbes, Synced, VentureBeat, BetaKit, ZDNET and PC Magazine. Like us, folks in the AI community were grappling for a way to deploy their skills against the pandemic and COVID-Net seemed one straightforward way to do it. 

But what exactly was the point behind an AI tool that could identify COVID given its small, but still non-zero, margin of error? After twenty months, it can be hard to recall the diagnostic challenges at onset of the crisis. Below is an extract from one of the many proposals we authored at the time:

At present, there are simply not enough materials and resources to meet the screening demands for COVID-19 using the standard Polymerase Chain Reaction (PCR) test. As has been widely documented, the PCR approach, though reliable, is not straightforward and requires specialized skills, laboratory equipment, and particular chemicals. With diminishing supplies and an increasing number of cases, patients are waiting days to get results. 

This limited screening velocity presents challenges to both containing the pandemic and identifying those most affected by the virus. The proposed project is an important step in creating an alternative testing method that will allow assessment centers to triage patients more rapidly, increasing the rate of diagnosis and patient flow and enable hospitals to better utilize resources as the pandemic continues.

Initially, then, COVID-Net was intended to supplement the definitive but more arduous PCR test. Per an interview I gave two weeks after we released the source code “given the limitations around the classical test, the chemicals one needs, the lack of resources, the expertise you need to carry it out, the time it takes to around a result, this is a complementary tool that will give you a statistical answer in a very quick period of time.”

By providing a rapid, if probabilistic, method of diagnosis the argument went, we could triage vulnerable populations more effectively to help curtail the spread of the virus (e.g., “there is a 93% chance you’ve contracted COVID, self-isolate until we can administer a more robust test”). For marginalized and isolated communities with limited resources, COVID-Net proved a useful tool to this end and we received word the system had found its way to India, Malaysia, Italy, the United Kingdom, and even the Australian Outback where government officials were evaluating the pandemic’s propagation to the remotest parts of the country.  

***

In light of the overwhelming – and unexpected – response to the initiative, we assigned a small, part-time team to work on the project with their counterparts at the University of Waterloo. The result was a steady stream of enhancements per the updates below:

What sticks out the most during those initial months was the unspoken camaraderie that developed around the initiative. Amidst the frightening and fluid circumstances of the pandemic our contributors went above and beyond, showering COVID-Net with their time and expertise in ways that significantly improved the system. 

Our corporate partners also stepped up with resources, support, and general goodwill. HPE, Intel, NVidia and ARM all generously provided hardware, software tools, and logistical support at various points in the project, resulting in an array of case studies that exemplified the system’s versatility: 

In the months that followed, COVID-Net evolved from a narrow technical experiment into a robust clinical tool with broad application. The following diagram, extracted from our latest paper on the topic, illustrates the comprehensive and present state of the system for COVID-19 CXR clinical decision support. 

Figure 1: The current COVID-Net system for CXR  

There are four parts to the COVID-Net system for CXR clinical decision support: 

  1. The COVIDx dataset – a large and diverse corpus of CXR images used to train the AI.  At present, the repository comprises 16,560 images from 15,528 patients across 51 countries, making it the one of the largest datasets in open access form. 
  2. The COVID-Net model – the deep neural network responsible for case detection.  The model currently differentiates between positive COVID-19 infections, pneumonia infections and normal patients.  As of this writing, the model has an accuracy of 93.3% with a sensitivity of 91.0% and a positive predictive value of 98.9%.  
  3. The COVID-Net S model – a deep neural network responsible for scoring the lung disease severity for COVID-positive patients.  The predicted scores can be used by clinicians and healthcare workers to get a better understanding of the patient’s disease stage and progression, and use this knowledge for individualized patient care decisions as well as individualized treatment planning (e.g., oxygen therapy, ventilator use, etc.).
  4. The COVID-NET User Interface – a report generation and case management interface that allows medical professionals to query, view and analyze patient scans in a streamlined fashion. 

Figure 1 also illustrates the evolving clinical benefits of the system: far from just a complementary diagnostic tool, COVID-Net is now used to grade disease severity and assist with patient treatment planning.  Specifically, the system supports numerous tasks around the clinical decision support workflow using automatic report generation features to assist clinicians with treatment decisions.  For example, a clinician using the COVID-Net system to assess a new patient would first evaluate their X-ray data using the COVID-Net classification model; in the case where this patient is COVID positive, the clinician would then use the COVID-Net S model to assess disease severity, and could plan accordingly for ICU admission if the COVID diagnosis is severe.  

Our efforts around COVID-Net culminated in a highly acclaimed paper in Nature that described the innovative aspects of the system. Of particular importance was the way in which our Explainability technology could identify the virus’s repository markers and manifestations per the figure below:

Figure 2: COVID-Net highlighting critical factors for COVID-positive patients

The benefits of this feature – pinpointing the critical factors associated with COVID-19 – were twofold. First, it allowed clinicians to validate that the predictions made by COVID-Net aligned with their intuitive and expert understandings. Second, it unearthed new insights into the visual indicators behind COVID viral infections, which health professionals could integrate into their own diagnostic processes to improve screening velocity and accuracy.

In tandem, these benefits exemplify the power of Explainable AI (XAI): it enables the subject matter expert to validate the system is making the right decisions for the right reasons, while potentially enlightening that same individual on their very area of expertise.  This iterative back-and-forth – the incremental sharpening of knowledge and understanding between human and machine – provides a compelling guidepost for our AI-filled future.  That this theme was captured in Nature by way of COVID-Net seemed a worthwhile culmination of the project we started twelve months before. 

***

Two strategic partnerships were key in advancing COVID-Net to the extent that we did. 

First, Lockheed Martin, who we announced a strategic collaboration with shortly after the pandemic began, allocated significant funding to the project under Canada’s Industrial Benefits Policy (ITB) program.  This capital allowed us to dedicate a full-time team to the project to coordinate research, conduct user testing, and build out the end-to-end system illustrated in Figure 1.  

It also underscored the importance of the ITB policy to organizations like ours, in which aerospace and defense companies can deploy non-dilutive funding to Canadian startups in the interest of the common good to drive economic growth.  Lorraine Ben, the CEO of Lockheed Martin Canada, emphasized this point in a joint article in the Canadian Defense Review:

“At Lockheed Martin Canada, the work we do is laying a foundation for decades to come in terms of innovation, job growth, STEM development, economic prosperity and security. Partnerships play a critical role in this effort, just like this exciting and timely initiative with DarwinAI. In a most challenging year, it is inspiring to see how Canada’s Industrial and Technological Benefits Policy drives enduring investments for Canadian industry and how it can accelerate solutions to the most complex challenges.”

Simply put, COVID-Net would be a very different – and likely very limited – beast were it not for Lockheed Martin and the Canadian government that approved the funding. 

Our second integral partnership was our collaboration with Red Hat and Boston Children’s Hospital (BCH) to make the system accessible to clinicians and health care professionals. In the early days, COVID-Net was an unwieldy assortment of technical assets that had to be wired together and orchestrated to generate results. This is unsurprising given its academic roots, but it was daunting and of limited use to folks on the front line. 

In cooperation with the Fetal Neonatal Neuroimaging and Developmental Science Center (FNNDSC) at BCH, our team developed the initial GUI that sat atop the hospital’s ChRIS system using Red Hat’s OpenShift framework. For our team, it was thrilling to work with the experts at a leading institution and understand the nuances of the virus from a diagnostic point of view. Clinicians from other institutions eventually joined the effort and helped transform our initial front-end into the slick and intuitive user interface depicted in Figure 3.

Figure 3: The COVID-Net User Interface

The screenshot above illustrates some of the key features of the COVID-Net system:

  • Creating new predictive analyses (top)
  • Finding and examining previously-completed analyses
  • Examining radiological images corresponding to an analysis using the COVID-Net UI radiology viewer (bottom)
  • Generating a PDF report of a predictive analysis

In combination, these features allow a clinician to assess a new patient for COVID-19, review the patient’s medical images and compare against other patients that have previously been assessed, and easily share the results with other medical professionals.   

***

In reflecting on our company’s journey around COVID-Net, the sentiments are numerous: awe with its creators, who designed and developed the system selflessly during a time of international crisis; gratitude to its contributors, who deployed their time and talents to enhance the project in ways too numerous to mention; appreciation for our corporate partners, whose resources and funding transformed an academic initiative into an enterprise-grade application; and finally, unmitigated respect for the front-line workers – clinicians, radiologists, nurses and other health care practitioners – whose cooperation gave us a small window into their on-the-ground efforts that continue to alleviate vast amounts of human suffering during these challenging times. 

Today, COVID-Net has found a suitable home at the VIP Lab at the University of Waterloo. The latest project repository can be accessed here, and curious readers are encouraged to continue to experiment and build upon its solid foundation.

And, even the entrepreneurs, our team has repurposed the core technology that enabled COVID-Net to commercial ends. Most notably, through our manufacturing solutions for quality inspection, which detects anomalies and defects for parts and goods with unerring precision, much like COVID-Net analyzes a CRX scan. Yet a hint of the humanitarian remains: one of our most exciting projects involves an inventive and forward-looking use-case around food security and insect agriculture (video here).   

***

Much time has passed since the world was flipped on its head in March of 2020 due to the Covid-19 pandemic. The challenges and changes persist. But our work continues. The story goes on. And, to paraphrase a US president, an angel still rides in the whirlwind and directs this storm.

In Solidarity,

Sheldon Fernandez

CEO, DarwinAI

Nov 2021