Note: this was written in October 2021, and has only been lightly redacted. I feel enough time has passed that it can now be put into the global memory bank without significant risks to myself or others involved. 

Scattered notes on my thoughts about our COVID response support, leaning mostly on my close connection to the Washington state work (with sprawl). I’m purposefully being vague about who did what because I am not sure I remember well and would rather slight everyone.

Team structure

  • We had a “cell” structure for the analytic & modeling work. Each cell had all the skills needed for their job – data munging, modeling, writing, presentation.
  • Support functions were shared across cells: relationship management, communications, project management, strategy management.
  • This spanned across organizational boundaries, with some people forming critical links between cells.
  • No one had complete leadership of anything (except for maybe me for a few weeks at the beginning of the crisis). This was a very good thing for doing the work, but not ideal for long-term strategic evolution and alignment.
  • No one sat at “just one level”, but rather people had roles at a couple different levels in anything they were involved with. This had the huge benefit of propagating contextual knowledge very easily across the team for the most part – with transient exceptions, no one lacked awareness of how they fit in and most people could independently adjust to spend their time where it was most useful within their purview. The downside had largely to do with burnout – thriving in multiple roles for a long time during an emergency is impossible – and groupthink (we didn’t have an effective “red-team”, nor much feedback about strategy that had legitimacy in the eyes of the people doing the work). I think we did okay on scientific groupthink because of the team size, way we talked on slack, and twitter omnivory, but it was fragile.
  • I have thought that if one was to replicate our team, it would be with more cells doing roughly the same thing for different stakeholders, rather than multiple teams working on the same thing. In other words, I do think formal model ensembles are a bad use of resources, whereas a shared “campfire circle” with groups working in different local contexts would be more useful. Idiosyncratic support from a shared intellectual culture to many stakeholders is more useful than statistically optimal support for fewer stakeholders.

    IDM internal - modeling

  • Dominated early on by few, strong personalities. I’m personally proud of being one of them, but the speed of organizational mobilization was frustratingly slow. Also, I think by September 2020, this led to a lot of sub-optimal lock-in where we didn’t always pivot effectively to more impactful things (but also, sometimes you can’t pivot if you care about relationships – NGO hit and run isn’t a good thing either). Again, lack of organizational support played a huge role here – there was no feedback or support process to handle burnout and tunnel vision until some of us started foisting that on our management, after which I feel like the support got much better. (I’m curious if this is a shared perspective or if I’m overgeneralizing from my experience and perception.)
  • I personally think the model triage process went very well (although I know not all agree), but the model support process is very hit-or-miss. In March 2020, we had seven candidate modeling approaches with different levels of maturity, usability, team support. We paired that down to four, with a plan to put the most mature one to bed because of usability and support limitations. Another flagship platform got triaged to data-light use cases because it couldn’t assimilate rich intrahost data and thus was only useful for analyses where hyperlocal detail wasn’t relevant. Covasim flourished due to dynamic dev team and also excellent usability, but dev culture excluded some insightful researchers. RAINIER was the high-risk play that proved very efficient and usable for situational awareness and learning from surveillance data, but to this day relies exclusively on one person.
  • One unifying principle was different models had to say the same thing about stuff both could speak to. This put a lot of pressure on calibration and team unity, and some projects that fell by the wayside did so because they couldn’t fit into this way of doing things.
  • Focus on situational awareness and scenarios over forecasting was key to being insightful and helpful. Build mental models with quantitative models. Closed loop control.

    IDM internal - analytics

  • Data exploration and visualization skills at the speed of thought are not evenly distributed. And the very high pace of covid policy research led us to lean on the same small group over and over again.
  • Would be very nice to figure out how to distribute this burden better.
  • (I’m not sure how to define this, but) we often took a transmission-informed approach to data analysis. Or rather, our perspective on data exploration was asking what it might be able to tell us about transmission. That focus is different the classic descriptive epi. It helps define good modeling questions and gives a more actionable lens on disease burden. Seems like a “cultural” thing that distinguishes us and other scientists like us from the MPHs.

    IDM internal - equity research

  • I have complicated feelings about our work in this space that I hope others can help me understand.
  • With the support of our close DOH colleagues, we were the first group in WA to publicly analyze burden by race, and I think this pressure accelerated public reporting by DOH and PHSKC.
  • We were very careful to avoid making causal claims the COVID data and observational study designs couldn’t directly support. I’m still not sure if we were too cautious or did right in a bad environment. It was the reputationally-safe thing to do, but it meant there were a lot of issues we couldn’t publicly weigh in on where we had a clear picture of what was needed and no one with a platform like ours was saying. Missed opportunities I remember are (1) pushing the state to get n95s to farm and warehouse workers (summer 2020), (2) taking the schools modelings work from (now proven correct) “here’s what’ll happen under different scenarios” to “implement the damn mitigation scenarios and get kids back in school”, (3) pushing the association between income, mobility, and burden as evidence that more workplace mitigation action was needed, as well as more support for isolation outside the home (knowing that workplaces are only part of the correlation). While we did share our science with the public and discuss these implications with the state, we did not use our public voice to advocate for any of this, and I think that was a mistake.
  • Why Washington state? I do not have a galaxy-brain vision of impact, and so I feel good about how much effort we focused on WA, but I can understand this is contentious. Our stated reasons were (1) it’s our home and our neighbors need us, (2) it’s the best access to data and learning we’ve ever had for any disease, and learning is generalizable (3) the policy/gov’t partners actually wanted our help – clear access to levers of power, (4) we can leave an institutional legacy of better modeling and analysis behind at DOH, which is rare in US public health. We did have an under-resourced LMIC effort in 2020, but there were few clear pathways to impact anywhere to my knowledge, so I’m not sure that more effort there, at least early, would’ve been worthwhile.
  • I do think the promise of generalizable learning is playing out in our 2021 portfolio. The work now, especially in covasim, is heavily informed by stuff we couldn’t easily learn from the data available from most LMIC partners. And with poor population data, the feedback to get intrahost stuff right would’ve been weaker (compare our modeling to others working in specific settings to see the difference in rigor).
  • (There’s a lot here I can’t speak to but others can!)

    IDM public outreach - writing

  • While I still feel scientific journal articles are completely the wrong format for the work we did, I now think it was a mistake to put so little on medrxiv. We went with a strategy based on Imperial College and later done very well by Public Health England, but by virtue of being in a backwater state, we did not garner the platform they had. Medrxiv had much more visibility.
  • We also severely handicapped ourselves by being so sterile on twitter. Granted, I was very uncomfortable in that medium, but I still wonder how different it would’ve been with different institutional instincts to back us.
  • All of this is an artifact of coming out of the shadows in a big way under huge pressure. IDM had a very quiet public presence before covid and none of the muscles to be public players. And so we missed a lot of the natural growth that would’ve happened March to June 2020. And after that, BMGF instincts kept us from growing further, and our moment had passed anyway. But, given that I still think we are among the most insightful on earth about COVID (at least in 2020), I regret that we didn’t embrace a bigger platform when it was there for us to have.
  • Our writing did serve the policy partners and local media well.

    IDM - public health interface

  • Multilayer network. Strong, barrier-free connections to the analytic folks doing the number crunching, data management, etc, and connections to people with power and authority. Being effective requires good relationships at both levels, and good relationships within hierarchies. Trust is essential to have distributed authority.
  • Tons of plumbing work by IDM. Dedicated people to manage relationships and communications. It’s a full-time job that most orgs are missing at the level in the org we operate at.
  • Really important that partner organizations want outside help and are willing to find ways to make it work.
  • Really important we were willing to teach and share credit. While I’m sure there are some folks who felt we had too dominant a presence, we mostly felt an esprit de corps with everyone we worked with across organizations, and believe they felt the same. This fostered a collaborative culture in which everyone was learning from everyone else, both with technical partners and policymakers. This made it possible for us to remove ourselves from roles as they became redundant with stuff done by other organizations, while we continued to stay at the edge of the science.

Other Notes: Correspondence with an interviewer

Q: “We would like to know why you were willing to work with the policymakers. Specifically, was it a matter of chance, personality, or something initiated from the top leadership? We are asking this question to identify potential infrastructure that has to be put in place to redeploy effective modeling-policy partnerships in the future.”

Great question, and I appreciate the clarification about identifying infrastructure as part of the motivation. As they say, I did not have time to make this letter shorter…

Speaking for myself and what I know of my closest colleagues, the first reason, and probably most obvious, was that we wanted to help. We knew we had specific expertise that wasn’t widespread in public health or government, but that was very necessary for making sense of a new infectious disease, with messy data, large uncertainties, and very consequential decisions in a situation where waiting for more information was itself a decision with serious consequences. The core of that expertise comes from, collectively, having spent a lot of time working to understand the statistics and population dynamics of infectious disease transmission across multiple pathogens. That gave us prior knowledge to more reliably interpret SARS-CoV-2 data and to make well-informed inferences about what was happening and was likely to happen soon. This expertise is most common among modelers who focus on these aspects of the problem, and is distinct from understanding sources of transmission (classic shoe leather epi), clinical disease (doctors), societal factors that systemically influence risk (epidemiology of inequity), public perception and democratic leadership (elected gov’t), etc. We were confident we could help policymakers focus on the most important dynamics of SARS-CoV-2 that could be influenced through their actions.

Second, there was an important top-down component from policymakers that, for me, had a lot to do with why I ended up working a lot in Washington state and not somewhere else. Early in the pandemic, my close colleagues and I were talking to lots of people in lots of places, trying to help wherever we could and wherever there was someone we could connect with. But over March and April 2020, for me, it became clear that WA-DOH was one of two very receptive partners who wanted our help and were willing to try to overcome institutional barriers to work together. The other was Public Health Seattle & King County, where I had a prior connection through the Seattle Flu Study. WA-DOH saw our early work with King County and reached out to us to talk with them. We worked with both, but over the first few months of the epidemic, it became clear there was less need for what we offered at PHSKC than WA-DOH, and there were more people to help across Washington State than just King County, and so my efforts flowed toward the state relationship over time. In both cases, the top leadership of those gov’t orgs fully supported that their staff should build a relationship with us, including taking the time to set up legal data sharing agreements on the fly.

Related was the culture within these organizations. It was also important to find the places in the org where collaboration was natural. For various reasons, not everyone below leadership was equally willing to work with us. I don’t have much more to say on this right now without speculating on people’s motives, but being clear about where modelers need to plug in is an important part of any future system that works better. I say more about this later in the CDC comments.

I guess overall, one could say there was a “speed dating” aspect to the early days of the pandemic. There was a lot of quick identifying of compatibility that then got more systematized over the first few months. That systematizing may have made it easier to onboard others, and so perhaps other modelers not at IDM have different experiences to share.

Personality played a part in this. Work style compatibility certainly, but also communication skills (despite this email LOL). I think one of my strengths is being able to translate from technical model-speak to intuitive but still scientifically-rigorous messages, and that helped our gov’t and policy partners understand how they could use our information. I saw presentations by similarly capable modelers have less impact because they couldn’t adapt their message to a non-academic non-modeler audience. And while I do it well in conversations and presentations, I’ve seen many others who do it better in writing and on twitter. Similarly, it works better when policymakers ask tight, answerable questions that that directly inform their mental models and/or specific decisions. Gov. Inslee in particular was good at this, which made it easier for us to know how to frame the science so it could be understood.

One thing to be wary of is systematizing the wrong thing. This may be out of scope, but I want to briefly mention the CDC covid modeling activities we chose not to participate in (except for one person attending meetings to see what we weren’t missing). This was the specific early project I’m thinking of https://www.cdc.gov/coronavirus/2019-ncov/hcp/planning-scenarios.html, although we’ve also mostly stayed out of forecasting https://www.cdc.gov/coronavirus/2019-ncov/science/forecasting/forecasting.html. Early in the pandemic, CDC collaborated with some researchers to set up a large US modeling group. It was built out of the flu forecasting infrastructure that CDC has been organizing for a few years, and so it had a head start. But we (IDM) quickly saw it wasn’t gonna be useful for anything (in my opinion). Two biggest reasons. 1) There is a thick glass ceiling for modeling at CDC, and the group’s principal advocate isn’t high enough in the hierarchy to have much influence on decisions. 2) In order to get many modelers to move together, they simplified the questions to the point of uselessness. Pandemic planning scenarios that took 3 months to formalize were a joke when new data about what was actually happening was being produced every day. The effort should’ve been spent on distributing modeling resources to help states instead of wasting a bunch of modelers on consensus multi-model ensemble exercises that help no one (and if the policy person thinks that’s helping them, I contend they don’t know what they need!).

My point is CDC kind of had a system for working with external modelers, but it was set up in a way that isolated all that activity from decision making. It would be a huge mistake to formalize that system. There needs to be buy-in from the decision makers at the top. Or rather, there needs to be a real pathway for information to flow up to the top. Token gestures that are poorly supported internally to public health are worse than nothing, as they consume people’s effort that could be spent elsewhere for no gain.

For attribution, please cite this work as:

Famulare (2025, Nov 7). Retrospective on the IDM COVID response (emphasis domestic). Retrieved from https://famulare.github.io/2025/11/07/Retrospective-on-the-IDM-COVID-response.html.

<
Previous Post
Teaching hypervigilance to LLMs
>
Blog Archive
Archive of all previous blog posts