Ultragenyx Pharmaceutical Inc. (RARE) Earnings Call Transcript & Summary

Good morning and good afternoon depending on where you are. Welcome to the World Orphan Drug Congress and our session entitled New World of Clinical Development for Rare Diseases. I'm thrilled to be with you today. As [ Karen] mentioned, my name is Scott Schliebner, Senior Vice President, PRA Health Sciences, and I'm joined by a fantastic panel today to discuss some of the changes ahead of us and what this new world of rare disease clinical development looks like. So thank you for joining us, and we will turn it over to our panel to introduce themselves. And Marshall, perhaps we could start with you.

Hey everybody. It's great to be back to the Orphan Drug Congress. It's been an interesting year. Good to be talking to everybody again. My name is Marshall Summar. I am the Director for the Rare Disease Institute at Children's National Hospital. I've been in rare disease field since about 1985 as a clinician, biochemical/clinical geneticist. Also done a lot of work with the National Organization for Rare Disorders and then with FDA and NIH on different rare disease topics. And I'll pass it on to my next person, Alison.

I'm Alison Skrinar with Ultragenyx Pharmaceutical. I am a developmental psychologist by training, and I have been with Ultragenyx for about 10 years, but have been in rare disease drug development for 20 years, beginning at Genzyme. And I have a group of about 12 people at Ultragenyx who specialize in clinical trial design and end point selection.

Yes, hi. My name is Dr. Aswin Chandrakantan. I'm the Chief Medical Officer at Komodo Health. I am a doctor by background. I've spent the last 15 years in my career developing health care data analytics software as well as data platforms. A number of the life sciences companies as well as patient advocacy groups that Komoda currently works with are focused on rare diseases. And we've made an incredible set of capabilities and also selection algorithms that are really helping to power a number of the trials in this space. So excited to just talk about some of the trends today and also some of the challenges as well as the progress the industry has made in the last 5 to 10 years or so. Thank you for having me. And I'll pass it on to -- sorry, go ahead.

Sorry. I'm Wendi Lau. And I have been in the pharmaceutical industry for over 30 years in various roles within the operational -- operations within clinical trial research, within clinical operations, medical writing; and then more recently was involved with implementation of new technology within Astellas and helping with additional innovative ideas that we could implement to help improve the way we conduct clinical trials. And I am very excited to be here. I have some experience in orphan drug diseases, and I look forward to having this conversation with this panel.

Fabulous. Thanks, Wendi. Thanks, everyone. I'm really excited for the different perspectives and experience you bring to our panel today. Marshall, let's get started with you, if you don't mind. The pandemic, of course, has done a lot of different things. It's been a disruptor. It's been an accelerant for a lot of things. It's also shined spotlight on disparities, inequities. And as we're still in the midst of it, yet there is light at the end of the tunnel, it's impacted so much of what we do with clinical trials and drug development. What do you see that we kind of carry forward from COVID into the future to improve clinical trials for patients?

Well gosh, it's been a heck of a year. I think everyone would admit. And I think from a lot of clinical trials standpoint, this will go down as the great year in protocol deviations since we can't bring patients in very often to do things. I think there are some lessons that we learned. We actually got to run some experiments we might not otherwise have run, about how we see patients and how we see them for clinical trials. And I'm hoping that we can carry these lessons forward. Just looking at it from a clinical standpoint, during the year last year, we had to flip almost 100% of our rare disease patient visits to remote visits, to telemedicine visits. And what we found is those are quite successful. You can get very good data. You can get very good visits with these things. And I think one of the biggest lessons we're going to learn from this is we have been bringing in the patients far too often into the physical facilities that is safe for them, whether there's COVID or not and I think also really impacting our efficiency. We saw huge drops in no-show rates because when you're calling someone at home, it's a lot harder for them to miss an appointment. We saw drops in admission rates. Without having to bring patients into health care facilities, it's safer for the patients. They're not -- if you've been to a children's hospital, you're typically going to catch the cold that's going around. By not bringing the patients into those facilities, we actually saw lower admission rates. From a clinical trial standpoint, I think that's useful because you can have less variation of patients who are sort of exposed to fewer environmental agents like viruses, things like that. I think that's really good. The other thing. I think there's a potential for some little cost savings in clinical trials. If we can reach digitally into patients' homes, electronics are a lot cheaper than driving to the hospital, the loading and unloading, taking off from work, waiting in the waiting room, everything else. Now obviously, sample collection can be challenging, but there's so many options now where we can start to collect samples in new creative ways. It doesn't have to be blood, there's a lot of different things you can do to collect samples. I think collecting a lot of our case report information, things like that can be done digitally quite safely. I'm looking forward. I actually definitely won't go and ask what one has to say about that, but I think there's a lot of these opportunities. So my fear is the rubber band will snap back, and we'll go back to the way we're doing things. And I think frankly if we do that, we will have lost an incredible opportunity to reduce the cost of clinical trials, improve it for patients. And there's one last issue too, which comes to access. Most of our patients who are coming into clinical trials, you have to open, for a disease, a lot of centers, so you can get one close to the patient. But when you're going digitally, those electrons are really fast and they're just as fast wherever you're going. So we can see patients in a broader geographic distribution, patients who might not have been able to participate before now can participate and for rare disease, you need every patient you can get. So for me, using these telemed digital tools and things like that is a huge positive coming out of this. We've just got to hang on to it.

Thanks, Marshall.

I'm turning it over to the rest of the group to comment and tell me how I'm wrong or right.

Yes, please. Does anyone else like to add to Marshall's thoughts?

Yes, Marshall, I think that one of the things that we have to recognize is from an industry perspective is despite there's this initial trepidation about having patients have more visits at home versus in the clinic because you worry about the quality of the data. And I think that from my perspective working in patient-focused drug development, you're actually getting patients in their real state. We do a lot of assessments in the clinic, and people are nervous. They've traveled. They don't feel well. They've slept in a strange bed. They're on a different schedule. They are receiving invasive procedures. And then we're asking them about their quality of life and how they're feeling. And so I think it's really emphasizing us that if we do want real-world evidence and we do want to collect data that reflects how the patient is feeling and functioning in response to an intervention, one of the best ways to get that data is to do it on a normal day. And a clinical trial visit day is not a normal day.

Great points. Thank you. When we talk to patients, we hear that travel and time is the biggest burden when it comes to participating in a clinical trial. So we've seen some positives from the pandemic, positives in terms of impacting clinical trials. Wendi, I wanted to direct the question over to you. So taking some of the learnings we've seen, trying to prevent Dr. Summar's rubber band from snapping back and to be able to continue with the progress, how do we continue to make clinical trials both simple and attractive for patients to continue to participate in?

Yes, I mean I think a lot of the opportunity there is in listening to the patients themselves to see what makes most sense for them, what matters to them. Really designing that trial from the beginning with them in mind, and having their opinion in there and finding out what the burden will be. I mean I think to the point that we just discussed, we put a lot of burden into our studies in terms of being on-site and going to the clinic and spending hours at the clinic to do your assessments. And if there are opportunities to be more flexible, at least in how those assessments are captured or where those assessments are captured, it really helps that participant to really have more of a real-world experience and evidence that would be helpful and clinically meaningful to them. I know FDA is looking at some of those opportunities as well. It's not just about that scientific end point, improving the efficacy, but really helping to make sure that the clinical benefit is there for the participants and getting their perspective when you're designing the trial. Even the investigator perspective, everybody who's going to be involved, how do we make sure that the burden is as low as possible so that we can get those participants into the study and keep them in that study.

Yes, let me comment on what Wendi said. One of the things that we've observed from a clinical standpoint is our working families using these digital tools and digital access, means we can get enrollment for families that it was too much burden to before. Single working parents can't really take the time off to come into clinic, spend hours waiting to get in, be seen, do the questionnaires, do all of those things. But early evening at home, then they can do that without risking their job. So for me, there's a little bit of a leveling effect of some of this that folks, who before really couldn't afford to participate or couldn't get the access, can actually gain access. And I think that's actually another upside of the whole thing.

And I'd just like to make one more point. I think I struggle with this myself. I would present a challenge to industry, when you have a clinical trial, to really look at all of the assessments and determine if that's really necessary to establish the safety and efficacy of our product? Or is it something that because we're trying to go so quickly in rare disease drug development is an exploratory thing, is it something that we're really trying to understand as we move forward with diseases that we really know so little about? If you don't need it to establish the safety and efficacy of the product, then maybe it's too burdensome. I mean I think if everybody goes through and looks at what's nice to have and what must have, perhaps we can cut down on not only the number of visits, but also just the burden of the study visit themselves. How much blood do you really need? How many MRIs do you need? And I think it benefits everyone to really start to be a little bit more nimble and efficient with the conduct of it.

It's a great point, Alison. I mean I just wanted to chime in on that as well. Because I think that a lot of times, we have all of our different sort of drug development functional groups around a virtual table really wanting to include all the different assessments, measurements, time points that we all really want all this data. And I think when you -- people talk about patient-focused perspectives. For me, patient focus is thinking about that end user, right? Not necessarily, let's collect everything at every time point, but what's that experience like for the patient? Is this realistic? Is it feasible? The mantra of "if you build it, they will come," I think is really in the rearview mirror, right? We need to be thinking about 2-parent working families, Marshall. Or a lot of times, we know we're enrolling entire families in the trial, whether it's a caregiver or a child or a family. There's a lot of burden there that is hard for people to endure when they're already coping with a rare disease. So the setting is set for us to really capitalize on sort of this momentum and the changes we're seeing. So Aswin, I'd love to turn it over to you and see if you could share some perspective on maybe what are some of the changes you see kind of coming in the next few years in the rare disease space?

Yes, so I think there's 4 macro trends that are top of mind for me. First is the role of data, and we'll actually talk about that in a little bit. Secondly, focus on the patient-centric trial design. And there's a lot of good ways to do it, and there's a lot of token ways to do it, and distinguishing those. The third, I think Marshall actually hit on the head, which was talking about access, which is the focus on diverse populations and addressing access issues. There's a significant portion of the population. Like, look at this outside of Chicago, I can tell you that there's probably never been a patient recruited for a clinical trial there, even though there's a ton of disease burden here. And that for us is an introspection that we need as an industry. The fourth is I think -- and this is going to come up, I think, a little bit later on in our conversation around external control arms, because I do see that very much changing how the rare disease recruitment landscape goes. So just to drill into this, the role of data for me is making sure that we're studying the journeys of patients in the past in order to inform decisions of the future. And so for me, they cluster around -- like we actually give respect to the patient when we make sure that they actually don't hit the barriers that other patients have actually hit in their disease area. And so a lot of that can be around making better decisions around feasibility in site selection, making better decisions and using data tools to drive real-time recruitment. Oftentimes, the issue itself is I can't -- the patient is stuck in the diagnostic journey for the diagnostic odyssey in rare disease. And you're trying to close the gap between effective prevalence and prevalence. And that is actually -- I see that as like something that the life science and the industry can marshal and help address and it can be addressed through data. On the trial design piece, I think decentralization is probably the biggest lever there, which is we didn't actually need as much [indiscernible] as we need and we didn't actually need as much in time. And if you want real-world evidence, then keep patients in their native real-world settings. Really, a lot of the folks that here are kind of already resonated with that theme. And the last piece for me is around the external control arms, which I believe that and long-term outcome studies, every single life sciences company that I'm working with is making huge investments in building recruitment or patient consent and data sets. Patients have a voice, and advocacy groups are super strong in rare disease. And we can partner with them in order to find better diagnostic modalities, identify patients that are stuck in the diagnostic odyssey, drive education and adoption that sometimes the trial itself is the standard of care and there's no other care or care therapeutic options. And so I see this really nice confluence of factors coming together and coalescing in the next 5 years. And a lot of these technologies are right here and right now. And it's us to actually carry that into all the work that we do daily.

Let me comment on -- Aswin, that was really neat. One thing I do want to kind of bring to the table on this discussion is when we're looking at longitudinal data collections in rare disease, these are small groups of patients, and this data is highly precious to the entire community. What I really urge, whether you're in pharmaceutical development, research or anything, design these things for long term. Don't design them for one trial, don't design them so that you get your data before you're going to do a clinical trial. Because the data that's collected is going to drive treatment that may have nothing to do with what you're doing. It's going to drive your incidence and prevalence data. And we found that the best repositories for these typically are often patient advocacy groups. There's great ways to partner with those now that are very cost effective for doing these things. But the worst thing actually happened now is a lot of data is collected. It's really high-quality data. And no one ever gets to see or use it to try and help the patients after the trial stops. That to me, that's something I actively won't -- I try to not do.

Great point. Thanks, Marshall. Wendi or Alison, would you like to -- anything you'd like to add in here?

I think that one of the things I will say is I second the importance of observational data, but I will say it's expensive to do an observational study. And so who has the money and the time and the resources to do it? The industry does. So when you find yourself in a pre-competitive setting, we all want and need observational study data to present to regulators in order to defend our study design and to gain agreement. And what we all want at the end of the day is for patients to have access to be able to make a decision about whether or not they want access to a safe and effective treatment. So again, I think industry needs to work together, support [indiscernible] group resource consortiums to collect this data and then make a commitment that we can all use it. Because in that way, if all of that observational data can be used by anybody who is actively investing in rare disease drug development, then everybody wins. Because the patient has quicker access, and regulators are able to make more informed decisions about the risk/benefit profile by understanding the burden of disease in the [indiscernible]

Completely agree with you, Al.

Great points. Collaboration, right, and so building on the collaboration theme and working together industry and patients. Alison, let's talk a little bit more about patients. And I think your role as a psychologist is really interesting in terms of the insights you bring. How do sponsors work with patients? How do they better like resolve some of the concerns that patients and families may have about clinical study?

Well, I think one of the most important things is that in the 20 years that I've been doing this, that wall is coming down. And so industry is now able to engage patients. And I think the most important thing that's coming out of patient-focused drug development is we have a unique ability to work with patients and to make them an active partner in our clinical trials. We think it's super exciting. The science is super exciting, we think. We get this mentality sometimes that people are going to be lined up out the door to try this. But gene therapy, for example, can be very scary. Intrathecal administrations with anesthesia can be very scary. Infusions that are biweekly that take 4 hours, so you're talking about at least one workday every other week plus a day in advance that to do study visits, then you're talking about something that becomes too cumbersome for patients and their families. So I think one of the most important things that we can do is before finalizing the protocol, have an advisory group. And is this something that you think that you can do? Or is this something that's not -- is I think a lot of times, the idea of participating in a clinical trial and getting being one of the first people to get it is exciting, but it's also scary and it's also burdensome. And so if these protocols are so intensive and patients tell you, "This is not something that I think that I can do," then what you're going to be doing is struggling with recruitment and retention. And in rare disease, that's the worst thing that can happen to you because we realize so much on adding access to the patients, getting enough of them and getting them to stay into a trial for the duration. So we need to design these trials in a way that reflects patients' concerns, before we go ahead and push them through and then wonder why people aren't enrolling.

Yes.

Alison, I couldn't have put it better. I have on top of that, one thing you should do with your clinical protocol is add up the number of commitment hours for that family based on that, include travel time and everything else, you'll be appalled sometimes at what you end up with that. And if you start keeping that in mind, I think that's another place where we can see -- I've had families say, "We don't want to do that because we'll be living at the clinic the full time."

Right.

"We can't do that anymore."

Right, right. I mean it's -- the amount that we bake into some of these studies, we touched earlier what's nice to have versus what's critical. I try to have my team, we focus exclusively on how do you reduce the burden of participation. How can we make it a little easier? Marshall, your point is really good. I mean a lot of times, we're not quantifying the level of burden for patients where, again, our translational group wants these end points and these data points, and we want to build everything in there. We've tried to develop a tool that -- to assess the burden of participation, to try to kind of build on this idea, Alison, of maybe a panel or a focus group upfront before studies finalized. How do we -- let's take a look at it first, so that we're not overcoming those hurdles later. Let's try to remove some of them upfront, so that we're not part of the 50% of trials that are behind in enrollment schedule right now. So there's a lot of work to be done there, there's a lot of progress to be made. And so Aswin, I wanted to turn back to you and talk a little bit about -- we talked a little bit about some of the changes you see coming in the next few years. This idea that rare diseases is a little bit of an innovation lab, necessity being the mother of invention. We have to do things differently here. How do you -- how do rare diseases encourage innovation or kind of stretch what we're doing in this space?

Yes. First, I think on the regulatory side, it's definitely made the regulators look at like what can we reasonably expect in terms of population size, in terms of power of studies in order to meet the bar? So I think that's actually, I think for us it's been very helpful because that has a lot of consequential downstream impact on the entire industry at large. And if I were to be candid, all of life sciences is becoming rare disease over time. Like 15 years ago, I was like, "Okay, I need a breast cancer patient." Now it's like, "I need a metastatic breast cancer patient that's triple receptor negative and failed 2 lines of therapy." That is starting to feel like a rare disease. And that's actually really for me the actualization of personalized medicine, which is actually quite nice. But it also puts a lot of stress on the traditional model. "Okay, I'm just going to go to MSK, and I'm going to go to MD Anderson, and I'm going to go to these big sites." And that's where the big patient pools are. But when you have a 5 or 6 clinical trials that are essentially competitive, that could be challenging in the regular large life sciences indications. In the rare disease space, that model may not even be the right model to work, which is you can't expect all the patients to funnel into these big academic centers. So decentralization is also here to stay, which I'm actually very excited around. And I think the other pieces are we can't use what I consider the greatest insult to any patient, rare disease or not, to not having learned from the journey of the past of other patients. If we know so much more about rare disease in the last decade than we've ever known about in the last 200 years. And so for me, I think the actualization of what we have and the benefit of the patient is very, very close, but not quite there. So it's going to take an active charge in terms of making decentralization very much a norm, collecting data in real-world evidence environments. It's also going to be with the recognition that all patients don't make it to the academic center. And that's okay. They're going to be out in the community. And we need to find -- we need to be able to bring care and we need to be able to bring experimental modalities to them rather than bringing them to us. And so I feel like there's almost -- I fear using this word, but there's almost a hubris of, well, orphan diseases patients know the clinical trials are sometimes the only choice in terms of averting significant morbidity or mortality. And I think that we over-index on -- everything that we do in terms of data collection is in the service of the patient and their population, but sometimes it's actually not. And so we need to make that right trade-off between everything from specificity and sensitivity when we built data algorithms. Two, what is the concierge version of taking a patient through clinical trials? And how can we meet the safety and efficacy barriers that are important for us as an industry while preserving access speed. And anytime we're adding something extra, those are potentially life-saving therapies that have taken an additional 6 months, an additional year or an additional half a decade to make their way into the hands of other patients that are not receiving those therapies. So when we look at things in terms of opportunity cost as an industry, I think we're starting to now understand what -- our choices are quite consequential. So let me pause there. I'd love to get a reaction from the group.

So Aswin, and kind of that's something we've been playing with a little bit as a concept of a more hub-and-spoke model. I think a completely decentralized model is kind of hard to do because you do need one place kind of pulling things together. But the model we currently use in rare diseases is we'll set up almost 20 or 30 [ clinical ] sites to get 20 patients. And of course, the time, cost and everything else and that can be quite some substantial. And I can give real-world examples of just that. I wonder if there's a way to start to evolve towards a model where we can keep the patient as much as possible in their traditional clinical setting? Partnering -- part of the study, do the collections that have to be done there, but then run things from a more central standpoint. I don't mean so much a CRO model, but something where the study revolves. What you all -- what does everybody think about that?

More convenient for patients in many ways. Probably I guess, I've heard from sites, we've explored models like this too. A little bit of a challenge of respond who -- that's like responsibility for the patient and maybe who's sort of watching out for safety overall when you start having diffusion of responsibility to home health care nursing and maybe a satellite clinic and a main clinic. There are some things that need to be worked out. But I think that that's -- the paradigm hasn't changed much in multiple decades, right? So I think that's not a huge quantum leap. I think we should -- we owe it to patients to be able to figure out some of those logistics, I think?

Yes.

Yes, I think that there's a huge change management component to that, and it's not just about the investigational site in that perspective. But also for some of the bigger sponsors who are maybe going into this foray is really understanding how that model would work and how, from a quality perspective and everything, that we're really making sure that there's opportunity to do that in a way that meets the needs of the sponsor, but also the regulators. And I think that it's possible. We just need to work through some of those logistics.

Yes, we have an old phrase where I grew up, which is 2 tons of fertilizer in a 1-ton truck.

Yes.

A lot of the trials when we're trying to execute them, it kind of feels like that. Like we're trying to really overstuff these things. And on the families, remember with a lot of these rare disease families, they are burdened already just from a day-to-day dealing with this. So I think we've got to -- I think if we want to improve our enrollments, our retention in these studies. And also remember that the consumers are becoming a lot more sophisticated about which studies they'll participate in. And for a lot of these studies, they won't have multiple choices, and particularly with the gene therapy models, it's not like they can pick one, try it this year and then try the other one in the next year. It's kind of sampling without replacement, Aswin, if I use that properly. I think -- but I think that's something we really have to consider. Or Alison, if you want to -- or actually all 3 can [ diss ] my statistics.

Yes. Well, I like that the term decentralized is now part of our vocabulary. If you rewind 15, 18 months ago, I feel like a lot of us were talking about new clinical trials coming on board. And trying to bring that sort of perspective of, "Let's decentralize it a little bit. Let's make this easier on families and patients, right?" And at least the conversations I was having 15 months ago were of the vein of, "Well, that's interesting, but call us back when you've done that 10 times. It seems a little new. It seems a little risky, that's different, risk-averse pharma," et cetera. Now we moved through the pandemic, we've had to adopt some of these things to just get by, right? And so we want that rubber band to not snap back. So this idea of technology being there in decentralized models and platforms being really everywhere right now, really it's a matter of which sort of model you want to choose. Also, I wanted to ask you a question about the use of that technology and some of these new paradigms, obviously, we can make it more convenient for patients. What are some of your thoughts on how to make that more patient-friendly using this decentralized model?

Well, I think the great thing is that there's a lot of technology out there that would allow us to collect data in a real-world setting. The challenging part of that is it's expensive. If you're talking about devices, who's going to maintain those devices? Who is going to provide support to patients and families when those devices don't work? And then what you do with the mass amounts of data that these devices product? So when you have a wearable, how do you know how to filter that out? And which data points are the clinical meaningful ones that you're going to present to the agency to use as evidence of the response to treatment? And so the technology is there, and the feasibility is there, but actually making it work is still a work in progress. We have to figure out how to -- we have a decentralized model, how to make sure that we are implementing something that is actually usable by patients, that is producing a reasonable amount of data and giving us data fields that allow us to say, yes, this particular intervention is clinically meaningful in a real-world setting because we're seeing changes in these parameters. So when you have a wearable that collects 200 data fields, what are the 3? Because we have issues of multiplicity. You can't look at everything. You can't present everything. But what are -- and that requires that you be proactive, start pilot testing these, figuring out what the issues are, figuring out what the sites are going to have to do, what the vendors are going to have to do, to make sure that you're getting data. Missing data is really such a challenge for us, particularly in rare disease. So it's definitely a work in progress. I think that the extensive volume of data and the clinical meaningfulness of the data are daunting. And so that's something that has to be negotiated ahead, but this is where it's going. And so I think we have to do our part to help figure out what to do with this data, how to position it and what it really means in the lives of the patients.

Alison, can I ask you a question? So we find wearables problematic in the clinical world for the reasons you just pointed out. So we actually don't use them that much. One place we're trying to figure out though is the concept that we'll be using video imagery, particularly with movement disorders, neurodevelopmental disorders and then also online instruments, in other word. Obviously, historically kind of you have a neuropsychologist or psychologist do evaluation with patients [indiscernible] in a somewhat artificial environment. How do you see those developing?

So one of the things, I think video is going to be extremely important, so I'll tackle that one first. We have 3 neurodevelopmental disorders in our pipeline, and they affect children. And those children do not -- are not able to communicate. And so if there are changes in response to an intervention, whether it be in communication or behavior or cognition or gross or fine motor, you're not going to be able to elicit that data realistically in a clinical setting. Who's going to pick up on tiny changes? It's going to be the parents and the caregivers. Where are they going to pick that up? In the home setting. So video is going to be extremely important in capturing early and qualitative changes in response to an intervention. I think one of the things that you can do is create a standardized video protocol and ask patients -- the caregivers to video tape certain activities of daily living at the beginning, and then periodically, so they're capturing the same particular activities. And then you give those videos, they're submitted through an app, to blinded readers, who are blinded to the treatment status of the patient. And that can be a really powerful end point if it's done correctly. Because it does allow you to have blinded central readers who are not aware of the study time point or the treatment status of the patient. I think that's going to be hugely important, particularly -- and even with movement disorders, it's literally a moving target, right? You're not going to have that event happen -- you're going to be lucky enough to have it to happen in the clinic where you can assess it. So that's extremely important. The ePROs as we talked about earlier, quality of life is -- the quality of your life is not good on the day that you're there for a clinic visit. But if you are looking at quality of life when patients are back in their home settings, you can use an ePRO to collect that type of data several days in a row before and after the intervention. And it can be pretty easy for patients to use and give you very valuable data. I think for one of our disease programs, the patients told us that they had pain. But they also told us that, that pain was greater in the evening after they had been weightbearing all day long. And so the best way to capture the impact of the intervention on pain was to use an ePRO and capture that data at night when that patient was actually [ feeling ]. And they told us that. And we didn't -- we weren't able to implement that. But moving forward, that's a huge lesson learned. The patient told you exactly what causes pain and when they feel that pain. But you wouldn't be able to do that in the clinic. But that's a perfect example of where an ePRO can actually give you the ability to have accurate data that you're looking for.

And a great example of how patients can influence the design and really guide you on what they're experiencing, right? So that's -- that's key, and that's an awesome example. So talking a little bit about wearables, talking a little bit about -- so there's this convenience factor of being able to collect data remotely, but you're also pointing out the volumes of data that are going to come with some of these Bluetooth-enabled devices, et cetera, that are becoming more and more pervasive. So I'm wondering if -- just to kind of pull you back into the conversation, Wendi, we talked a little earlier about natural history studies, working with advocacy groups, collaborating with industry. Can you give us a little bit of your perspective on the use of technology, and even perhaps some of these wearables in that natural history setting to sort of get a good sense of sort of natural course of disease?

Yes, I mean I think finding those opportunities to use technology to help capture the information, whether it's in part of a clinical trial or even a part of normal capturing of the participant or the patient in daily life is -- I think that's going to be helpful for us. It's not just about implementing technology for a clinical trial. But I think to Marshall's point earlier, how do we get that technology, if it's going to be helpful, if it's providing useful end points, if it's a way to measure and find value in how the participant is feeling or the patient is feeling over time or how they're doing, how do we get that into normal daily practice within the medical institutions and by the physicians themselves or health care providers? I think if there are opportunities for us to do that and really help prevent issues that may be cropping up and identify those signals that would lead to those issues, or find ways to monitor and manage their -- their symptoms or what's going on, I think that is something for the future that would be helpful to all of us and really help start documenting some of that, not just what's happening in the clinical trial, but the natural history of things. And videos are great for that. AI shouldn't necessarily be used in the context of clinical trials, how do we do that with normal health care. And my -- thinking through some of the things that you guys are talking about, my dream would be that it's not just readers, people watching the videos, but how do we use AI to start looking at those videos. If we can [indiscernible] errors with AI, why can't we use AI to help document and manage transition or change in people at how they are videoed and what their movements are or their actions are over time? There's got to be an algorithm that are out there that we can use to help document some of that in a more objective manner.

I'll just jump in here. I very much agree with Wendi. I feel like we know more about online shoe line than we know about health care. And I think it's specifically pointed, especially for the rare disease group, where it's like you really need to know as much as you possibly can. And so one of the things that I oftentimes think about is pyramid, which is there's data, there's information, there's knowledge, and there's insights. And so when we talk about data, we're actually talking about many, many different stacks and applications and domain-specific expertise. They are used as lenses to actually understand and digest an action against the data. And so oftentimes, I think it [ adds ] much to what Alison was saying, Fitbit is generating hundreds of data points every single second. That's just data. And then to manage it, put it into a structure, becomes information, you apply domain expertise to it. And now you're like, "Okay, now I actually understand specific movement patterns in, say, a Parkinson's patient over a specific period of time or a patient that has a mental health illness." And then you get to the inside, which is like, "Okay, well, now I know what to do differently based on the -- based on what I've collected." And so I think for us, we actually use the term data to actually mean many, many, many different things. And I would also say that placing the burden on life sciences industry or folks who are not deeply embedded in data science is actually a misuse of the data. And it's also a misuse of the person's time on the other side, trying to actually understand it. This is where technologies and processes, where you ask intelligent questions from a domain perspective, and then the machine takes data, makes it information. The domain expert makes information and then submits it into knowledge. And then artificial intelligence can say, "Okay, what do you do differently? Where -- is there a patient that is potentially -- that has a rare disease within the specific catchment area." And so I think it's helpful to think about what is the role of people versus machines, and are we actualizing the potential of what we call data out there.

Great points. Thanks for adding that, Aswin. I'm trying to keep an eye on our time here. We have a few minutes left. There's one important topic that I think would be fun to talk about building upon data, building upon patient-focused drug development. And that is the use of control arms in studies, right? So this is a fun point of debate and a challenge, I think, for all of us in the rare disease space, right? So FDA wants control-arm patients. We don't want patients on a placebo, certainly. There's real-world evidence of synthetic control-arm options that are starting to get utilized. It's a really interesting point here. Marshall, could you share a little bit of your perspective on this topic?

I know you were going to throw that one to me. Here's what I see from the patients I work with and then also from the trials we're running, particularly with some of our ultrarare diseases, these patients all talk to each other. They look to gain in the system, so that they have -- they minimize their chances of being in a control arm, whether themselves or their child. And as a parent, you can really kind of understand why they would want to do that. And we see patients not enrolling in trial, waiting for either compassionate use or some other aspect of the trial where they can be assured that they know what they're going to get. So there's a little bit of this. The patients are aware of this. I think the agency is where -- I know a lot of the docs and some of them I had the pleasure of working with professionally over the years in the orphan products [indiscernible] at FDA, and they get this too. The one piece of advice I would say for folks entering into this is go talk to the agency before you even start designing your clinical trial. Visit early, visit often. And then also one of the points I think everyone in this group has made is that having a good broad set of data about this disease that is ongoing, longitudinal in nature with really good depth to it, that has almost a more generic approach to the entire disease is always going to serve you well. Because if you can use that to develop your synthetic control arm from the natural history of the disease by actually knowing it, I think that makes it much more likely you're going to be able to look at doing control those clinical trials. One of the things we ran into with Dr. Gottlieb when he was FDA Director and we were working on some of our NORD natural history studies is he said, "I want to control this clinical trials in rare disease." I think -- I won't speak for Janet Woodcock, but I Janet probably wants the -- I think we all want the same thing. But at the same time, you will also run into elements at the FDA, that cycle, the gold standard is a double-blind placebo-controlled trial. That's why I think you need to go talk to them before you put into paper on what your clinical protocol is going to look like.

I couldn't agree more. I mean what we owe ourselves and the regulators and the patients and the providers, and frankly, the payers, is an adequate and well-controlled study. But what does that mean? And does it have to be the standard randomized double-blind, placebo-controlled trial with a p-value of less than 0.05, when we know that, that may not be feasible? And FDA does have a guidance document, and it does provide alternatives to that. And I think if we start talking to patients, to potential sites and realize what some of the challenges are going to be, we can propose some of those alternative designs and work with the agency to get agreement before the trial has started. But I think that especially for the ultra -- ultra-rare, how would you ever do a randomized double-blind placebo-controlled trial with 12 patients? So there's -- we have to start getting creative for ourselves, for the regulators and for the patients that we can actually give everybody what they're looking for, which is an adequate and well-designed study.

And remember too, as part of -- I believe, that it was [ 21st century or orphan ], they're supposed to have patient representatives in the evaluation committees at FDA. That's actually in the law, in statute right now. It doesn't happen as often as it showed, but I think it does have real value.

Great points. Thank you all for your contributions. Well, I think we are wrapping up our session here. Thanks for sharing ideas on kind of a volatile, debatable topic here around synthetic control arms and how do we go forward with that. But clearly in this kind of new world of clinical development that we're in, we've seen a lot of, again, disruptions. We've seen a lot of accelerants. We've seen a lot of adoption of new technologies and really kind of the emphasis on patients. You all had some great perspectives on how we capitalize on this great progress. But at the same time, there's some challenges with data collection, some challenges with involving patients, there's some challenges with selecting end points and deciding how much or little data to be -- to be collected. So really, I think interesting times ahead, and I appreciate everyone's thoughts and contributions today. Thank you for joining our panel. Thank you to our attendees for joining today's session. And we will look to the Q&A section of the chat, and see if we can address some of your questions next. So thank you very much.