Backblaze, Inc. (BLZE) Earnings Call Transcript & Summary

Hi, everyone. Welcome back to Drive Stats. Today, it's our special edition once a year where we get to talk about the full year of 2025. So as a reminder, I'm Stephanie Doyle. I'm a technical storyteller here at Backblaze, and I get to carry drive stats forward with my partner in crime over here, Pat?

Yes. I'm Pat Patterson, Chief Technical Evangelist. And yes, I run the queries and wrangle the databases. And yes, Stephanie and I work as a team producing all of this data every quarter. So before we get started, just some quick housekeeping. The webinar is being recorded. So if you need to drop off for whatever reason, you can come back, pick it up tomorrow after the recording is posted online. Please do ask questions at any time during the webinar. [Operator Instructions] And please don't miss the attachment. So we've attached what we got PDFs, I can look over here, actually. We've got the year-end report link, the archive link to subscribe to newsletter and the drive stats home base, where you can find statistics going back for 13 years.

13 years?

Yes. So what is Drive stats? If you're joining us for the first time, you might be thinking, what are they talking about? Well, since 2013, we've collected, curated and published raw device metrics. So what we're doing is for every drive that we are running in production, we are collecting every day, the drive serial number, model ID, the smart attributes and all important, whether the drive failed that day. And that's what lets us put together these annualized failure rates and other statistics to present here in our report. So every quarter, we do this, but this is our annual report. So you get a little bit more data, we look at 2025, the whole year as well as just the quarter snapshot. So where are we? We're currently spinning over 330,000 hard drives. So in the quarter, we saw nearly 1,000 drive failures. And if you can do mental math, you can get to the number that's down on the bottom right there. We saw an annualized failure rate of 1.13% in the quarter. And that was -- we love this, drive days. Drive Day is a drive running for a day. So 1,000 drives run for a day is 1,000 drive days, 1 drive runs for 1,000 days is 1,000 drive days. So that's how we measure the volume of statistics we're collecting. And you can see there that the drive population, it's actually quite interesting because we've got Seagate and Toshiba with about 1/3 each. And then if you look at HGST, that was Hitachi Global Storage Technologies. That's like a legacy brand that was absorbed into Western Digital. That actually makes up another 1/3. So really, this drive population is pretty much split in 1/3 between Seagate, Toshiba and Western Digital, if you count HGST. And we also have the annual stats and the lifetime stats we'll be digging into, and you'll see those annual failure rates are a little bit different. Now Steph, this is just because we're looking at different time periods, right? The quarterly one is -- bounces around a bit more.

Yes, yes. Each time you -- when you think about creating boundaries around these things, quarterly, annual and lifetime, it's not just that they ran for a quarter. It's also that we have different exclusions set up. So your quarterly, that's 250 drives minimum to make it on to the chart as well as a certain amount of drive days. Annual, it's 500 drives. Am I -- actually, I think annual is -- we've got the exclusions listed on each. I do this every time. But the point is that we try and give you more confidence with each additional interval because if you create a higher minimum standard, then what we actually see is less volatility in those AFRs, which is something we've visualized in this presentation throughout. So we'll be able to talk a little bit more about that when we get into it.

And naturally, there's more drive days incorporated in the data. So we're spreading that -- information is more kind of spread out.

Absolutely.

So Steph, take us into the numbers. What are we looking at here?

Yes. So this is the quarterly data. And like we said, this is your lowest bar to reach on to a chart that we track here. So you'll see that this quarter, we had an incredibly low annualized failure rate that was 1.13%. That's the lowest I've seen in well over a year. But it does represent a pretty solid fluctuation. If you look at the quarterly AFRs for this year, we've bounced around a little bit. So 1.42%, 1.36%. Last quarter, we were pretty high, and this quarter, we're pretty low. We've talked previously in other reports about how we define a failure and what that means from a data engineering perspective. But the important part here is that every time we're talking about quarterly failure, actually any kind of failure rate, we're talking about real failures. So if you go back and you look at some of our older drives up here, anything that's an age in months of 100 months or 90 months, some of those are actually being filtered out of the drive population through our normal migration process without ever failing. So we only classify a true failure as a failure, and that means that the drive has stopped working for whatever reason. And we end up with some really cool highlights because of that. So first of all, we've got our first 26-terabyte drive, always really cool to see those high-capacity drives coming on. There's implications for the drive fleet and what that means for parity, but that's a conversation for another day. But as we've seen over the years, drives have gotten bigger and the cost per terabyte has gone down. So that certainly is something you're seeing in buying trends for our data centers as well. And then we've got the sort of the Honor Roll as we're calling it this quarter, where you see 0s and 1s, so no more than 1 failure. And I'm going to kick it over to Pat to talk about whether this matters or not.

It's kind of a bit of fun. Actually, if we go -- you're there already. So yes, it's kind of a bit of fun. So if we look 0 failures by definition means you have 0 annualized failure rate. But if we look at the drive counts here, for some of these, they are very low. I mean, 187 drives, 247 drives, that's like almost nothing compared to some of our hard drives in operation. In fact, if I go back a couple of slides, we can see we've got 40,000 of the 16-terabyte Toshibas here in -- towards the bottom of the table. So really, these are a bit of fun, but it lets us call these out. And in some cases, like the Seagate 12-terabytes and the Western Digital 26-terabytes, we're getting towards 100,000 drive days, over 1,000 drives in operation. So we're starting to see some significance there. 1 failure out of 1,200 drives. We're hopeful that, that drive will continue to perform, but we're literally in the first quarter of operation there. You see we've -- 1,200 drives is a vault. So if you're wondering what I'm talking about, our storage is split into storage servers, which we used to call pods. I'm not sure they're officially pods anymore, but each pod has 60 drives and 20 pods chain together in what we call a vault. So that's our deployment unit. So we've deployed 1 vault of these 26-terabyte drives. And we've seen 1 drive failed out of that vault. So that's pretty encouraging news, a 0.4 annualized failure rate is pretty good.

Yes. I think we both -- when we saw that 1,201, we were really excited to contextualize the stat because when we're talking about normal migration process, you try to deploy a full vault of drives at once. So we were like, oh, look at that, you can see very cleanly 1,200 full vault, 1 failure, 1,201.

Yes. It's always quite gratifying when the numbers fall out like that. Over time, we replace drives as they fail. It's not always possible to replace them with the same exact model. So you do get into this situation where there's like 465 dives of this model and 112 drives of that model, these 16s they're only different in an ultimate character of the model ID. So those are mixed up together in a vault. But it's really nice when you see, okay, we deployed 1,201 and 1 failed, so we still got 1,200.

Moving forward. So what I always like to do is we talked about our Honor Roll, so to speak. But there's also -- any time you see a significantly high failure rate, we want to talk about that, too. So what you see when you're looking at this HGST, which I do every time, this 8-terabyte drive, we saw a 10.29% failure rate. That's pretty high. And you can see from the historic numbers like this actually isn't a vault that's performing badly traditionally. So that says that something happened. And now, of course, any time I get into investigation, I'm actually looking back at the quarter before. So somebody has already done this. Somebody saw that there was an issue and looked into it. And when I went ahead and checked it out, what happened with this drive was it was a little bit hard to identify. There weren't any issues with temperature we could see. There was a potential that maybe because this is all actually 1 vault of drive, so there's something going on with the chassis or vibration. But frankly, with the age of the drive, it's over 7.5 years old. We just said, all right, this drive is telling us it's time to rotate out. So instead of spending a lot of time to really get down to what happened or even doing something like replacing the chassis, we just said time to migrate. So we'll see that one come out of the population as well. And the other interesting one was this Toshiba 16-terabyte drive, which if anybody was here last quarter, we actually had a huge spike in that drive here, the 16.95% and had done some investigation and saw that there was some very normal updating going on in one of our data centers that read as a failure. And at that time, I said, this is going to come back down as we see this work settle out. So we always like to report back in. It's still a little high, 4.14% is a little high, but I wouldn't call it an outlier, although I didn't run a quartile analysis this time around. But it's definitely coming back down. We're probably seeing the tail end of that work and then it will be coming [indiscernible].

Exactly. Yes, I guess the work spilled over into the beginning of the quarter, and that's what we're just seeing as it settles back down.

Yes, absolutely. All right. So moving on, let's talk annual data. Pat, do you want to take it away from here?

Yes. So with the annual data, we're looking at essentially the same set of drives, but spread over 12 months instead of 3. So what you'll see is a little bit of variance in the numbers, obviously, around 4x as many drive days, around 4x as many failures. I don't think, it's -- I don't know if we've ever had a 0-failure drive model across an entire year.

I'm not sure. We certainly didn't this year, but that would be a fun query to run. We should check that out.

Right. And what we -- again, like very small drive numbers here, we see 1 failure. We see this -- since this 26-terabyte drive was deployed in Q4, this row was exactly the same. That's the quarter snapshot. But all of this is across the whole year. And yes, we see much more -- much less variance in the AFRs between drive, it's kind of smoothing out over time. So I think the highest there is just about 6% for this one and that's the drive that we were just talking about. I think that's the one with that elevated rate. So yes, I mean, that's the annual data, not a lot to pick out, I think, in the table. But we do have drives that averaged less than 2 failures a quarter. So this HGST, 4 terabytes, some of these Seagates. So those are our -- those are the ones that we like to look at and say, hey, that's great. These drive models are where we're seeing the least trouble.

Yes. But it is also just like we're talking about how sometimes the numbers can be a little deceiving. It's worth saying that it's not evenly distributed. So I believe that top model there, the HGST, the 4 terabyte, they had like three failures 1 quarter and then the rest of them 0, 0 and 1 or something like that. So the history of each drive, I think it's something that we bang this drum quite a bit where we're saying that models have a lot of model to model, you really can't compare because they have a lot of like very internal variance and sometimes it's hard to tell what actually is creating a failure. And so that's, I think, the most fun part when we slice the data differently with different exclusions or look at things like this because you end up always coming back to the fact that the more precise you can be with the line item data, the more you really understand a single model.

Right. And really one character difference in the model ID, it can be -- it can behave as a completely different drive. There may be firmware changes. There may be -- may even be like, I don't know, a different motor or something like that as they went from one model ID to another. So it's even hard to generalize across same capacity, same manufacturer and extensively the same drive. When you go buy it, it just says BarraCuda 16 terabyte or whatever. But it's -- really, you've got to look at that model ID to get the whole picture.

Absolutely. And I think what's fun about the annual data, too, is that we actually compared this too. We wrote another article at the end of the year last year that was based on some internal metrics about where our data center techs were spending work. So we were able to sort of compare not just like this, you can see the total time spent where people were working for data center, this is in hours. And one note there, Canada, we actually deployed that in March. So it's looking like it has low hourly metrics. But in fact, we actually tracked the launch a little bit separately from our normal work center activities. So Canada is -- we love all our data centers, but I think that Canada is getting a fair read here or an unfair read here. But kind of no surprises here. U.S. West is our largest footprint, right? And then you've got a lot of ongoing work. You can see where things were happening. And then comparing that to what the most replaced drive was by capacity, I thought was very interesting. And you'll note that in this data, you're going to see some drive sizes that don't show up on those annual and quarterly, and lifetime charts. And that's because this is inclusive of all of the drives in our data centers. So this would include boot drives and SSDs and all that other stuff and drives or there's too few of them to make it into the quarterly data yet. So if you see that, that is why. But being that we are interested in who's doing what, you can see that this sort of makes our 12 terabyte and 14 terabyte and 16 terabyte drives look bad. And that's not really fair because these are raw numbers, and we know that we have a solid concentration just by quantity of those drives, but also that they're aging. So they look like they're doing bad, but in fact, it's probably the fact that we just, by the numbers, have more of them in our drive fleet.

And I'm going to say, if we just go back for one second. I think -- I mean, I'm going to guess. I haven't analyzed it specifically, but I'm going to guess that a lot of these replacements -- these replacements because of failure or just replacements?

Just replacements. So this could be normal migration activity we're seeing here.

Exactly. Yes, that's what I was going to say on the 8-terabyte drives, that is what we're seeing there is likely the migration just as vaults age, and we do these planned migrations, what we call CVT - Cluster, Vault, Tome. I think we've got a whole article about that.

We do. Great article.

But we migrate your data from maybe, whatever, 5-, 6-year old vault, whatever, onto a new vault. And that frees up rack space. We migrate from an 8, say, to a -- let's make the math easy. We migrate from an 8 to a 24 and we can have that same mass data in 1/3 of the footprint in the data center. So that's one of the motivations as well as drives aging and not wanting to get into like huge numbers of failing drives. It's just more space-effective to have the data on higher capacity drives.

Yes. Storage density, definitely something we're interested in. And I think that when we move forward, looking at what our drive capacity is versus the population, it's funny because this sort of shakes out almost like a bell curve, right? So we did a number of drives here where it's 26.1%, 54% or 51% or so on the 14 terabytes to 16 terabytes and then 20 terabytes. So validating our earlier claim that like by the numbers, you just have more drives in the 14 terabytes to 16 terabytes, that's definitely sure. But the other thing I think that's interesting here is if you think about the age of the drives and compare the age of the drives by the percentage, you're going to see different -- it may look like a bell curve in terms of the numbers, but we know that we're shifting to those higher capacity drives because the 20 terabytes are younger. And the other thing here, too, is speaking to your point about storage density, Pat, is that 22% of our drive fleet is 20 terabyte plus drives. That represents a lot more capacity than the 26% of 0 to 12 terabyte drives. So just some interesting notes there when you think about how we're presenting this data to you. We're giving you a raw number of drives, but there's other things to think about as well.

Yes. There's quite a lot of nuance in the numbers.

Certainly. And then this is a comparison of our last 3 years of annual data. I always love this as a top-level view. Pat, do you want to take us through this?

Yes, absolutely. So what you're seeing here is like from the highest level view, our annual failure rate has been dropping quite nicely. So 2023, 1.70% and then to 1.57% in the next year. And then for 2025, 1.36%, which is great news. We're having to swap out many fewer failed drives. And you're seeing -- this is an interesting view as well as you see these new drives pop up. So you see this 26 terabyte drive there, the 24 terabyte coming in last year. So what is that 4,000 sets, 4 volts of the Toshiba 24. And then interestingly, you've got -- well, we've got Seagate 24 terabyte as well, but you've got lower capacity new drives coming in. And this is just -- okay, why are we buying a new model of Toshiba 16 terabyte. Well, the older generation just ages out of the inventory. And we buy the closest to swap in, which is the '09 Variant rather than the '08. And that's why you see these new drives joining the fleet. So a couple of those, yes, just if you're interested, why is this showing up? Well, that's the Seagate 14 and Toshiba 16 that we're just buying the current drive model.

Yes, which I think we haven't done this, but I'm actually now thinking that would be an interesting comparative analysis for that model-to-model stuff you're talking about because what we're saying there is basically they're the next generation of the same drive. So if anybody wants to -- I see this come up in the comments section all the time, like I can't find this one with that one character difference or how are these two 16 terabytes distinct? And when you see something like this, that might be a good way for you as a person buying drives to sort of compare drive model over time with those slight differences and see what works for you because obviously, a data center use case, very different than a consumer use case, which will…

Very, very. We spin our drives 24/7.

They never stopped working. All right. And I think one of the cool things just reflecting back on Pat's point about how we've been seeing the drive or the AFR change over time as this is the 4 years. So it's a little bit of a cheat because I gave you 3 years on the first chart previous, but this is 2022 to 2025, and we can see really that we're returning to this 1.37% rate. What does that mean for the quality of the hard drives, probably not much, right? Like we're just seeing same drives performing differently over time. And I like to use these moments to always reflect that like when we talk about these numbers, just like I said, we can remove drives without failure. What we have here is a little bit of a curated number in some ways, right? Like low failure rates in our data centers are reflection of all the work that those data center techs are doing on an ongoing basis and that our storage engineering team is doing on an ongoing basis to make sure that we have a healthy drive population.

And it's impressive as well that the annualized failure rate reduced despite those couple of drives where we kind of saw incidents that raised their failure rate temporarily. Even with that in the numbers, -- it's come down from the previous year. So that's quite gratifying.

Yes. So shout out to the folks doing the work. We love it.

[indiscernible]. We love you.

So we've got about 5 minutes here. So we'll go through the lifetime data somewhat quickly, but we don't have to be superfast about it. But I do see there's a question or two, so we want to get to that if we can. So lifetime data, 1.30% is our failure rate here. And just like with the other tables, you'll see maybe fewer models here, and that's just because they haven't reached the minimum standards to be on this table yet.

To explain what we mean here, this one is a little bit different in that rather than being like a quarter or a year, what we're saying is for these drive models that we're currently using, if we go back over the lifetime of all of those drives like within those models, what annual failure rate do we see? So this is like trying to really bring together the most data. And you'll see some of these like with 80 months or 90 months or even 100 months here that we've -- that's the count of like all of those drives that we've ever deployed.

Yes. Yes. If I were to make an analogy, I feel like this is the -- it goes down on your permanent record number.

So yes, lifetime AFR is 1.30%. It doesn't tend to change much because it is such a large data set. In fact, let me just go back. 557 million drive days. It's pretty significant data. And we got a new entry, that 24-terabyte. We deployed another 2 vaults bringing us to like 4 vaults total, as I mentioned, 4,800 drives and some change, some failed drives there. So we see new drives coming into this lifetime table every few quarters.

Yes. And I think it's also worth calling out that like we're talking about how volatile the AFRs can be depending on how we shape the numbers, right? The lifetime AFR has been extraordinarily consistent. I think the last 4 quarters, we've basically gone up or down by 0.1%, which speaks to the fact that you -- when you collect enough data, you can really sort of find the sweet spot in the numbers.

Yes. Absolutely. We've got a bunch of links. So as I mentioned, there are attachments below the screen there, below the video output. So the year-end report, you can see all the blog posts in the series, join our newsletter and visit our home base. If you want to contact us, you can e-mail us at [email protected]. We're on the socials, just get in touch with Backblaze and the comments section on the blog post, we love to see your comments on the blog. So with that, we do have -- we do have a couple of questions. So Abraham is a student sharing his information with this class. Awesome. Since we're covering RAID and -- okay, so Abraham's class is covering RAID and its level, how does the RAID level used affect its statistics? Does it affect them at all? That's a really good question. Now RAID, we're not actually using RAID in our data centers. We use an algorithm called Reed-Solomon erasure coding, where we split the incoming data into so many chunks and add a number of parity chunks that is variable. And so we split, say, 15 data chunks, 5 parity into 20. So the number that we're seeing there can be -- is what we're reporting on here. But RAID tends to be smaller numbers of drives linked together. And I don't know, would you see any difference, I don't think so. I mean I think it's more in the usage patterns, the volume of data being going back and forth from the drives maybe that's moving the drive heads across the data rather than the RAID level itself.

And I went ahead and linked one of Pat's articles that he [indiscernible] previously about NAS -- specific to NAS, but RAID levels, right? Because I think it does a really great job with the visuals in that article. But one thing about RAID drives and specific to NAS is that you're really just talking about different redundant storage patterns, right? So when you think about creating a full backup or what it would mean to restore, this is really giving you a logical pattern to how to store your data. And to Pat's point, we use Reed-Solomon at a code level for that. But if you're talking about applying hardware failure rates to what you would want with a RAID configuration. Really, you're just talking about how many drives do you need to achieve, whatever rate configuration you would want and how much redundancy do you need to build in for the 1 drive failing. So you can always add another drive, as I say, and create a different configuration if you want more confidence.

And one last question. Well, it's more of a suggestion than a question. When we report on those 0 failure drives, we should order them by the number of drives deployed because then you see like the real stars, if there's 1,000 drives, 2,000 drives with no failures, that's kind of more significant, more interesting than 200 drives. So yes, I think we'll probably do that and start reporting on those drive numbers as well as the 0 just so you've got a little bit more context there.

Yes, I totally agree.

With that, I think we have…

I think we've got one more in here from [indiscernible], and I know we're about a minute over. So if anyone needs to drop, feel free. But just to answer this question here, how do we achieve which mechanism and means that we do to replace or migrate the data from the drive when it crushes without -- I'm assuming that crashes without the user noticing that something is wrong. Well, the first thing is we're a heavily surveilled environment. But I think that, that gets into actually to Jose's question about redundancy, right? We compared with what a consumer does or needs, you're usually talking about having 1 drive that may or may not fail. Maybe you have a few in connection. But within our data centers, we have data so that we can rebuild a file from as few as 1 parity shard, which -- and we store that 20 times across the data center in different configurations, which is that Reed-Solomon erasure code we're talking about. So if you want to check all those things out, I linked to those resources above as well. And the other thing is that what we mentioned earlier about the CVT framework we use to be able to migrate data effectively, be proactive about it, it's a really interesting process to look into.

Yes. And also, I think we've talked before about what makes a failure. And we'll watch statistics like -- so those smart stats I mentioned. We'll watch statistics like unrecoverable errors. And if those start to show a pattern, we don't wait for the drive to actually crash and then say, I can't give you the data. We swap it out before then so that we get ahead of that event. And often, we can clone the drive, which is a lot faster than rebuilding it from the erasure coding and get it back into -- get that data back up to its reliability level much faster.

Absolutely. So thanks for the questions, everybody. We're going to go ahead and jump off since we're a little bit over. But we always appreciate the interaction. We always appreciate you showing up, and I'll see you next time.

All right. Bye for now.