XZ Utils Backdoor Explained- How to Mitigate Risks | Episode #7

Hey, everybody.

Welcome to another episode of John Has Trust Issues where I discuss issues related to Zero Trust and authorization in a few minutes.

My name is John Martinez and I'm the technical evangelist over at StrongDM, and my trust issues stem from that robotic vacuum that's just going all over the house, but always seems to chase me around the house trying to eat my toes or something.

I know the dog and the cat really hate that vacuum.

Anyway, so let's get onto today's subject.

And I'm talking about the XZ util backdoor that was discovered in one of the compression in the, in the XZ compression Library on Linux.

And oh my goodness, uh, where do I start with this one?

So much happening with this event, with this incident.

First, it, it's really all about trust and really, you know, that that's a huge part of the reason why I bring it up, in my John has trust issues episode here is because there's such a big human element to this incident, this particular back door.

So, first of all, just a little bit of background.

The XZ project, XZ util project, or lib, LZMA, is so critical to that it's actually linked, to System D, which for us, Linux old Unix gray beards is in it basically, I mean, it's the core of the operating system and it really only had one maintainer.

I mean, that, that's what's so amazing about this.

The Linux Gods decided that, you know, it, having one maintainer for such a critical but uncelebrated part of the operating system needed more than one contributor to it.

So, somebody decided to step up and become also a contributor to this, but get this, this contributor was gaining the trust of the project for two years, and they even contributed fixes.

Now, that's the long game. We don't know whether this, uh, this contributor was a nation state actor.

There's some suspicion around that, but we do know that they were gaining the trust up to the point where they eventually brought in the back door.

It wasn't from what we read, what I've read so far, it wasn't actually part of the, the core part of the library code itself, but it was part of CMake,and it just built, and really at the end of the day, this was so insidious and so pervasive and so under the view of a lot of people that were actually bringing this library into, uh, the distributions, the Linux distributions.

Now, what happened is that a Postgres developer over at Microsoft named Andress Fre, and I'm, I'm sure I’m butchering the name.

So, uh, apologies to Andres about that.

But Andres happened to stumble across an issue where SSH was just a little bit slow.

We're talking, you know, a little bit less than 600 milliseconds, slow for an SSH session.

And that's when Andres started digging into the issue, starting to try to figure out why is my SSH session, uh, so slow?

And so digging, and digging, digging.

Now remember, Andres is a Postgres developer, so not a security person per se, but eventually was able to follow the trail to where the compression library, was linked to system D, and in some distribution, system D is linked to an open SSH server.

And so it went right.

And so at the end of the day, the back door allowed, remote execution, remote access to SSH servers.

And that really is where the crux of the technical part of the issue is.

But really, you know, at the end of the day, we're talking about the human element.

Just again, I'll use this word again, the insidious insidiousness of this particular method of getting in, being around for two years, contributing valid code, valid fixes, to the project.

I mean, it, it was just really such a crazy operation.

I think the saving grace though here was that it didn't make it into any of the actual release Linux distributions.

We were, we were talking about only pre-release or version next parts of the distribution.

And we do know that Fedora, Debian, arch Linux, et cetera, and eventually the government, and I'm talking about CSA here, they issued critical alerts on the 29th, shortly after Andres discovered the back door.

But wow, I mean, the, this really is that crazy.

So, um, a couple of the, a couple of the articles we'll link to in the comments and the notes here for the video, but really at the end of the day, this could have been a bad one.

The internet really could have melted down.

I mean, I think a lot of us in this Linux and security world really are saying, sounding the alarm bells.

And, you know, there's a much bigger topic here that we can discuss, that we can have discussions around.

But definitely it all points back to the human element of this things like, you know, more than one maintainer to a Linux project, a core Linux project, more than one maintainer to any, open source project, really at the end of the day.

And at the end, also, part of this is an element of are we asking as corporations, are we asking as an industry as a whole, asking too much from our open source developers?

Which by the way, most of 'em are volunteers, right?

Are we asking too much of our volunteers where we're demanding?

And really at the end of the week, we gotta rethink this because Linux is so key and so core to a lot of different things are in our industry, but really we dodge the bullet.

And, and just, just as an aside, I really love what some of the distributions are calling their patches to this.

CIA called for a rollback, in the CVE it also calls for a rollback as well.

But I really like what we've done here where we're patching pre unknown state, where the backdoor was at, that we think, that's version five four or 5.1.

I just saw one of the, one of the d one of the patch links that said that the new, the new name of the patch or the roll forward, which is really a rollback is 5 6 1 plus really 5.4, 0.5 0.1.

It really is a rollback.

So we dodged the compression apocalypse.

It could have happened. How do I mitigate, you know, and you're thinking, you know, in practically in my environment, how do I mitigate something like this from happening again?

The key here is open SSH server.

So what does that look like?

What does a mitigation for this particular issue look like?

And let's go over a couple of things here.

You know, number one, goes without saying update your Linux boxes like now, especially if you have Linux boxes that are using up and coming releases.

You know, some of us do like to live on the bleeding edge, you know, definitely this is one area where if you are running a future version or a pre-release version of a Linux distribution, definitely patch, next move away from SSH Bastion hosts.

And, and, you know, this is being, being at my job here at Strong dm, we are huge believers of this, where, you know, you don't need Bastion hosts anymore.

There's so many issues that happen with Bastion hosts, namely Port 22 open to the world.

If that rings a bell, uh, you'll know that, you know,this particular back door could have gotten you if you did have, a Port 22 bastion host open to the world for sure.

Finally, don't expose your credentials.

I'm not talking just about usernames and, and passwords.

I'm talking about private keys.

172

00:08:33.585 --> 00:08:36.325

Um, uh, if someone were to, to pop your, your SSH server without an invite, they, they happen to exercise this particular back door.

I mean, they would've been snooping around for other things.

And Bastion host definitely is right for that.

Even if you encrypt the encrypt the, the actual credentials, et cetera, it would've been a bad problem for you.

So really, those three things are, are a good start.

I'm gonna link in, in the video as well, an article, a blog that I wrote about the backdoor.

So definitely read that as well.

Read a lot of the other links, including the email thread that happened between Andres and some others.

A very interesting reading for you to educate yourself on it and figure out a little bit more of what really this XU utils lib, L-X-M-L-A, lm NOP compression Library backdoor is all about.

And just really quickly, a production note here for me to talk to you about.

No penguins were injured in the making of this episode, just so you know.

And though that's it, there's another episode of John Has Trust Issues.

This episode was brought to you by StrongDM, the modern access management platform that enables continuous zero trust authorization for all of your infrastructure.

Thanks for watching. Have a great day.