Espressif hired a bunch of Rust devs years ago and they have been quietly doing great work supporting Rust on their chips ever since. Still haven’t seen any other chip companies get close.

https://developer.espressif.com/blog/2025/10/esp-hal-1/

lol no you don't #privacy #cookies

Watching all of the random things that people have been saying about AWS's #outage yesterday reminded me of a discussion that I had with one of my teams while working at Google.

We were doing longer-term planning, and people were proposing multi-year goals. One (moderately senior) teammate wanted a goal of roughly "MTTR decreases X% per quarter".

So, that sounds nice in theory but it's not how mature services really work. As you fix the easy bugs, you get fewer and fewer trivial outages. "Admin typed dumb thing" mostly goes away with better checking and deployment policies. "Partial backend failure caused cascading failure" is mostly handled by avoiding patterns that cause cascading failures, and then dealing with partial failures as best as they can be handled. "Trivially bad software release broke things" gets handled by improved testing and canarying over time.

Unfortunately, once you get rid of the easy outages, you're left with *weird* stuff. I/O patterns that trigger latent firmware bugs in SSDs, causing accelerated failure fleet-wide, with a multi-year lead time on replacements. Datacenter fires. Natural disasters. CPU bugs. ROMs that get overwritten by excessive reading. Software bugs that cut across 4 or more services and somehow manage to find decade-old fatal flaws. Overloading some resource *that no one knew existed* (per-second-level domain HTTP cookie jar size, undocumented stateless router hardware state limits). Or (one of my favorites) BGP stops converging correctly because several racks were too heavy and their plastic wheels had cracked (yes, really!).

The sorts of things that you *can't* fix fast, because no one even has a good model for what is happening, and none of the usual quick fixes (roll back, drain, loadshed, etc) are helpful.

In this specific service's case, we weren't *quite* to the maturity level where I expected MTTR to start rising, but we were getting close. And, frankly, we didn't track MTTR very closely anyway.

Reading takes on AWS's outage like "when we had our own datacenters, we never had long outages without any ETA for recovery" mostly just means that you never had any of the really *fun* problems.

Remember, the reward for a job well-done is a new, harder job.

@me One of my first Delphi apps was a fake "system is locked" window with a password prompt -- to steal a BIOS password or something like that.
Worked just fine with some "social engineering" on my mom 😅

It's becoming increasingly clear to me that Reflect Orbital's fucking stupid giant mirror satellite, with absolutely NOTHING useful to offer, which will cause countless safety issues, ecological disasters, and destroy the night sky, is going to launch.

A bunch of astronomers and I have sent out a fact sheet about them to a bunch of journalists, but very few are going to write about this. So, let me try posting it all here.

Here's what I know about Reflect Orbital and all the downsides:

One of the challenges I really faced when I transitioned to staff engineer, was realizing that the fundamental nature of my role had changed. It was no longer my primary responsibility to solve the problems. Instead it was my responsibility to help people understand the problem more clearly and then enable and empower them to solve it.

Picard management tip: "That's the way it's always been" is never an adequate reason to continue doing something.

The ghost haunting my house is a sleepy one

Picard management tip: Debate honestly. The goal is to arrive at the truth, not at your preconception.