Hey everyone! Now that we're two decades past it, let's have a heart to heart chat about the Y2K bug.
Some of you might not be old enough to remember the panic articles and news stories surrounding this, and some of us were old enough that we've forgotten the details, but the gist was this: many commercial computer applications, such as bank and medical record software, used to be written to store dates with a 2-digit year, and assumed that everything took place between 1900 and 1999.
The problem was then, of course, what happened on January 1, 2000? The 2-digit year would become 00, and lots of things would begin to break.
Imagine your last mortgage payment was in 1999, your next mortgage payment is due in 2000? Well, what's the difference between 99 and 00?
It looks like your most recent payment is almost 100 years away from now, but since you can't have made a payment in the future, that payment must have been 100 years AGO. Better start assessing fees and foreclosing.
Y2K was a huge issue in computing that was replicated among many systems worldwide. The reason why the world at large never saw anything bad occur was that a lot of people worked extremely hard to update afflicted systems.
Developers who had worked on mainframe systems writing code in languages that hadn't been used in one of more decades came out of retirement and worked overtime.
And once it was fixed... that was it. Nothing happened. Nobody ran articles talking about this enormous effort.
These days a lot of people seem to believe that Y2K was a doomsday hoax. A big nothing hyped up by fear merchants.
It wasn't. It was a nearly catastrophic occurrence from widespread and short-sighted decisions.
That time it was something that could be fixed with some last minute crunches. Next time, probably not.
@kelbesque I remember agencies like the Red Cross recommended stockpiling water, food and cash
@kelbesque Well said, and as you say, it's irritating when people now suggest it was over-hyped. One company at which I worked had a dedicated team working for 4 years to resolve the issues. At another company we brought forward the replacement fate of our core systems. Both would have succumbed to the millennium bug if it wasn't for all that work.
@kelbesque well said! I was kinda hoping 2038 was gonna be an issue but I doubt it lol
@amphetamine @zenhob my guess is that it really depends on what they're actually using the dates for. I don't know enough about what satellites are doing to talk to that, but for processing signals on the receiving end, we can probably work around that using a local timestamp conjoined to the signal's reported time.
Anything the satellite needs datetime for locally would be a problem. 😬
@kelbesque @amphetamine @zenhob Fortunately, satellites are actually pretty brainless. They would virtually never be keeping anything like a datetime in the first place; they'd be keeping relative uptimes / mission time, and the ground can easily reboot them to 0 if necessary.
(Except for positioning satellites, which *are* fancy clocks. Fortunately they actually thought of rollovers there, though it can give ground devices trouble.
@Nentuaby @kelbesque @amphetamine @zenhob GPS satellites use cesium clocks, which are a different order of beast. They're very precise at counting upwards, but don't maintain much of an internal state. The almanacs maintained and transmitted by the birds are recalculated with known numbers of iterations, largely independent of the clocks.
Also... Their daily maintenance logs are public and published for receiver manufacturers. They are not perfect machines...
most IoT devices are 32 bit, and running 32 bit Linux
literally the only OS that didn't update their 32 bit variants' time code
even if someone could convince Linux kernel devs to update, it's way too late: most IoT devices are created (churned out) by Web agency like companies, who hop from project to project, until they tank
@hirojin @amphetamine @kelbesque @zenhob I'm almost done with the kernel now as of linux-5.6, but it's been many years, and then there is all of user space. A lot of the expensive machines will be updated at any cost, and most of the cheap stuff will have long broken down before it starts hurting.
My worry is that we are already too late for the stuff in the middle: as embedded Linux is slowly moving towards arm64 hardware, the deployment of 32 bit hardware is at its peak, and a lot of the affected machines still running in 2038 have already been deployed.
@kelbesque Relatrd Fun Fact: Our old Tandy 1000 HX from 1987 was Y2K-compliant. We actually tested setting the date once and it worked. We sold it before the year 2000 ever came around, though.
@kelbesque Roll on the 2038-calypse!
LB : This is a thing I think about not-infrequently. I was just starting to develop my understanding of the infrastructure of the internet and computers. The whole thing sounded ridiculous at the time (already-off-the-grid friends spent the next five years working through the rice and flour they'd stockpiled). I learned much more later about the work and potential impact of if the work hadn't been done.
There's now a whole generation of folks /starting work/ who never heard about it at all.
@kelbesque We had one customer, a financial institution, that did terrible things when the mainframe switched to "00", it prefixed it with "19" before kicking it over to an audit tracking system which processed that date work for digits. Bad things happened that too four months to fix.
I worked sixty hour weeks for years learning obscure programming languages to fix our customer bugs. I also got very good at writing code parsers and generators.
@kelbesque I'm really happy to see you talking about this as the large effort that it was. A huge effort went into not only doing it, but making sure it happened w/out causing panic. I'd like to add that it wasn't just consumer or business software what were at risk, but also the nuclear attack early warning systems of a variety of countries which was a huge freaking deal.
@kelbesque A related thing I found interesting: this bug is often associate with the millennium but it would have happened at the turn of any century. For example, we would have had a "Y1900" bug if we'd invented computers 100 years earlier.
@kelbesque I was in elementary school at the time, and while everything you said is true
what filtered down to us was that planes were going to literally fall out of the sky at the stroke of midnight, or that nuclear warheads would detonate in their silos, like they were all armed and on countdown timers or something.
there were definitely *also* overhyped doomsday hoaxes going around.
It's bullshit that people weren't credited with having fixed the problem (which is why so many people were led to believe it wasn't a problem in the first place), but I guess it would've been too much for the news reporters of the 90s/00s to spin a hero piece about programmers.
@kelbesque I was doing database support at the time, not coding, but I don’t remember any last minute crunches either. The big OS and database vendors mostly patched out Y2K issues well in advance, and most places were doing Y2K audits a year or two ahead, for legal and insurance reasons.
And the issues, when they happened, mostly took a while to show up - dodgy data spotted several months into the new year rather than sudden catastrophic failures.
@kelbesque And honestly? It didn’t take that much nous to do a text search for bad date formats in the code, even when it wasn’t being regularly updated.
@ghost_bird It doesn't take a lot of looking at retweets from people who were coding during that time to see there were a lot of people who had different experiences with it. Were different systems afflicted to different levels of severity with different levels of risk? Definitely. Was every codebase easy to grep on, debug, and test? Not a chance.
@kelbesque All true, but but I think you underestimate how much of the work was auditing, not fixes, and how much was done well in advance.
@kelbesque to be fair, though, it didn't help that a lot of the media around the y2k bug massively overstated the risks. No, your wristwatch wouldn't explode. No, nuclear weapons were not going to launch because the control systems are confused about the year. Your home stereo was never going to magically fail
It was a huge deal. It was expensive. But it was never a "doomsday" scenario, and I think that's a lot of where the "hoax" feeling comes from
@kelbesque yeah, I've had to correct that belief a number of times on tumblr
There's an episode of the Dilbert cartoon where Dilbert has to work overtime to fix the Y2K bug. Even though I think it does make the joke that the threat was overblown, it's ironically the most honest dipiction of what happened.
@kelbesque I was doing developer support for a company with a lot of telecom customers during that period - our team went to work on Dec 30th and didn't leave until Jan 2nd just in case all our efforts for a few years prior had missed something and one of our customers experienced any y2k issues. We didn't get any calls, but we were ready.
@kelbesque A guy I talk with at Starbucks came out of retirement to fix a bunch of Cobalt systems
LB reminds me that I've heard comparisons to Y2K from gammons saying that anyone who thinks Brexit is anything less than perfect are fear-mongering in a similar way.
It really does us a disservice to misunderstand Y2K as much as we do, because it lets bastards like them off the hook when we really are headed for disaster: The massive efforts that went into avoiding Y2K aren't happening re Brexit, and they'd need to happen much more widely across society to avoid this disaster.
@kelbesque apparently places that didn't bother to prepare also didn't have anything happen? I read that Italy took almost no precautions.
@kelbesque Can confirm. One of the things I did to put myself through undergrad was reversing and patching code running on VAXen for a power company. Also saw what happened when a rollover test failed.
@kelbesque A bunch of overhype to sell advertising and various suvivalist gear is far more plausible than a massive coordinated effort to upgrade infrastructure.
That was my first steps into the corporate/consulting world.
I mainly helped keep power grids up/supplied, and kept people getting paid.
I felt like those were pretty good critical systems to tackle.
@kelbesque oh goshhhh I have a blog post almost ready about this very thing!! Turns out, next payment is due every ~decade; and there's another one coming up around 2039 (32bit Unix epoch, thanks C)! And some more regular maintenance every ~230 years or so (64bit signed nanoseconds, thanks go)!
@kelbesque i remember half-hoping our entire banking infrastructure would literally light on fire
Thank you for this insightful reminder of a tricky time in our history.
It isn't in the nature of "news" to do headlines unless they are warnings, preferably ominous ones.
The Y2K aftermath had nothing which qualified as "Breaking News" (to use today's TV tag line).
Thanks to all the pros who figured out how to keep us from a massive barrage of told-you-so headlines.
@kelbesque do you know of any accounts of people who actually had to fix such code for $big_organizations? I haven't been able to find any that have any amount of technical detail. I wonder if they are buried beneath really scary NDAs.
screaming @ all of these (from https://www.fastcompany.com/90439521/the-weird-wonderful-world-of-y2k-survival-guides-a-look-back )
Y2K Family Survival Guide w/ a foreword by Leonard Nimoy 😂 https://youtu.be/EEhEQEG43RU
Y2K: The Movie (1999) https://youtu.be/FuG1O_PTZio
@kelbesque There are a few still around who remember a time when memory was so expensive that saving two bytes on every date stored made the difference in being able to run a program or not.
@papa but you're only saving those bytes if you don't encode in Unix time and thus have to encode a date as its string representation. 😓
@papa or even just like, use those two bytes to store a 16-bit integer of years. 65535 years would have been a lot more resilient. 😰
@kelbesque Unix? What's that?
@kelbesque Also many programmers in the 1960s who never dreamed anyone would be using their hacks thirty years down the line.
@mike Oh word? I had no idea!
@mike Is there anything else in the thread that I wrote about the topic that you'd like to summarize for me?
Onster Farm is the official Mastodon instance of Doctective Jake Peralta. In this house, we use alt text.