- This status page actually identified the outage: https://hackernews.onlineornot.com/ - Pages by Hund and Statuspal did not show the outage.
- The last post before the outage was https://news.ycombinator.com/item?id=46301823 (1:39:59 PM GMT). The last comment was https://news.ycombinator.com/item?id=46301848 (1:41:54 PM GMT).
- There was an average of ~4 seconds per comment just prior to the outage. Based on this, HN likely went down at 1:41:58 PM GMT.
(The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)
In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.
I'll add more as we find out more, but it probably won't be till later this afternoon PST.
We all knew that but I haven't seen any confirmation before this.
So don’t beat yourself up please.
When I worked for “SaaS unicorn” we typically had multiple levels of escalation, and acknowledging would have done nothing because the alarm would continue firing until fixed. Not sure what’s changed in 15 years of ops, I had assumed it would be better now- I can’t imagine silencing an alert totally by acknowledging it- if its still occurring.
I’m totally fine with how you handled it, if anything I am thankful. But that seems to be a system I would improve if I had the time.
“mute” is different than “resolve” to me, and both should exist. (Where mute is an acknowledgement of an issue as ongoing.)
Try opening HN -> it's down, better check HN to see everyone talking about a major website being down -> Try opening HN -> loop
That was a few hours ago. I'm glad this loop is broken.
https://www.proginosko.com/leechblock/
You'll still open new tabs and go to HN, but you'll be reminded quickly, and every day can be downtime day \o/ (for you, personally)
It's like they say: "Your demons will comfort you when no one else will. That's why it's so hard to get rid of them"
youd go through that effort when you could have just stopped though.
If I could snap my fingers and break toxic habits and patterns, I would have done so decades ago :)
That's so refreshing in terms of being a user-focused feature, and yet it stands in sharp contrast against today's engagement-hyperfocused climate. I never would have thought to look on a website's own settings page to limit my access to that same website.
I love it, thank you for pointing me to this!
Did it like 5 times during that 1h-ish outage. :(
I know dang basically works tirelessly to not change the format in order to not induce those addictive patterns
but yet here we all are
It's understandable to be addicted. Lol.
I visit this place multiple times a day.
i cant find the link, but there was a post about how to "be nice" and it was a revelation to a worrying amount of "geniuses" on here. bare in mind the sum total of the advice was "be nice, dont be rude"
But also, people like me. Be careful what you choose to believe on this website
This was especially obvious during Covid, I even stopped visiting because the comment section was so crazy.
Nice joke!
At least, I hope it was a joke...
https://x.com/HNStatus
Is there a better place to check, beyond a basic down detector that may provide more insight or signal that the outage is acknowledged?
Seems to reset it on the web view, too.
I didn't read the post text, it's identified there haha, my bad! I wish the text post text wasn't grey, I gloss over it too easily.
I suppose you could also just clear your HN cookies in regular browsing window, but then when they fix it you'd have to log in again.
It's not that much different from HN, come to think of it.
(ha, ha)
It did work without being logged on. The auth service appeared to be down as the log in attempt (just showing the page) failed.
I believe it's because they accept user reports.
https://hackernews.onlineornot.com/incidents/yaz-eOJeARBL
https://downforeveryoneorjustme.com/hacker-news
Strangely, nothing from the statuspal, which is the first google result
https://hacker-news.statuspal.io/
on edit: ok others pointed out it was cached pages I saw. explains it.
It's down about 8.4 minutes per week. On 26% of days it doesn't work at least once, and on 12% of days it has more than one consecutive failed check. The longest uptime streak was 24 days
I've been keeping track since exactly 2 years (to the day!) because I was surprised that it seemed briefly down for me on a daily basis. Was I getting unlucky and hitting it every time, or was it just down very often? Nobody posted anything so I started answering the question for myself :p
I've been meaning to post the tracker to HN but there's a pesky bug I want to fix: the "is it currently down" stat. I don't know how this is beyond me but something in the code bugs out. So this is my first time posting about it
https://x.com/paulg/status/1953289830982664236?s=46
I'm sure it's a coincidence but it started working again shortly after emailing [email protected]
I'm still impressed nonetheless.
I'd like to know what caused the outage and how it could have been prevented, for learning purposes.
You can just look at them, turn on showdead in your profile and you'll see a bunch of flag-killed comments in this discussion by whatevermrfukz. No need for a plugin or scraper.
Anyway, glad to see you back.
Paris 1812.
Cheers from France.
HN was down about an hour ago.
Glad to see it back !
Cheers.
Working with full dates in the HTML and doing a tiny JavaScript that calculates the "minutes ago" would actually be a neat improvement.
After more than an hour I thought, "wow this is pretty harsh" and "so much of my exposure to learning things is directly tied to HN posts". I was lost lol.
Edit: Now it happens again. Knee jerk defenses all the way down.
Being "voted to -2" doesn't necessarily mean you were wrong (it often correlates though). People might just think it wasn't relevant in whatever context you posted it in
I often find it hard to tell what makes people think something I write is not helpful (or sometimes also a comment someone else made) and thus appreciate comments that clarify constructively. It can also help to ask for clarification if you're particularly surprised about the votes on a given post