Another GitHub outage in the same day(githubstatus.com)
208 points by Nezteb 3 hours ago | 128 comments
- noodlesUK 2 hours agoCan someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable?
I moved my company over to GH enterprise last year (from AzDO) and I'm considering moving us away to another vendor altogether as a result of the constant partial outages. Things that used to "just work" now are slow in the UI, and GH actions fail to schedule in a reasonable timeframe way more than they ever used to. I enjoy GH copilot as much as the next person, but ultimately I came to GH because I needed a git forge, and I will leave GH if the git forge doesn't work.
[-]- sobjornstad 2 hours agoI second this. GitHub used to be a fantastic product. Now it barely even works. Even basic functionality like the timeline updating when I push commits is unreliable. The other day I opened a PR diff (not even a particularly large one) and it took fully 15 seconds after the page visually finished loading -- on a $2,000 dev machine -- before any UI elements became clickable. This happened repeatedly.
It is fairly stunning to me that we've come to accept this level of non-functional software as normal.
[-]- matthewisabel 7 minutes agoHey from the GitHub team. Outages like this are incredibly painful and we'll share a post-mortem once our investigation is complete.
It stings to have this happen as we're putting a lot of effort specifically into the core product, growing teams like Actions and increasing performance-focused initiatives on key areas like pull requests where we're already making solid progress[1]. Would love if you would reach out to me in DM around the perf issues you mentioned with diffs.
There's a lot of architecture, scaling, and performance work that we're prioritizing as we work to meet the growing code demand.
We're still investigating today's outage and we'll share a write up on our status page, and in our February Availability Report, with details on root cause and steps we're taking to mitigate moving forward.
- HoldOnAMinute 2 hours agoThe trend of "non-functional software" is happening everywhere. See the recent articles about Copilot in Notepad, failing to start because you aren't signed in with your Microsoft Account.
We are in a future that nobody wanted.
[-]- amarant 2 hours agoNot quite everywhere. There's a common denominator for all of those: Microsoft.
Their business is buying good products and turning them into shit, while wringing every cent they can out of the business. Always has been.
They have a grace period of about 2-4 years after acquisition where interference is minimal. Then it ramps up. How long a product can survive once the interference begins largely depends on how good senior leadership at that product company is at resisting the interference. It's a hopeless battle, the best you can do is to lose slowly.
[-]- Andrex 1 hour agoThings don't always ramp up after 2-4 years. Sometimes MS just kills the project or company after that period of time.
See also their moves in the gaming industry.
- its_magic 1 hour agoI for one am shocked--SHOCKED, I say!--to learn that anything bad could happen as a result of a) putting everything in "the cloud" and b) handing control over the entire world's source code to the likes of Microsoft.
Who could have POSSIBLY foreseen any kind of dire consequences?
- bonesss 2 hours agoThis thread has complaints about software coming from the same supplier both degrading.
The person(s) who wanted this want Azure to get bigger and have prioritized Azure over Windows and Office, and their share price has been growing handsomely.
‘Microslop’, perhaps, but their other nickname has a $ in it for a reason.
- habitable5 2 hours ago> We are in a future that nobody wanted.
some people wanted this future and put in untold amount of money to make it happen. Hint: one of them is a rabid Tolkien fan.
[-]- b00ty4breakfast 43 minutes agothe irony of Tolkien being associated with a techno-dystopia makes me nauseous
- cyanydeez 2 hours agoRent seekers paradise (ft copilot)
- its_magic 1 hour agoLaughs in my own Linux distro
- michaelcampbell 2 hours agoMS PM's wanted it, got their OKR's OK'd, got their bonuses, and moved on.
- dylan604 2 hours ago> We are in a future that nobody wanted.
Nor deserved.
[-]- heliumtera 2 hours agoThen why is it the future we have?[-]
- its_magic 1 hour agoIt was a complete accident. Nobody could have foreseen it. We are currently experiencing the sudden discovery that Microsoft is an evil corporation and maybe putting everything in the cloud wasn't the best move after all.
- timacles 46 minutes agoLet’s just say there are a couple of guys, who are up to no good. And they started making trouble in our neighborhood.
jokes aside it’s all because of hyper financial engineering. Every dollar every little cent must be maximized. Every process must be exploited and monetized, and there are a small group of people who are essentially driving all this all across the world in every industry.
- dev_l1x_be 55 minutes agoSo React rewrite did not help after all? Imagine, one of the largest software tool companies on Earth cannot reliably REbuild something in React. I lost count of the inconsistency issues React introduced.[-]
- catigula 52 minutes agoReact isn't causing these issues.[-]
- dev_l1x_be 4 minutes agoGood to know. So it only causes the UI inconsistency bugs.
- sodapopcan 2 hours agoYa, it really was one of the most enjoyable web apps to use pre-MS. I'm sure there are lots of things that have contributed to this downfall. We certainly didn't need bullshit features like achievements.[-]
- noodlesUK 2 hours agoEven just a year or two ago its web interface was way snappier. Now an issue with a non-trivial number of comments, or a PR with a diff of even just a few hundred or thousand lines of changes causes my browser to lock up.[-]
- sodapopcan 2 hours agoBut even clicking around tabs and whatnot is noticeably slower. It used to be incredibly snappy.
- samgranieri 2 hours agoI've been a GitHub user since the very early days. I had a beta invite to the service. I really wish they didn't swap out the FE for a React FE.
They need to start rolling back some of their most recent changes.
I mean, if they want people to start moving to self hosted GitLab, this is gonna get that ball rolling.
[-]- throw20251220 44 minutes agoGitLab is slower for me than that React GH app. Why would I move to GitLab?
- blibble 53 minutes ago> GitHub used to be a fantastic product. Now it barely even works.
it's almost as if Microsoft bought it, isn't it?
- kimixa 2 hours agoWe loved Github as a product when it needed to return or profit beyond "getting more users".
I feel this is just the natural trajectory for any VC-funded "service" that isn't actually profitable at the time you adopt it. Of course it's going to change for the worse to become profitable.
[-]- tibbar 2 hours agoGitHub isn't VC funded at the moment, though. It's owned by Microsoft. Not that this necessarily changes your point.
- notpushkin 2 hours agoI don’t get it. Why making the UI shittier would possibly lead to more profit?[-]
- kimixa 11 minutes agoIt seems most of the complaints are about the reliability and infrastructure - which is very much often a direct result of lack of investment and development resources.
And then many UI changes people have been complaining about are related to things like copilot being forcibly integrated - which is very much in the "Microsoft expect to gain a profit by encouraging it's use" camp.
It's pretty rare companies make a UI because they want a bad UI, it's normally a second order thing from other priorities - such as promoting other services or encouraging more ad impressions or similar.
- bigbuppo 2 hours agoNot going to happen. This is terminal decline. Next step is to kill off free repos, and then they'll start ratcheting up the price to the point that they have one small dedicated engineering team supporting each customer they have. They will have exactly one customer. At some point they'll end up owned by Broadcom, OpenText, Rocket, or Progress.[-]
- tazjin 19 minutes agoKilling off free repos is not going to happen. That would be a suicide move on the level of the Digg redesign, or Tumblr's porn ban.
It kind of would be good for everyone if they did do it though. Need to get rid of this monopoly, and maybe people will discover that there are alternatives with actually good workflows out there.
[-]- bigbuppo 2 minutes agoThey are owned by Microsoft. When has Microsoft ever had a good idea?
- tibbar 2 hours agoGithub used to publish some pretty interesting postmortems. Maybe they still do. IIRC that they were struggling with scaling their SQL db and were starting to hit the limits. It's a tough position to be in because you have to either to a massive migration to a data layer with much different semantics, or you have to keep desperately squeezing performance and skirting on the edge of outages with a DB that wasn't really meant to handle what you're doing with it now. The OpenAI blog post on "scaling" Postgres to their current scale has much the same flavor, although I think they're doing it better than Github appears to be doing.
- co_king_3 2 hours ago> Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable?
It's Microsoft. A reliable product is not a reasonable expectation.
- markus_zhang 1 hour agoMaybe take the initiative and move your own first? It definitely would have a bigger effect than begging here.
- wnevets 2 hours ago> Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable?
They claim that is what they are doing right now. [1]
[1] https://thenewstack.io/github-will-prioritize-migrating-to-a...
[-]- semiquaver 2 hours agoZero indication that migrating to azure will improve stability over the colos they are in now. The outages aren’t caused by the datacenter, whatever MS execs say.[-]
- Andrex 1 hour agoWasn't the last one even caused by Azure?
- amluto 2 hours agoThe problem with the GH front end being an unbelievably bloated mess will not be even slightly improved by moving to Azure.
- skywhopper 2 hours ago"Migrating to Azure" is, unfortunately, often the opposite of "delivering a reliable product".
- jbreckmckye 2 hours agoAs an aside, God, Azure DevOps, what a total pile of crap that product is
My "favourite" restriction that an Azure DevOps PR description is limited to a pathetic 4000 characters.
[-]- OkayPhysicist 34 minutes agoMy favourite restriction is the fact that colored text doesn't work in dark mode. Why? Because whatever intern they had implement dark mode didn't understand how CSS works, and just slapped !important on all the style changes that make dark mode dark, and thus overwrite the color data.
I ended up writing a browser extension for my team to fix it, because the boss loved to indicate stuff with red/green text.
- dylan604 2 hours agoAmazon's deprecated CodeCommit is limited to 150 chars like it's an old SMS or Tweet.[-]
- jbreckmckye 2 hours agoHa! Nice. I never worked with CodeStar / CodeCommit. Was it pretty bad?[-]
- dylan604 2 hours agoThat's going to depend on each user's demands. The PR message limit is the biggest pain for me. I don't depend on the UI very often. I'm not trying to do any CI/CD nonsense. I just use it as a bog standard git repo. When used as that, it works just fine for me
- noodlesUK 2 hours agoIt shows you the level of quality to expect from a Microsoft flagship cloud product...[-]
- jbreckmckye 2 hours agoSo I work for a devtools vendor (Snyk) and 6 months ago I signed into Azure DevOps for the first time in my life
I couldn't believe it. I actually thought the product was broken. Just from a visual perspective it looked like a student project. And then I got to _using_ the damn thing
[-]- noodlesUK 2 hours agoIt's also completely unloved. Even MSFT Azure's own documentation regularly treats it as a second class citizen to GitHub. I have no idea why they don't just deprecate the service and officially feature freeze it.
Honestly that's the case with a lot of Azure services though.
[-]- easton 2 hours agoIt's the boards. GitHub issues doesn't let you do all the arcane nonsense Azure DevOps' boards let you do.
- tibbar 2 hours agoYou would kind of expect with the pressure of supporting OpenAI and GitHub etc. that Azure would have been whipped into shape by now.[-]
- semiquaver 2 hours agoAZDO has been in KTLO maintenance mode for years.
- rvz 2 hours agoYou might as well self-host at this point as that is far more reliable than depending on GitHub.
Additionally, there is no CEO of GitHub this time that is going to save us here.
So as I said many years ago [0] in the long term, a better way is to self host or use alternatives such as Codeberg or GitLab which at least you can self host your own.
- kevmo314 2 hours agoI wonder if GitHub is feeling the crush of fully automated development workflows? Must be a crazy number of commits now to personal repos that will never convert to paid orgs.[-]
- 1f60c 2 hours agoIME this all started after MSFT acquired GitHub but well before vibe coding took the world by storm.
ETA: Tangentially, private repos became free under Microsoft ownership in 2019. If they hadn't done that, they could've extracted $4 per month from every vibe coder forever(!)
[-]- dizhn 7 minutes agoIs someone who is not really using github's free service losing something important?
- winddude 2 hours agoI was wondering about that the other day, the sheer amount of code, repos, and commits being generated now with AI. And probably more large datasets as well.
- reactordev 2 hours agoThis is the real scenario behind the scenes. They are struggling with scale.[-]
- jbreckmckye 2 hours agoHow much has the volume increased, from what you know?[-]
- reactordev 2 hours agoOver 100x is what I’m hearing. Though that could just be panic and they don’t know the real number because they can’t handle the traffic.[-]
- bredren 2 hours agoAn anecdote: On one project, I use a skill + custom cli to assist getting PRs through a sometimes long and winding CI process. `/babysit-pr`
This includes regular checks on CI checks using `gh`. My skill / cli are broken right now:
`gh pr checks 8174 --repo [repo] 2>&1)`
Error: Exit code 1 Non-200 OK status code: 429 Too Many Requests Body: { "message": "This endpoint is temporarily being throttled. Please try again later. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service)", "documentation_url": "https://docs.github.com/graphql/using-the-rest-api/rate-limits-for-the-rest-api", "status": "429" } - chasd00 1 hour agoSo much for GitHub being a good source of training data.
Btw, someone prompt Claude code “make an equivalent to GitHub.com and deploy it wherever you think is best. No questions.”
- jbreckmckye 2 hours agoOne hundred? Did I read that right?[-]
- 9cb14c1ec0 1 hour agoYes, millions of people running code agents around the clock, where every tiny change generates a commit, a branch, a PR, and a CI run.[-]
- neuropacabra 33 minutes agoI simply do not believe that all of these people can and want to setup a CI. Some maybe, but even after the agent will recommend it only a fraction of people would actually do it. Why would they?
- reactordev 2 hours agoThere’s a huge up tick in people who weren’t engineers suddenly using git for projects with AI.
This is all grapevine but yeah, you read that right.
- dwoldrich 34 minutes agoLive by the AI Agent hype, die by the AI Agent crush.
- danhon 20 minutes agoIsn't github in the middle of their (latest) attempt to migrate to Azure?[0]
[0]: https://www.theverge.com/tech/796119/microsoft-github-azure-...
- falloutx 2 hours agoWe can all chill for couple weeks, Github guys take your time. Infact, don't even worry about it.
- dec0dedab0de 55 minutes agoI still say that mixing CI/CD with code/version control hosting is a mistake.
At it's absolute best, everything just works silently, and you now have vendor lock-in with whichever proprietary system you chose.
Switching git hosting providers should be as easy as changing your remotes and pushing. Though now a days that requires finding solutions for the MR/PR process, and the wiki, and all the extra things your team might have grown to rely on. As always, the bundle is a trap.
[-]- bamboozled 46 minutes agoI don't think any of this was a mistake ;) Lock-in was by design.
- sisve 21 minutes agoI moved everything on github to a self hosted foregjo instanse some days ago. I really did not do anything. Created some tokens so that CC could access github and forgejo and my dns API. Self hosting is so much simpler and easier with AI. Expect more people to self host small to medium stuff.
- Kovah 2 hours agoI consider moving away from Github, but I need a solid CI solution, and ideally a container registry as well. Would totally pay for a solution that just works. Any good recommendations?[-]
- adamcharnock 2 hours agoWe can run a Forgejo instance for you with Firecracker VM runners on bare metal. We can also support it and provide an SLA. We're running it internally and it is very solid. We're running the runners on bare metal, with a whole lot of large CI/CD jobs (mostly Rust compilation).
The down side is that the starting price is kinda high, so the math probably only works out if you also have a number of other workloads to run on the same cluster. Or if you need to run a really huge Forgejo server!
I suspect my comment history will provide the best details and overview of what we do. We'll be offering the Firecracker runner back to the Forgejo community very soon in any case.
- joeskyyy 2 hours agoLong time GitLab fan myself. The platform itself is quite solid, and GitLab CI is extremely straightforward but allows for a lot of complexity if you need it. They have registries as well, though admittedly the permission stuff around them is a bit wonky. But it definitely works and integrates nicely when you use everything all in one!
- dylan604 2 hours agoShould our repos be responsible for CI in the first place? Seems like we keep losing the idea of simple tools to do specific jobs well (unix-like) and keep growing tools to be larger while attempting to do more things much less well (microsoft-like).[-]
- tibbar 2 hours agoI think most large platforms eventually split the tools out because you indeed can get MUCH better CI/CD, ticket management, documentation, etc from dedicated platforms for each. However when you're just starting out the cognitive overhead and cost of signing up and connecting multiple services is a lot higher than using all the tools bundled (initially for free) with your repo.
- tibbar 2 hours agoLots of dedicated CI/CD out there that works well. CircleCI has worked for me
- import 55 minutes agoGitea / forgejo. It supports GitHub actions.
- hhh 1 hour agoGitLab, best ci i’ve ever used.
- cyanydeez 2 hours agoGitLab can be selfhosted with container based CI and fairly easy to setup CE[-]
- IshKebab 2 hours agoCE is pretty good. The things that you will miss that made us eventually pay:
* Mandatory code reviews
* Merge queue (merge train)
If you don't need those it's good.
Also it's written in Ruby so if you think you'll ever want to understand or modify the code then look elsewhere (probably Forgejo).
- Kenji 54 minutes ago[dead]
- vampiregrey 3 hours agoAt this point, GitHub outages feel closer to cloud provider outages than a SaaS blip. Curious how many people here still run self-hosted Git (GitLab / Gitea) vs fully outsourcing version control.[-]
- neilv 2 hours agoYay for GitLab and Forgejo/Gitea.
My previous two startups used GitLab successfully. The smaller startup used paid-tier hosted by gitlab.com. The bigger startup (with strategic cutting-edge IP, and multinational security sensitivity) used the expensive on-prem enterprise GitLab.
(The latter startup, I spent some principal engineer political capital to move us to GitLab, after our software team was crippled by the Microsoft Azure-branded thing that non-software people had purchased by default. It helped that GitLab had a testimonial from Nvidia, since we were also in the AI hardware space.)
If you prefer to use fully open source, or have $0 budget, there's also Forgejo (forked from Gitea). I'm using it for my current one-person side-startup, and it's mostly as good as GitLab for Git, issues, boards, and wiki. The "scoped" issue labels, which I use heavily, are standard in Foregejo, but paid-tier in GitLab. I haven't yet exercised the CI features.
- arthur-st 2 hours agoSelf-hosted Gitea is a good time if you're comfortable taking care of backups and other self-hosting stuff.
- betaby 2 hours agoSelf hosted GitLab is absolutely worth it.[-]
- edverma2 2 hours agoI was just looking into this today but it seems pricey. $29/user/month for basic features like codeowners and defining pr approval requirements. Going with Forgejo.[-]
- 1f60c 2 hours agoWait, what? So you're on the hook for backups, upgrades, etc. and you have to pay them for the privilege? I thought GitLab was free as in speech and beer.[-]
- cyberax 2 hours agoIt's an Open Core model. You can deploy the free version, but it lacks some pretty important features like SSO.
But that $30 per month per user is also the cost for their cloud-hosted version. It also includes quite a bit of CI/CD runtime.
- vampiregrey 2 hours agoI think i will slowly start moving to self hosted git intra at my homelab.
- sam_lowry_ 2 hours agoSelf-hosted git is absolutely worth it.
- monkaiju 2 hours agoor forgejo![-]
- DeepYogurt 2 hours agoForgejo should 100% be people's default for self hosting
- zhouzhao 2 hours agoYeah man. Forgejo (albeit it being a weird name from a language that nobody wants to use), is doing very well in my homelab.
When I worked at the univerity we used Gitea.
Every job outside of univerity I had used Gitlab self hosted. While I don't like the UI or any aspect of Gitlab a lot, it gets the job done.
[-]- zer00eyz 2 hours agoI use Gitea already... I haven't seen Forejo before today. Im now curious if it is worth the switch.[-]
- terminalbraid 2 hours agoForejo was originally forked from Gitea
- blibble 2 hours agoforgejo doesn't need half a supercomputer to run it
- atonse 53 minutes agoI'm starting to wonder if people doing what were previously unconventional workflows (which may not be performance optimized) are affecting things.
For example, today, I had claude basically prune all merged branches from a repo that's had 8 years of commits in it. It found and deleted 420 branches that were merged but not deleted.
Deleting 420 branches at once is probably the kind of long tail workflow that was not worth optimizing in the past, right? But I'm sure devs are doing this sort of housekeeping often now, whereas in the past, we just never would've made the time to do so.
- ariedro 2 hours agoIt would be interesting to have a graph showing AI adoption in coding against the number of weekly outages across different companies. I am sure they are quite correlated.[-]
- the_real_cher 2 hours agoI bet there's other factors that are correlated as well!
- thomasfromcdnjs 2 hours agoSomeone needs to make an mcp server for my claude so it can check if services are down, it goes stir crazy when github is down and adds heaps of work around code =D
- bstsb 2 hours agomy four-core VPS running a Git server has higher uptime than GitHub at this point
(although admittedly less load and redundancy)
[-]- chilipepperhott 2 hours agoDoes redundancy even matter if the end result is still poorer uptime?
- elzbardico 56 minutes agoYeah, Vibe code more github![-]
- neuropacabra 31 minutes agoSo far it feels they are vibe coding it day and night lol…probably with GitHub Copilot
- devy 2 hours agoThey were talking about prioritizing migration into Azure for a long while now. Not sure this incident today is related.
https://thenewstack.io/github-will-prioritize-migrating-to-a...
And coincidentally, an early CircleCI engineer wrote an article about GitHub Action (TLDR: don't use GitHub Action for CI/CD!)
https://www.iankduncan.com/engineering/2026-02-05-github-act...
[-]- baq 1 hour ago> TLDR: don't use GitHub Action for CI/CD!
You should reach the same conclusion by trying to use it for this purpose, but also indeed for any purpose at all. Incidents that make you unable to deploy making all your CD efforts pointless are only the cherry on top.
- alexellisuk 2 hours agoI’m seeing 429s cascading downloading things like setup-buildx on self hosted runners. That seems odd/off.
Anyone else having issues? It is blocking any kind of release
- nhuser2221 2 hours agoI am glad I have finally started self hosting my own git server, and stop worrying about github :-)
- varispeed 2 hours agoDid they replace developers and devops with openclaw?
- WhyNotHugo 2 hours agoHow is this "news" when it comes up multiple times a week?
It's just "yet another day of business as usual" as this point.
- rvz 2 hours agoA great time to consider self hosting instead. Since there is no CEO of GitHub to contact anymore.
A prophecy that was predicted half a decade ago [0] which is now more important then as it is now today.
- musha68k 2 hours agoRadicle moment.
- ChrisArchitect 2 hours ago[dupe] Discussion: https://news.ycombinator.com/item?id=46946827[-]
- rpns 1 hour agoNot quite, that one is an earlier outage while this one started at (or a bit before) 19:01 UTC.
The history for today is a bit of a mess really: https://www.githubstatus.com/history
[-]- ChrisArchitect 35 minutes agoThey are all being discussed in that thread, the submitted url is just one of the various incident links on the day. Duplicate discussion.
- esafak 1 hour agoNo, it's a new outage -- that's the point! Check the URLs.[-]
- ChrisArchitect 35 minutes agoThat's not the point. The point is it's a duplicate discussion of one of a number of incident links being discussed, all over there.
- heliumtera 1 hour agoRemember the other day when a bunch of yous were making fun of zig moving away from GitHub? Now suddenly you all say this is not the future you wanted.
Everyday you opt in to get wrecked by Microsoft.
You all do realize you all could, for a change, learn something and never again touch anything Microsoft related?
Fool me once...
[-]- TacticalCoder 9 minutes ago> You all do realize you all could, for a change, learn something and never again touch anything Microsoft related?
I learned that lesson in the 90s and became an "ABM" (Anything But Microsoft).
People sadly shall never learn: Windows 12 is going to come out and shall suck more than any previous version of Windows except Windows 11, so they'll see it as progress. Then Windows 13 is going to be an abysmal piece of crap and people shall hang to their Windows 12, wondering how it's possible that Microsoft came out with a bad OS.
There are still people explaining, today, that Microsoft ain't all bad because Windows XP was good (for some definition of good). Windows XP came out in late 2001.
Stockholm syndrome and all that.
- skywhopper 2 hours agoThis is the predictable outcome of subordinating the GitHub product to the overarching "AI must be part of everything whether it makes sense or not" mandate coming down from the top. It was only a year ago that GitHub was moved under the "CoreAI" group at Microsoft, and there's been plenty of stories of massive cost-cutting and forcing teams to focus on AI workflows instead of their actual product priorities. To the extent they are drinking their own Kool-Aid, this sort of ops failure is also an entirely predictable outcome of too much reliance on LLM-generated code and workflows rather than human expertise, something we see happening at an alarming scale in a number of public MS repos.
Hopefully it will get bad enough fast enough that they'll recognize they need to drastically change how they are operating. But I fear we're just witnessing a slow slide into complacency and settling for being a substandard product with monopoly-power name recognition.