I hope there will be an update where when my RAM gets full my PC doesn't freeze and becomes unusable... I remember that Linux and Windows do this in different ways and Windows doesn't have the problem.
Have you tried setting up systemd-oomd [1]? It works by killing the offending process (hopefully) before a true OOM.
When I run in a frozen system because of RAM, I'm a happy user of the Magic SysRq 'f' [2], it bypasses most kernel scheduling so it works even on frozen systems
Its not enabled by default, but enabling MGLRU setting /sys/kernel/mm/lru_gen/min_ttl_ms to 70 has entirely resolved the issue for me. It _does_ usually pick firefox or vscode to kill, but my system remains interactive, with no need for earlyoom.
From my experience, Windows does freeze and become completely unusable too, just not usually from RAM usage hitting 100%.
I don't use Windows much, nor do I care much about that environment, but from what I've seen it seems to keep RAM usage below 100% most of the time. What I do see pretty often is the drive getting stuck at 100% usage instead, which makes the whole system ridiculously unusable anyway.
Windows does freeze and become unstable, but it's usually because of some ring 0 errors not getting trapped appropriately.
But Windows has always handled both OOM and out of disk space very well. The system will be extremely sluggish, but it typically continues operating.
The Linux design is to keep using memory, then push to swap, and then when you OOM you hard lock. The built in kernel OOM module can miss when RAM usage spikes rapidly. You can enable OOM monitors like systemd-oomd or earlyoom, but they do not run by default, and their behavior is to term the offending process.
The thing about Windows is that when you request memory, you're only granted memory if it can be guaranteed in the first place. The application will get a "not enough memory" error. On Linux, you're permitted to request more memory than the system actually has, with the idea that you won't actually use that much memory. It's optimistic that way. But if you do use it, then you're screwed.
edit: I meant that regularly hitting OOM on a production server is usually more of a provisioning/resource-management issue. Obviously, how the OS handles OOM once it happens is still its responsibility.
Running out of memory on a linux server sounds more like a skill issue than something comparable to how a desktop OS should behave :\
Windows uses eager memory allocation, and applications will die when they allocate (and think image editor loads a beefy image) or fail to start when it runs out of memory.
Linux uses lazy allocation and overcommits, so what dies isn’t necessarily what you would expect thanks to the OOM killer.
With a fixed swap size windows doesn’t necessarily slow down as it runs out of memory.
YMMV if you don’t have 64 gigs of RAM and a 64 gig swap.
Windows will aggressively page out as you approach RAM usage, MacOS does the same but based on activity iirc. On Linux if you just set up paging (forgot how i did this on arch) you won't have any issues.
Me too, but you can improve the behavior yourself too. It's been an issue with desktop linux and the default settings for a long time. You could maybe tune OOM killer settings, but another option is to use a package like earlyoom: https://man.archlinux.org/man/earlyoom.1.en
Configure overcommit_memory, overcommit_ratio, admin|user_reserve_kbytes, min_free_kbytes, oom_kill_allocating_task, oom_score_adj; if you're on a dev box make it such that even if you approach memory limits to just instantly start killing processes rather than swapping. It's not a solution for servers, but for dev environments it's basically perfect.
So normally if you overcommit it might be undefined behavior if you not have enough memory. However if everything is fine pods can preempt based on prio to new nodes.
With the right settings only the faulty app might fail. Without it it can be a real pain to find the rouges
Windows does freeze when a Chromium-based browser decides to hug the pc.
Chrome/Vivaldi are the only browsers where, when i have a long HN page open, when i resize it, my laptop freezes for like a minute. Not a full freeze, the mouse still moves, but like it's the year 1999 with a refresh rate of 4 seconds.
That's because windows always uses swap by default, and your system will freeze if you run out of swap which is usually 33 gigabytes, or ir your drive and ram are full.
I stopped using swap almost a quarter of century ago, and since then I have not seen a PC with Linux freezing because of full RAM.
Before the recent increase in RAM prices, it was not difficult to have a generous amount of RAM in a PC, that would make very unlikely the out-of-memory situations.
When OOM really happened, it was much better to become aware of this immediately, and reduce the number of concurrent threads for a running job, or whatever caused that, instead of having reduced performance or even freezing.
Mannnnn.... I've done similar and have NOT had the same experience. The problems I've had are usually because I'm loading stuff in a weird disk-memory-cpu trifecta and they all really want to push the remaining 30GB stuff into a database. Then it's off to ctrl-alt-f5 to wait for gnome to crash so I can auth in and kill the ravaging process.
My PC doesn't have swap and it freezes for about 20 s when I run out of memory until the OOM killer finishes its job, usually by killing Code or more rarely Firefox. I'll take this over the machine swapping so slowly that I need to hard reset anyway, but I'd appreciate it if the OS could stay more responsive when it happens.
Unfortunately Linux makes it difficult to fully get away from swapping because any file-backed page is eligible to be evicted, and even if you have "disabled swap" the kernel will cheerfully page-out your program, even if it needs to be immediately paged in again to continue. The only way to stop it from doing so is to remap all executable mappings into anonymous memory, a thing which is possible but which the loader and linker do not make easy.
This type of thrashing was the most common OOM issue I saw on my servers, the system would effectively deadlock until I pulled the plug. In my setup this seemed to be caused by slow-ish disks, and using a small amount of zram made things more CPU-bound / less IO-bound. I haven’t seen the problem since, the OOM killer does what I expect.
Anyways, you've unlocked an old memory of mine, where Windows would crap itself due to low RAM and the fonts and interface elements would render in a "compatible" way, like using "fixedsys" font everywhere.
It happens sometimes, even in the present day: almost certainly it means something leaked and hit 9999 GDI objects (or user objects?) -- you can add those columns in task manager. Killing the offending process should fix it
Are these directly comparable, though? Doesn't Windows default to using free space on the system drive to grow the swap file in case of memory pressure? Linux has you set up swap files/partitions explicitly.
Also the way memory allocation in Windows works is that if it doesn't actually have the physical RAM (extended by pagefile) then the program will crash.
If you disable pagefile (like I did for some of my servers) and your program mallocs more than what you have in available space (even without ever dirtying a page) you will observe this.
I'm sure that this helps Windows a lot, even if it's not counted as actually used until the page is dirty.
Linux programs very often have virtual addresses many times the amount of physical ram, because there's never been any restriction. It's then very easy to just malloc huge chunks and use what you need and don't care about it too much.
Especially with many "tiny" allocations, python for example has huge sized objects which consume gobs of RAM dynamically, so any long running python process not only fragments memory but ends up having a bunch of objects consuming virtual memory...
idk why I felt the need to rant about this, but it's a difference that I've noted.
Yes, Windows doesn't overcommit memory. In the default config Windows is allowed to grow the page file as much as it wants, and empty pages are cheap to reserve in the page file (no need to actually write those zeros, you just need to reserve space for them). So the end effect is comparable, with the difference that software shouldn't just go ahead and ask for gigabytes of memory it doesn't actually need
I prefer the Windows approach, it's more predictable and has better behavior under memory pressure. But it can cause issues with software written under the assumption that the OS uses memory overcommit
Solaris family kernels move this problem to run/allocation time, instead of waiting for the system to become unusable before trying to do something about it. Kinda nice
One of the biggest reason I migrated to Codex. I'd like to see each agentic CLI process as a simple building block, and having 100 claude session means I sometimes run out of memory (and I have 256GB of them). Codex never have this problem.
The communication is more direct too with GPT5.6 family of LLM. No more belt and suspenders.
When my Intel Mac ran out of disk space once, it became impossible to clear the space without rebooting (due to APFS's annoying journaling crap or whatever -- it would literally say "rm: can not remove [whatever]: no space left on device" or something), which caused Google Chrome to wipe all my local storage and cookies because it loves to do that when you run out of disk space, and there's no way to get that data back even if you do somehow clear space with the browser still running. I've been bitten by this many times on at least four different platforms. I wish it could just keep everything on disk instead of deleting it all? Like anything that deletes my data automatically can just go die in my opinion. This is one of the reasons I don't use Google Chrome anymore, another is how it deletes "old" history too (after 3 months) even if you have terabytes free, which is just so offensive I can't even explain it. Ungoogled Chromium lets you turn some of this stuff off, but when I moved to Mac I just started using Safari.
The only way to save my file system was to boot into recovery mode and run a verification with Disk Utility, then I was able to remove some files and get my space back.
Great article! I share the same hunch as the author does; when allocating memory ultimately the application itself is in the best position to inform the kernel about the desired stickiness to VRAM. The best a kernel can do is guessing.
As a side note, it strikes me how much we owe to young trans people for low level performance engineering.
Of course. These are the people that the industry needs who understands proper performance-level engineering.
The commits they authored in [0] match and all of what they have mentioned in the article and this is how you know the author knows exactly what they are talking about and have explained it well.
Puts all the vibe-coders, and the so-called "AI Engineers" to shame. I know that if I ask about the basic system design of amdgpu and drm (direct rendering manager) infrastructure in Linux, a small handful will be able to explain it without using AI or googling.
I am inclined to rather point out remarkable things than unremarkable things. There seems to be relatively many trans people in this niche.
If I see a soil that is unusually rich in iron, I would be more inclined to point that out. If I come across a piece of soil that is as average as the 98% of the surrounding lands I wouldn't be less likely to point out "hey look, this soil is unremarkable average."
Also, it was just an aside from me. It is unclear what your problem is exactly, and I am also not sure if a tech forum could provide counsel for that.
I imagine "we" "owe" a lot more to young men or just men in general for the same, but I guess there's no virtue signaling points to be harvested from making that observation.
It seems my praise was hurting you. Let me praise some people from an unrelated project: we owe a lot to the greybeards keeping Scribus alive. And you are right, we owe a lot to all kinds of people.
To quote myself from another comment: «I am inclined to rather point out remarkable things than unremarkable things. There seems to be relatively many trans people in this niche.» That's all. I didn't know this required a trigger warning.
> virtue signaling points
I didn't know about them but you do. Can I sell them to you?
I see all the comments about how Windows and Linux handle OOM, but I suppose I'll also mention macOS.
On my M4 Max Mac Studio if I try to load too big of an AI model with protections off, the desktop starts glitching back and forth between the past few hundred frames. It looks bad when it happens but CtrlC still works to kill llama.cpp and if you were using LM Studio, SSH also works. Once you kill the offending process, or if macOS does it for you, the desktop comes back immediately.
On my M4 Max MacBook Pro, if I run or even load an LLM more than a couple times, even a small one, general desktop performance starts to get really, really bad until a reboot and I don't know why.
Genuinely, everything starts feeling super sluggish, even long after the process has terminated -- cmd+tab gets slow, apps constantly hang for a few seconds at a time, the cursor position starts lagging and videos are slow to pause/play -- and the only way I've found to fix it is to reboot the entire machine. Clearing memory, restarting apps etc. doesn't work.
Don't get me wrong, the performance and QoS system works really well while the model is still in memory, but after a few cycles of loading and unloading, performance just ends up getting bad in general. Doesn't matter if I'm loading it with llama.cpp, MLX, native Metal, if I start using the GPU for machine learning the machine just gets really upset.
This sucks man... macOS is not supposed to need reboots like that, my Intel Mac could run for months on end without slowing down at all and I could push it as hard as I wanted. As far as I can tell, this behavior is new as of either macOS 26 or the 27 beta. Apple really has been betraying their true Unix roots lately.
Same, I loaded a model that should have fit fine in my available memory and macOS progressively got more broken in the weirdest ways, like my AirPods connecting but being unresponsive and not being able to be configured. I think it started killing internal processes that failed to come back.
I am on 15.7.9 and I have had uptimes in the high 50s with weekly local model usage, I even tried GLM5.2 streaming from SSD and I have not noticed any slowdowns after at all. I guess you are right about 26 and 27. Dang, the more I hear about 26 I am just so glad that I did not update. What version of macOS are you on? 26 or 27? Have you considered doing a custom backup and restoring manually to 15.7?
Unfortunately I don't have the spare 8 TB. Been unemployed for 6 months, have not gotten a single offer in that time, + memory/storage crisis, etc. Shit's fucked.
I don't think I would downgrade though. I just hope this gets fixed someday... I can deal with the terrible memory bandwidth but I don't like to reboot.
(That said, macOS is so, so much better at reboots than Windows)
Gosh, 7.2 literally just dropped with a bunch of awesome of performance/gaming related improvements (large folios, cache-aware scheduling, improved MGLRU reclaiming, Fair GPU Scheduler etc)... and I already can't wait for 7.3 to come out.
Meanwhile in the Windows world, users hate updates... Like I genuinely can't think of a single instance that made users exclaim, "oh boy I just can't wait for the next Patch Tuesday!".
I don't use windows too often, but I remember I was happy with an update a year or two ago which allows changing the volume by scrolling on the tray icon.
Now, of course, it still doesn't work with a touchpad, but baby steps, right?
I don't know if reverting features and then users being glad when they implement them again years later is great proof that people "like" Windows updates as a general statement
Not to start a flame war, but people who move their taskbar to the left edge of the screen are extremely sinister and should not be trusted, because they probably use tabs instead of spaces, and vi instead of emacs. Putting your taskbar on the right edge is fine though, as is using spaces and emacs. Putting the taskbar on the top or bottom just means you're gay, not that there's anything wrong with that. ;)
Tabs on the other hand (and of the other type) should be able to move freely between all edges of all windows.
The best thing about Win10 reaching EOL is not being at the mercy of Windows Update any more. It was by far the biggest risk to having a functioning system.
MS should either fix the RCEs in Windows 10 (it is unsupported but the defects were already there when it was still supported, just undiscovered), or they should refund their customers (since they never managed to release a defect-free version).
Windows is such a horrible OS to use when you work in a small company that isn't big enough to have IT to shield you from all the horrible stuff, like in a small studio. Normally I work in Linux, sometimes need to boot Windows just to do some export or something, then goal is to boot into Linux again ASAP. But time and time again fucking Windows forces me to prepping updates when shutting down for next boot, with literally no way around it, even weird terminal incantations can't stop this, nor changes in the registry, it just refuses to shut down without prepping to apply updates on the next boot.
Which means next time I have something to do on Windows that in reality takes ~30 seconds once I'm in, it'll instead take 15 minutes because Windows decides that my time is worth nothing to them.
Add in that I use Windows maybe once a month or something, and every single experience with using Windows for me is this fucked up process of time wasting.
This sounds like the perfect use case for a simple, straightforward VM. Dual booting is dangerous, time consuming, and makes it inconvenient to share clipboard, stuff... why don't you use a Windows VM instead? You can freeze a VM (saving state) rather than restarting it. Windows won't bat an eye since it's not being shut down or suspended. No longer unexpected 15 minutes pauses - restart it when you have time to spare, and keep working in your host Linux OS while the Windows VM does its shenanigans.
Fair point, the old school service packs and hotfix rollups were cool. But I think that sort of enthusiasm towards updates ended with XP, at least it did for me, because I switched to Linux permanently after Windows 7 came out.
XP was indeed when I stopped caring about Windows updates, SP2 was a huge deal - after that they where either "eh" or actively things I didn't want.
To be honest I think the reason I stopped care was more that I started using Linux as an OS in the 90's and over time more and more of my computing life was on Linux (except gaming) so by 2004 (or a little earlier) Windows was just for games.
Being able to preview media files directly in the preview pane of explorer was mind blowing to me. This feature alone made me love ME, let alone the interface customisation it allowed.
We did, and there was some cool stuff coming out of MS now and then. I remember when the first Power Toys came out - there's nothing really magic in current ones though.
I might be in the minority here, but some of the latest Windows updates that dropped in the Insider channel are really cool, and I'm excited to see those coming to main.
They seem to be focused on performance improvements because of the MacBook Neo pressure and RAM crisis, but right now, I'm mostly excited for the right-click menu and taskbar improvements. You can already do this with third-party software, but it's not the best experience unfortunately.
Are they finally going to fix the issue of having introduced a second right click menu that forces me to click “show more options” literally every single time I want to do anything?
Sadly I’m in a locked down corporate environment, so I can’t edit the registry. IT even closed my ticket when I asked for them to make the edit on my behalf.
I don’t use Windows on any personal machines, so there’s no way for this to benefit me. I hope it helps others who are frustrated by this though!
I'm not so annoyed by it except that I liked the old design (smaller text) better. What puzzles me more is why it takes >1000ms to open a simple right click menu? And even after it opening, it often has to "Load" some options? In what world does it take over a second to open a simple popup on modern hardware?
What, you don't find it charming that they shoehorned in the old right click menu without even changing its appearance to match the rest of Windows 11? It really shows how much thought Microsoft is putting into Windows now.
Phoronix.com is my go to place for good news. It always gives me a sense of progress and makes me feel humble since so many smart people share their work
Any improvements with fractional scaling? Can anyone please link to any material reg the work being done for improving fractional scaling in Linux; it would be great if it could match the rendering quality of macOS.
Also, I wouldn't hold the rendering quality of macOS as a high standard at all, since all it does is render at a multiple and then downsample, wasting energy and introducing blurriness.
Perhaps it's subjective, I have multiple portable displays including a e-paper monitor and I find that for low resolutions the HiDPI in the macOS produces more legible text in those monitors.
In my laptop 200% fractional scaling of the built-in QHD+ OLED display on KDE Plasma is great, but when I connect my portable 1080p displays or e-paper monitor (UXGA) the fractional scaling of it is not as good as when compared to when the same monitors are used with macOS (M4) in HiDPI mode under lower resolutions.
As the other comment says, I suspect it to do with GPU as the built-in displays are usually driven by intel and external displays are always driven by my Nvidia 4090.
But surprisingly, when I drive my e-paper monitor using original Jetson Nano (Ubuntu 20.04 ESM / Nvidia Maxwell); the fractional scaling is quite good perhaps even better than macOS.
IME it works fine on Intel iGPUs, too. I even use multiple ratios with no issue (constant 125% on the laptop, 100% on an external screen and 200% on a different screen at work).
Contrary to Windows, this also works well when hotplugging screens. Menus don't end up a blurry mess, and apps seem to work fine. On Windows, even some 1st-party apps are borked (task manager comes to mind).
`gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer', 'xwayland-native-scaling']"
`
That stops being necessary as some point around 49/50 I think.
I use arch based linux. This year updates have broken video decoding twice. Font rendering once (still not fixed without manually deleting files.). And broken the whole system due to there not being a mechanism that makes sure all QT libraries upate at the same time once or twice. That is all one system.
Meanwhile 4 separate windows pcs have not had problems with updates for at least 6 years
It's not fair comparing Arch and Windows as they are two very different types of OSes. Arch is bleeding edge and it is completely normal and expected to be prone to breakages. It's not meant for the average PC user, it's meant for experienced users who're comfortable with troubleshooting technical issues, reporting bugs, and know and are happy to downgrade packages/rollback to previous snapshots etc. On the upside, Arch also gets fixes before anyone else.
If you want a stable Linux experience though, pick something like Fedora, or even better, pick an immutable distro like Aurora or Bazzite, if you really, really care about stability.
I use Arch exclusively and haven't had any of the issues you describe.
That said, I do occasionally deal with quirks due to the rolling distro nature of Arch. The benefits far outweigh the occasional upgrade pain IMO though.
How would you detect broken video decoding or font rendering? I don't know what those are. I don't think i've had any issues this year (Maybe i had do do some pacman-wrangling once this year, but i'm slow to update, so that was probably my fault...)), that I noticed anyway, but maybe everything's broken and i just can't tell.
A curious thing happens in my Ubuntu 24 system, maybe that's just the normal OoM killer algo but no matter what process is hogging, Firefox is killed. Chrome eating too much memory? Firefox dies. Too many apps open? Firefox dies. Its funny.
I doubt it makes much of a difference, and you can always manually manage what data lives in the GPU when if you 100% have to overcommit. Games have a much larger and more diverse set of objects in the VRAM, and their usage is less predictable, so manual scheduling of the memory is infeasible typically.
If you have model layers/experts that reside in CPU RAM, it's generally better to do that part of inference on the CPU than pay the cost of shipping them over to the GPU. LLM inference is generally bottlenecked by data-movement, not compute. There may be a limited exception for prefill or perhaps decode of very wide batches, where shuffling the data around may be justified.
Great article. I find that I learn something every time I read a post about linux kernel work.
I guess an LRU with priority would handle VRAM for games pretty decently without going getting too application specific.
What about VRAM to Disk specifically NVME, would direct to disk be feasible for large workloads, I know it is used for streaming in assets directly via. PCIE, but i wonder how the performance would be on compute workloads running with NVME as a swap for GPU VRAM.
This post focuses on performance, but what about crashing due to apps seeing no RAM available to allocate once VRAM is filled up?
I remember having this problem when I used an NVIDIA 750 Ti with 2 GB of memory. Just opening a few Firefox or Ghostty terminals (both are GPU accelerated) would result in those apps crashing or not being able to render their contents. Sometimes the compositor would crash or become unstable too (kwin, niri, etc.). I had to reboot every few hours. Complete system instability.
This problem only happened with Wayland, when I switched to X windows these problems went away.
Since then I switched to an AMD RX 480 (8 GB) GPU and never had an issue here with Wayland, if GPU memory gets full, system memory gets seamlessly used instead. It felt like with the NVIDIA card, it never allocated back to system memory if no VRAM was available. This was only a problem on Linux since that same NVIDIA card was fine in Windows for 10+ years with zero issues related to instability around VRAM allocation.
Tons of people reported the same issue on Linux on NVIDIA's forums for years.
It's got to be an edge case for Wayland to be used with a 2GB VRAM card? Targeting recent cards and letting people using older hardware install x-windows, or whatever, seems sensible.
You say (in your blog post) "Linux" didn't work well so you reverted to "Windows", but as is so often the case you buried the distro/version info. You don't seem to have been using Win11 (later you mention Win7 & Win10Pro) ... So it's cutting edge software didn't work on my 2014 system, but older software did. The Linux/Windows aspects aren't particularly relevant AFAICT.
Nvidia refused for many years to properly support Linux, whilst AMD have supported it. So no surprise if a Nvidia card didn't work well/stably but an AMD one did.
> You say (in your blog post) "Linux" didn't work well so you reverted to "Windows", but as is so often the case you buried the distro/version info.
Not sure what you mean. The post directly mentions I'm using Arch and was using the official drivers from NVIDIA (the latest at the time). I also didn't switch back to Windows. Although now the experience is much better with an AMD card. Totally worth the switch.
As for 2 GB on Wayland, I think it affects other cards too, but a low GB card feels it more. If you have 8-16 GB of VRAM you might not ever notice it's a problem since you have enough of a buffer to open up many more things.
Your last sentence sums it all: AMD is the only serious option on Linux. Not perfect, but far better than Nvidia, thanks to their open drivers that anyone (and especially Valve) can improve.
Just search KWin Nvidia memory leak or Wayland Nvidia memory leak or kwin wayland Nvidia memory leak. It’s constantly issues and I remain convinced some intermixing of the two + monitors is at fault.
I gave myself 32GB of swap since SSDs are quite fast and the pages sent to disk seem to intelligently be rarely referenced ones. Feels amazing to run 4 VMs and 12 Claude Code instances on my mid tier hardware
Can you explain a little more please. I thought that graphics processing would revert to CPU if the GPU is maxed out and so swap might ultimately help? Is there no general GPU-swap mechanism? If CPU threads are competing for GPU resource does the GPU getting paged-out?
I suppose this could be an historic thing, GPUs can have a type of DMA now, and disk access is much faster too, so paging perhaps makes sense when it didn't before?
AMD's GPU do use system memory seamlessly as a fallback, but NVIDIA's don't, at least not back in January 2026. Hundreds of people reported it in their forums too, or variations of the problem.
Great writeup, gpuvis looks particularly interesting and glad the kernel is providing tracepoints for performance events.
> Not only does the display hardware like scanned-out images to be in VRAM, it also completely skips past the GPU’s virtual memory architecture and works with physical addresses exclusively.
Well there's your problem. Only so smart your memory management can be when you have to pay the cost of doing it manually. Although presumably this only applies to a small fraction of the VRAM?
> Although presumably this only applies to a small fraction of the VRAM?
They did mention they saw 4GiB of eviction for a single 32MiB scan out image.
So while I would call the image allocation small, it seems to cause an avalanche of evictions. Amplified by the fact that each frame has one of these images, though I expect subsequent frames might have a better chance of already fitting into evicted space.
What I don't exactly understand is: doesn't it make sense to always reserve the contiguous physical memory for this case and not allow anything else to be put in it?
Or alternatively, instead of evicting it all, can you move the data around in physical memory while updating the page table, so that you can clear a large enough continuous block of physical memory?
Such great work!
Happens quiet regularly I hit a game just realizing seconds later I haven't shut down my local LLM yet. At least punishment will be less harsh. Lets see, maybe some games in window mode will just work fine.
I worked on a PC/PS4/Xbox One game once. I wish I remembered more details, but at some point when looking at memory usage I found a 2048x2048 (or something like that) texture for something on a character. This texture was never anywhere near occupying the full screen. It got downsized once I told one of the artists, but I wouldn't be surprised if some games are inadvertently shipping with that kind of waste.
The release version of City Skylines 2 had enormous issues due to this problem. Some tiny details, like a pile of wood behind a shed, had ridiculous amount of pixels. I do believe that should be pretty much fixed in that title nowadays, however.
This is a nice blog and it makes sense to me now. As a gamer and linux user myself, I've previously had to do tweaks and go-arounds without really understanding what was going on behind the scenes. :)
Honestly, swap thrashing on low vRAM setups has been painful forever. Glad to see Linux kernel optimizations targeting memory pressure where it actually hurts.
When I run in a frozen system because of RAM, I'm a happy user of the Magic SysRq 'f' [2], it bypasses most kernel scheduling so it works even on frozen systems
[1]: https://man.archlinux.org/man/systemd-oomd.8
[2]: https://en.wikipedia.org/wiki/Magic_SysRq_key
Docs https://docs.kernel.org/admin-guide/mm/multigen_lru.html#thr...
I don't use Windows much, nor do I care much about that environment, but from what I've seen it seems to keep RAM usage below 100% most of the time. What I do see pretty often is the drive getting stuck at 100% usage instead, which makes the whole system ridiculously unusable anyway.
But Windows has always handled both OOM and out of disk space very well. The system will be extremely sluggish, but it typically continues operating.
The Linux design is to keep using memory, then push to swap, and then when you OOM you hard lock. The built in kernel OOM module can miss when RAM usage spikes rapidly. You can enable OOM monitors like systemd-oomd or earlyoom, but they do not run by default, and their behavior is to term the offending process.
The thing about Windows is that when you request memory, you're only granted memory if it can be guaranteed in the first place. The application will get a "not enough memory" error. On Linux, you're permitted to request more memory than the system actually has, with the idea that you won't actually use that much memory. It's optimistic that way. But if you do use it, then you're screwed.
sysctl -w vm.overcommit_memory=2 sysctl -w vm.overcommit_ratio=50
It is probably a bad idea to do that though as it will limit the amount of virtual address space an application can use
Running out of memory on a linux server sounds more like a skill issue than something comparable to how a desktop OS should behave :\
Linux uses lazy allocation and overcommits, so what dies isn’t necessarily what you would expect thanks to the OOM killer.
With a fixed swap size windows doesn’t necessarily slow down as it runs out of memory.
YMMV if you don’t have 64 gigs of RAM and a 64 gig swap.
https://kubernetes.io/docs/concepts/scheduling-eviction/pod-...
https://kubernetes.io/docs/concepts/configuration/manage-res...
https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#... (New feature)
So normally if you overcommit it might be undefined behavior if you not have enough memory. However if everything is fine pods can preempt based on prio to new nodes.
With the right settings only the faulty app might fail. Without it it can be a real pain to find the rouges
Chrome/Vivaldi are the only browsers where, when i have a long HN page open, when i resize it, my laptop freezes for like a minute. Not a full freeze, the mouse still moves, but like it's the year 1999 with a refresh rate of 4 seconds.
Before the recent increase in RAM prices, it was not difficult to have a generous amount of RAM in a PC, that would make very unlikely the out-of-memory situations.
When OOM really happened, it was much better to become aware of this immediately, and reduce the number of concurrent threads for a running job, or whatever caused that, instead of having reduced performance or even freezing.
Maybe not in the same way then.
Anyways, you've unlocked an old memory of mine, where Windows would crap itself due to low RAM and the fonts and interface elements would render in a "compatible" way, like using "fixedsys" font everywhere.
But only in veeery old Windows, think Win95B or the like.
But yeah it’s completely ridiculous and frustrating that the default is to just lock up your DE
If you disable pagefile (like I did for some of my servers) and your program mallocs more than what you have in available space (even without ever dirtying a page) you will observe this.
I'm sure that this helps Windows a lot, even if it's not counted as actually used until the page is dirty.
Linux programs very often have virtual addresses many times the amount of physical ram, because there's never been any restriction. It's then very easy to just malloc huge chunks and use what you need and don't care about it too much.
Especially with many "tiny" allocations, python for example has huge sized objects which consume gobs of RAM dynamically, so any long running python process not only fragments memory but ends up having a bunch of objects consuming virtual memory...
idk why I felt the need to rant about this, but it's a difference that I've noted.
I prefer the Windows approach, it's more predictable and has better behavior under memory pressure. But it can cause issues with software written under the assumption that the OS uses memory overcommit
https://www.kernel.org/doc/html/latest/mm/overcommit-account...
all these electron apps and invisible virtual machines (looking at you Claude) really don't help.
The communication is more direct too with GPT5.6 family of LLM. No more belt and suspenders.
What’s a decent amount? No idea. Don’t test it :)
The only way to save my file system was to boot into recovery mode and run a verification with Disk Utility, then I was able to remove some files and get my space back.
As a side note, it strikes me how much we owe to young trans people for low level performance engineering.
The commits they authored in [0] match and all of what they have mentioned in the article and this is how you know the author knows exactly what they are talking about and have explained it well.
Puts all the vibe-coders, and the so-called "AI Engineers" to shame. I know that if I ask about the basic system design of amdgpu and drm (direct rendering manager) infrastructure in Linux, a small handful will be able to explain it without using AI or googling.
[0] https://gitlab.freedesktop.org/pixelcluster/kernel/-/commits...
wtf?
If I see a soil that is unusually rich in iron, I would be more inclined to point that out. If I come across a piece of soil that is as average as the 98% of the surrounding lands I wouldn't be less likely to point out "hey look, this soil is unremarkable average."
Also, it was just an aside from me. It is unclear what your problem is exactly, and I am also not sure if a tech forum could provide counsel for that.
To quote myself from another comment: «I am inclined to rather point out remarkable things than unremarkable things. There seems to be relatively many trans people in this niche.» That's all. I didn't know this required a trigger warning.
I didn't know about them but you do. Can I sell them to you?On my M4 Max Mac Studio if I try to load too big of an AI model with protections off, the desktop starts glitching back and forth between the past few hundred frames. It looks bad when it happens but CtrlC still works to kill llama.cpp and if you were using LM Studio, SSH also works. Once you kill the offending process, or if macOS does it for you, the desktop comes back immediately.
Genuinely, everything starts feeling super sluggish, even long after the process has terminated -- cmd+tab gets slow, apps constantly hang for a few seconds at a time, the cursor position starts lagging and videos are slow to pause/play -- and the only way I've found to fix it is to reboot the entire machine. Clearing memory, restarting apps etc. doesn't work.
Don't get me wrong, the performance and QoS system works really well while the model is still in memory, but after a few cycles of loading and unloading, performance just ends up getting bad in general. Doesn't matter if I'm loading it with llama.cpp, MLX, native Metal, if I start using the GPU for machine learning the machine just gets really upset.
This sucks man... macOS is not supposed to need reboots like that, my Intel Mac could run for months on end without slowing down at all and I could push it as hard as I wanted. As far as I can tell, this behavior is new as of either macOS 26 or the 27 beta. Apple really has been betraying their true Unix roots lately.
I am on 15.7.9 and I have had uptimes in the high 50s with weekly local model usage, I even tried GLM5.2 streaming from SSD and I have not noticed any slowdowns after at all. I guess you are right about 26 and 27. Dang, the more I hear about 26 I am just so glad that I did not update. What version of macOS are you on? 26 or 27? Have you considered doing a custom backup and restoring manually to 15.7?
I don't think I would downgrade though. I just hope this gets fixed someday... I can deal with the terrible memory bandwidth but I don't like to reboot.
(That said, macOS is so, so much better at reboots than Windows)
But my GLM5.2 is not 8TB, it is only around 200GB and works pretty well. (0.9tok/sec from a 40Gbps NVMe)
I used https://huggingface.co/anemll/GLM-5.2-sidecar
Meanwhile in the Windows world, users hate updates... Like I genuinely can't think of a single instance that made users exclaim, "oh boy I just can't wait for the next Patch Tuesday!".
Now, of course, it still doesn't work with a touchpad, but baby steps, right?
I cant wait for the number one gripe of all my users to finally be catered for.
Tabs on the other hand (and of the other type) should be able to move freely between all edges of all windows.
https://news.ycombinator.com/item?id=39432170
Which means next time I have something to do on Windows that in reality takes ~30 seconds once I'm in, it'll instead take 15 minutes because Windows decides that my time is worth nothing to them.
Add in that I use Windows maybe once a month or something, and every single experience with using Windows for me is this fucked up process of time wasting.
To be honest I think the reason I stopped care was more that I started using Linux as an OS in the 90's and over time more and more of my computing life was on Linux (except gaming) so by 2004 (or a little earlier) Windows was just for games.
I still remember how Windows ME looked at first, the feeling of something better. That didn't last long
https://en.wikipedia.org/wiki/Microsoft_Plus!
They seem to be focused on performance improvements because of the MacBook Neo pressure and RAM crisis, but right now, I'm mostly excited for the right-click menu and taskbar improvements. You can already do this with third-party software, but it's not the best experience unfortunately.
So much of windows 11 configuration is removing things you never asked for and putting things back.
Source: https://learn.microsoft.com/en-us/answers/questions/2287432/...
I don’t use Windows on any personal machines, so there’s no way for this to benefit me. I hope it helps others who are frustrated by this though!
https://wayland.app/protocols/fractional-scale-v1
Also, I wouldn't hold the rendering quality of macOS as a high standard at all, since all it does is render at a multiple and then downsample, wasting energy and introducing blurriness.
As the other comment says, I suspect it to do with GPU as the built-in displays are usually driven by intel and external displays are always driven by my Nvidia 4090.
But surprisingly, when I drive my e-paper monitor using original Jetson Nano (Ubuntu 20.04 ESM / Nvidia Maxwell); the fractional scaling is quite good perhaps even better than macOS.
Contrary to Windows, this also works well when hotplugging screens. Menus don't end up a blurry mess, and apps seem to work fine. On Windows, even some 1st-party apps are borked (task manager comes to mind).
Gnome >= 47 is fine, you will need to do:
`gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer', 'xwayland-native-scaling']" ` That stops being necessary as some point around 49/50 I think.
Meanwhile 4 separate windows pcs have not had problems with updates for at least 6 years
If you want a stable Linux experience though, pick something like Fedora, or even better, pick an immutable distro like Aurora or Bazzite, if you really, really care about stability.
That said, I do occasionally deal with quirks due to the rolling distro nature of Arch. The benefits far outweigh the occasional upgrade pain IMO though.
Majority of Windows users are not programming enthusiasts and just don't care.
OS works. Programs continue to work. Good. Keep it that way.
What does this mean for compute workloads? Specifically, LLM inference.
Does it mean anything at all, or is this purely a games-thing?
[1]: https://github.com/rfjakob/earlyoom
[2]: https://man.archlinux.org/man/systemd-oomd.8
Just be careful where you click
[1]: https://en.wikipedia.org/wiki/Xkill
I guess an LRU with priority would handle VRAM for games pretty decently without going getting too application specific.
What about VRAM to Disk specifically NVME, would direct to disk be feasible for large workloads, I know it is used for streaming in assets directly via. PCIE, but i wonder how the performance would be on compute workloads running with NVME as a swap for GPU VRAM.
4x as slow in absolute best case, NVMe drives have 4 PCIe lanes usually
I remember having this problem when I used an NVIDIA 750 Ti with 2 GB of memory. Just opening a few Firefox or Ghostty terminals (both are GPU accelerated) would result in those apps crashing or not being able to render their contents. Sometimes the compositor would crash or become unstable too (kwin, niri, etc.). I had to reboot every few hours. Complete system instability.
This problem only happened with Wayland, when I switched to X windows these problems went away.
Since then I switched to an AMD RX 480 (8 GB) GPU and never had an issue here with Wayland, if GPU memory gets full, system memory gets seamlessly used instead. It felt like with the NVIDIA card, it never allocated back to system memory if no VRAM was available. This was only a problem on Linux since that same NVIDIA card was fine in Windows for 10+ years with zero issues related to instability around VRAM allocation.
Tons of people reported the same issue on Linux on NVIDIA's forums for years.
I wrote a lot more details and recorded demo videos around 9 months ago here https://nickjanetakis.com/blog/gpu-memory-allocation-bugs-wi..., but since then a lot of these things have been resolved from switching over to AMD.
You say (in your blog post) "Linux" didn't work well so you reverted to "Windows", but as is so often the case you buried the distro/version info. You don't seem to have been using Win11 (later you mention Win7 & Win10Pro) ... So it's cutting edge software didn't work on my 2014 system, but older software did. The Linux/Windows aspects aren't particularly relevant AFAICT.
Nvidia refused for many years to properly support Linux, whilst AMD have supported it. So no surprise if a Nvidia card didn't work well/stably but an AMD one did.
Not sure what you mean. The post directly mentions I'm using Arch and was using the official drivers from NVIDIA (the latest at the time). I also didn't switch back to Windows. Although now the experience is much better with an AMD card. Totally worth the switch.
As for 2 GB on Wayland, I think it affects other cards too, but a low GB card feels it more. If you have 8-16 GB of VRAM you might not ever notice it's a problem since you have enough of a buffer to open up many more things.
Here’s one for example: https://forums.developer.nvidia.com/t/kwin-6-7-x-causes-mass...
Just search KWin Nvidia memory leak or Wayland Nvidia memory leak or kwin wayland Nvidia memory leak. It’s constantly issues and I remain convinced some intermixing of the two + monitors is at fault.
It's GPU drivers freaking out when they run out of VRAM, but it might only be specific to NVIDIA cards since AMD seems to handle this better.
But, if it's driver specific that means it could be fixed at the kernel level I suppose, especially since Windows handles it seamlessly?
I suppose this could be an historic thing, GPUs can have a type of DMA now, and disk access is much faster too, so paging perhaps makes sense when it didn't before?
AMD's GPU do use system memory seamlessly as a fallback, but NVIDIA's don't, at least not back in January 2026. Hundreds of people reported it in their forums too, or variations of the problem.
> Not only does the display hardware like scanned-out images to be in VRAM, it also completely skips past the GPU’s virtual memory architecture and works with physical addresses exclusively.
Well there's your problem. Only so smart your memory management can be when you have to pay the cost of doing it manually. Although presumably this only applies to a small fraction of the VRAM?
They did mention they saw 4GiB of eviction for a single 32MiB scan out image.
So while I would call the image allocation small, it seems to cause an avalanche of evictions. Amplified by the fact that each frame has one of these images, though I expect subsequent frames might have a better chance of already fitting into evicted space.
What I don't exactly understand is: doesn't it make sense to always reserve the contiguous physical memory for this case and not allow anything else to be put in it?
Game developers often aren't super careful with this stuff, and there could be literal gigabytes of data which isn't used at all.