Handling OOM livelocks is exciting, and they have a good explanation for why the current OOM-killer fails:
> One usecase is avoiding OOM hangs/livelocks. The reason these happen
is because the OOM killer is triggered by reclaim not being able to
free pages, but with fast flash devices there is always some clean
and uptodate cache to reclaim; the OOM killer never kicks in, even as
tasks spend 90% of the time thrashing the cache pages of their own
executables. There is no situation where this ever makes sense in
practice.
After reading it, I realized I was actually hoping for information at a lower level than VM for memory pressure. Actually finding live, actionable information about DRAM bandwidth usage, delays caused by the hardware system of caches including TLBs, L1/2/3 caches and main memory contention, etc. I have not found that existing tools are insufficient in monitoring/dealing with VM swapping - OTOH I usually seek to keep that at zero and leave a little swap just to allow for some chance of alerting and recovery before OOM killer kicks in.
How much time does each CPU core spend running/waiting for L1/L2/L3/DRAM? How often are those stalls due to cross-core contention for the same bit of memory? Which execution units are in use? What's the limiting factor on throughout?
That's what I can think of in the first two minutes, anyhow. It all comes down to the last one.
Curious that there is no mention of the existing "memory" cgroup. On some desktop Linux, you'll find it here:
ls -l /sys/fs/cgroup/memory/
The 000-permission 'pressure_level' file controls asynchronous notifications to apps, advising prompt shedding of load. This is apparently the mechanism alluded to in a Googler's recent blog post, writing from the point of view of Go server coding: https://news.ycombinator.com/item?id=17551012
> One usecase is avoiding OOM hangs/livelocks. The reason these happen is because the OOM killer is triggered by reclaim not being able to free pages, but with fast flash devices there is always some clean and uptodate cache to reclaim; the OOM killer never kicks in, even as tasks spend 90% of the time thrashing the cache pages of their own executables. There is no situation where this ever makes sense in practice.
After reading it, I realized I was actually hoping for information at a lower level than VM for memory pressure. Actually finding live, actionable information about DRAM bandwidth usage, delays caused by the hardware system of caches including TLBs, L1/2/3 caches and main memory contention, etc. I have not found that existing tools are insufficient in monitoring/dealing with VM swapping - OTOH I usually seek to keep that at zero and leave a little swap just to allow for some chance of alerting and recovery before OOM killer kicks in.
That's what I can think of in the first two minutes, anyhow. It all comes down to the last one.
https://github.com/facebookincubator/oomd
There was also more minimal proof-of-concept example posted by Endless OS guys:
https://gist.github.com/dsd/a8988bf0b81a6163475988120fe8d9cd