I had an idea to write a filesystem filter driver which would let you configure path remapping rules of sorts, depending on the application. Things like:
- %userprofile%\.vscode -> %appdata%\vscode
- %CSIDL_MYDOCUMENTS%\Call of Duty -> %userprofile%\Saved Games\Call of Duty
Because my documents and home directories filling up with a bunch of garbage which has a designated place on the filesystem filled me with impotent rage. I scaffolded out a project to write a filter driver in rust, read through the minifilter documentation, realized how much work it was going to be, and gave up.
I have made my peace with the fact that a windows system is just going to be filled with garbage.
Windows is not the only operating system treating the user's filesystem as a dumping ground. I don't know how many times I keep deleting .DS_Store, .fseventsd and extra files named ._xxxx but that doesn't keep Apple from dumping them all over my filesystem.
At least macOS has one place to install applications, one place for the user's documents, and most apps somehow respect them. Then, you have designated dumping grounds like ~/Library that contains tons of junk I have no idea whether I need or not.
There are basically two ways to organize a file system, by subject and by purpose.
Linux prefers by-purpose organization, so executables go in /bin (or at least something ending in /bin), configuration goes into /etc, libraries go into /lib and so on. This simplifies discovery and cross-application data management, the answer to "how do I find a library" is just "search those 5 directories, if it's not there then it's missing", but it complicates package management and makes it too easy for garbage to accumulate. It's also better for GUI apps, it's far easier to make a music app that lets you think of artists, albums and playlists, not directories, if all your music is in one place.
Mac and Windows (mostly) prefer by-package organization, where everything related to a single application is contained inside that application's folder. This makes discovery harder, as now e.g. finding all applications that can open a particular file format requires you to check every single one. However, it makes package maintenance easier, removing an app just removes all its files. This approach complicates GUI applications and makes it impossible to escape the directory hierarchy, you can no longer make a nice music app if music files are strewn all around.
No system clearly falls in one bucket or the other, the Windows registry is partially organized by purpose, so are parts of Mac OS's Library folders, while Linux has package managers like Nix, which organize by package.
Ideally, we'd do by-package organization, and have some sort of per-user / per-process "union directories" (a bit like the /bin in Plan9) that would provide easy lookups for downstream applications.
Flatpak is winning on Linux because when every app has its own dependency chain of packages it becomes totally impossible to run older versions of an app or to run multiple versions side-by-side. Proper sandboxing is way too hard when files get strewn all over your system. rpm/deb packages only work for utilities and core libraries where you're happy to run whatever the latest version is. For everything else you need some combination of containerization/virtualization for security and robustness.
There are several ways of accomplishing this in user mode.
The Detours library lets you attach a custom DLL during the launch of any process, intercepting Win32 calls to file system APIs by writing a custom hook.
There's the built-in "compatibility shims", which are borderline undocumented, but they work similarly. They can toggle compatibility flags and redirect file and registry paths. Conveniently, these are designed to trigger only for specific EXEs based on heuristics.
To support technology like App-V and later Docker containers, Windows has an entire API surface for "virtualising" the filesystem and the registry, as well as any other NT kernel namespace.
Then there's the User Mode Filesystem and probably more approaches I'm not aware of or that I forgot about...
Can you describe those options further or send references to where I can read more, specially about the API for App-V you mention? I was looking, like OP, into writing a minifilter to overlay files and folders for game modding purposes, mostly (I know of Detours - but I think injecting is somewhat dirty), but I'd rather it be done in usermode given driver signing costs and all.
You're looking for AppCompat shims. I used them to make old applications work on newer versions of Windows (in the context of educational programs that K-12 teachers wanted to keep as they moved from Win9X to NT-based Windows). Here's a decent starting place: https://techcommunity.microsoft.com/blog/askperf/demystifyin...
Thanks. A looong time ago I looked into accomplishing it using AppContainers but that approach looked even more confusing and difficult than the kernel driver.
I just completely ignore the My Documents toxic waste dump, and use a different place (with a shorter path) for files I actually care about. I might have created directory junctions to redirect one or two programs that hardcode their data location.
You have to go pretty out of your way as a user to actually see those transaction logs though vs what GP is talking about with blatant pollution the user is actually expected to see and interact with.
These files correspond to the HKCU registry hive. There is really no better place to put them, because even the location of %LOCALAPPDATA% and %APPDATA% can be controlled... in the registry.
Of course there is a better way, don't give up so easily. One immediate improvement - use a subfolder for organization, so just 1 "garbage" item instead of many.
Funnily enough, Windows itself does stuff like this to allow applications written for Win9x (where e.g. Program Files were writable, so many apps would create files in their install folder). This is what the stuff in ~/AppData/Local/VirtualStore is.
Marking files as hidden seems easier on Windows. I find it much more annoying to hide files on Linux (especially folders like ~/snap which can't be renamed).
If there's anything I've heard about drivers, though, it's that filesystem drivers are particularly annoying to write.
If you want to simply hide the files from view, you may have an easier time writing a shell plug-in. You have to deal with COM, but you're less likely to take down the whole system.
I wish there was something like that for Linux so programs would stop littering my home directory with stuff that should clearly go into .config or .local.
> I have made my peace with the fact that a windows system is just going to be filled with garbage
Or, you know, create a "%USERPROFILE%\Desktop\Actual Documents" folder and set that as the default open/save location in the applications you care about?
The pollution of the "My Documents" folder is unfortunate, yet nothing new, and not exactly limited to Windows (I mean, even on supposedly-perfect MacOS, I have to tolerate various game save crap under 'Documents', not to mention various 'helpful' vendor detritus), and not something that you would want to write a device driver for.
(That being said, hello Rust people, welcome to the Windows kernel! Personally, I've been enjoying the freedom to write C# drivers for years, with a minimal C++ wrapper, and I'm looking forward to the fruits of your creativity!)
Their point is that the files shouldn’t be there in their home directory in the first place. Your approach would still leave symlinks/junctions in their homedir. Their proposed driver-based approach would allow apps to think they’re writing to your homedir when they’re actually writing under appdata, leaving the homedir clean.
> - %CSIDL_MYDOCUMENTS%\Call of Duty -> %userprofile%\Saved Games\Call of Duty
That is a fight not worth fighting. Even Microsoft does not give a care in the world when deciding on storage locations and they put files directly in your %USERPROFILE% folder. Often also naming them like Linux/Unix dotfiles. I have opened tickets, pointing them to Microsoft documentation clearly stating that this is not how it should be, but they just don't care.
Still, it is infuriating seeing so much willful ignorance, when choosing the correct location is a decision just as quick as choosing the wrong location.
Apps are terrible at this on Windows. Sometimes it's in %USERPROFILE%, sometimes in one of the three %appdata% folders, then My Docs, other times in %ProgramData%, and even worse, %PUBLIC%.
There's a registry for a reason, but even Microsoft doesn't use it for x-platform apps.
The only way I can see it getting solved is the OS sandboxing everything behind the scenes, redirecting everything the application spews in random locations to The One True Profile Folder per application. Even then it wouldn't be without challenges, off the top of my head some method if you actually wanted an application to get out of its sandbox and look at the real documents folder, another method for App1 to look at files in App2's sandbox (and detecting when this is desired), or what to do if an application is able to change how it identifies itself to the OS (new versions?).
The registry hasn't been the recommended place for Win32 app settings for well over two decades now. The guidance is to only use it for system stuff that's already these, and have config files in the standard designated locations under %AppData% (depending on whether they are machine-specific or user-specific; ideally, apps are supposed to support both).
I was replying to the portable application part. If the registry were accessible through the filesystem interface that might be easier to deal with through standard APIs. Like a piece of python has no problem reading from procfs or sysfs on linux.
What is meant is that instead of ~/.app/config, you are getting now ~/.config/app/config.
It’s much cleaner and it comes with a more logical and useful split of config and cache stuff, e.g. you can just skip .cache when rsync-ing to a remote storage instead of having a long list of per-app escapes…
Not really. Now when I want to uninstall something, I have to look in multiple places for its artifacts, rather than just nuking .<appname>. And there is no universal pattern of use. It's even worse than /usr vs /opt.
This is an issue that really can't be fixed imo.
If use the scheme "app/storage" then migration/deletion of singular apps is easy, but if you want to touch just a part of every program (e.g. delete the cache) then its filled with the same tedium you were initially complaining about with regards to uninstalling.
On the other hand, the scheme "storage/app" allows you to "nuke" the cache for every installed program without touching the config at all (while lessening the ease of complete removal).
As I don't usually remove programs (or install new ones, for that matter) the scheme that works best for me and many others is "storage/app". In an ideal world a hybrid scheme might be possible, but as it stands now it is not an option and one way has to be chosen over the other.
I actually really like the way git does it, where it reads each of these in order, last one wins.
- Default configuration compiled in
- Global configuration
- Per-user configuration
- Per-project configuration
You can opt-in to however much configuration complexity you need. Just cloning the occasional thing? Don't bother configuring, the global defaults are probably good enough. Simple commits and pushes? Make a user-level dotfile in your $XDG_CONFIG_HOME to set some basic info. Complex per-project aliases? If you want that, go ahead and opt in.
Contrast that with programs that just dump their whole default config into my home dir on first run. Just filling up with nonsense, often no way to tell what I changed or didn’t.
That is good one, I agree. I have seen some odd cases, where the configuration options were hided too well when they were compiled to the binary itself. But that is probably an another issue.
Hardly makes a difference when the outcome is the same.
Also no one edits text files by hand, we use tools for that, tools that manipulate what is anyway a stream of bytes, whose meaning is given by whatever tool is used to manipulate such stream of bytes.
I haven't heard anything about the kernel but they're still trucking along with using Rust for other sensitive low-level stuff, most recently their secure enclave firmware:
Interesting! This looks very different from embedded drivers, which I've done a lot of in rust. Those are mostly reg reads, writes, bit shifting, DMA, and data sheet references.
The code here looks to be essentially C with different syntax - every function marked unsafe, all resources manually managed. Sorry to be blunt, but what's the point of this?
And I don't know how I missed this, but attempting to use the `UNICODE_STRING` returned in `string_to_ustring` is a guaranteed use after free. If you're interested in writing Windows kernel code, this is not the place to start.
I'm glad to see that Microsoft is investing in Rust bindings for WDK[0], but browsing the repo, there's really no point in using this over C since they haven't bothered to invest in safe, Rust native bindings. The kmdf example[1] is like 50% "SAFETY:" comments because they're stuck using the straight C bindings for every WDK API.
`boost_write` doesn't appear to validate the length of the user supplied buffer before casting and dereferencing either, so that's a kernel-mode OOB read. Not sure how exploitable this actually is though.
Yeah, I totally get that, but I suppose my argument would be is if you're going to bother writing your driver in Rust, until there are more mature Rust bindings for the OS interfaces, you might as well only write your most safety-sensitive business logic in Rust, and then write all the interfacing with the OS in C.
> wdk: Safe idiomatic bindings to APIs available in the Windows Development Kit (WDK)
And then if you look at that crate they've only implemented dbg_break, print, spinlock, and timer.
I'd take this as a sign that the bindings are a work in progress.
If you're going to write your driver in Rust... honestly I'd recommend just writing the `wdk` crate bindings you need for your driver and probably even upstreaming them. Wrapping FFI bindings in safe abstractions is usually pretty easy - but it's definitely the case that without them rust doesn't give you much.
This sample is using `wdk_sys` bindings directly without wrapping them... which is basically never the recommended way of interacting with the FFI in rust.
Also a ton of ALL CAPS TYPES. Are we seriously throwing out all of the standard Rust naming conventions to adopt the ancient Windows naming-as-typing crap?
That doesn't bother me per-se - those all caps names are pretty much all directly from the standard C bindings, and it makes sense to preserve that naming for the sake of having that 1:1 mapping with the ground truth C definition.
The actual issue here is that this "simple driver in Rust" is having to touch those direct C bindings at all - if Microsoft is going to advertise that they have support for writing drivers in Rust, that should presumably mean an API surface that's native to the language.
That way functions can be marked as accurate/"ok" to call in safe code by the author of the bindings. They could absolutely not be safe; in that case, the binding author is in error marking it so.
- %userprofile%\.vscode -> %appdata%\vscode
- %CSIDL_MYDOCUMENTS%\Call of Duty -> %userprofile%\Saved Games\Call of Duty
Because my documents and home directories filling up with a bunch of garbage which has a designated place on the filesystem filled me with impotent rage. I scaffolded out a project to write a filter driver in rust, read through the minifilter documentation, realized how much work it was going to be, and gave up.
I have made my peace with the fact that a windows system is just going to be filled with garbage.
At least macOS has one place to install applications, one place for the user's documents, and most apps somehow respect them. Then, you have designated dumping grounds like ~/Library that contains tons of junk I have no idea whether I need or not.
I actually like that simplicity more than how many things FHS describes for the various Linux distros: https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
Like, when a developer just wants to ship a product and doesn't have that much free time, it's understandable that tidy things will be overlooked.
Maybe something like https://www.gobolinux.org/at_a_glance.html would help with that.
Linux prefers by-purpose organization, so executables go in /bin (or at least something ending in /bin), configuration goes into /etc, libraries go into /lib and so on. This simplifies discovery and cross-application data management, the answer to "how do I find a library" is just "search those 5 directories, if it's not there then it's missing", but it complicates package management and makes it too easy for garbage to accumulate. It's also better for GUI apps, it's far easier to make a music app that lets you think of artists, albums and playlists, not directories, if all your music is in one place.
Mac and Windows (mostly) prefer by-package organization, where everything related to a single application is contained inside that application's folder. This makes discovery harder, as now e.g. finding all applications that can open a particular file format requires you to check every single one. However, it makes package maintenance easier, removing an app just removes all its files. This approach complicates GUI applications and makes it impossible to escape the directory hierarchy, you can no longer make a nice music app if music files are strewn all around.
No system clearly falls in one bucket or the other, the Windows registry is partially organized by purpose, so are parts of Mac OS's Library folders, while Linux has package managers like Nix, which organize by package.
Ideally, we'd do by-package organization, and have some sort of per-user / per-process "union directories" (a bit like the /bin in Plan9) that would provide easy lookups for downstream applications.
More, if you count UN*X binaries.
And yet far too many apps dare to put their files there. Drives me absolutely insane.
You must be using finder a lot. When working from terminal, none of that happens. Frankly, no clue why it bothers people.
> Shortcut to Documents (OneDrive - Personal)
Oh good, now all that garbage is being synced across machines. Lovely.
The Detours library lets you attach a custom DLL during the launch of any process, intercepting Win32 calls to file system APIs by writing a custom hook.
There's the built-in "compatibility shims", which are borderline undocumented, but they work similarly. They can toggle compatibility flags and redirect file and registry paths. Conveniently, these are designed to trigger only for specific EXEs based on heuristics.
To support technology like App-V and later Docker containers, Windows has an entire API surface for "virtualising" the filesystem and the registry, as well as any other NT kernel namespace.
Then there's the User Mode Filesystem and probably more approaches I'm not aware of or that I forgot about...
If there's anything I've heard about drivers, though, it's that filesystem drivers are particularly annoying to write.
If you want to simply hide the files from view, you may have an easier time writing a shell plug-in. You have to deal with COM, but you're less likely to take down the whole system.
Or, you know, create a "%USERPROFILE%\Desktop\Actual Documents" folder and set that as the default open/save location in the applications you care about?
The pollution of the "My Documents" folder is unfortunate, yet nothing new, and not exactly limited to Windows (I mean, even on supposedly-perfect MacOS, I have to tolerate various game save crap under 'Documents', not to mention various 'helpful' vendor detritus), and not something that you would want to write a device driver for.
(That being said, hello Rust people, welcome to the Windows kernel! Personally, I've been enjoying the freedom to write C# drivers for years, with a minimal C++ wrapper, and I'm looking forward to the fruits of your creativity!)
mklink /d link_src target_dir
mklink /J link_src target_dir
This has good discussion on the plus and minus of both. https://superuser.com/questions/343074/directory-junction-vs...
That is a fight not worth fighting. Even Microsoft does not give a care in the world when deciding on storage locations and they put files directly in your %USERPROFILE% folder. Often also naming them like Linux/Unix dotfiles. I have opened tickets, pointing them to Microsoft documentation clearly stating that this is not how it should be, but they just don't care.
Still, it is infuriating seeing so much willful ignorance, when choosing the correct location is a decision just as quick as choosing the wrong location.
And better know where they come from, as there is no clean way to safely remove them.
Apps are terrible at this on Windows. Sometimes it's in %USERPROFILE%, sometimes in one of the three %appdata% folders, then My Docs, other times in %ProgramData%, and even worse, %PUBLIC%.
There's a registry for a reason, but even Microsoft doesn't use it for x-platform apps.
There are rules when to use what, but usually folks rather code away instead of reading documentation.
The problem, as usual, is that apps ignore it and do whatever they want. Such as e.g. using dotfiles directly in ~ instead.
It’s much cleaner and it comes with a more logical and useful split of config and cache stuff, e.g. you can just skip .cache when rsync-ing to a remote storage instead of having a long list of per-app escapes…
As I don't usually remove programs (or install new ones, for that matter) the scheme that works best for me and many others is "storage/app". In an ideal world a hybrid scheme might be possible, but as it stands now it is not an option and one way has to be chosen over the other.
Which one is better - a configuration file in some place or default configuration buried inside binary blob?
I actually really like the way git does it, where it reads each of these in order, last one wins.
- Default configuration compiled in
- Global configuration
- Per-user configuration
- Per-project configuration
You can opt-in to however much configuration complexity you need. Just cloning the occasional thing? Don't bother configuring, the global defaults are probably good enough. Simple commits and pushes? Make a user-level dotfile in your $XDG_CONFIG_HOME to set some basic info. Complex per-project aliases? If you want that, go ahead and opt in.
Contrast that with programs that just dump their whole default config into my home dir on first run. Just filling up with nonsense, often no way to tell what I changed or didn’t.
TO DO. /s
Also no one edits text files by hand, we use tools for that, tools that manipulate what is anyway a stream of bytes, whose meaning is given by whatever tool is used to manipulate such stream of bytes.
Makes me want to run off a read-only DVD.
Add a cron job to delete them ? /s
Almost 2 years ago they said "36,000 lines of code including a systemcall" [1], I'm curious how that project has progressed.
[1] https://www.thurrott.com/windows/282471/microsoft-is-rewriti...
https://techcommunity.microsoft.com/blog/windows-itpro-blog/...
Maybe nowadays there is more to it.
[0] https://github.com/microsoft/windows-drivers-rs/
[1] https://github.com/microsoft/windows-drivers-rs/blob/main/ex...
A driver that actually does something interesting would be able to have unsafe interfaces at the boundary, and more typical Rust code inside.
> wdk: Safe idiomatic bindings to APIs available in the Windows Development Kit (WDK)
And then if you look at that crate they've only implemented dbg_break, print, spinlock, and timer.
I'd take this as a sign that the bindings are a work in progress.
If you're going to write your driver in Rust... honestly I'd recommend just writing the `wdk` crate bindings you need for your driver and probably even upstreaming them. Wrapping FFI bindings in safe abstractions is usually pretty easy - but it's definitely the case that without them rust doesn't give you much.
This sample is using `wdk_sys` bindings directly without wrapping them... which is basically never the recommended way of interacting with the FFI in rust.
The actual issue here is that this "simple driver in Rust" is having to touch those direct C bindings at all - if Microsoft is going to advertise that they have support for writing drivers in Rust, that should presumably mean an API surface that's native to the language.
That way functions can be marked as accurate/"ok" to call in safe code by the author of the bindings. They could absolutely not be safe; in that case, the binding author is in error marking it so.
It reads pretty naturally to me as referring to the implementation of the driver.
There's a lot more complexity in writing FFI code - you have to think very carefully about everything you do. Case convention is a triviality here.
So you can use TRect and TPoint and pass those to PtInRect[1] which expects RECT and POINT respectively.
[1]: https://learn.microsoft.com/en-us/windows/win32/api/winuser/...
Being totally migrated to Linux by then I refused to use Windows for writing as well as building it, so I worked hard to build it with MSYS.
Long story short, I made it, and the driver worked great.
I think I had to write a patcher for the resulting PE (.sys) to get it to actually load.
Fun times.
It seems to be WordPress.com.
They probably rely heavily on caching and CDN.
Makes me sad that I just didn't stick to my WordPress blog 10 years ago.