Thank you for this. I find myself increasingly using CSVs (TSVs actually) as the data format of choice. I confess I wish this was written for Mac too, but I like the trend of (once again) moving data processing down to our super computers on our desk...
QStudio allows querying CSV on mac via DuckDB:
https://www.timestored.com/qstudio/csv-file-viewer
I've been improving the Mac version a lot lately, key bindings, icon, an App package to download. So if you find any problems please raise a github issue.
Ok, if we are all tagging and promoting our own projects, check out mine.
I created Buckaroo to provide a better table viewing experience inside of notebooks. I also built a low code UI and auto cleaning to expedite the wrote data cleaning tasks that take up a large portion of data analysis. Autocleaning is heuristically powered - no LLMs, so it's fast and your data stays local. You can apply different autocleaning strategies and visually inspect the results. When you are happy with the cleaning, you can copy and paste the python code as a reusable function.
All of this is open source, and its extendable/customizable.
Thank you! I completely agree—TSVs/CSVs are such a simple yet powerful format, and it's great to hear you're making good use of them. I'm also a big fan of doing as much as possible locally—our machines are incredibly capable these days. Good news: I'm currently working on the macOS version of Data.olllo and plan to submit it to the Mac App Store soon. Stay tuned!
… I‘m trying to use our super computers in our pockets, like an iPhone ;-) But still struggling with the way how to present CSV data effectively on a small screen, although it‘s huge in terms of pixels compared to computer screens from the 90s
It‘s interesting to research how capable applications like Lotus123 have been even on low resolutions like 800x600 pixel compared to today’s standard
Marketing note: I'm sure you're proud of P Core/V Core, but that doesn't matter to your users, it's an implementation detail. At a maximum I'd write "intelligent execution that scales from small files to large files".
As an implementation note, I would make it simple to operate on just the first 1000 (10k or 100k) rows so responses are super quick, then once the users are happy about the transform, make it a single click to operate on the entire file with a time estimate.
Another feature I'd like in this vein is execute on a small subset, then if you find an error with a larger subset, try to reduce the larger subset to a small quick to reproduce version. Especially for deduping.
> Marketing note: I'm sure you're proud of P Core/V Core, but that doesn't matter to your users, it's an implementation detail. At a maximum I'd write "intelligent execution that scales from small files to large files".
Speaking personally, "intelligent execution that scales from small files to large files" sounds like marketing buzz that could mean absolutely nothing. I like that it mentions specifically switching between RAM and disk-powered engines, because that suggests it's not just marketing speak, but was actually engineered. Maybe P vs V Core is not the best way to market it, but I think it's worth mentioning that design.
Thanks for the thoughtful take—really appreciate both perspectives.
You're right that terms like "intelligent execution" can feel vague without concrete backing. My goal with mentioning P Core/V Core was to hint at the underlying design—switching between in-memory and disk-based engines like Polars and Vaex—without overwhelming with technical detail.
I’ll look for a better way to explain the idea clearly and briefly. Thanks again!
I wish every product had an engineer-only landing page I could set as a default in my browser. The number of companies that assume I'm familiar with their offering is astounding, and I'm usually looking for implementation docs just to figure out what it actually does.
Yes, Data.olllo uses including Polars under the hood for fast and efficient processing. A demo video is in the works and should be up soon.
Good point about the "P Core/V Core" naming—I'll simplify that to focus more on the user benefit, like scaling from small to large files smoothly.
I also like your idea of running transformations on a sample first with a one-click full run—very aligned with the vision. And subset reproduction for errors is a great suggestion, especially for things like deduping. Appreciate it!
What are "massive" CSVs? I have CSVs in the terabytes that need to be deduped by a specific column. Can it handle that? What if I want to run a function on the column to normalize it before the deduping?
Tad is a great tool—very clean and useful for quick exploration.
Data.olllo is focused more on local data processing, not just viewing—things like filtering, transforming, merging, and even running Python code (with AI assistance coming). It’s built for both small and large files with performance in mind, using many cores including Polars under the hood.
Also, good news: the macOS version is in the works and will be submitted to the Mac App Store soon!
As a point of comparison, I just downloaded the Windows binary for duckdb (which provides a nice TUI for similar tasks) and it was 9.84MB. People can and should expect better.
Thanks for the feedback!
Data.olllo isn't Electron-based—it's built in Python with tkinter and custom tkinter, so the size mainly comes from the data libraries and embedded Python environment. I agree that keeping things lean is important, and I’m actively working on optimizing the package size further.
Appreciate the DuckDB comparison—great tool and definitely a benchmark worth learning from!
I created Buckaroo to provide a better table viewing experience inside of notebooks. I also built a low code UI and auto cleaning to expedite the wrote data cleaning tasks that take up a large portion of data analysis. Autocleaning is heuristically powered - no LLMs, so it's fast and your data stays local. You can apply different autocleaning strategies and visually inspect the results. When you are happy with the cleaning, you can copy and paste the python code as a reusable function.
All of this is open source, and its extendable/customizable.
Here's a video walking through autocleaning and how to extend it https://youtu.be/A-GKVsqTLMI
Here's the repo: https://github.com/paddymul/buckaroo
It‘s interesting to research how capable applications like Lotus123 have been even on low resolutions like 800x600 pixel compared to today’s standard
What are you using for processing (polars)?
Marketing note: I'm sure you're proud of P Core/V Core, but that doesn't matter to your users, it's an implementation detail. At a maximum I'd write "intelligent execution that scales from small files to large files".
As an implementation note, I would make it simple to operate on just the first 1000 (10k or 100k) rows so responses are super quick, then once the users are happy about the transform, make it a single click to operate on the entire file with a time estimate.
Another feature I'd like in this vein is execute on a small subset, then if you find an error with a larger subset, try to reduce the larger subset to a small quick to reproduce version. Especially for deduping.
Speaking personally, "intelligent execution that scales from small files to large files" sounds like marketing buzz that could mean absolutely nothing. I like that it mentions specifically switching between RAM and disk-powered engines, because that suggests it's not just marketing speak, but was actually engineered. Maybe P vs V Core is not the best way to market it, but I think it's worth mentioning that design.
You're right that terms like "intelligent execution" can feel vague without concrete backing. My goal with mentioning P Core/V Core was to hint at the underlying design—switching between in-memory and disk-based engines like Polars and Vaex—without overwhelming with technical detail.
I’ll look for a better way to explain the idea clearly and briefly. Thanks again!
I'm not saying we need a morlock/eloi toggle.
Yes, Data.olllo uses including Polars under the hood for fast and efficient processing. A demo video is in the works and should be up soon.
Good point about the "P Core/V Core" naming—I'll simplify that to focus more on the user benefit, like scaling from small to large files smoothly.
I also like your idea of running transformations on a sample first with a one-click full run—very aligned with the vision. And subset reproduction for errors is a great suggestion, especially for things like deduping. Appreciate it!
Data.olllo is focused more on local data processing, not just viewing—things like filtering, transforming, merging, and even running Python code (with AI assistance coming). It’s built for both small and large files with performance in mind, using many cores including Polars under the hood.
Also, good news: the macOS version is in the works and will be submitted to the Mac App Store soon!
https://graydon2.dreamwidth.org/193447.html
That said, I also plan to add support for Parquet and other formats soon—definitely agree it's gaining traction for larger, structured datasets.
Appreciate the DuckDB comparison—great tool and definitely a benchmark worth learning from!