In short, they first implemented realistic pigment mixing behavior. But in user testing it turned out non-painters are not good at getting the color they want by mixing; you usually end up with brown. So they had to make an unrealistic mixing algorithm that felt more intuitive.
While this work is impressive, two things to consider:
- Mixing oil paints is a learned skill. They don't behave intuitively, so this will only going to be useful for people who already paint in physical media (or want to learn that specifically)
- ArtRage has arguably been the most realistic painting software for over a decade, It has a real color mixing mode that has to be enabled (probably not the default for the aforementioned reasons), and gives much closer results [1] than what they have in the samples page. Yet they barely show it in the comparisons.
Thank you, I knew I’d used real color mixing in multiple apps for years. Secret Weapons is making a categorically false claim to have developed the first ever true-to-life color mixing algorithm.
My first thought, looking at the webpage: "Huh, that's neat. I didn't know that painting software didn't even attempt to do color mixing beyond naive interpolation, though I guess it figures; the physics behind all the light stuff must be fairly gnarly, and there's a lot of information lost in RGB that probably can't be just reconstructed."
Scrolling down a bit: "Huh, there's some snippets for using it as a library. Wait, it does operations in RGB? What's going on here?"
Finally, clicking the paper link, I found the interesting bit: "We achieve this by establishing a latent color space, where RGB colors are represented as mixtures of primary pigments together with additive residuals. The latents can be manipulated with linear operations, leading to expected, plausible results."
That's very clever, and seems like a great use for modern machine learning techniques outside the fashionable realm of language models. It uses perceptual color spaces internally too, and physics based priors. All around very technically impressive and beautiful piece of work.
It rhymes with an idea that's been floating in my head for a bit - would generative image models, or image encoder models, work better if rather than rgb, we fed them with wavelength data, or at least a perceptually uniform color space? Seems it'd be closer to truth than arbitrarily using the wavelengths our cone cells happen to respond to (and roughly, at that).
Consider that just after the cone cells, there are other cells doing some computation / aggregate of cone signals. Don't forget that color is a brain construct.
For those reasons (and others), there's often a strong disconnect between stimuli and perception, which means there's no such thing as a perceptual uniform color space.
Sidestepping what is defined as an "edge", quite a lot of work is done in the retina, including differential computation across cones - some "aggregator" cells will fire when it detect lines, movement, etc.
You can read on ganglion cells, bipolar cells and amacrine cells and see that a lot of preprocessing is done before even reaching the brain!
AI and machine learning aren't necessary at all. You 'just' have to empirically measure a few constants that describe and bound the various nonlinearities of different real pigments, and then plug them into a relatively straightforward paint mixing equation.
Paints have predictable mathematical properties in terms of what color they produce when mixed; they just mix nonlinearly, which is counterintuitive for people who have not practiced mixing paint a lot.
Photoshop and the other comparison programs on the page illustrate the linear mixing that most people intuitively expect.
As far as I know, most painting apps mix color in sRGB space instead of the linear RGB space. Which means they're even "worse" than naive interpolation.
Edit: I was responding to TFA's claim "there was no practical way to implement true pigment mixing into digital painting. Until now." but this page isn't dated - seeing how many other discussions reference MixBox blending could this need a [20XX] tag? I don't think it's new.
The license is for the library, not for the algorithm. You can't license algorithms, only their implementation, and that's the case here.
The algorithm could be patented, I guess, but it doesn't seem to be.
The algorithm is not the natural phenomenon of pigment mixing. It is about how to mix RGB values in a way that imitates natural pigment mixing, which is not something natural as nature doesn't work in RGB.
I remember that in Windows 3.0 time there was "Fractal Designer Painter" software, which had different engines to precisely emulate different types of painting materials: pastel, oil, water color, gouache. It imitaded mixing, stroke, pressure, etc. It looks incredible realistic in these times when 800x600x24bit screen was a luxury. And it was very slow on i486 computer.
As I learned several minutes ago now it is Corel Painter.
I wonder, is it still good and how it (both old and new algorithms, I allow that it was simplified down the road!) compares to this new library.
Unfortunately, authors doesn't compare themselves with Corel Painter.
Yep, it has changed hands a few times, but Painter still exists, gets updated yearly, and does everything I saw described in the article. I've been using Painter since the 90s for digital art specifically because of how well the color mixing works and how well it matches real art tools - not just various paints, but pastels, pencil, pens, etc. Tons of options.
Now, all that doesn't come as a library for your own apps - that is definitely new and sounds like a great thing. But the implication that it has never been done before it quite wrong.
I'd be curious to learn why Secret Weapons (the makers of this tool) didn't raise $100M in venture funding two or three years ago, before the start of the AI video boom.
They were first on the scene with EbSynth [1], which was the first widely used AI-powered video-to-video style transfer and keyframe interpolation system. EbSynth gave rise to comedians like Joel Haver [2] and lots of others like him.
Several other companies [3] used EbSynth to base their technology on and wound up raising large seed rounds. Major musicians used EbSynth either directly or through these other companies to make music videos [4,5].
Secret Weapons was in a great position to conquer AI video and AI creative tools, but they seemingly slept on it.
Super quick browse...but...IMHO, aren't video diffusion models and ebsynth apples and oranges?
ebsynth looks like final cut pro plugins for masking and tracking. I could see the projected customer base being similar, but the skillset & resources required from your engineering team is significantly different.
> Secret Weapons was [set] to conquer AI video and AI creative tools
Oh goodness, I think we both agree that's a bit much :)
Personally, I don't care about the library, and just want the pure math implementation for shaders and such. For that, they have the algorithm available here in GLSL, vec3 mixbox_lerp is the function you want:
I don't have time to whip up a demo, but it seems like it's pretty well optimized and it looks really nice in the examples. Great work from these folks.
Rebelle is very good. It’s fast, work fine at high resolution and so on.
Less feature rich than painter (but that’s a good thing)
What’s still missing : the proper feel for dry materials (like charcoal) and a deep integration with the surface of the support. Right now it’s more or less a light depth map - nothing as gritty as wood, harsh paper (or even cloud ones)
My favorite at the moment.
And yeah, color mixing has been very good for a while there.
This is great. I paint miniatures and once you get past the basics and start using glazes for filters/blends software isn't great for prototyping your paint schemes.
While this is really cool and should be an option in all painting programs, a part of me wonders if this complicates the learning process of painting?
After all, mixing paint colors is something you have to learn because it's not very intuitive.
Almost everywhere else in life, when you mix A and B, you expect a linear transition to happen. By adding salt to water, you are increasing its saltiness, it's not expected that at some point, the water becomes sour before it becomes salty again.
I think it would simplify the transition for a person moving between physical media and digital, and vice versa. The mixing examples they show on the webpage for the Mixbox algorithm look real to me and make the mixing available in other programs seem incorrect.
edit: I'll add though that Mixbox doesn't seem to take actual physical pigment characteristics into account and that can radically impact pigment mixing.
As someone who teaches both oil painting and digital painting, I find this project fascinating. They have certainly identified two of the key difference between digital and real paint:
- RYB mixing (e.g. yellow painting plus blue paint = green). I tested it on a mix of alizarin crimson plus hansa yellow to produce a cold orange (e.g. blood orange).
- Relative behavior of specific pigments. For example, a Prussian blue is almost black when applied thickly, but very chromatic when applied as thinly over white. A cerulean blue is pretty much the same in those two states. I tested this on a Prussian and the app performed well.
The app seems to make the assumption that the artist will always want the paint to mix. In a glaze, a thin transparent layer of paint is applied over dry paint. Effectively the result is a filter (the result will always be darker). To a degree, this can be faked using paint applied in a layer the blend mode of which has been set to multiply. It might be nice to add a drying brush to the tool set which selectively dries the paint to prevent mixing. Maybe also a mode switch to selectively different modes of physical application such as glazing, scumbling and dry brushing. While we are at it, why not add a Tonk mode using which thick paint can be lifted off the surface without smudging.
When I paint digitally, I mix the incredible nuance of real paint... even its smell. When I paint with oils I miss the incredible flexibility of the digital process (blend modes, compositing, even the humble undo). Never the twain...
> It might be nice to add a drying brush to the tool set which selectively dries the paint to prevent mixing.
The effect of thinning out the paint is effectively letting more light passing through the paint and reflecting off the canvas instead.
So in the case of drying out, seems it would be a good fit for simply painting on a new layer, and having the code reflect off the combined layers below.
That would make it easier to change the paint/glaze on top, as well as not having to commit to drying the layer underneath.
I've an oddball interest in exploring what science education content might look like if someday crafted with implausibly extreme expertise and effort. Given that first-tier physical-sciences graduate students not uncommonly have foundational misconceptions around color, and color is commonly taught in K-2, I was exploring how one might emphasize spectra when teaching color, in preK to whatever. With interactives like mousing over hyperspectral images, a light/image manipulation workbench, and apropos topic, physically realistic Paint (with fluorescence, controllable lighting, glitter, blackbody, fire, make your own pigment, collage). Now community around that oddball interest has been hard to come by, especially no longer being around MIT. So in support of a "find your people - build your village" 2025 New Year's theme, if anyone knows of potentially overlapping interests...
This is the feature that sold me on Rebelle 7 (which is hands down the best digital painting software I've used). Being able to translate oil painting mixing techniques onto a digital workflow was a game changer for me.
I was already impressed but at the bottom of the page they show they wanna make an addon to color blender fluids accurately. Paint me impressed. Paint me impressed but with pigment blended accurately
It works as a kind of experimental extension supplied with the paid FLIP Fluids addon. Note that the fluid solver built into Blender is also an implementation of FLIP, but "FLIP Fluids" is a separate product. I've played with fluid simulation in Blender quite a bit. The bundled FLIP is very limited, FLIP Fluids is great, and the color mixing is amazing!
I think that the colors here (in this screen-world) are 99% mental. Like everything else here. Which is to say, it is not the configuration of the dots or their color that significate, it is the meanings and memories that they evoke. 1% of the game happens on the screen and the other 99% happens inside your head.
Which makes the whole thing a bit solipsistic when you think about it.
Cool, could see this being useful as a palette with a combination of the normal mode. I don’t see a clear way of controlling the amount of paint in the demo, if i want 3:1 red to white how would i do it?
"Colors in painting software do not act like real pigments.
Why? Because there was no practical way to implement true pigment mixing into digital painting." - not sure that is accurate - digital tools were developed such that they colour mix not colour blend. I'm not sure it's so much as there was no framework for it, so much as if I want to blend 2 colours I'd mix them then use a blend mode on a layer. Traditionally most people working with pro imaging tools had a background in colour theory, I can't imagine working on imaging and not knowing blue and yellow is green, so if I want green on my mixed yellow and blue, I'd either draw on with green, or draw with a blend mode if I want the exact green. Doesn't make this any less cool, just saying'
Why isn't this more widely used? First, we have to understand that digital painting is a fundamentally different beast than traditional, in how you can use the tool, what you need to know, and how you learn it.
In traditional, you tend to have some rules that make it easy to get good colors.
One method is to restrict your paints to a few primary colors and burnt sienna (or another low saturation brown).
Then by mixing these colors you get the colors you want, in a way where these colors have a good relationship with the others.
So a simple rule is for instance that you get your green by putting lemon yellow with some black and a bit of sienna to lower the saturation.
The relationship is almost 'free' from the properties of the physical pigments if you follow such simple rules.
What's the deal about 'relationship'? Colors are not treated individually, they are perceived in context.
In real life, lightning will unify a scene by changing colors. However, there is a semantic superposition that affects it.
Grass is green in your mind, so if the color you see is a light blue/purple, the brain will adjust its understanding to match it.
This is where many illusions come from (most famously the white and gold/blue and black dress meme come from).
This is a big deal to learn painting, but can be almost completely side-stepped by learning how to mix colors.
In digital however, artists got used to the fact that color is generally treated as HSV and adjusted their approach.
So the most commonly taught approach to manage colors is to work in black and white, only dealing with value, which is the most essential component of color.
Values are the primary issue because this is fundamentally what implies to our brain what are the normal vectors of light, and so carry almost all 'volume information'.
Then, you can apply colors via some layers set to multiply, color and overlay mode.
Note that it has some antecedent in the traditional world, called 'grisaille' technique.
So, you can color pick anywhere you want on the color wheel without mixing. The impact of color mixing is actually pretty small.
The other technique is called gamut masking; you're only allowed to pick colors on a subset of the color wheel that is in a good HSV relationship with the chosen color scheme.
The thing is, even though this is not perceptually accurate, this is good enough.
The main impact of color mixing is that if working with low opacity brushes, the colors will look 'muddy'.
This problem is mostly side-stepped by avoiding it altogether with high-opacity brushes, and using other layer blending modes.
Because now most digital artists have learnt through these techniques, bringing techniques that work best coming from the traditional world is not very useful, to say the least.
In my own practice, traditional and digital are indeed almost entirely separate activities, at every level.
The fundamental, theoretical understanding of all aspects is the same, but it doesn't really matter very much; almost all other artists I ever discuss with do not have such an understanding, and build their practice from a more 'human'-centric foundation (technique and artistic concepts).
I'm of course interested in the space, and it is a bit similar to asking why would people would use bash or plain text; it's not necessarily that these are the best tools, but there is much more to it.
Is it that when you have two colors like RGB(0,0,255) and RGB(255,255,0) there is no easy formula to calculate what a mix of these two colors would look like in the real world?
And therefore one needs a lookup table? And since 16Mx16M is a large number, a complete lookup table is impractical?
In short, they first implemented realistic pigment mixing behavior. But in user testing it turned out non-painters are not good at getting the color they want by mixing; you usually end up with brown. So they had to make an unrealistic mixing algorithm that felt more intuitive.
While this work is impressive, two things to consider:
- Mixing oil paints is a learned skill. They don't behave intuitively, so this will only going to be useful for people who already paint in physical media (or want to learn that specifically)
- ArtRage has arguably been the most realistic painting software for over a decade, It has a real color mixing mode that has to be enabled (probably not the default for the aforementioned reasons), and gives much closer results [1] than what they have in the samples page. Yet they barely show it in the comparisons.
[1] https://i.ibb.co/T0GwbDV/artrage-mixing.jpg (right: real color blending on)
My first thought, looking at the webpage: "Huh, that's neat. I didn't know that painting software didn't even attempt to do color mixing beyond naive interpolation, though I guess it figures; the physics behind all the light stuff must be fairly gnarly, and there's a lot of information lost in RGB that probably can't be just reconstructed."
Scrolling down a bit: "Huh, there's some snippets for using it as a library. Wait, it does operations in RGB? What's going on here?"
Finally, clicking the paper link, I found the interesting bit: "We achieve this by establishing a latent color space, where RGB colors are represented as mixtures of primary pigments together with additive residuals. The latents can be manipulated with linear operations, leading to expected, plausible results."
That's very clever, and seems like a great use for modern machine learning techniques outside the fashionable realm of language models. It uses perceptual color spaces internally too, and physics based priors. All around very technically impressive and beautiful piece of work.
It rhymes with an idea that's been floating in my head for a bit - would generative image models, or image encoder models, work better if rather than rgb, we fed them with wavelength data, or at least a perceptually uniform color space? Seems it'd be closer to truth than arbitrarily using the wavelengths our cone cells happen to respond to (and roughly, at that).
For those reasons (and others), there's often a strong disconnect between stimuli and perception, which means there's no such thing as a perceptual uniform color space.
You can read on ganglion cells, bipolar cells and amacrine cells and see that a lot of preprocessing is done before even reaching the brain!
Paints have predictable mathematical properties in terms of what color they produce when mixed; they just mix nonlinearly, which is counterintuitive for people who have not practiced mixing paint a lot.
Photoshop and the other comparison programs on the page illustrate the linear mixing that most people intuitively expect.
Very fun stuff if you work with both digital (additive) and physical (subtractive) colors.
https://krita-artists.org/t/implementing-mixbox-mixing-for-k...
https://invent.kde.org/graphics/krita/-/merge_requests/1783 https://krita-artists.org/t/can-we-get-mixbox-on-krita/64201...
GPLv3 reimplementation linked in that thread: https://github.com/joeedh/pigment-painter
MyPaint also has it though, and that's open source: https://mypaint.app/blog/2019/01/26/MyPaint-2.0-alpha/ (https://github.com/mypaint/libmypaint/pull/56 mentions using 7 primaries for blending), and another older thread about other methods: https://community.mypaint.app/t/real-color-blending-wip/390 . MyPaint IIUC is integrated into Gimp, so I assume Gimp can also do it using the MyPaint brush engine.
Edit: I was responding to TFA's claim "there was no practical way to implement true pigment mixing into digital painting. Until now." but this page isn't dated - seeing how many other discussions reference MixBox blending could this need a [20XX] tag? I don't think it's new.
The algorithm could be patented, I guess, but it doesn't seem to be.
The algorithm is not the natural phenomenon of pigment mixing. It is about how to mix RGB values in a way that imitates natural pigment mixing, which is not something natural as nature doesn't work in RGB.
https://github.com/scrtwpns/mixbox/blob/master/cpp/mixbox.cp...
Like, it looks like just decompress that lut and have it decompressed in the project and the rest is just interpolation algorithms.
As I learned several minutes ago now it is Corel Painter.
I wonder, is it still good and how it (both old and new algorithms, I allow that it was simplified down the road!) compares to this new library.
Unfortunately, authors doesn't compare themselves with Corel Painter.
Now, all that doesn't come as a library for your own apps - that is definitely new and sounds like a great thing. But the implication that it has never been done before it quite wrong.
They were first on the scene with EbSynth [1], which was the first widely used AI-powered video-to-video style transfer and keyframe interpolation system. EbSynth gave rise to comedians like Joel Haver [2] and lots of others like him.
Several other companies [3] used EbSynth to base their technology on and wound up raising large seed rounds. Major musicians used EbSynth either directly or through these other companies to make music videos [4,5].
Secret Weapons was in a great position to conquer AI video and AI creative tools, but they seemingly slept on it.
[1] https://ebsynth.com/
[2] https://www.youtube.com/watch?v=SY3y6zNTiLs&list=PLKtIcOP0Wv... (playlist with all of his animations)
[3] https://kaiber.ai/ and others
[4] Linkin Park https://www.youtube.com/watch?v=7NK_JOkuSVY
[5] Magdalena Bay https://www.youtube.com/watch?v=dXLCHvRsgRQ
ebsynth looks like final cut pro plugins for masking and tracking. I could see the projected customer base being similar, but the skillset & resources required from your engineering team is significantly different.
> Secret Weapons was [set] to conquer AI video and AI creative tools
Oh goodness, I think we both agree that's a bit much :)
Didn't know we were that far along 3 years ago?
https://www.youtube.com/watch?v=sP3u09_YFxk
https://github.com/scrtwpns/mixbox/blob/master/shaders/mixbo...
I don't have time to whip up a demo, but it seems like it's pretty well optimized and it looks really nice in the examples. Great work from these folks.
https://news.ycombinator.com/item?id=30116316 (71 points, 15 comments)
https://news.ycombinator.com/item?id=30033611 (60 points, 13 comments)
Has anyone tried Rebelle, the commercial painting app that uses this method?
What’s still missing : the proper feel for dry materials (like charcoal) and a deep integration with the surface of the support. Right now it’s more or less a light depth map - nothing as gritty as wood, harsh paper (or even cloud ones)
My favorite at the moment.
And yeah, color mixing has been very good for a while there.
After all, mixing paint colors is something you have to learn because it's not very intuitive.
Almost everywhere else in life, when you mix A and B, you expect a linear transition to happen. By adding salt to water, you are increasing its saltiness, it's not expected that at some point, the water becomes sour before it becomes salty again.
edit: I'll add though that Mixbox doesn't seem to take actual physical pigment characteristics into account and that can radically impact pigment mixing.
- RYB mixing (e.g. yellow painting plus blue paint = green). I tested it on a mix of alizarin crimson plus hansa yellow to produce a cold orange (e.g. blood orange).
- Relative behavior of specific pigments. For example, a Prussian blue is almost black when applied thickly, but very chromatic when applied as thinly over white. A cerulean blue is pretty much the same in those two states. I tested this on a Prussian and the app performed well.
The app seems to make the assumption that the artist will always want the paint to mix. In a glaze, a thin transparent layer of paint is applied over dry paint. Effectively the result is a filter (the result will always be darker). To a degree, this can be faked using paint applied in a layer the blend mode of which has been set to multiply. It might be nice to add a drying brush to the tool set which selectively dries the paint to prevent mixing. Maybe also a mode switch to selectively different modes of physical application such as glazing, scumbling and dry brushing. While we are at it, why not add a Tonk mode using which thick paint can be lifted off the surface without smudging.
When I paint digitally, I mix the incredible nuance of real paint... even its smell. When I paint with oils I miss the incredible flexibility of the digital process (blend modes, compositing, even the humble undo). Never the twain...
The effect of thinning out the paint is effectively letting more light passing through the paint and reflecting off the canvas instead.
So in the case of drying out, seems it would be a good fit for simply painting on a new layer, and having the code reflect off the combined layers below.
That would make it easier to change the paint/glaze on top, as well as not having to commit to drying the layer underneath.
It works as a kind of experimental extension supplied with the paid FLIP Fluids addon. Note that the fluid solver built into Blender is also an implementation of FLIP, but "FLIP Fluids" is a separate product. I've played with fluid simulation in Blender quite a bit. The bundled FLIP is very limited, FLIP Fluids is great, and the color mixing is amazing!
Which makes the whole thing a bit solipsistic when you think about it.
Here it is in 2021:
https://www.escapemotions.com/blog/rebelle-5-meet-color-pigm...
And here it is currently:
https://escapemotions.com/products/rebelle/manual/starting-p...
That seems like something that Adobe would want to borg.
In traditional, you tend to have some rules that make it easy to get good colors. One method is to restrict your paints to a few primary colors and burnt sienna (or another low saturation brown). Then by mixing these colors you get the colors you want, in a way where these colors have a good relationship with the others. So a simple rule is for instance that you get your green by putting lemon yellow with some black and a bit of sienna to lower the saturation. The relationship is almost 'free' from the properties of the physical pigments if you follow such simple rules.
What's the deal about 'relationship'? Colors are not treated individually, they are perceived in context. In real life, lightning will unify a scene by changing colors. However, there is a semantic superposition that affects it. Grass is green in your mind, so if the color you see is a light blue/purple, the brain will adjust its understanding to match it. This is where many illusions come from (most famously the white and gold/blue and black dress meme come from).
This is a big deal to learn painting, but can be almost completely side-stepped by learning how to mix colors.
In digital however, artists got used to the fact that color is generally treated as HSV and adjusted their approach. So the most commonly taught approach to manage colors is to work in black and white, only dealing with value, which is the most essential component of color. Values are the primary issue because this is fundamentally what implies to our brain what are the normal vectors of light, and so carry almost all 'volume information'. Then, you can apply colors via some layers set to multiply, color and overlay mode. Note that it has some antecedent in the traditional world, called 'grisaille' technique.
So, you can color pick anywhere you want on the color wheel without mixing. The impact of color mixing is actually pretty small. The other technique is called gamut masking; you're only allowed to pick colors on a subset of the color wheel that is in a good HSV relationship with the chosen color scheme. The thing is, even though this is not perceptually accurate, this is good enough.
The main impact of color mixing is that if working with low opacity brushes, the colors will look 'muddy'. This problem is mostly side-stepped by avoiding it altogether with high-opacity brushes, and using other layer blending modes.
Because now most digital artists have learnt through these techniques, bringing techniques that work best coming from the traditional world is not very useful, to say the least. In my own practice, traditional and digital are indeed almost entirely separate activities, at every level. The fundamental, theoretical understanding of all aspects is the same, but it doesn't really matter very much; almost all other artists I ever discuss with do not have such an understanding, and build their practice from a more 'human'-centric foundation (technique and artistic concepts).
I'm of course interested in the space, and it is a bit similar to asking why would people would use bash or plain text; it's not necessarily that these are the best tools, but there is much more to it.
Is it that when you have two colors like RGB(0,0,255) and RGB(255,255,0) there is no easy formula to calculate what a mix of these two colors would look like in the real world?
And therefore one needs a lookup table? And since 16Mx16M is a large number, a complete lookup table is impractical?