Carmack on star fields in VR

(twitter.com)

123 points | by tosh 504 days ago

12 comments

  • jws 504 days ago
    He touches on line graphics as having the same problem rendering. I run into that when making strip charts.

    In general, the higher level your graphics API, the harder it is to get horizontal and vertical lines to draw on the physical pixels. At the frame buffer level it is trivial. Cocoa and UIKit it is possible with some effort. SwiftUI it is impossible (you will end up using the escape hatch to Cocoa or UIKit). HTML is hopeless.

    When printing meant writing some Postscript it was a little work, but doable. Getting it onto a printer through a modern printer API is essentially impossible.

    Fundamentally this is why most graphs you see online look like they were drawn with a worn out felt tip marker. It hides the sins at the expense of limiting data density and precision.

    • bombcar 504 days ago
      • xnzakg 504 days ago
        This was fun, at first I was wondering why none of the examples were working properly until I remembered I was using a 4k screen on a windows machine, which set the scaling to 150% by default. After setting the scaling to 100% the demos suddenly made sense. Also, interesting how changing the gamma setting on my screen also changed the results...
    • kllrnohj 504 days ago
      It's hard in HTML due to the clunky way it handles density scaling. You wont have any issue doing razer sharp pixel alignment on systems that are natively density aware, like Android, as then the rendering stack can just work in actual pixels like it should, and density handling is done by the layout system instead.

      So it's not about the number of layers of abstraction or general "high level"ness, but rather if those abstractions are trying to cover up mistakes made elsewhere.

    • bno1 504 days ago
      Sounds like you want font hinting but for vector graphics.
    • Cullinet 504 days ago
      Thank you for your report. I have to represent scaled graphs also online via browsers and had been curious about the implementation of scaling through APIs and ultimately browser based drivers. My concern is that the human brain is very much better at comprehending spatial dithering, critical for e.g. price trend extrapolation, than the open literature gives credit. Vital research by CSERIAC at Wright Patterson was declassified to sell the post SDI COTS reboot, but heavily reclassified after 9/11. This has been to the great detriment of the human computer interaction research community which really didn't survive into this century.
      • cma 504 days ago
        Can you unacronym this?
        • tmtvl 504 days ago
          I think you know what APIs are, CSERIAC is the Crew System Ergonomics Information Analysis Center, SDI is the Strategic Defence Initiative, and COTS means Commercial Off The Shelf, as in literally buying equipment from regular commercial vendors rather than doing the whole massive DoD contract thing.
          • GuB-42 504 days ago
            And DoD is the Department of Defense ;)

            About acronyms, I remember a company that won a bid for a company I worked with. Part of the reason they won is that their proposal was very clear, and one of the stated reason was the lack of acronyms.

            To this day, I try to avoid using acronyms as much as possible. I also remember an argument at another company, someone was complaining of problems on the "VMS", the other was calling bullshit. It took a few minuted before they realized they were talking about two different things with the same acronym. There are only 17576 3-letter acronyms, if 200 of these are used, the chance of a collision is about 2/3 (see: birthday paradox), and that's assuming a uniform distribution, in reality, it is worse. Misunderstandings are inevitable.

            • milesvp 503 days ago
              It’s even worse tham that. Many letters can easily be confused for different letters, hence the reason for phonetic alphabets. I had one company who had an external system known as OAS, and some manager who maybe should have known about OAS, but had never worked with it decided to create an acronym OIS. Luckily the two systems were rarely talked about in the same context, but it still took me lots of mental effort to know which of the 2 systems various individuals were talking about in meetings when either were mentioned.
    • omeze 504 days ago
      I don't quite understand this (not a graphics dev) -- if you have the ability to get the viewport size in a browser, and if the display is running at its native resolution, why do you not have pixel-perfect accuracy?
      • turtledragonfly 504 days ago
        It depends on what the various different layers in between you and the gfx hardware are doing.

        For instance, see the discussion on this "monitor calibration" shader: https://www.shadertoy.com/view/NsS3Rw

        In that case, FireFox does not 100% provide some of the needed APIs in order to get pixel-perfect rendering. There's nothing you can do to fix that yourself.

           "...Anyways, long story short, we need Firefox to implement 'devicePixelContentBoxSize' before we can have pixel exact rendering, deterministically, in that browser."
        
        That's just one example.

        For something delicate like pixel-perfect rendering, every layer in the chain from hardware to OS -> libraryX -> libraryY -> libraryZ -> browser -> your code, all need to handle coordinates and thigs _perfectly_. If any one of those links in the chain messes things up, or doesn't provide the correct rounding, etc. then you are stuck.

        It's not so terribly hard to do, but it has subtleties and edge cases, and everyone involved must handle them correctly. Also, if they handle them not-correctly, the result is still "good enough" for many purposes, so the fix doesn't get prioritized.

        • zamadatix 504 days ago
          FWIW Firefox has supported that for a ~year now and you could fix it yourself it was just more work than the prebuilt way.
          • turtledragonfly 504 days ago
            Ah, good to know. I'm not involved in that project, just had the link handy as a relevant discussion. I'll wait for the ShaderToy devs to fix it, I guess (:
      • kevin_thibedeau 504 days ago
        Floats get rounded to an off-by-one pixel.
        • zamadatix 504 days ago
          I'd be amazed to see a resolution/dpi combo that results in a 1 pixel error from rounding and equally amazed that it isn't far from the same combo that would cause a rounding error outside of a browser.
          • fizzynut 504 days ago
            The issue in a browser is that you have a default zoom that is often not 100% and you can also "zoom" in/out of web pages which by will cause blurriness or overly sharp images. If you decide to ignore that in favor of the native resolution, then the content size is likely too big or small for what the user expects and it's not clear what the user wants it to look like or scale to even if you did implement that too because of the huge variety in screen sizes and ratios the content could be viewed from.

            A native application does not generally have these issues because by nature of being native it was built for the form factor it is being used on.

            • zamadatix 503 days ago
              Be it macOS, Windows, iOS, or Android they all have the same problem with native apps. There is nothing about code being native that changes the user wanting content on their device to be larger or smaller.
          • jws 504 days ago
            I find it to more about a thin line being rendered as a little bit of color in two adjacent lines of pixels. The result is a messy blur instead of a crisp line. If any layer of the libraries and APIs doesn’t pass up the coordinate system mappings, like say a window doesn’t report its exact screen space position to its content, then you lose access to the information to register the pixels precisely.
            • zamadatix 504 days ago
              That can definitely happen between but as a loss of information between layers that don't support DPI fully (e.g. Windows 7 type DPI information which isn't per screen aware) or are intentionally ignoring physical pixel information (e.g. percentage based sized/positioned elements) not float precision loss. Float precision loss is definitely a thing but even a 32 bit float is integer perfect to 8,388,608 meanwhile a 7,640 pixel screen (1,000 times smaller) is considered extraordinarily large. Fractions close to 0 (such as APIs that use a 0-1 range for the screen or DPI represented a a percentage of a standard value) have a similar accuracy.

              I'm not even sure on Windows there is a way to render anything

    • naikrovek 504 days ago
      > When printing meant writing some Postscript it was a little work, but doable. Getting it onto a printer through a modern printer API is essentially impossible.

      Brother printers (at least some) support BR-script which is Postscript without the trademark violation. It's supported at the driver level, I think, which would render the PS on the PC then send the rendered document as one of the hardware-supported languages like PCL.

      I mention this as an interesting bit of info, and not to refute anything you said

    • galangalalgol 504 days ago
      If you netcat a ps file to the ip of the printer that should still work yes?
  • graffix 503 days ago
    > Stars are perfect point light sources to the eye, they will not cover multiple pixels unless you are doing an intentional bloom effect

    On any real camera, stars always cover multiple pixels (the entire screen strictly speaking). This is described by a point-spread function (PSF), where the "least blurry" physically realizable PSF is the Airy disk. The bloom effect occurs due to the long tails of the PSF. Inasmuch as stars appear as perfect points to the eye, it's down to perceptual limits and (I suspect) heavy filtering in the visual cortex.

    Not saying Carmack is wrong about VR and video game rendering here. Just that if you're trying to replicate real sensors, the right way is precisely what he says to avoid. You must somehow draw

    > large, dim stars

    During a recent effort to implement physically-accurate star rendering at work, I had access to a photograph of the night sky. Bright stars visibly and clearly covered 3x3 pixels- more in reality, due to the above.

    • kridsdale1 501 days ago
      I guess as a VR dev you have to first decide, are you making a first person perspective for a human with eyes, or for a pair of camera lenses, with all the mechanical and optical artifacts that comes with that? Humans don’t see lens flare. But our cinematic cultural library now makes us expect them in games, even if the character is biological and not wearing glasses.
  • pyrolistical 504 days ago
    He is talking about actual star fields.

    Not to be confused with the upcoming game https://en.m.wikipedia.org/wiki/Starfield_(video_game)

    The title should fix the capitalization and use two worlds

  • VanillaCafe 504 days ago
    I can't seem to find the reference, but I seem to recall that Valve had specialized rendering logic for the power lines in Half Life 2?

    I presume that's basically the same fundamental problem as rendering line art and star fields...

  • supernova87a 504 days ago
    Can anyone explain this a bit more?

    > Stars (along with line art) are one of the most blatant tests for gamma correct rendering — if everything shimmers as you pan around, they are filtering in gamma space instead of linear space. It should all be rock solid.

    Something to do with as the underlying values are shifted between pixels (as you pan around), they increase/decrease pixel brightness inconsistently if you're mapping the intrinsic values to some curve before calculating the appropriate pixel brightness? (I'm unfamiliar with "gamma correct rendering")

    • whatshisface 504 days ago
      If a star is halfway between two pixels, its brightness should be distributed evenly between both pixels so that the total emitted light from that area of your screen is the same as if it was all on one pixel. Applying a nonlinear function to the values will usually mess that up.
    • pornel 504 days ago
      Human perception of brightness is not linearly proportional to the amount of light emitted (if you're in a dark room with 1 candle, and light 1 more candle this makes a big difference, but 100 → 101 candles is imperceptible. Physically both cases add the same amount of light).

      The relationship between perceived brightness and physical amount of light is approximately quadratic (~doubling amount of light is perceived as one incremental brightness step), which is called the gamma curve.

      So in graphics you have to choose whether your numeric brightness values are on a scale relevant to humans (gamma compressed values), or whether they're on a scale that better models physical properties of light (AKA linear light).

      This is generally confusing, and a lot of software gets it wrong (doesn't correct for the gamma curve where needed). For example, try blurring RGB red and green. In many programs it will give you a brown color, which is not the color of mixing red and green light, but a bad math on gamma-compressed values: √(A + B) ≠ √A + √B

      • shmageggy 504 days ago
        > The relationship between perceived brightness and physical amount of light is approximately quadratic (~doubling amount of light is perceived as one incremental brightness step), which is called the gamma curve.

        This sounds logarithmic to me, similar to the decibel system for audio loudness perception

      • supernova87a 504 days ago
        Thanks!
  • himinlomax 504 days ago
    He mentions gamma space and linear space; wouldn't the actual solution for all these filtering problems to move to floating point buffers? Haven't SGI patents expired by now?
  • the__alchemist 504 days ago
    If anyone hasn't played Space Engine, check it out! Labor of love.
  • hoten 504 days ago
    What does it mean to filter in gamma space vs linear space?
    • dTal 504 days ago
      "Filtering" is a graphics term of art that roughly means "interpolation".

      Imagine you want to represent a star with a single max brightness pixel. Suppose you want to smoothly pan that pixel to its neighbor. If you simply switch the pixel off and switch the neighbor on, it will look jumpy. So what you want to do is gradually dim the first pixel, while gradually brightening the second, so that the intensity smoothly moves over instead of jumping.

      The catch: at all times, the sum of the light intensity from both pixels must be constant. You want the star to move, not flicker. Seems easy, right? Subtract 1 from the first pixel and add 1 to the second, keeping the sum the same?

      Wrong! The relationship between the actual light emitted by a pixel and the number we use to represent it in software is not linear. It generally follows a power law, usually with an exponent of about 2.2, since this is a more efficient representation given that human perception also follows a power law. If 255 were actually twice as many photons as 127, you would find that they didn't actually look all that different - the range 0-255 would skew towards most values looking pretty bright. Instead the values "encode" the real brightness, and if you do math on these encoded values you are operating in "gamma space".

      So, in order to correctly do our 1-pixel panning, we need to "decode" our pixel value to linear brightness by taking the logarithm (base 2.2), add 1 / subtract 1 there, and then raise it back up. This is what Carmack means by "filtering in linear space". Obviously, many programmers are not aware of the requirement to do this, and just treat the pixel values as if they were real brightness values. Result: flicker.

      • hoten 504 days ago
        Thanks for breaking it down!

        It seems like this was a necessity to allow for a larger range of representable colours in a single byte. Is this system largely a carry over from more constrained hardware days, or is it still appropriate for modern hardware? Like, we could represent colors with a few more bytes and toss out the gamma encoding to get back to the same level of control. Or am I overlooking more fundamental reasons why this is a good abstraction?

        • midnightclubbed 504 days ago
          Still appropriate for modern hardware. Despite hardware getting faster, display resolutions and refresh rates have also increased to eat the memory bandwidth gains. Especially so at VR resolutions.

          Modern rendering engines will use a mixture of 8bit, 10bit, and 16bit per color channel buffers - the final 'lit' color buffer is where you will generally write fp16 (8bytes per pixel) linear HDR color values, if you can spare the performance hit, as this is where you need the highest range.

        • dTal 504 days ago
          It's a good abstraction, because most of the time you care about perceived brightness. With gamma encoding, 255 looks twice as bright as 127. You are going to need to deal with that perceptual power law at some point no matter what representation you use, so you may as well pick the efficient one. It's probably not a good tradeoff to make everyone deal with a color scale that doesn't look right, and is inefficient, just to make intensity operations linear so that unwary programmers don't trip.
    • DiabloD3 504 days ago
      Light displayed on computer monitors are not linear, instead follow a power law transformation, also known (less precisely) as a non-linear transfer function. Pixels should only be stored in gamma (non-linear) space for final display; all math on them should be performed in linear space instead.

      sRGB has a badly defined gamma space of an encoding of { [0-15] (linear), [16-255] (power of 2.4 over [0-239] with an offset of 16) } with a decoding of 2.2 (but explicitly taking into account "dark room" ambient lighting, not the actual display response). This is often quoted as "power of 2.2 encoding and decoding", and this has never been correct. For modern viewing conditions with a 100 nits monitor in an ambient condition of "bright room", the proper interpretation of sRGB is to read as "actual monitor response of 2.4", which leads to....

      bt1886 defines this as 2.4 encoding combined with a 2.2 and 2.4 decoding where the crossover point is calculated in a way that takes into account actual black levels observed on the device to maintain perceptual quality while avoiding black and white crush (devices with actually zero black (ex: CRT, Plasma, OLED) would just be pure 2.4 decoding). This allows all commercial content (NTSC, PAL, and BT709/BT2020) to display favorably no matter how badly it was mastered originally and no matter how bad the viewing conditions actually are.

      BBC, taking into account "ye average display over the past 30 years", encodes as 2.35 to take into account NTSC, PAL, 2.2, 2.4, and actually-sRGB displays. This is arguably the most correct approach: bt1886 and sRGB/bt709 on displays that are too dark will be too dark but perceptually correct on displays that are too bright; "encode as 2.2" (which is still wrong) will be very dark on every display; 2.35 will look correct on "wrong, but too dark", "wrong, but too light", "actually sRGB", and "actually bt1886" monitors.

      There also is a third (fourth?) standard of 2.6 that is purely found in extremely low display light (40 nits or less), zero ambient completely blacked out conditions, used for movie theatres, due to perceptual changes in human eyesight; sRGB, bt1886, and 2.35 would all be too dark and/or too undersaturated here otherwise.

  • sylware 504 days ago
    BTW, do we get a native elf/linux proper build?

    ...

  • HackerNot 504 days ago
    undefined
  • phendrenad2 504 days ago
    > Stars are perfect point light sources to the eye

    But that isn't true rofl. Stars are, in fact, (extremely) large spheres, and can take up more than one pixel if your pixels are small enough.

    • GuB-42 504 days ago
      He said "to the eye".

      But indeed, not all stars are "point light": the sun as seen from the earth is not. And it is a good thing, because otherwise that would make our sun like a laser, our retinas won't last long...

      • robocat 504 days ago
        > our retinas won't last long

        I think you are mistaking cause end effect. If the sun were a point source, the layout of our eyes (and our behaviours) would have evolved differently to compensate.

        Alternatively, if the diameter of the sun (or the apparent diameter of the sun) quickly changed that much, our eyesight is the least of our worries.

    • fshbbdssbbgdd 504 days ago
      Other than the sun, are there any stars that should take up multiple pixels at the angular resolution available with current display technology?
      • sbierwagen 504 days ago
        Alpha Centauri is 4.344 light years away and about 0.00930093 AU in diameter, and if my math is right should subtend 3.12x10^-25 arc seconds.

        The Oculus Quest 2 has a horizontal FOV of 97 degrees. In order for α Centauri to cover two pixels, you would need a display with 6.2179487x10^26 horizontal pixels. 621 yottapixels. Something like a hundred sextillion times "4K" resolution. And that's linear resolution, tag on some more orders of magnitude for display area.

        There is a reason parallax is used to measure distance to stars, rather than directly measuring their visual diameter.

        • jws 504 days ago
          The atmosphere blurs the starlight in the last miles of its journey. We’ll need a astrophotographer here, but I’m pretty sure Capella, in October, in the norther latitudes is larger than a human eye pixel equivalent. The atmosphere really messes with it at that time. I’ve watched it across a cool,lake with warm surface air through binoculars and it was flashing red to blue at about 8 times a second and seemed large. It’s a bright star so remains visible after being spread.
          • robocat 504 days ago
            Furthermore they are presuming that the eye’s optics and “pixels” are idealised: they have rather a myopic point-of-view.

            Even in outer space, with no atmospheric aberration, there will be some blur for anyone with less than absolutely perfect eyesight.

            There will be a point-spread-function because our optics are not ideal (especially with bright point sources), and blur will result from a variety of causes.

            • sbierwagen 504 days ago
              I poked the numbers above into a Rayleigh criterion calculator and got a number measured in thousands of light years? My comment on if my math was correct was prescient. 10^-25 showing up should have been a hint to myself, whoops.

              (The calculated horizontal resolution required for a 2 pixel wide star from the original post should instead be 100,009,624)

              The corrected number for star apparent diameter should be 0.006983 arcsec. Depending on how finely you want to resolve the disk, that would take somewhere between a 70 to 140m wide telescope. (With 400nm light)

              The largest optical telescope currently under construction is 39.3m wide: https://en.wikipedia.org/wiki/Extremely_Large_Telescope

            • scatterhead 504 days ago
              > there will be some blur for anyone with less than absolutely perfect eyesight.

              Computer graphics are meant to be perceived through human eyeballs. So we can count on the eyes continuing to blur point lights whether you're looking at a screen or at outer space.

              • yencabulator 503 days ago
                It's not that simple. Computer graphics are (currently) displayed on a flat 2D plane at some single distance from the eyeball. Typical vision defects affect different distances differently. Think "reading glasses" vs "driving glasses".

                Also, the game might be simulating an experience from a character's point of view, and expect the player to be wearing glasses to correct their personal vision defects (to the extent possible).

    • justsomehnguy 504 days ago
      Nope.

      They are too far away.

      It's clearly evident if you look at them in a telescope, especially in observatory.

  • abudabi123 504 days ago
    NASA should get gaming on this outreach operation potential and continue-on John Carmack's tradecraft. Seed $57 million to John Carmack's Kerbal Space Program to define and support "perfect" star fields in the compositor.
    • Cullinet 504 days ago
      undefined
      • interroboink 504 days ago
        You got me curious on the creation of NASA. I read through the Executive Order[1] you mentioned (pretty short).

        I'm not sure I quite see the "psycho political" parts you indicate. Seems fairly straightforward, with quotes like "A science research agenda", "the exploration of technologies, demonstrations, and strategies, ... that could be used for sustainable human and robotic exploration", "encourage the interest of Americas youth in studying and pursuing careers in mathematics, science, and engineering...", and so-on.

        Certainly there was a political context in which that order was given, but ... am I missing something?

        [1] https://www.nasa.gov/missions/solarsystem/aldridge_executive...

      • Cullinet 504 days ago
        apologies for the rapid appending of non destructive edits to my above comment, but does what I wrote warrant negative moderation? Are any log my stated facts incorrect? I can understand if cursory reading might lead to a inaccurate bias confirmation of anti American sentiment by someone who is (always understandably) biased against criticism of the American system, but I am a ageing hawk in every way, of no liberal leanings in politics or geopolitical views of any kind. Humanly I'm not at all aggressive, but there hasn't been such a shortfall in strategic stance by America in my nearing retirement lifetime. I advocate the separation of military and scientific space endeavour. The current commercial efforts offer just such a opportunity for so doing. For those who have personal or directly informed recollection, NASA and indeed the early moon mission objectives polled most unfavourably across red and blue States for some years.