Model Evaluation Table

Q: Is real world fog still a good priod for foundation models?

princeton_stf · depth

We do see, however, deterioration (e.g. see muses evaluation, rds-camera noise model)

Model Evaluation Table

Fog density in SeeingThroughFog vs. Muses

Using the sparse LiDAR scans, and combining them with some heuristics from DCP and contrast mapping, even the "dense" split showed a much more "clear" split than the fog-split of muses.

image.png
princeton_stf (light) distribution
image.png
princeton_stf (dense) distribution
image.png
Distribution over muses
{
    "daytime": {
        "day": false,
        "night": true
    },
    "fog": {
        "no": true,
        "yes": {
            "denseFog": false,
            "lightFog": false
        }
    },
    "infrastructure": {
        "highway": false,
        "inCity": true,
        "suburban": false
    },
    "point_removed": 12,
    "precipitation": {
        "no": false,
        "yes": {
            "rain": true,
            "snow": {
                "heavySnow": false,
                "lightSnow": false
            }
        }
    },
    "roadState": {
        "dry": true,
        "fullSnow": false,
        "partialSnow": false,
        "wet": false
    },
    "sidewalkState": {
        "clean": true,
        "partialSnow": false,
        "snowCovered": false
    },
    "tunnel": false,
    "twilight": false
}

Open / Going on

  • Check original marigold by training on noisy data (exlusively)
  • See at what noise level our model begins to fail
Step Intuition
Foggy (SeeingThroughFog ~ 13k) Keep the real camera/fog look instead of synthesizing everything.
Estimate pseudo-depth Use LiDAR + depth completion to know where fog should accumulate.
Clean unreliable LiDAR Filter for valid LiDAR areas
Estimate local airlight Dense fog color/brightness varies across sky, lamps, road, and cars.
Add incremental fog Increase fog density on top of existing fog. (+ camera noise model / heterogenous fog etc.)