No jobs
Back to SD 2.1. Mari

Train: Optimised single_pass — charbonnier ramp + sky0.7 area-pooled + 50% full-noise (C2) d7f75b23

Supersedes config 93 (“L1@10 + sky0.7 + 50% full-noise (C)”, run 2026-07-28_10-02-59_4779) after a conceptual review of commit 3cbdf26. Keeps the intent of (C) — supervised sky at 0.7 and 50% of examples at the zero-terminal-SNR endpoint — and fixes four issues that made it uninterpretable or actively harmful. 1. Prediction loss: the hard L2→L1 step at epoch 10 is replaced by an L2→Charbonnier ramp over epochs 10–14. A hard swap roughly doubles both primary losses in one epoch (E|d| = sqrt(2/pi)*sqrt(E[d^2])), shifting the primary/auxiliary balance and gradient clipping at the same instant; pure L1 also has non-vanishing gradient at the optimum, which leaves a dither floor — the opposite of the local-structure goal. Charbonnier is quadratic near zero and linear on large residuals. 2. LR schedule: constant_with_warmup → cosine_with_warmup. A constant LR is what makes the L1-family gradient floor bite; the template's own comment already recommends cosine here. 3. Latent weight pooling: new loss.depth_weight_pooling=area. With min pooling a latent cell mixing valid depth and sky collapses to the sky weight, so 0.7 attenuated exactly the thin structures against sky that this experiment targets. Area pooling keeps the all-or-nothing gate on genuinely missing pixels. 4. VKITTI2 valid_max_depth 300 → 650. VKITTI2 stores 16-bit centimetre depth, so its sky sentinel is 655.35 m; at 300 m the above_max fallback relabelled all real geometry between 300 m and the sentinel as sky and, at sky weight 0.7, actively taught it the 800 m override. Also enables depth.neutralize_invalid_depth on both training sources: under log_depth, clamping mapped 0/negative/-inf pixels onto the NEAR plane (1 m), and the VAE encoded that into neighbouring supervised latents despite zero loss weight. RDS retains valid_max_depth == sky_depth_override == 800, so its above_max sky mask may be empty. This is left unchanged deliberately because the raw RDS sky sentinel was not verified; training now warns at startup and logs realized per-source mask coverage each epoch. Check the rds `sky` fraction on epoch 0 before comparing sources. Requires the reviewed fixes on the training branch (depth_weight_pooling, charbonnier registry entry, neutralize_invalid_depth, all-breakpoint schedule validation, sky-inclusive smoothness region, and the depth_sky/nonsky prediction-loss split).

training failed

Output

pending d7f75b23
Launch Output
Waiting for SLURM job to start...

Failure Metadata

failed
interactive.monitor.poll INT_EXIT_CODE_NONZERO

Interactive command exited with code 255

Context JSON
{
  "reason": null,
  "exitCode": 255,
  "remotePid": 3483876,
  "computePid": null,
  "allocationJobId": null
}

Datasets

14
Dataset clear_root_rds
Type rgb
Split full
Path /cluster/work/igp_psr/drothenpiele/data/rds/rgb.zip/
Dataset depth_root_rds
Type depth
Split full
Path /cluster/work/igp_psr/drothenpiele/data/rds/depth.zip/
Dataset intrinsics_root_rds
Type intrinsics
Split full
Path /cluster/work/igp_psr/drothenpiele/data/rds/calibration.zip/
Dataset train_hazy_root_rds
Type rgb
Split train
Path /cluster/work/igp_psr/drothenpiele/data/rds/gloomy_fixed/foggy_rgb.zip/
Dataset val_hazy_root_rds
Type rgb
Split val
Path /cluster/work/igp_psr/drothenpiele/data/rds/gloomy_fixed/foggy_rgb.zip/
Dataset clear_root_vkitti2
Dataset VKITTI2
Type rgb
Split full
Path /cluster/work/igp_psr/drothenpiele/data/vkitti_2.0.3_rgb.zip
Dataset depth_root_vkitti2
Dataset VKITTI2
Type depth
Split full
Path /cluster/work/igp_psr/drothenpiele/data/vkitti_2.0.3_depth.zip
Dataset intrinsics_root_vkitti2
Dataset VKITTI2
Type intrinsics
Split full
Path /cluster/work/igp_psr/drothenpiele/data/vkitti_2.0.3_textgt.zip/
Dataset train_hazy_vkitti
Type rgb
Split train
Path /cluster/work/igp_psr/drothenpiele/data/vkitti_next/gpu_2/foggy_rgb.zip/
Dataset val_hazy_vkitti
Type rgb
Split val
Path /cluster/work/igp_psr/drothenpiele/data/vkitti_next/gpu_2/foggy_rgb.zip/
Split muses_rgb
Dataset muses
Type rgb
Split fog_day
Path /cluster/work/igp_psr/drothenpiele/data/muses/frame_camera_trainvaltest.zip/
Split muses_sparse_depth
Dataset muses
Type sparse_depth
Split full
Path /cluster/work/igp_psr/drothenpiele/data/muses/lidar_trainvaltest.zip/
Split muses_intrinsics
Dataset muses
Type intrinsics
Split full
Path /cluster/work/igp_psr/drothenpiele/data/muses/frame_camera_trainvaltest.zip/
Split muses_extrinsics
Dataset muses
Type camera_extrinsics
Split full
Path /cluster/work/igp_psr/drothenpiele/data/muses/frame_camera_trainvaltest.zip/

Execution Artifacts

2
run.sh

Published Exports

Launch-Owned Exports

These semantic handles are what downstream pipelines should resolve against. Auto-published exports come from config-template metadata captured on this launch.

Published

This launch does not publish any exports yet.

Raw Artifacts

Parameters

278

Typed Parameters

Dataset clear_root_rds
real-drive-sim / full (4)
Dataset depth_root_rds
real-drive-sim / full (5)
Dataset intrinsics_root_rds
real-drive-sim / full (6)
Dataset train_hazy_root_rds
real-drive-sim / train (195)
Dataset val_hazy_root_rds
real-drive-sim / val (196)
Dataset clear_root_vkitti2
VKITTI2 / full (1)
Dataset depth_root_vkitti2
VKITTI2 / full (2)
Dataset intrinsics_root_vkitti2
VKITTI2 / full (12)
Dataset train_hazy_vkitti
VKITTI2 / train (216)
Dataset val_hazy_vkitti
VKITTI2 / val (238)
Split muses_rgb
muses / fog_day (77)
Split muses_sparse_depth
muses / full (76)
Split muses_intrinsics
muses / full (82)
Split muses_extrinsics
muses / full (81)

Simple Parameters

clip
true
crop
center
seed
42
tracker
wandb
use_ema
false
gpu_type
rtx_4090:1
job_name
sd_train_concat
norm_max
1
norm_min
-1
run_name
joint-opt-charbonnier-sky-area-noise
tmp_size
10G
adam_8bit
true
data_kind
real_drive_sim
ema_decay
0.9999
ema_dtype
float32
log_every
10
max_depth
800
min_depth
1
adam_beta1
0.9
adam_beta2
0.999
batch_size
16
clear_root
4
depth_mode
metric_log
depth_root
5
image_size
[384, 768]
joint_mode
single_pass
log_images
true
num_epochs
200
output_dir
/cluster/scratch/drothenpiele/SD21/exp_1
time_limit
2-00:00:00
mem_per_cpu
8G
num_workers
4
adam_epsilon
1e-8
adam_foreach
false
aspect_ratio
[1, 2]
camera_model
pinhole
conditioning
concat
lambda_depth
1
lambda_point
0.0
min_lr_ratio
0.01
project_name
joint-dehazing
use_geometry
false
warmup_ratio
0.05
weight_decay
0
cpus_per_task
8
lambda_camera
0.0
lambda_dehaze
1
lambda_eg_ssi
0.0
learning_rate
0.00004
max_depth_rds
800
max_grad_norm
1
min_depth_rds
0.00001
val_hazy_root
104
camera_enabled
true
freeze_encoder
false
log_visibility
true
num_log_images
4
use_depth_head
true
val_batch_size
10
camera_fallback
invalid
depth_head_type
base_residual
enable_xformers
true
euler_train_dir
/cluster/work/igp_psr/drothenpiele/data/out/train/sd21-joint
lambda_depth_tv
0.0002
mixed_precision
fp16
prediction_type
v_prediction
source_kind_rds
real_drive_sim
train_hazy_root
103
visibility_cmap
viridis
camera_embedding
sine
camera_emit_rays
false
camera_mlp_ratio
4
camera_num_heads
4
camera_token_dim
256
depth_noise_type
annealed_multires
lambda_log_depth
0.0
lr_schedule_type
cosine_with_warmup
min_max_quantile
0.02
no_decay_enabled
false
pretrained_model
sd2-community/stable-diffusion-2-1
sky_depth_meters
800
camera_num_layers
2
camera_num_tokens
4
conv_in_init_mode
joint_marigold
geometry_lambda_z
0.15
geometry_timestep
0
joint_weight_psnr
1
joint_weight_ssim
10
lambda_depth_head
0.2
lambda_lambda_mse
0.0
lambda_visibility
0.05
max_depth_vkitti2
650
min_depth_vkitti2
0.00001
sampling_strategy
source_balanced
source_weight_rds
1
depth_ensemble_tol
0.001
lambda_uncertainty
0.0
model_depth_output
vae_decode
val_every_n_epochs
1
camera_feature_pool
avg
depth_ensemble_size
1
geometry_lambda_phi
1
num_inference_steps
25
rds_val_max_samples
128
sampling_epoch_size
all
save_every_n_epochs
2
source_kind_vkitti2
vkitti2
use_visibility_head
true
warmup_start_factor
0.001
camera_metadata_root
null
conv_in_target_scale
0.7071
external_camera_mode
none
geometry_loss_robust
charbonnier
lambda_vae_depth_rec
0.0
lambda_visibility_tv
0.005
sky_depth_meters_rds
800
camera_branch_enabled
true
camera_embedding_size
latent
depth_gradient_robust
charbonnier
depth_multires_levels
4
encoder_learning_rate
0.000007
geometry_head_enabled
false
geometry_lambda_theta
1
lambda_depth_gradient
0.05
lambda_geometry_total
0.0
source_weight_vkitti2
1
use_cross_task_fusion
true
vae_decoder_trainable
false
visibility_rank_pairs
128
weight_decay_backbone
0
weight_decay_no_decay
0
zero_grad_set_to_none
true
camera_feature_sources
auto
camera_metadata_format
euler_loading
depth_ensemble_max_res
1024
depth_smoothness_space
latent_x0
freeze_cross_attention
true
gradient_checkpointing
true
inference_depth_output
vae_decode
lambda_geo_consistency
0.0
lambda_visibility_rank
0
use_task_skip_adapters
false
visibility_rank_margin
0.05
camera_embedding_stride
8
camera_prediction_space
residual_intrinsics
depth_ensemble_max_iter
2
depth_multires_strength
0.9
depth_smoothness_scales
4
geometric_pairs_enabled
false
joint_noise_correlation
0
joint_weight_delta1_pct
0.5
keep_last_n_checkpoints
5
validation_depth_output
vae_decode
visibility_target_gamma
4
visibility_warmup_steps
1000
vkitti2_val_max_samples
0
cross_task_fusion_blocks
[0,1,2,3]
cross_task_fusion_detach
false
cross_task_fusion_kernel
adaptive
depth_ensemble_reduction
median
depth_normalization_type
log_depth
geometry_camera_encoding
sine
geometry_head_num_blocks
2
joint_weight_abs_rel_pct
0.5
sky_depth_meters_vkitti2
800
camera_intrinsics_loading
hierarchical
depth_diagnostics_enabled
true
depth_head_residual_scale
0.05
geometry_depth_convention
z_depth
geometry_metric_depth_max
800
geometry_metric_depth_min
1
num_inference_steps_depth
20
vae_decoder_learning_rate
0.000001
visibility_apply_to_depth
false
camera_residual_logit_clip
4
depth_head_hidden_channels
64
depth_resize_interpolation
nearest
depth_smoothness_normalize
true
detach_rays_for_point_loss
true
enable_efficient_attention
true
lambda_visibility_preserve
0.05
visibility_apply_to_dehaze
true
visibility_apply_to_fusion
false
visibility_hidden_channels
32
visibility_target_quantile
0.9
weight_decay_depth_decoder
0
weight_decay_geometry_head
0
camera_require_for_training
true
camera_sine_num_frequencies
null
checkpoint_selection_metric
valset/muses/mae
cross_task_fusion_dilations
[1,2,4]
depth_decoder_learning_rate
0.00002
emit_camera_full_resolution
true
geometry_eg_ssi_edge_source
hazy
geometry_encoder_input_mode
hazy_hazy
geometry_head_learning_rate
0.00004
geometry_output_uncertainty
true
geometry_ray_representation
unit
gradient_accumulation_steps
1
validation_geometry_enabled
false
weight_decay_dehaze_decoder
0
camera_output_representation
angles
camera_valid_for_approximate
false
cross_task_fusion_directions
["rgb_to_depth","depth_to_rgb"]
dehaze_decoder_learning_rate
0.00002
depth_head_context_dilations
[1, 2, 4]
depth_smoothness_edge_source
clear
depth_smoothness_edge_weight
10
geometry_head_residual_scale
0.05
lambda_depth_edge_smoothness
0.002
lambda_rgb_depth_consistency
0.1
weight_decay_geometry_camera
0
depth_diagnostics_edge_source
dehazed
depth_diagnostics_edge_weight
10
depth_head_base_upsample_mode
bilinear
geometry_camera_learning_rate
0.00004
geometry_head_hidden_channels
128
geometry_sine_num_frequencies
64
visibility_decoder_grad_scale
1.0
camera_approximate_fov_degrees
60
checkpoint_selection_direction
min
cross_task_fusion_learned_gate
true
geometry_use_deterministic_vae
true
native_crop_lower_region_ratio
0.05
depth_multires_downscale_factor
2
lambda_depth_gt_edge_smoothness
0
validation_geometry_camera_mode
predicted
validation_geometry_output_mode
geometry_head
camera_include_input_in_encoding
null
camera_intrinsics_metadata_scope
intrinsics
cross_task_fusion_gate_init_bias
0
cross_task_fusion_sender_lowpass
false
depth_head_use_depthwise_context
true
geometric_pairs_views_per_sample
2
geometry_condition_on_visibility
false
geometry_detach_encoder_features
true
cross_task_fusion_hidden_channels
64
depth_diagnostics_normalize_depth
true
native_crop_centered_region_ratio
0.0
native_crop_lower_region_position
center_bottom
rgb_depth_consistency_edge_weight
4
rgb_depth_consistency_weight_mode
hallucinated
geometric_pairs_emit_pair_metadata
true
geometry_include_input_in_encoding
false
lambda_depth_multiscale_smoothness
0.001
depth_ensemble_regularizer_strength
0.02
native_crop_centered_region_position
center
cross_task_fusion_attention_kv_tokens
1024
cross_task_fusion_attention_max_block
-1
depth_diagnostics_road_edge_threshold
0.08
geometry_head_detach_camera_embedding
true
depth_diagnostics_road_bottom_fraction
0.4
geometry_encoder_no_grad_when_detached
true
cross_task_fusion_sender_lowpass_kernel
3
depth_diagnostics_high_frequency_kernel
5
validation_geometry_compute_edge_metrics
true
validation_geometry_compute_point_metrics
false
validation_geometry_max_points_per_sample
20000
validation_geometry_save_geometry_samples
true
validation_geometry_compute_camera_metrics
true
validation_geometry_compute_resolution_sweep
false
validation_geometry_max_point_metric_samples
8
validation_geometry_save_point_cloud_samples
false
validation_geometry_compute_uncertainty_metrics
true
Raw JSON
{
  "clip": "true",
  "crop": "center",
  "seed": 42,
  "tracker": "wandb",
  "use_ema": "false",
  "gpu_type": "rtx_4090:1",
  "job_name": "sd_train_concat",
  "norm_max": 1,
  "norm_min": -1,
  "run_name": "joint-opt-charbonnier-sky-area-...

Events

Launch Events

0
No launch events recorded.
Euler View - ML Experiment Monitor