I tried modifying the reconstruction loss such that values that are pushed out of bounds do not contribute to the loss and it works as expected in tensorflow after training an autoencoder. However, when I recreated it in pytorch, the resulting image seem to still stick close to the bounds as if normal mse or mad is used.
I have made two different versions of the loss in pytorch. One is a direct translation from tensorflow, and the other clips the loss itself. Neither has the same behavior as it did in pytorch.
Values stick close to the bounds with PyTorch: 
Values vary generously out of bounds with TensorFlow: 
# Tensorflow Implementation def relaxed_reconstruction_loss(y_true, y_pred, minval=-1.0, maxval=1.0, lp=1): clip_mask = tf.logical_or( tf.logical_and(y_true <= minval, y_pred <= minval), tf.logical_and(y_true >= maxval, y_pred >= maxval) ) y_pred_clipped = tf.where(clip_mask, tf.clip_by_value(y_pred, minval, maxval), y_pred) return tf.reduce_mean(tf.reduce_sum(tf.abs(y_true - y_pred_clipped) ** lp, axis=(1, 2, 3))) # Pytorch Implementation A def relaxed_reconstruction_loss(y_true, y_pred, minval=0.0, maxval=1.0, lp=1): clip_mask = ((y_true <= minval) & (y_pred <= minval)) | ((y_true >= maxval) & (y_pred >= maxval)) y_pred_clipped = torch.where(clip_mask, torch.clamp(y_pred, minval, maxval), y_pred) loss = torch.abs(y_true - y_pred_clipped) ** lp return loss.sum(dim=(1, 2, 3)).mean() # Pytorch Implementation B def relaxed_reconstruction_loss(y_true, y_pred, minval=0.0, maxval=1.0, lp=1): loss = torch.abs(y_true - y_pred) ** lp at_min = (y_true <= minval) & (y_pred < minval) at_max = (y_true >= maxval) & (y_pred > maxval) relaxed_mask = at_min | at_max loss = loss * (~relaxed_mask).detach() return loss.sum(dim=(1, 2, 3)).mean() What error am I making in the pytorch version to make it behave the same with tensorflow?
