A neural network wins only in training

A neural network achieves much lower training loss than a linear model but no consistent outer-period improvement. What would justify keeping it?

Retain it only if additional independent evidence supports useful net performance or another operational benefit that outweighs complexity. Compare regularisation, sample-size sensitivity, stability, inference cost and failure modes under the same protocol. A more expressive architecture is not itself a benefit. Record unsuccessful experiments so repeated architecture search does not silently turn the outer periods into tuning data.