Finite-difference gradient check

Question

Implement a central finite-difference gradient approximation for a scalar objective on a floating-point vector. Why should the step not be arbitrarily tiny?

Short answer

Use symmetric perturbations in each coordinate. Truncation error falls as the step decreases, but floating-point cancellation eventually grows; compare several sensible step sizes and use absolute and relative tolerances.

Original question and answer written by SelfMadeQuant.com.

See the hint and the full worked solution in the question bank (free account).

Open the worked solution

Lessons behind this question

The Supervised Learning Problem and the ML Pipeline
Matrix Algebra: Operations, Rules, and the Transpose
Kernel, Image, and the Rank-Nullity Theorem

Start with a free lesson: Basic Statistics and Regression is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.

Read it free