Let X⊆R,X \subseteq \mathbb{R}, f:X→Rf: X \rightarrow \mathbb{R} be a function and x0∈Xx_0 \in X be a limit point of X.X. The function ff is differentiable at x0x_0 if and only if the limit of the difference quotient f(x)−f(x0)x−x0\frac{f(x)-f(x_0)}{x-x_0} as x→x0x \rightarrow x_0 exists.

Let X⊆R,X \subseteq \mathbb{R}, f:X→Rf: X \rightarrow \mathbb{R} be a function and x0∈Xx_0 \in X be a limit point of XX at which ff is differentiable. The derivative of ff at x0,x_0, written df(x0)dx\frac{df(x_0)}{dx} or f′(x0),f'(x_0), is the limit of the difference quotient.

df(x0)dx:=lim⁡x→x0f(x)−f(x0)x−x0\frac{df(x_0)}{dx} := \lim_{x \rightarrow x_0} \frac{f(x)-f(x_0)}{x-x_0}

If the limit does not exist then df(x0)dx\frac{df(x_0)}{dx} is undefined and ff is not differentiable at x0.x_0.

Let X⊆RX \subseteq \mathbb{R} and f:X→Rf: X \rightarrow \mathbb{R} be a function. The function ff is differentiable on XX if and only if ff is differentiable at every x0∈X.x_0 \in X.

Let X⊆RX \subseteq \mathbb{R} and f:X→Rf: X \rightarrow \mathbb{R} be differentiable on X.X. The derivative of ff is the function f′:X→Rf': X \rightarrow \mathbb{R} that maps each x∈Xx \in X to the derivative of ff at x.x. Writing the nearby point as x+hx+h with step size h→0h \rightarrow 0 gives the equivalent form

df(x)dx=lim⁡h→0f(x+h)−f(x)h.\frac{df(x)}{dx} = \lim_{h \rightarrow 0} \frac{f(x+h)-f(x)}{h}.

The derivative is also written ddxf(x).\frac{d}{dx}f(x).

Let X⊆R,X \subseteq \mathbb{R}, f:X→Rf: X \rightarrow \mathbb{R} be a function and x0∈X.x_0 \in X. If ff is differentiable at x0x_0 then ff is continuous at x0,x_0, so a function that is differentiable on XX is continuous on X.X. The converse is false: f(x)=∣x∣f(x)=\lvert x\rvert is continuous at 00 but not differentiable there, since ∣x∣x\frac{\lvert x\rvert}{x} is 11 for x>0x>0 and −1-1 for x<0x<0 and so has no limit as x→0.x\rightarrow 0.

Let X⊆RX \subseteq \mathbb{R} and f:X→Rf: X \rightarrow \mathbb{R} be differentiable on X.X. The second derivative of ff at x,x, written f′′(x)=d2f(x)dx2,f''(x)=\frac{d^2f(x)}{dx^2}, is the derivative of f′f' at xx wherever f′f' is differentiable. The kk-th derivative of f,f, written f(k)(x)=dkf(x)dxk,f^{(k)}(x)=\frac{d^kf(x)}{dx^k}, is defined recursively as the derivative of f(k−1)f^{(k-1)} with f(0)=f.f^{(0)}=f.

Let X⊆R,X \subseteq \mathbb{R}, f:X→Rf: X \rightarrow \mathbb{R} be a function and k∈N.k\in\mathbb{N}. The function ff is kk times continuously differentiable on X,X, written f∈Ck(X),f\in C^k(X), if and only if f(k)f^{(k)} exists and is continuous on X.X. The set C0(X)C^0(X) is the set of continuous functions on X.X. The function ff is smooth on X,X, written f∈C∞(X),f\in C^{\infty}(X), if and only if f(k)f^{(k)} exists for every k∈N.k\in\mathbb{N}.

Let X⊆R,X \subseteq \mathbb{R}, f,g:X→Rf,g:X\rightarrow\mathbb{R} be differentiable at x∈Xx\in X and c∈R.c\in\mathbb{R}. Derivatives of constants, sums and scalar multiples satisfy the following rules.

ddxc=0,ddx(f(x)±g(x))=f′(x)±g′(x),ddx(c f(x))=c f′(x)\frac{d}{dx}c=0,\qquad\frac{d}{dx}\left(f(x)\pm g(x)\right)=f'(x)\pm g'(x),\qquad\frac{d}{dx}\left(c\,f(x)\right)=c\,f'(x)

Let n∈N.n\in\mathbb{N}. The power rule states that the derivative of xnx^n is nn times xn−1.x^{n-1}.

ddxxn=n xn−1\frac{d}{dx}x^n=n\,x^{n-1}

The same formula holds for any real exponent nn when x>0.x>0. Together with the rules for sums and scalar multiples, every polynomial is differentiable on R.\mathbb{R}.

Let X⊆RX \subseteq \mathbb{R} and f,g:X→Rf,g:X\rightarrow\mathbb{R} be differentiable at x∈X.x\in X. The product rule states that f gf\,g is differentiable at xx with

ddx(f(x) g(x))=f′(x) g(x)+f(x) g′(x).\frac{d}{dx}\left(f(x)\,g(x)\right)=f'(x)\,g(x)+f(x)\,g'(x).

Let X⊆RX \subseteq \mathbb{R} and f,g:X→Rf,g:X\rightarrow\mathbb{R} be differentiable at x∈Xx\in X with g(x)≠0.g(x)\neq 0. The quotient rule states that fg\frac{f}{g} is differentiable at xx with

ddx(f(x)g(x))=f′(x) g(x)−f(x) g′(x)g(x)2.\frac{d}{dx}\left(\frac{f(x)}{g(x)}\right)=\frac{f'(x)\,g(x)-f(x)\,g'(x)}{g(x)^2}.

Let X,Y⊆R,X,Y\subseteq\mathbb{R}, f:X→Yf:X\rightarrow Y be differentiable at x∈Xx \in X and g:Y→Rg:Y\rightarrow\mathbb{R} be differentiable at f(x).f(x). The chain rule states that the composition g∘fg\circ f is differentiable at xx with

ddxg(f(x))=g′(f(x)) f′(x).\frac{d}{dx}g\left(f(x)\right)=g'\left(f(x)\right)\,f'(x).

Let XX be an interval and f:X→Rf:X\rightarrow\mathbb{R} be continuous and strictly monotonic. The inverse function rule states that if ff is differentiable at xx with f′(x)≠0f'(x)\neq 0 then the inverse function f−1f^{-1} is differentiable at y=f(x)y=f(x) with

ddyf−1(y)=1f′(f−1(y)).\frac{d}{dy}f^{-1}(y)=\frac{1}{f'\left(f^{-1}(y)\right)}.

Let X⊆R,X \subseteq \mathbb{R}, f:X→Rf:X\rightarrow\mathbb{R} be a function and x0∈X.x_0\in X. The point x0x_0 is a local maximum of ff if and only if there exists δ>0\delta>0 such that f(x)≤f(x0)f(x)\leq f(x_0) for all x∈Xx\in X with ∣x−x0∣<δ.\lvert x-x_0\rvert<\delta. The point x0x_0 is a local minimum of ff if and only if there exists δ>0\delta>0 such that f(x)≥f(x0)f(x)\geq f(x_0) for all x∈Xx\in X with ∣x−x0∣<δ.\lvert x-x_0\rvert<\delta.

Let X⊆R,X \subseteq \mathbb{R}, f:X→Rf:X\rightarrow\mathbb{R} be a function and x0∈X.x_0\in X. The point x0x_0 is a global maximum of ff if and only if f(x)≤f(x0)f(x)\leq f(x_0) for all x∈X.x\in X. The point x0x_0 is a global minimum of ff if and only if f(x)≥f(x0)f(x)\geq f(x_0) for all x∈X.x\in X.

Let X⊆R,X \subseteq \mathbb{R}, f:X→Rf:X\rightarrow\mathbb{R} be a function and x0∈X.x_0\in X. The point x0x_0 is an extremum of ff if and only if it is a local or global maximum or minimum of f.f.

Let X⊆RX \subseteq \mathbb{R} and x0∈X.x_0\in X. The point x0x_0 is an interior point of XX if and only if some open interval (x0−δ,x0+δ)(x_0-\delta,x_0+\delta) with δ>0\delta>0 is contained in X.X.

Let X⊆R,X \subseteq \mathbb{R}, f:X→Rf:X\rightarrow\mathbb{R} be a function and x0∈Xx_0\in X be an interior point of XX at which ff is differentiable. Fermat’s theorem states that if x0x_0 is a local extremum of ff then the derivative of ff at x0x_0 is zero.

f′(x0)=0f'(x_0)=0

Let X⊆R,X \subseteq \mathbb{R}, f:X→Rf:X\rightarrow\mathbb{R} be a function and x0∈X.x_0\in X. The point x0x_0 is a stationary point of ff if and only if ff is differentiable at x0x_0 with f′(x0)=0.f'(x_0)=0. Stationary points have the following properties:

  • a stationary point need not be an extremum: f(x)=x3f(x)=x^3 has f′(0)=0f'(0)=0 but 00 is neither a local maximum nor a local minimum;
  • an extremum that is not an interior point, or at which ff is not differentiable, need not be a stationary point;
  • the extrema of ff on [a,b][a,b] are found among the stationary points, the endpoints and the points of non-differentiability.

Let f:[a,b]→Rf:[a,b]\rightarrow\mathbb{R} be continuous on [a,b][a,b] and differentiable on (a,b)(a,b) with f(a)=f(b).f(a)=f(b). Rolle’s theorem states that the derivative of ff is zero at some point strictly between aa and b.b.

∃ c∈(a,b):f′(c)=0\exists\,c\in(a,b) : f'(c)=0

Let f:[a,b]→Rf:[a,b]\rightarrow\mathbb{R} be continuous on [a,b][a,b] and differentiable on (a,b).(a,b). The mean value theorem states that the derivative of ff equals the average rate of change of ff over [a,b][a,b] at some point strictly between aa and b.b.

∃ c∈(a,b):f′(c)=f(b)−f(a)b−a\exists\,c\in(a,b) : f'(c)=\frac{f(b)-f(a)}{b-a}

Let XX be an interval and f:X→Rf:X\rightarrow\mathbb{R} be differentiable. The mean value theorem has the following consequences:

  • if f′(x)=0f'(x)=0 for all x∈Xx\in X then ff is constant on X;X;
  • if f′(x)≥0f'(x)\geq 0 for all x∈Xx\in X then ff is monotonically increasing, and if f′(x)>0f'(x)>0 for all x∈Xx\in X then ff is strictly increasing;
  • if ∣f′(x)∣≤L\lvert f'(x)\rvert\leq L for all x∈Xx\in X then ff is Lipschitz continuous on XX with Lipschitz constant L.L.

Let X⊆R,X \subseteq \mathbb{R}, f:X→Rf:X\rightarrow\mathbb{R} be a function and x0x_0 be an interior stationary point of ff at which f′′(x0)f''(x_0) exists. The second derivative test states that x0x_0 is a local minimum if f′′(x0)>0f''(x_0)>0 and a local maximum if f′′(x0)<0.f''(x_0)<0. If f′′(x0)=0f''(x_0)=0 the test is inconclusive.

Let f,gf,g be differentiable on an open interval containing x0,x_0, except possibly at x0x_0 itself, with g′(x)≠0g'(x)\neq 0 on that interval away from x0.x_0. L’Hôpital’s rule states that if the limit of the ratio of derivatives exists, and either lim⁡x→x0f(x)=lim⁡x→x0g(x)=0\lim_{x\rightarrow x_0}f(x)=\lim_{x\rightarrow x_0}g(x)=0 or ∣g(x)∣\lvert g(x)\rvert grows without bound as x→x0,x\rightarrow x_0, then the limit of the ratio of the functions equals the limit of the ratio of their derivatives.

lim⁡x→x0f(x)g(x)=lim⁡x→x0f′(x)g′(x)\lim_{x\rightarrow x_0}\frac{f(x)}{g(x)}=\lim_{x\rightarrow x_0}\frac{f'(x)}{g'(x)}

Here ∣g(x)∣\lvert g(x)\rvert grows without bound as x→x0x\rightarrow x_0 means that for every M>0M>0 there exists δ>0\delta>0 with ∣g(x)∣>M\lvert g(x)\rvert>M whenever 0<∣x−x0∣<δ.0<\lvert x-x_0\rvert<\delta.

Let XX be an interval, x0∈Xx_0\in X and f:X→Rf:X\rightarrow\mathbb{R} be kk times differentiable at x0.x_0. The Taylor polynomial of degree kk of ff at x0x_0 is the polynomial whose derivatives at x0x_0 up to order kk agree with those of f.f.

Tk(x):=∑i=0kf(i)(x0)i!(x−x0)iT_k(x):=\sum_{i=0}^{k}\frac{f^{(i)}(x_0)}{i!}\left(x-x_0\right)^i

Let XX be an interval, x0∈Xx_0\in X and f:X→Rf:X\rightarrow\mathbb{R} be kk times differentiable at x0x_0 with Taylor polynomial Tk.T_k. The remainder of the Taylor polynomial is the difference Rk(x):=f(x)−Tk(x).R_k(x):=f(x)-T_k(x).

Let XX be an interval, x0∈Xx_0\in X and f:X→Rf:X\rightarrow\mathbb{R} be kk times differentiable at x0.x_0. Taylor’s theorem states that the remainder vanishes faster than (x−x0)k\left(x-x_0\right)^k as x→x0,x\rightarrow x_0,

lim⁡x→x0Rk(x)(x−x0)k=0,\lim_{x\rightarrow x_0}\frac{R_k(x)}{\left(x-x_0\right)^k}=0,

and that if ff is k+1k+1 times differentiable on the open interval between x0x_0 and xx then there exists cc strictly between x0x_0 and xx with

Rk(x)=f(k+1)(c)(k+1)!(x−x0)k+1.R_k(x)=\frac{f^{(k+1)}(c)}{(k+1)!}\left(x-x_0\right)^{k+1}.

The case k=0k=0 is the mean value theorem.

Let XX be an interval, x0∈Xx_0\in X and f:X→Rf:X\rightarrow\mathbb{R} be smooth at x0.x_0. The Taylor series of ff at x0x_0 is the power series centred at x0x_0 whose coefficients are the derivatives of ff at x0x_0 divided by factorials.

∑i=0∞f(i)(x0)i!(x−x0)i\sum_{i=0}^{\infty}\frac{f^{(i)}(x_0)}{i!}\left(x-x_0\right)^i

Let XX be an interval, x0∈Xx_0\in X and f:X→Rf:X\rightarrow\mathbb{R} be smooth at x0.x_0. The function ff is analytic at x0x_0 if and only if its Taylor series at x0x_0 converges to f(x)f(x) for all xx in some open interval containing x0.x_0. A smooth function need not be analytic, since its Taylor series may converge to a different function.

Let ∑n=0∞an(x−c)n\sum_{n=0}^{\infty} a_n\left(x-c\right)^n be a power series with radius of convergence R>0.R>0. The term-by-term derivative of the power series is the power series obtained by differentiating each term.

∑n=1∞n an(x−c)n−1\sum_{n=1}^{\infty} n\,a_n\left(x-c\right)^{n-1}

A power series is differentiable on (c−R,c+R),\left(c-R,c+R\right), its derivative is its term-by-term derivative and the term-by-term derivative has the same radius of convergence.

ddx∑n=0∞an(x−c)n=∑n=1∞n an(x−c)n−1\frac{d}{dx}\sum_{n=0}^{\infty} a_n\left(x-c\right)^n=\sum_{n=1}^{\infty} n\,a_n\left(x-c\right)^{n-1}

Repeating this, a power series is smooth on its open interval of convergence, its coefficients are ak=f(k)(c)k!,a_k=\frac{f^{(k)}(c)}{k!}, so the series is its own Taylor series at c,c, and it is analytic there. Differentiating the exponential, cosine and sine series term by term gives

exp⁡′=exp⁡,cos⁡′=−sin⁡,sin⁡′=cos⁡.\exp'=\exp,\qquad\cos'=-\sin,\qquad\sin'=\cos.

By the inverse function rule, the derivative of the natural logarithm is ddxlog⁡ex=1x\frac{d}{dx}\log_e x=\frac{1}{x} for x>0.x>0. Since exp⁡′=exp⁡>0\exp'=\exp>0 on R,\mathbb{R}, the exponential function is strictly increasing, and the real power ax=exp⁡(xlog⁡ea)a^x=\exp\left(x\log_e a\right) has derivative axlog⁡eaa^x\log_e a for a>0.a>0.

Let x‾=(x1,…,xn)∈Rn\underline{x}=\left(x_1,\ldots,x_n\right)\in\mathbb{R}^n and y‾∈Rn.\underline{y}\in\mathbb{R}^n. The Euclidean distance between x‾\underline{x} and y‾\underline{y} is the square root of the sum of the squared differences of their coordinates.

∥x‾−y‾∥:=∑i=1n(xi−yi)2\lVert\underline{x}-\underline{y}\rVert:=\sqrt{\sum_{i=1}^{n}\left(x_i-y_i\right)^2}

The Euclidean length of x‾,\underline{x}, written ∥x‾∥,\lVert\underline{x}\rVert, is the Euclidean distance from x‾\underline{x} to 0‾.\underline{0}.

Let X⊆Rn.X\subseteq\mathbb{R}^n. The set XX is open if and only if every point x‾∈X\underline{x}\in X has a δ>0\delta>0 such that every y‾\underline{y} with ∥y‾−x‾∥<δ\lVert\underline{y}-\underline{x}\rVert<\delta lies in X.X.

Let X⊆RnX\subseteq\mathbb{R}^n be open, f:X→Rf:X\rightarrow\mathbb{R} be a function, x‾∈X,\underline{x}\in X, i∈{1,…,n}i\in\{1,\ldots,n\} and e‾i∈Rn\underline{e}_i\in\mathbb{R}^n be the ii-th standard basis vector. The partial derivative of ff with respect to xix_i at x‾\underline{x} is the derivative of ff along the ii-th coordinate with the other coordinates held fixed.

∂f(x‾)∂xi:=lim⁡h→0f(x‾+h e‾i)−f(x‾)h\frac{\partial f(\underline{x})}{\partial x_i}:=\lim_{h\rightarrow 0}\frac{f\left(\underline{x}+h\,\underline{e}_i\right)-f(\underline{x})}{h}

Let X⊆RnX\subseteq\mathbb{R}^n be open, f:X→Rf:X\rightarrow\mathbb{R} be a function and x‾∈X\underline{x}\in X be a point at which all partial derivatives of ff exist. The gradient of ff at x‾\underline{x} is the vector of partial derivatives.

∇f(x‾):=(∂f(x‾)∂x1,…,∂f(x‾)∂xn)\nabla f(\underline{x}):=\left(\frac{\partial f(\underline{x})}{\partial x_1},\ldots,\frac{\partial f(\underline{x})}{\partial x_n}\right)

Let X⊆RnX\subseteq\mathbb{R}^n be open, f:X→Rf:X\rightarrow\mathbb{R} be a function and x‾0∈X\underline{x}_0\in X be a point at which the gradient of ff exists. The point x‾0\underline{x}_0 is a stationary point of ff if and only if ∇f(x‾0)=0‾.\nabla f(\underline{x}_0)=\underline{0}. For n=1n=1 this is the earlier definition of a stationary point.

Let X⊆RnX\subseteq\mathbb{R}^n be open, f:X→Rf:X\rightarrow\mathbb{R} be a function, x‾∈X\underline{x}\in X and v‾∈Rn.\underline{v}\in\mathbb{R}^n. The directional derivative of ff at x‾\underline{x} in the direction v‾\underline{v} is the derivative of ff along the line through x‾\underline{x} in the direction v‾.\underline{v}.

Dv‾f(x‾):=lim⁡h→0f(x‾+h v‾)−f(x‾)hD_{\underline{v}}f(\underline{x}):=\lim_{h\rightarrow 0}\frac{f\left(\underline{x}+h\,\underline{v}\right)-f(\underline{x})}{h}

The partial derivatives are the directional derivatives along the standard basis vectors.

Let X⊆RnX\subseteq\mathbb{R}^n be open, f:X→Rf:X\rightarrow\mathbb{R} be a function and x‾∈X.\underline{x}\in X. The function ff is differentiable at x‾\underline{x} if and only if there exists g‾∈Rn\underline{g}\in\mathbb{R}^n such that the error of the linear approximation f(x‾)+∑i=1ngihif(\underline{x})+\sum_{i=1}^{n}g_i h_i vanishes faster than ∥h‾∥.\lVert\underline{h}\rVert.

lim⁡h‾→0‾∣f(x‾+h‾)−f(x‾)−∑i=1ngihi∣∥h‾∥=0\lim_{\underline{h}\rightarrow\underline{0}}\frac{\left\lvert f\left(\underline{x}+\underline{h}\right)-f(\underline{x})-\sum_{i=1}^{n}g_i h_i\right\rvert}{\lVert\underline{h}\rVert}=0

Differentiability of a function of several variables has the following properties:

  • if ff is differentiable at x‾\underline{x} then g‾=∇f(x‾),\underline{g}=\nabla f(\underline{x}), ff is continuous at x‾\underline{x} and every directional derivative exists with Dv‾f(x‾)=∑i=1n∂f(x‾)∂xivi;D_{\underline{v}}f(\underline{x})=\sum_{i=1}^{n}\frac{\partial f(\underline{x})}{\partial x_i}v_i;
  • the existence of all partial derivatives at x‾\underline{x} does not imply differentiability or even continuity at x‾.\underline{x}.

Let X⊆RnX\subseteq\mathbb{R}^n be open and f:X→Rf:X\rightarrow\mathbb{R} be a function. The function ff is continuously differentiable on X,X, written f∈C1(X),f\in C^1(X), if and only if all partial derivatives of ff exist and are continuous on X.X. A continuously differentiable function is differentiable on X.X.

Let X⊆RnX\subseteq\mathbb{R}^n be open and f‾=(f1,…,fm):X→Rm\underline{f}=\left(f_1,\ldots,f_m\right):X\rightarrow\mathbb{R}^m be a function whose components fj:X→Rf_j:X\rightarrow\mathbb{R} are differentiable at x‾∈X.\underline{x}\in X. The Jacobian matrix of f‾\underline{f} at x‾\underline{x} is the m×nm\times n matrix Jf‾(x‾)J_{\underline{f}}(\underline{x}) whose entry in row jj and column ii is the partial derivative of the jj-th component with respect to the ii-th coordinate.

[Jf‾(x‾)]ji:=∂fj(x‾)∂xi\left[J_{\underline{f}}(\underline{x})\right]_{ji}:=\frac{\partial f_j(\underline{x})}{\partial x_i}

The rows of the Jacobian matrix are the gradients of the components of f‾.\underline{f}.

Let X⊆RnX\subseteq\mathbb{R}^n and Y⊆RmY\subseteq\mathbb{R}^m be open, f‾:X→Y\underline{f}:X\rightarrow Y be differentiable at x‾∈X\underline{x}\in X and g‾=(g1,…,gp):Y→Rp\underline{g}=\left(g_1,\ldots,g_p\right):Y\rightarrow\mathbb{R}^p be differentiable at f‾(x‾).\underline{f}(\underline{x}). The multivariate chain rule states that g‾∘f‾\underline{g}\circ\underline{f} is differentiable at x‾\underline{x} with

∂(gk∘f‾)(x‾)∂xi=∑j=1m∂gk(f‾(x‾))∂yj∂fj(x‾)∂xi.\frac{\partial\left(g_k\circ\underline{f}\right)(\underline{x})}{\partial x_i}=\sum_{j=1}^{m}\frac{\partial g_k\left(\underline{f}(\underline{x})\right)}{\partial y_j}\frac{\partial f_j(\underline{x})}{\partial x_i}.

Entry by entry this is the product of the Jacobian matrices, Jg‾∘f‾(x‾)=Jg‾(f‾(x‾))Jf‾(x‾).J_{\underline{g}\circ\underline{f}}(\underline{x})=J_{\underline{g}}\left(\underline{f}(\underline{x})\right)J_{\underline{f}}(\underline{x}). In particular, if x‾(t)=(x1(t),…,xn(t))\underline{x}(t)=\left(x_1(t),\ldots,x_n(t)\right) is differentiable at tt and ff is differentiable at x‾(t)\underline{x}(t) then

ddtf(x‾(t))=∑i=1n∂f(x‾(t))∂xidxi(t)dt.\frac{d}{dt}f\left(\underline{x}(t)\right)=\sum_{i=1}^{n}\frac{\partial f\left(\underline{x}(t)\right)}{\partial x_i}\frac{dx_i(t)}{dt}.

Let X⊆RnX\subseteq\mathbb{R}^n be open, f:X→Rf:X\rightarrow\mathbb{R} be a function and i,j∈{1,…,n}.i,j\in\{1,\ldots,n\}. The second-order partial derivative of ff with respect to xix_i and then xjx_j is the partial derivative of ∂f∂xi\frac{\partial f}{\partial x_i} with respect to xj.x_j.

∂2f(x‾)∂xj∂xi:=∂∂xj(∂f(x‾)∂xi)\frac{\partial^2 f(\underline{x})}{\partial x_j\partial x_i}:=\frac{\partial}{\partial x_j}\left(\frac{\partial f(\underline{x})}{\partial x_i}\right)

Let X⊆RnX\subseteq\mathbb{R}^n be open, f:X→Rf:X\rightarrow\mathbb{R} be a function and x‾∈X\underline{x}\in X be a point at which all second-order partial derivatives of ff exist. The Hessian matrix of ff at x‾\underline{x} is the n×nn\times n matrix Hf(x‾)H_f(\underline{x}) of second-order partial derivatives.

[Hf(x‾)]ij:=∂2f(x‾)∂xi∂xj\left[H_f(\underline{x})\right]_{ij}:=\frac{\partial^2 f(\underline{x})}{\partial x_i\partial x_j}

Let X⊆RnX\subseteq\mathbb{R}^n be open and f:X→Rf:X\rightarrow\mathbb{R} be a function. The function ff is twice continuously differentiable on X,X, written f∈C2(X),f\in C^2(X), if and only if all second-order partial derivatives of ff exist and are continuous on X.X.

Let X⊆RnX\subseteq\mathbb{R}^n be open and f∈C2(X).f\in C^2(X). Schwarz’s theorem states that the order of differentiation does not matter, so the Hessian matrix is symmetric.

∂2f(x‾)∂xi∂xj=∂2f(x‾)∂xj∂xi  ∀ i,j\frac{\partial^2 f(\underline{x})}{\partial x_i\partial x_j}=\frac{\partial^2 f(\underline{x})}{\partial x_j\partial x_i}\,\,\forall\,i,j

Let X⊆RnX\subseteq\mathbb{R}^n be open, f∈C2(X),f\in C^2(X), x‾∈X\underline{x}\in X and h‾∈Rn\underline{h}\in\mathbb{R}^n with x‾+h‾∈X.\underline{x}+\underline{h}\in X. The second-order Taylor expansion of ff at x‾\underline{x} states that

f(x‾+h‾)=f(x‾)+∑i=1n∂f(x‾)∂xihi+12∑i=1n∑j=1n∂2f(x‾)∂xi∂xjhihj+R2(h‾)f\left(\underline{x}+\underline{h}\right)=f(\underline{x})+\sum_{i=1}^{n}\frac{\partial f(\underline{x})}{\partial x_i}h_i+\frac{1}{2}\sum_{i=1}^{n}\sum_{j=1}^{n}\frac{\partial^2 f(\underline{x})}{\partial x_i\partial x_j}h_i h_j+R_2(\underline{h})

where the remainder satisfies lim⁡h‾→0‾R2(h‾)∥h‾∥2=0.\lim_{\underline{h}\rightarrow\underline{0}}\frac{R_2(\underline{h})}{\lVert\underline{h}\rVert^2}=0. The first-order terms are the directional derivative Dh‾f(x‾)D_{\underline{h}}f(\underline{x}) and the second-order terms are the quadratic form 12h‾⊺Hf(x‾)h‾\frac{1}{2}\underline{h}^\intercal H_f(\underline{x})\underline{h} of the Hessian matrix.