矩阵求导法则与性质

介绍矩阵求导法则，以及常用的求导公式、迹函数、行列式求导结论

矩阵求导法则

矩阵求导应该分为标量求导、向量求导、矩阵求导三个方面来介绍，公式繁多，但仔细看看其实是有规律可循的。

标量求导

无论是矩阵、向量对标量求导，或者是标量对矩阵、向量求导，其结论都是一样的：等价于对矩阵（向量）的每个分量求导，并且保持维数不变。

例如，我们可以计算标量对向量求导：

设$y$为一个元素，$x^T = [x_1...x_q]$是$q$维行向量，则：

$$\frac{\partial y}{\partial x^T} = [\frac{\partial y}{\partial x_1}...\frac{\partial y}{\partial x_q}]$$

向量求导

对于向量求导，我们可以先将向量看做一个标量，然后使用标量求导法则，最后将向量形式化为标量进行。

例如，我们可以计算行向量对列向量求导：

设$y^T=[y_1...y_n]$是$n$维行向量，$x=[x_1,...,x_p]$是$p$维列向量，则：

$$\begin{align}\frac{\partial y^T}{\partial x} &=& [\frac{\partial y_1}{\partial x}...\frac{\partial y_n}{\partial x}] \nonumber \\ &=&\begin{bmatrix}\frac{\partial y_1}{\partial x_1}& ... &\frac{\partial y_n}{\partial x_1} \\ ... & ... &... \\\frac{\partial y_1}{\partial x_p} &... &\frac{\partial y_n}{\partial x_p} \end{bmatrix} \nonumber\end{align}$$

矩阵求导

与向量求导类似，先将矩阵化当做一个标量，再使用标量对矩阵的运算进行。

例如，我们可以计算矩阵对列向量求导：

设$Y =\begin{bmatrix}y_{11} &... & y_{1n} \\... &... &... \\ y_{m1} & ...&y_{mn} \end{bmatrix}$是$m\times n$矩阵，$x=[x_1,...,x_p]$是$p$维列向量，则：

$$\frac{\partial Y}{\partial x} = [\frac{\partial Y}{\partial x_1},...,\frac{\partial Y}{\partial x_p}]$$

矩阵微积分

常见求导性质

实值函数相对于实向量的梯度

设$f(x) = x = [x_1,...,x_n]^T$

$$\frac{\partial f (x)}{\partial x^T} = \frac{\partial x}{\partial x^T} = I_{n\times n}$$

$$\frac{\partial (f (x))^T}{\partial x} = \frac{\partial x^T}{\partial x} = I_{n\times n}$$

$$\frac{\partial f (x)}{\partial x} = \frac{\partial x}{\partial x} = vec(I_{n\times n})$$

$$\frac{\partial (f (x))^T}{\partial x^T} = \frac{\partial x^T}{\partial x^T} = vec(I_{n\times n})^T$$

其中，$vec$表示向量化矩阵，按列将矩阵表示为向量，具体可见Wikipedia。

常见性质

1. $f(x) = Ax$，则
   ∂f(x)∂xT=∂(Ax)∂xT=A ∂ f ( x ) ∂ x T = ∂ ( A x ) ∂ x T = A

   $$\frac{\partial f (x)}{\partial x^T} = \frac{\partial (Ax)}{\partial x^T} =A$$

2. $f(x) = x^TAx$，则
   ∂f(x)∂x=∂(xTAx)∂x=Ax+ATx ∂ f ( x ) ∂ x = ∂ ( x T A x ) ∂ x = A x + A T x

   $$\frac{\partial f (x)}{\partial x} = \frac{\partial (x^TAx)}{\partial x} =Ax+A^Tx$$

3. $f(x) = a^Tx$，则
   ∂aTx∂x=∂xTa∂x=a ∂ a T x ∂ x = ∂ x T a ∂ x = a

   $$\frac{\partial a^Tx}{\partial x} = \frac{\partial x^Ta}{\partial x} =a$$

4. $f(x) = x^TAy$，则
   ∂xTAy∂x=Ay ∂ x T A y ∂ x = A y

   $$\frac{\partial x^TAy}{\partial x} = Ay$$

   ∂xTAy∂A=xyT ∂ x T A y ∂ A = x y T

   $$\frac{\partial x^TAy}{\partial A} = xy^T$$

5. df(X)=tr((∂f(X)∂X)TdX) d f ( X ) = t r ( ( ∂ f ( X ) ∂ X ) T d X )

   $$df(X) = tr((\frac{\partial f(X)}{\partial X})^T d X)$$

6. 矩阵微分也满足线性法则、乘积法则。
7. 矩阵的逆的微分
   d(X−1)=−X−1(dX)X−1 d ( X − 1 ) = − X − 1 ( d X ) X − 1

   $$d(X^{-1}) = -X^{-1}(dX)X^{-1}$$

迹函数

迹函数相对于矩阵的梯度

$$\frac{\partial (tr (ZZ^T))}{\partial Z} = \frac{\partial (tr (Z^TZ))}{\partial Z} = 2Z$$

矩阵微分算子和迹算子的可交换性

$$d(tr(X)) = tr(d(X)) = \sum\limits_{i=1}^{n} dx_{ii}$$

常见性质

1. ∂tr(A)∂A=In×n ∂ t r ( A ) ∂ A = I n × n

   $$\frac{\partial tr(A)}{\partial A} = I_{n\times n}$$

2. ∂tr(AB)∂A=BT ∂ t r ( A B ) ∂ A = B T

   $$\frac{\partial tr(AB)}{\partial A} = B^T$$

3. d(tr(AXB))=tr(A(dX)B)=tr(BA(dx)) d ( t r ( A X B ) ) = t r ( A ( d X ) B ) = t r ( B A ( d x ) )

   $$d(tr(AXB)) = tr(A(dX)B) = tr(BA(dx))$$

   ∂tr(AXB)∂X=(BA)T=ATBT ∂ t r ( A X B ) ∂ X = ( B A ) T = A T B T

   $$\frac{\partial tr(AXB)}{\partial X} = (BA)^T = A^TB^T$$

4. d(tr(AX−1B))=tr(A(dX−1)B)=−tr(AX−1(dX)X−1B)=−tr(X−1BAX−1dX) d ( t r ( A X − 1 B ) ) = t r ( A ( d X − 1 ) B ) = − t r ( A X − 1 ( d X ) X − 1 B ) = − t r ( X − 1 B A X − 1 d X )

   $$d(tr(AX^{-1}B)) = tr(A(dX^{-1})B) = -tr(AX^{-1}(dX)X^{-1}B) = -tr(X^{-1}BAX^{-1}dX)$$

   ∂tr(AX−1B)∂X=−(X−1BAX−1)T=−X−TATBTX−T ∂ t r ( A X − 1 B ) ∂ X = − ( X − 1 B A X − 1 ) T = − X − T A T B T X − T

   $$\frac{\partial tr(AX^{-1}B)}{\partial X} = -(X^{-1}BAX^{-1})^T = -X^{-T}A^TB^TX^{-T}$$

5. ∂tr(XTX)∂X=2X ∂ t r ( X T X ) ∂ X = 2 X

   $$\frac{\partial tr(X^TX)}{\partial X} = 2X$$

   ​

行列式

行列式相对于矩阵的梯度

$$\frac{\partial |Z|}{\partial Z} = |Z|(Z^{-1})^T$$

微分形式

$$d|X| = tr(|X| X^{-1} dX)$$

常见性质

1. d|AXB|===tr(|AXB|(AXB)−1d(AXB))tr(|AXB|(AXB)−1A(dX)B)tr(|AXB|B(AXB)−1A(dX)) d | A X B | = t r ( | A X B | ( A X B ) − 1 d ( A X B ) ) = t r ( | A X B | ( A X B ) − 1 A ( d X ) B ) = t r ( | A X B | B ( A X B ) − 1 A ( d X ) )

   $$\begin{align} d|AXB| &=& tr(|AXB| (AXB)^{-1}d(AXB)) \nonumber\\&=& tr(|AXB| (AXB)^{-1}A(dX)B) \nonumber\\&=& tr(|AXB| B(AXB)^{-1}A(dX))\nonumber\end{align}$$

   ∂|AXB|∂X=|AXB|AT(BTXTAT)−1BT ∂ | A X B | ∂ X = | A X B | A T ( B T X T A T ) − 1 B T

   $$\frac{\partial |AXB|}{\partial X} = |AXB|A^T(B^TX^TA^T)^{-1}B^T$$

2. ∂|X|∂X=|X|X−T ∂ | X | ∂ X = | X | X − T

   $$\frac{\partial |X|}{\partial X} = |X| X^{-T}$$

3. ∂|XXT|∂X=2|XXT|(XXT)−1X ∂ | X X T | ∂ X = 2 | X X T | ( X X T ) − 1 X

   $$\frac{\partial |XX^T|}{\partial X} = 2|XX^T| (XX^{T})^{-1}X$$

reference

1. 矩阵的导数与迹