Jacobian Matrix

Gradient Vector

Hessian Matrix

Hessian Matrix is Jacobian of a gradient. 

Symmetric Matrix

• Matrix and its transpose are same

 

Hermitian Matrix

• Matrix is hermitian if A_transpose (A^T) = A_bar (Conjugate of complex no)

Positive Semi-definite Matrices

• One definition:
  • Matrix M ∈ L(V) is positive definite iff
    • M is symmetric
    • v^T * M * v >= 0 for all v ∈ V
• Now following are equal [0]
  • v^T * M * v >= 0 for all v ∈ V
  • All eigenvalues are non negative
  • There exists a matrix B such that B^T * B = M
• Application
  • A twice differentiable function of n variable is convex if and only if Hessian of it is positive semi-definite (PSD) [2] [3]

 

References

[0] : https://www.cse.iitk.ac.in/users/rmittal/prev_course/s14/notes/lec11.pdf

[1] : http://www.princeton.edu/~amirali/Public/Teaching/ORF363_COS323/F15/ORF363_COS323_F15_Lec2.pdf

[2] : https://wiki.math.ntnu.no/_media/tma4180/2016v/note2.pdf

[3] : https://math.stackexchange.com/questions/946156/proving-convexity-of-a-function-whose-hessian-is-positive-semidefinite-over-a-co