Deep Neural Network - cross entropy cost - np.sum vs np.dot styles

entropy-cost.md

Cross Entropy Cost and Numpy Implementation

Given the Cross Entroy Cost Formula:

where:

• J is the averaged cross entropy cost
• m is the number of samples
• super script [L] corresponds to output layer
• super script (i) corresponds to the ith sample
• A is the activation matrix
• Y is the true output label
• log() is the natural logarithm

We can implement this in Numpy in either the np.sum or np.dot styles, like this:

The cross entropy lost is defined as (using the np.sum style):

np sum style

cost = -(1.0/m) * np.sum(Y*np.log(A) + (1-Y)*np.log(1-A))

Note: A is the Activation Matrix in the output layer L, and Y is the true label matrix at that same layer. Both have dimensions (n_y, m), where n_y is number of nodes at output layer, and m is number of samples.

This is equivalent as the followings "dot product" style:

np dot style

cost = -(1.0/m) * (np.dot(np.log(A), Y.T) + np.dot(np.log(1-A), (1-Y).T))

Remarks

The np.sum method is probably easier to read. Computational efficiency wise I wonder which one is preferred? (requires validation)

Reference

• deeplearning.ai course

Cost calculation using dot has sum missing -
cost = -(1.0/m) *np.sum( np.dot(np.log(A), Y.T) + np.dot(np.log(1-A), (1-Y).T))

Dear @kumarsubodh, as the dot products are already scalars (single numbers) the np.sum (which sums the array of products in the other style - https://docs.scipy.org/doc/numpy/reference/generated/numpy.sum.html ) is not needed in "np dot style".