Stdlib intrinsics and activation function: call for feedback

1

Dear community, I have opened a couple of PRs on stdlib to include some intrinsic functions and activation functions that could be used as base for Neural Networks.

The intrinsic function PR contains right now only:

  • stdlib_sum and stdlib_sum_kahan: are equivalents of sum for real and complex arrays. The 1D implementations are based on fsum and fsum_kahan. The N-D implementations build on top of the (N-1)-D
  • stdlib_dot_product and stdlib_dot_product_kahan: are equivalents of dot_product for real and complex arrays. Based on fprod and fprod_kahan.

The activations PR contains the following functions for all real kinds: gaussian, elu, relu, gelu, gelu_approx, selu, sigmoid, silu, step, tanh (using a fast approximation), Softmax, LogSoftmax, Softplus, and their 1st derivatives *_grad.

I would like to collect your thoughts on how to move forward with both as I think they have points in common:

  • For the case of sum and dot_product, these implementations offer both fast and accurate results (at least for 1D). So, would it be interesting to expose them as sum and dot_product or does the prefix stdlib_ makes sense to avoid conflicts? should a different naming be used?
  • In case a fast but downgraded-precision version is implemented (like for tanh or erf) which naming would be appropriate? I’ve been using f* (Fortran Fast :smile: ?) ex: ftanh, ferf. Moving them to stdlib I thinking the naming should be agreed upon.
  • For the activation functions, should the hard coded gradients be there? better to drop this and only provide the functions?

Which other considerations would you recommend at this stage?

Thank you

4 Likes
2

Some of the activation functions have been implemented in @milancurcic neural-fortran code. You might be able to “borrow” some of the more important ones from there

1 Like