ENH: Implement automatic differentiation

[pchanial]

Use derivation rules to deduce the derivates of an operator.

[nbarbey]

We could implement it like this :

• add a diff method to operators
• implement functions defining differentiation rules on operators. eg : https://gist.github.com/1717713
• implement a flag / decorator to Operators to define them as differentiable
• update AdditionOperator, CompositionOperator to propagate the diff methods
• possibly the diff method is itself an Operator with a diff method, which could be used by a diff2 / hessian method, etc ...

[pchanial]

• I though about a '.D' property that would return the operator for the derivate, to be in line with the .C, .T and .H attributes. Have you checked that the differentiation rules in https://gist.github.com/1717713 also apply for operators of shape nxm ?
• I am not sure about the differentiable flag, because there's an ambiguity about its meaning: does it mean that the operator has no '.D' implementation or does it mean that mathematically, this operator cannot be differentiated ? This remark also goes for a possible 'inversible' flag. Currently the algebraic flags stand for mathematical properties (unlike inplace or inplace_reduction that depend on the implementation and that can be improved) that are translated into rules.
• For the hessian, we could have a D2 method which would take a vector as argument and return an operator ? Similar to the current '..._p' methods in Criterion (although it's not completely clear to me how these methods are used). The proper hessian could be obtained with '.D.D'.

[nbarbey]

The chain rule also apply in higher dimensions [1].
Interestingly the derivative of Ax is A^T [2]
I think this gives us all we need to implement automatic differentiation in a way that allow the definition of criterions as Operators.

[1] http://en.wikipedia.org/wiki/Chain_rule#The_chain_rule_in_higher_dimensions
[2] http://en.wikipedia.org/wiki/Matrix_calculus