February 26, 2015

Getting the Power of GPU for Deep Learning

install EPD
install bleeding edge theano

On my macbook pro it says my graphics card is Intel HD Graphics 4000 1024 MB. I was very disappointed that Apple just left an intel graphics card on a $2000 laptop. They could definitely do better. So I gave up on CUDA. Recently I found that two graphics cards on board!

February 17, 2015

Visualizing Optimization Algorithms

Here we look at a visualization of optimization algorithms:
- SGD (Stochastic Gradient Descent)
- Momentum
- NAG  (Numerical Algorithms Group)
- Adagrad (Adaptive Gradient Algorithm)
- Adadelta
- RMSProp (Divide the gradient by a running average of its recent magnitude)

What we are missing is the L-BFGS (Limited-memory BFGS)...