## Brooke Wenig

M.S. Computer Science (Distributed Machine Learning)

Fluent in 中文

brooke@databricks.com

## Schedule

Keras and Neural Network Fundamentals

MLFlow

CNNs and ImageNet

Transfer Learning and Deep Learning Pipelines

Horovod: Distributed Tensorflow

## Survey

Spark before?

Pandas/Numpy?

Machine Learning? Deep Learning?

Expectations?

## Course Objectives

Fundamentals of Deep Learning and best practices

Utilize Keras, Deep Learning Pipelines, and Horovod

Understand when/where to use transfer learning

## Course Non-Objectives

Demonstrate every new research technique/API

Detailed Math/CS behind the algorithms

## Machine Learning Overview

Supervised Learning

Unsupervised Learning

Reinforcement Learning

Classification

Regression

## Unsupervised Machine Learning

Learn structure of the unlabeled data

## Reinforcement Learning

Learning what to do to maximize reward

Explore and exploit

"All models are wrong; some models are useful."

## Accuracy

What if I told you I had a model that was 99% accurate in predicting brain cancer?

## Baseline Model

You ALWAYS want to have a baseline model to compare to

This should be a "dummy" model, i.e. a coin flip

## Model Selection?

Underlying data distribution

Some models are more costly to train

Need for interpretability?

## Right to Explanation

General Data Protection Regulation

# Deep Learning Overview

## What is Deep Learning?

Composing representations of data in a hierarchical manner

## Layers

Input layer (fixed)

Zero or more hidden layers

Output layer (fixed)

## Loss functions

Image credit F. Chollet

## Regression Evaluation

Measure "closeness" between label and prediction

• When predicting someone's weight, better to be off by 2 lbs instead of 20 lbs

Evaluation metrics:

• Loss: $(y - \hat{y})$
• Absolute loss: $|y - \hat{y}|$
• Squared loss: $(y - \hat{y})^2$

## Evaluation metric: MSE

$Error = (y_{i} - \hat{y_{i}})$

$SE = (y_{i} - \hat{y_{i}})^2$

$SSE = \sum_{i=1}^n (y_{i} - \hat{y_{i}})^2$

$MSE = \frac{1}{n}\sum_{i=1}^n (y_{i} - \hat{y_{i}})^2$

$RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^n (y_{i} - \hat{y_{i}})^2}$

## Train vs. Test MSE

Which is more important? Why?

## Keras

High-level Python API to build neural networks

Official high-level API of Tensorflow

Supports: Tensorflow, Theano, and CNTK

Has over 250,000 users

Released by François Chollet in 2015

Sigmoid

Tangent

ReLU

Leaky ReLU

PReLU

ELU

## Sigmoid

Not zero-centered

Image credit A. Karpathy

## Tangent

Zero centered!

BUT, like the sigmoid, its activations saturate

Image credit A. Karpathy

## ReLU

BUT, gradients can still go to zero

Image credit A. Karpathy

## Leaky ReLU

For x < 0: $$f(x) = \alpha * x$$ For x >= 0: $$f(x) = x$$

Image credit A. Karpathy

## Optimizers

Choosing a proper learning rate can be difficult

Image credit F. Chollet

Easy to get stuck in local minima

Image credit F. Chollet

## Momentum

Accelerates SGD: Like pushing a ball down a hill

Take average of direction we’ve been heading (current velocity and acceleration)

Limits oscillating back and forth, gets out of local minima

## Hyperparameter Selection

Which dataset should we use to select hyperparameters? Train? Test?

## Validation Dataset

Split the dataset into three!

• Train on the training set
• Select hyperparameters based on performance of the validation set
• Test on test set

## ImageNet Challenge

Classify images in one of 1000 categories

2012 Deep Learning breakthrough with AlexNet: 16% top-5 test error rate (next closest was 25%)

## VGG16 (2014)

One of the most widely used architectures for its simplicity

## Convolutions

Focus on Local Connectivity (fewer parameters to learn)

Filter/kernel slides across input image (often 3x3)

CS 231 Convolutional Networks
Image Kernels Visualization

## What do CNNs Learn?

Breaking Convnets

## Transfer Learning

IDEA: Intermediate representations learned for one task may be useful for other related tasks

## Naive Approaches

1. Hand curated
2. Aggregates

Problems with these approaches?

## Collaborative Filtering

k factors characterize the users and items (k << n)

## Better

Use the user + product factors as input to neural network

Build distributed neural network for end-to-end scalability!

# Horovod

## Horovod

Created by Alexander Sergeev of Uber, open-sourced in 2017

Simplifies distributed neural network training

Supports TensorFlow, Keras, and PyTorch

## All-Reduce


# Only one line of code change!
optimizer = hvd.DistributedOptimizer(optimizer)


## Horovod Estimator

Part of Databricks' Runtime for ML

Distributed Tensorflow training on Spark DataFrames

MLlib Estimator API

Specify model via tf.estimator model_fn
model_fn(features, labels, mode) → tf.EstimatorSpec

## Horovod Estimator

Shards data across nodes’ local disks

Trains a tf.estimator across nodes

Feeds TFRecord data to estimator

Automatic checkpointing, logging

Simultaneous model evaluation

## Horovod Lab

### Ok, so how do I find the optimal neural network architecture?

Neural Architecture Search with Reinforcement Learning