What I learned deploying my First Deep Learning Models to Heroku

For anyone trying to deploy their first TensorFlow models, this would probably be good documentation on the issues they may run to that are hardly mentioned in any tutorial out there. And I checked. This is not meant to teach you to deploy the models or even build the models.

Probably one of the most important parts of machine learning is when the models built are integrated with other products, services, websites, or apps one might use. This is what we mean when we say we deploy the models. A built model doesn’t contribute much if it remains as a file or in the notebooks it was made in.

Again, this article is not an introductory article on model deployment via Flask or even model building. A minimal understanding of ML models with TensorFlow, back end development with Flask would help. I simply would like to document the issues I faced while trying to deploy my Computer Vision models which had been on local storage for about a year to a web app via Flask and Heroku.

The models I built were with TensorFlow. See, I had the idea to bring three of my lightweight models (each .h5 model weighed ~100MB) to the same Flask app. I didn’t want to build a separate app for each model. I had a working knowledge of Flask for the past couple of months so it felt natural to bring my working models to life via Flask and also to extend my knowledge of the backend framework.

HTML Forms

The idea was to allow the user to upload an image from whatever device they used to the web app and the image would be preprocessed and fed to a neural network which would then run the prediction and redirect to a results page which would give the required prediction along with the image the user uploaded.

To do this, an HTML form is required.

This code snippet below was in: app/templates/birds.html. The part that matters is that it was in a folder named templates.

<form action="{{ url_for('upload_birds') }}" method="POST" enctype = "multipart/form-data">   
 <input type = "file" name = "file" />    
<br>    
<input type = "submit"/>
</form>

The part which reads url_for() is a Flask functionality that uses the name of the function upload_bird() to determine where the image uploaded to be sent. At first, I wrote the actual HTML url provided by the Flask server. It looked something like http://127.0.0.1:5000/birds-result. For clarity, this is how the upload_birds() function looks

The predict_bird() function was simply a function to preprocess the input file to the arrays expected by the TensorFlow model and to return a prediction. For all purposes, treat it like that.

I didn’t realize what issue that would be until I had deployed the model and saw I was being redirected to a local Flask server.

Displaying Images

While developing the Flask app, I run into a situation where the images uploaded would show on the HTML page but not on the page rendered via Flask. I had thought this was an issue with the way the path was written but I later saw from the internet I had to direct Flask to know where the static files are located.

app = Flask(__name__, static_folder="uploads")

The uploads folder is where the uploaded images from the users would go. To display the images to the user, the HTML for it looked something like

<img src="{{ url_for('static', filename=file_path) }}" alt="bird-picture" height="500" width="500"/>

The snippet above where in the birds-result html page which receives both the image uploaded and the prediction to display.

Requirements Headache

I had the habit of always working with my packages installed globally and so I didn’t work in a virtual env. To upload your pages to Heroku, you'd need a requirement.txt file which would allow the server to install all dependencies. Please start your project in a virtual env and when done, do a pip freeze > requirements.txt to the project root folder. I had to do things the hard way :(.

GIT LFS ISSUE

Uploading the .h5 models to GitHub for hosting proved a headache. I didn’t know GitHub has a size limit for uploading. All files larger than 100MB would need to be uploaded via GIT LFS which basically moves your files to another server and leaves the pointers to it in your repository.

First, you install GIT LFS. Write a .gitattributes file that tells GIT what files should be uploaded via GIT LFS. Now it’s important to add & commit that .gitattributes file first before adding other files/folders or you’d run to the issue described above again. I figured it out the hard way. My .gitattributes looked like this

*.h5 filter=lfs diff=lfs merge=lfs -text

For an entire folder, you’ll do something like

app/main/model_artifacts/**/* filter=lfs diff=lfs merge=lfs -text

You’ll see shortly why I needed to do that.

Again, add & commit the .gitattributes file before proceeding to add other files/folders.

TensorFlow Issue/Path Issue

I think this was the most frustrating issue I had since it led to a lot of other issues trying to fix it and everything.

Now, my Heroku app was deployed and ‘working’. I wanted to test my models but I kept running to Internal Server error issues. This frustrated me for a while. I checked the logs of the Heroku page with

heroku logs --tails

And saw that the server couldn’t find my TensorFlow model. For the larger part of three hours, I was thinking it was a TensorFlow bug and I was changing the version along with the version of the dependencies and then deploying again and again and again. I was wrong. I hit several issues here that I would also discuss shortly.

I thought it was a path issue and then replaced ‘\’ which my laptop uses to ‘/’ which Heroku uses. I was also wrong.

I also decided to convert my models from the .h5 file to TensorFlow’s SavedModel format. To do this, I first had to retrain my best models and then save them in the SavedModelFormat instead of .h5 but I realized I could do better & faster by simply loading the .h5 model and then saving as a SavedModel Format. This looked like

model = tensorflow.keras.models.load_model('birds.h5')
print(model.summary()) # to be sure
model.save(filepath)

However, I was also wrong. This was quite frustrating. This was where I learned to include the folders in the .gitattribute file as mentioned earlier.

It was much later and many Stackoverflow pages that I realized that the model files weren’t in the repo truly since GIT LFS had taken it elsewhere! This was described above. All I needed to do is let Heroku know I did this and point it to where the files are actually located. I’ll simply link the StackOverflow answer that helped me out since they explained it better. In summary, I had to link GIT LFS to Heroku (not done by default) by installing a buildpack for it and creating a personal access token, and including these in the Config vars on Heroku.

Slug Size Issue

While trying to upload my models to the Hub, and remember I had three of them, I kept running to the issue in which I exceeded Heroku’s limit of 500MB. This limit was to contain the installation of all dependencies and files. I knew this was a huge problem because TensorFlow was 400mb+. I downgraded to a later version for this reason and the reason discussed above, but it didn’t help until I remembered I was installing the full package which included the GPU version. Since I need TensorFlow for inference and not for training, I switched to the CPU version and it reduced the slug size but I was still above the quota.

At this point, I knew better than to give up. I tried a mixture of everything above and ran into a mixture of everything discussed. But I eventually decided on splitting the task. I created two web apps and I deployed two of the models to one of the apps and then deployed the third to another. I polished the switching by swapping the links with the links Heroku provided.

I’d include the actual error logs I had in this article but I was mostly concerned with the web app working than documenting at the time. Also, these are the errors I could remember off the top of my head and I suspect I had more. I hope this could help anybody who’s trying to deploy their first ML model to Heroku via Flask.