Your First Deep Learning Project Is an API Call Away

Published in

Major League Hacking

8 min readNov 15, 2018

In recent years, image recognition algorithms have made headlines around the world. The field of deep learning as a whole took off in 2012 when researchers from the University of Toronto demonstrated for the first time that deep neural networks could be used to classify images with very high accuracy. However, it’s easy to look at terms like “deep learning” and “convolutional neural network” and have no idea what they mean, or how to use them in your own projects.

Fortunately, Clarifai provides an API that does just this. In this tutorial, we’re going to learn how to create a model that recognizes different hackathon stickers in the span of a few minutes, so that you’ll be prepared to use Clarifai at your next hackathon.

Set up your development environment

In this tutorial, we’ll be creating a web app with Node.js. To get started, create a new folder and initialize your project. If you don’t have Node.js installed, make sure to download it here. It will install the npm command as well.

mkdir clarifai-example
cd clarifai-example
npm init

The “npm init” command will prompt you to answer a few questions; you may hit enter on all of them to accept the default values. Now that our project is set up, we’re going to install a couple of packages.

npm install clarifai express multer pug

This command installs four packages, all of which we’re going to use in this tutorial:

clarifai is Clarifai’s JavaScript client library
express is a popular web framework for Node.js
multer handles file uploads for Express web servers
pug is a template engine for Node.js

Create your frontend app

To demonstrate Clarifai’s image recognition API, we’re going to create an app where you can upload photos and have them classified on the spot. Begin by creating an HTML file called “index.html,” and paste the code written below.

<!DOCTYPE html>
<html>
<head>
  <link href="/css/style.css" rel="stylesheet" />
  <title>Image Recognition Demo</title></head><body>
  <div class="container">
    <h1>Image Recognition Demo</h1>
    <form action="/upload" method="post" enctype="multipart/form-data">
      <label for="photo" class="label-button">Upload Image</label>
      <input id="photo" type="file" name="photo" />
      <label for="submit" class="label-button">Submit</label>
      <input id="submit" type="submit" name="submit" />
    </form>
  </div>
</body>
</html>

Once this is done, create a folder called “public”. This is where we can keep static files, such as CSS and some JavaScript files. Inside public, create another folder called “css”. Inside this folder, create a file called “style.css”. Your file structure should now look like mine does in the picture to the left. After you finish doing this, paste the CSS code written here into “style.css”:

* {
  margin: 0;
  padding: 0;
}

html, body {
  background: rgb(239, 239, 244);
  font-family: Arial, sans-serif;
  text-align: center;
  width: 100%;
}

.container {
  background: #fff;
  border-radius: 8px;
  margin: 64px auto 0;
  padding: 24px 0 32px;
  width: 600px;
}

input {
  display: none;
}

.label-button {
  background: #1e90ff;
  border-radius: 4px;
  color: #fff;
  display: table;
  margin: 16px auto 0;
  padding: 8px 12px;
  cursor: pointer;
}

Now, we’re going to start building the meat and bones of our web app. Create a file called “index.js” in the root directory of the project (the same one as “index.html”), and paste the code written below.

const express = require("express")

const app = express()
const port = 3000

app.get("/", (req, res) => {
  res.sendFile("index.html", {
    root: __dirname
  })
})

app.use(express.static("public"))
app.listen(port, () => console.log(`Running on http://localhost:${port}/`))

Now, try typing “node index.js” into your terminal. If you open http://localhost:3000 in your browser, you should see the page we just created with the HTML and CSS files above. It doesn’t do anything yet, but we’ll get to that very soon!

To stop running your web server, you can press CTRL-C on your keyboard. You’ll need to do this every time you update your JavaScript code. If you want your web server to automatically refresh when you update your code, you may want to use something like nodemon. Now that we’re done making the frontend app, let’s get started with Clarifai!

Build your model with Clarifai

To get started with Clarifai, you’ll need to create an account and verify your email address. Once you’re done setting up your account, you should see a screen where you can manage your applications. For this project, we’re going to create a new application. I called mine “hackathon-stickers,” and kept the rest of the default settings, as can be seen in the image below.

Once you’re done with this, click “Create App,” and then click on the app that you just created. This is where you can configure the details your application. Click on “Explorer” to go to the model explorer, where we can start training the model itself.

Then, create the model. I called mine “stickers,” but if you change this, you’ll need to remember it for later.

Now, I’m going to upload the photos I’ve taken of a few of the stickers I’ve kept from MIT’s hackathon and career fair. Feel free to try this with photos you take of your own stickers, but if you’d like to use mine, you can download them here. Once all of your photos have finished uploading, assign “concepts” to all of your photos. Concepts are essentially labels; Clarifai’s algorithms use them to understand what is actually present in each picture. Below, on the top right, you can see an example of one I’ve labeled as “yelp.”

Once you’re done labeling all of your pictures, click on your model on the top left of the browser window.

Then, train your model by clicking the “Train Model” button shown in the picture below.

It will take a second, but once it’s done, you should see the status on the right side of your screen change to “Model trained successfully.” Now, we’re ready to finish up our web app.

Integrate Clarifai into your app

Start by going back to the “index.js” file. We’re going to add the code for the other dependencies that we installed earlier. Update your “index.js” file so it looks like the one below.

const express = require("express")
const Clarifai = require("clarifai")
const multer = require("multer")
const pug = require("pug")

const app = express()
const port = 3000

const upload = multer()
const resultsTemplate = pug.compileFile("results.pug")

const clarifai = new Clarifai.App({
  apiKey: "[API KEY GOES HERE]"
})

app.get("/", (req, res) => {
  res.sendFile("index.html", {
    root: __dirname
  })
})

app.post("/upload", upload.single("photo"), (req, res) => {
  const base64String = Buffer.from(req.file.buffer).toString("base64")

  clarifai.models.predict("stickers", base64String).then(
   response => {
      res.send(resultsTemplate({
        concepts: response.outputs[0].data.concepts,
        image: base64String
      }))
    },
    err => {
      console.log(err)
    }
  )
})

app.use(express.static("public"))
app.listen(port, () => console.log(`Running on http://localhost:${port}/`))

In the code above, you’ll notice that I left a placeholder for your API key. In order to find this, click on your name at the top right of your browser window, and click on “Settings.” Click on “API Keys” on the left side of your window, and copy the key that is displayed under “hackathon-stickers-all-scopes.” This should be pasted in place of that placeholder.

On the line with “clarifai.models.predict”, you’ll also notice that the name of the model is specified. Earlier, we created a model named“stickers,”but if you named your model something else, this is where you should put your model’s name.

The code above also references a “results.pug” template file, so you should create this file in the same directory as “index.js.” Your file structure should now look like mine does, shown above. The template describes the webpage that our results will be shown on, and is later translated into HTML. If you’d like to learn more about how the template file works, check out Pug here.

html
  head
    link(href="/css/style.css", rel="stylesheet")
  body
    img(src="data:image/png;base64," + image)
    each concept in concepts
      p #{concept.name}: #{concept.value}

Finally, add some CSS to the bottom of “style.css” so that our results page looks alright.

img {
  margin: 64px 0 16px;
  max-height: 192px;
}

Test your model

We now have a working model, and a web app that allows us to try it out. Type “node index.js” in your terminal, hit enter, and open http://localhost:3000 in your browser. Upload an image, click submit, and you’ll see what your custom model thinks it’s looking at.

You’ll notice that the words on the left are the concepts you created earlier, and each one has a number assigned to it. These are probabilities (from 0 to 1), and they correspond with Clarifai’s confidence of each model being present in the picture. Values close to 1 indicate high confidence, and values close to 0 indicate that the concept is almost definitely not present in the picture.

Next steps

Now of course, you probably won’t be identifying pictures of stickers at your next hackathon. The uses of image recognition are far-reaching and allow for endless possibilities. Here are a couple of examples of how image recognition systems have been useful for others, if you need some inspiration:

Another concept to keep in mind is that your model is only as strong as its training examples. If you look closely at the photos I used to train this model, you’ll notice that all of my training examples were taken at my desk. As a result, my model will be very good at classifying pictures of stickers on wooden backgrounds. However, if I were to take a picture of one of these stickers on a different background, at an angle, or with a different orientation, the model may still be able to classify it correctly, but its confidence will be significantly lower. Try this out for yourself!

Congratulations, you now have a working custom image recognition model! The code we’ve written will work the same way with other concepts and models, so feel free to try this out with other types of images. If you had any issues putting together the code written in this tutorial, you can download the resulting codebase here. Good luck at your next hackathon!