7742-G a Brief Introduction to Machine Learning: Google's Teachable Machine & Max 8

Let's use our knowledge of JavaScript and Node.js with Max 8 to use Google's playfully fun Teachable Machine as an input  into Max 8.

tm.jpg

Machine Learning is opening up endless possibilities for artists. Deep Fakes, Speech Recognition, Image Recognition and more are used by artists to create, develop, inspire and perform with live .  

Here is a great interview for ArtFuse with Christiane Paul, author of Digital Art , by Tansy Xiao.

“The Question of Intelligence — AI and the Future of Humanity” Interview with Curator Christiane Paul.

Janelle Shaw has made some hilarious examples of AI generated text.

For more examples of how machine learning is used by artists have a look at this Foundry article:Machine learning for artists!

There are Machine Learning packages that can be used with max.

Using the Max Package Manager you can download and launch these libraries!

ml.* (watch and learn with this tutorial for ml.markov)

the fantastic & dense MuBu by ircam.

and one compiled for Max and Pd (you can run on Raspberry pi) ml.lib.

Here is a glossary of terms that you may encounter when reading / discussing Machine Learning.

Machine Learning
Machine learning is a branch of artificial intelligence (AI) focused on building applications that learn from data and improve their accuracy over time without being programmed to do so. 

In data science, an algorithm is a sequence of statistical processing steps. In machine learning, algorithms are 'trained' to find patterns and features in massive amounts of data in order to make decisions and predictions based on new data. The better the algorithm, the more accurate the decisions and predictions will become as it processes more data.

 

Neural Network
Artificial neural networks (ANNs), usually simply called neural networks (NNs), are computing systems vaguely inspired by the biological neural networks that constitute animal brains.


Artificial intelligence and cognitive modeling try to simulate some properties of biological neural networks. In the artificial intelligence field, artificial neural networks have been applied successfully to speech recognition, image analysis and adaptive control, in order to construct software agents (in computer and video games) or autonomous robots.

Transfer Learning
Transfer learning, used in machine learning, is the reuse of a pre-trained model on a new problem. In transfer learning, a machine exploits the knowledge gained from a previous task to improve generalization about another. For example, in training a classifier to predict whether an image contains food, you could use the knowledge it gained during training to recognize drinks.

Transfer learning is mostly used in computer vision and natural language processing tasks like sentiment analysis due to the huge amount of computational power required.

Transfer learning has become quite popular in combination with neural networks that require huge amounts of data and computational power.

Here is a lesson on transfer learning for more information.

Tensorflow

TensorFlow is a free and open-source software library for machine learning. It can be used across a range of tasks but has a particular focus on training and inference of deep neural networks.


Tensorflow is a symbolic math library based on dataflow and differentiable programming. It is used for both research and production at Google.


TensorFlow was developed by the Google Brain team for internal Google use. It was released under the Apache License 2.0 in 2015.


TensorFlow.js is a library for machine learning in JavaScript. 
 

nerual.png

Google's Teachable Machine
Train a computer to recognize your own images, sounds, & poses.

 

A fast, easy way to create machine learning models for your sites, apps, and more – no expertise or coding required.

TM allows you to train a machine learning model in a browser: images, sounds, and poses.


You can upload and save your trained model and use it in Max 8 & Node.js!

(uploading it to Google's cloud services through the GUI)

Google's Teachable Machine trains our models to work with very large pre-made models that have been trained with thousands of images. This way we can use 25-100 images and have good results. If we were to do this from scratch we would need a much larger set of data!

If you want to learn and read about these existing models have a look at MobileNet and ImageNet:
MobileNets is a large model that we can use for our Models to learn from when training.

and

ImageNet, out of Stanford, is machine learning that can run in a browser.

Let's train an image model and export it for use in Max 8 to trigger events, etc...

 

Three steps: 


1. Data Collection. Provide material for us to train the model with. (I say start by  adding 25+ images.

Note: the more images the longer the training time!

2. Training: Train your Model. Don't Switch Tabs! Training is happening in your Browser! so don't check email while training, let  Google train the model based on your input and not be interrupted.

3. Upload your Model: Use Tensorflow.js for our upload. When you upload, use cloud upload, it is easier to work with your data. You can choose to work locally with the model but for this class we will use the online cloud server to store our models.

4. Save your Teachable Machine Model to your Drive (dropdown menu in the upper LEFT)

tm-codeImage.jpg

YOUR LINK

Once UPLOADED you will receive Javascript Code that we can add to our web pages hosted, similar to our Max & Node.js examples (from Class 11).

The code we are interested in is it the link offered to you once yoru model has been uploaded.

(see the arrow in the image above)

my model link looks like this:

 

https://teachablemachine.withgoogle.com/models/lXMwFdqKx/

We will add this link to the index.html file that is in our "node4max_TeachableMachine_BASIC" folder of our Class 12 files.

find this section of the Teachable Machine <script> tag:

    ///////CHANGE THE BELOW URL TO THE ONE PROVIDED BY EXPORTING YOUR MODEL!!!!!!!!!!!!
  // the link to your model provided by Teachable Machine export panel
  const URL = "https://teachablemachine.withgoogle.com/models/lXMwFdqKx/";

and replace this URL with the one from your uploaded Model

Now START the Max Node.js script by opening the "node4max-TM_Basic.maxpat" and navigate to the web page linked on the Max Basic Patcher. (make sure to install the required packages the first time you run the patcher)

 

http://localhost:3000/

Here's what I did to make this work by adding a line to use socket.io and node.js:

The exported JavaScript code from Google's TM will look something like this:

 <div>Teachable Machine Image Model</div>
<button type="button" onclick="init()">Start</button>
<div id="webcam-container"></div>
<div id="label-container"></div>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@1.3.1/dist/tf.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@teachablemachine/image@0.8/dist/teachablemachine-image.min.js"></script>
<script type="text/javascript">
    // More API functions here:
    // https://github.com/googlecreativelab/teachablemachine-community/tree/master/libraries/image

    // the link to your model provided by Teachable Machine export panel
    const URL = "https://teachablemachine.withgoogle.com/models/XX_YOUR_MODEL_XX/";

    let model, webcam, labelContainer, maxPredictions;

    // Load the image model and setup the webcam
    async function init() {
        const modelURL = URL + "model.json";
        const metadataURL = URL + "metadata.json";

        // load the model and metadata
        // Refer to tmImage.loadFromFiles() in the API to support files from a file picker
        // or files from your local hard drive
        // Note: the pose library adds "tmImage" object to your window (window.tmImage)
        model = await tmImage.load(modelURL, metadataURL);
        maxPredictions = model.getTotalClasses();

        // Convenience function to setup a webcam
        const flip = true; // whether to flip the webcam
        webcam = new tmImage.Webcam(200, 200, flip); // width, height, flip
        await webcam.setup(); // request access to the webcam
        await webcam.play();
        window.requestAnimationFrame(loop);

        // append elements to the DOM
        document.getElementById("webcam-container").appendChild(webcam.canvas);
        labelContainer = document.getElementById("label-container");
        for (let i = 0; i < maxPredictions; i++) { // and class labels
            labelContainer.appendChild(document.createElement("div"));
        }
    }

    async function loop() {
        webcam.update(); // update the webcam frame
        await predict();
        window.requestAnimationFrame(loop);
    }

    // run the webcam image through the image model
    async function predict() {
        // predict can take in an image, video or canvas html element
        const prediction = await model.predict(webcam.canvas);
        for (let i = 0; i < maxPredictions; i++) {
            const classPrediction =
                prediction[i].className + ": " + prediction[i].probability.toFixed(2);
            labelContainer.childNodes[i].innerHTML = classPrediction;
        }
    }
</script>

Let's add our exported models to the index.html file located in the "public" directory of today's lesson folder. Copy and paste the JavaScript from the Browser to Sublime Text.
We will then add one line of code to this JavaScript in order for the data to be sent to Max 8 via Socket.io.
At the bottom of the exported model JavaScript code you will notice a function 'predict'
the line:
async function predict() {        // predict can take in an image, video or canvas html element               const prediction = await model.predict(webcam.canvas); 
 for (let i = 0; i < maxPredictions; i++) {            const classPrediction =                prediction[i].className + ": " + prediction[i].probability.toFixed(2);            labelContainer.childNodes[i].innerHTML = classPrediction;        }    }
after the line "labelContainer.childNodes[i].innerHTML = classPrediction; "
we will now add the line below, note: this Prefixes all messages from our Model with TM for us to parse with the Route Object in Max, using socket.io:

socket.emit('message','/TM '+ classPrediction);  // here we send the data to Max

this will send our data out of the Browser and into Max.
So the complete function predict() would be:

 // run the webcam image through the image model
    async function predict() {
        // predict can take in an image, video or canvas html element
        const prediction = await model.predict(webcam.canvas);
        for (let i = 0; i < maxPredictions; i++) {
          const classPrediction =
          prediction[i].className + ": " + prediction[i].probability.toFixed(2);
          labelContainer.childNodes[i].innerHTML = classPrediction;
            socket.emit('message','/TM '+ classPrediction);  // here we send the data to Max

 

          }
        }

For other useful Machine Learning AI resources for making art:
ml5.js: a friendly machine learning for the web.
🎥ml5.js Transfer Learning
🎥ml5.js Feature Extractor

p5 processing


mlfA: Machine Learning for Arts

max AI Processes Music

GPT-3
writing poetry with GPT-3
gpt-3 tools created online