In this activity, we will be learning to use BlazePose, a software library that finds joints and other body landmarks in an image or video. We will use the p5.js library for creative coding to control it.

This is a programming-intensive activity, but don’t worry if you don’t have coding experience. We have done it on the first day of class with many students with a range of prior coding experience, from none to lots. Just remember to ask for help when you are stuck, so that this is a exercise in learning and not in frustration.

BlazePose is a software library that estimates the pose of a person in an image or video. It employs machine learning to compute the probable positions of 33 body landmarks, or skeleton keypoints. These keypoints are located at major skeletal joins (e.g. wrists, elbows, shoulders, knees) and facial features (eyes, ears, and mouth).

BlazePose was developed by Google. It can be used with programs written in JavaScript, or Python. In this activity, we will use the JavaScript version.

More details on how BlazePose works, including a list of keypoints, are in this blog post.

p5.js is a JavaScript library with the goal of making coding accessible to artists, designers, educators, and beginners.

p5.js is inspired by — and very similar to — Processing. If you know Processing, many of the concepts in p5.js will look familiar (and vice versa). Unlike Processing, p5.js uses JavaScript instead of Java, and runs in a web browser (including on mobile devices) instead of in a dedicated IDE.

In this activity, you will use the browser version of p5.js, which is maintained by the Processing Foundation. We recommend running it in Chrome, (we can try other browsers in class).

You will run our class “starter kit” project that uses BlazePose (it’s based on, and similar to, public demo code for BlazePose). You will become familiar with the browser interface to p5, and you will modify the program.

You will create an account on the P5 Editor site, so that you can create and save versions of the program to make your own projects using BlazePose.

Run BlazePose in your browser by opening the following URL. Chrome works better (runs at a higher frame rate) than Safari.

Stand back from the computer so it sees your whole body.

Questions:

Q. Try out different poses. What is BlazePose good at? What can’t it do?

Q. What frame rate (fps) does your computer report? (This is the number at the upper left corner.) Send this in Slack. Q. What effect does scoreThreshold have?

Take notes to answer in the homework assignment.

Now you’ve got BlazePose running. Try it, and explore how well it works with wherever your laptop camera is aimed. Re-explore some of the earlier questions and think of other ones.

Try out different poses. What is BlazePose good at? What breaks it?

If your physical situation allows, you can also try different backgrounds.

Objective: verify that you can create a new project and change the behavior and appearance of the BlazePose starter kit program.

Objective: Share your project

Objectives:

Make use of at least one p5.js function that is not used in the original code base, to spice up the visual display. Suggestions: use other shapes. Going beyond: use vertex, place images

BlazePose working tip: Make time to work with others in class, as it is easier to try out changes when one person is far from the camera and one is modifying the code. – Sorry, Spring 2022 class!

General working tip: Make small changes, test often, and save often!

Send a screenshot to Slack.

The code as distributed draws only keypoints with a certain confidence level. Change the confidence level to a different number. Does that have a noticeable effect?

1. Find the line in the sketch:

image(video, 0, 0);

“Comment it out” by prefixing the line with two forward slashes //. This disables it — it tells the computer not to execute it — but it leaves the code in a form where it can easily be enabled again.

// image(video, 0, 0,);

What do you expect to change?

Now view the program in the browser. What changed?

2. Right above (or below) the line that you commented out, add this line:

background(0, 0, 0);

This draws the background on each frame. The background is drawn with a color composed of three lights of color red, green, and blue (R, G, and B). In this case each light intensity is set to 0, so the background is drawn in black.

We could have just written background("black"). The rgb notation is a better setup for:

3. Replace the background line with:

background(0, 0, 0, 5);

The background is drawn with semi-transparent black, that lets most of the background show through. This color is 0.05 opaque, which is 5% opaque or 95% translucent.

Note that in addition to the pose.keypoints array, there is another array pose.keypoints3D . The objects in this second array have x and y properties like those in pose.keypoints, but they also have z properties. You can use the x, y, and z with a WebGL canvas to display the pose in 3D. Or, you can use the z value in other ways, such as to control the size or color of a point.

Whenever you run into an issue with a web application that you are developing, you should open the Developer Console and look for warnings. This page has instructions for opening the console on Chrome (and other browsers).

The following warnings are expected, and you can ignore them. You can distinguish these from other warnings because they are all about URLs whose names end in .map not being found (”HTTP error: status code 404”).

If you clicked “no” when the site asked to use the camera, you will need to go into Settings to give it permission. Follow these instructions.

If your computer has multiple webcams connected, Chrome may choose the wrong one. This can be the case if you have installed a virtual webcam such as Snap Camera or OBS, or have connected your computer to an external webcam or a display that includes one.

If this is the case, follow the instructions here to direct Chrome to use the appropriate camera. Then refresh the page.