How to use motion capture on ARKit to compute posture angle

5 min readJan 20, 2020

ARKit extends the application use cases. In a recent one, we leverage the power of a cutting-edge technology provided by ARKit3. The human motion capture. The goal of this development is to detect good/bad posture at work. We will mainly focus on the angle between the back and the thighs.

Posture at work

There are several studies about it, and we will start from this BBC article. It’s the this study that convince me the most.

Do not take this article as a state of art about posture at work. From what I found, there are some articles that says that the best angle between the torso and thigh is 90°. Other studies say that it is better to have an angle between than 90° and 120°. In this article we will only focus on the study of Dr Waseem Bashir.

Dr Waseem Bashir of the Department of Radiology and Diagnostic Imaging at the University of Alberta Hospital, Canada, who led the study, said: “Sitting in a sound anatomic position is essential, since the strain put on the spine and its associated ligaments over time can lead to pain, deformity and chronic illness.”
“As to what is the best angle between thigh and torso when seated, reclining at 135 degrees can make sitting more difficult as there is a tendency to slide off the seat: 120 degrees or less may be better.”

Posture and consequences on disc pressure.

Motion Capture on iOS

Which devices are eligible?

Even if only a subpart of Apple devices can run this technology, it represents millions of devices…

The minimal target version is iOS 13 with A12 chip on it:

iPhone XS, XS MAX,
iPhone XR
iPhone 11, 11 Pro, 11 Pro Max
iPad with A12 chip
next generations…

Features and limitations

With ARKit3 we can now track a single person in real time and retrieve the transformations of the joints in the camera feed. Only the back camera can be used for motion capture. Only a sub-part of the ARSkeleton3D joints are found from the camera feed. The other joints are here to ease the mapping between your application rigged mesh and the ARSkeleton3D.

The SDK provides multiple kind of ARAnchor:

The ARBodyAnchor, with which ARKit tracks the motion of a single person.
The ARPlaneAnchor, with which ARKit tracks flat surface.
The ARObjectAnchor, with which ARKit tracks real-world 3D objects. Use the RealityComposer application to create and export ARobjectAnchor asset.
The ARImageAnchor, with which ARKit tracks real-world images.
The ARFaceAnchor, with which ARKit tracks pose, topology and expression of human face. ARKit use the front built-in camera for this anchor.

ARKit uses delegation on your ARSCNView to feed your application with data about the skeleton.

arView.delegate = self

Speed up your development

For development, I massively used Record and Replay feature of Xcode.

And why don’t we use RealityKit?

I tried to retrieve the joint information from RealityKit framework, but I didn’t find a solution. If someone has a solution, I will be glad to hear about it.

The 3D skeleton

Retrieve data joint from the ARBodyAnchor

From the ARSkeleton3D object, we can get the 4x4 transformation matrix:

Of a named joint in the root referential of the skeleton.
Of a named joint in the parent hierarchy.

Rigged mesh and relationship with ARBodyAnchor

The next picture is a snapshot in the SceneKit Scene editor of a rigged 3D mesh provided by Apple in their demonstration. It’s the robot.usdz model. All the joints in this skeleton hierarchy are also the name of a joint provided by the ARSkeleton3D. For our use case, we will focus on the left leg joint and the spine 7 joint.

Retrieve the joints and compute the back/thigh angle

The goal of this use case is to give feedback through two channels:

Display the angle in a UILabel. The label is a child of the ARSCNView.
Display a 3D representation of the angle in the video stream.

The 3D representation of the angle is a circular arc. The geometry is recreated at each update, surely not the most efficient way…). The geometry is a SCNShape build with a UIBezierPath. The node is displayed in green if the posture angle is between 90 and 120 degrees, in orange otherwise.

To compute the angle between two 3D nodes, I use this basic operation on 3D vectors :

We get the left leg joint and the spine 7 joint in the root skeleton referential. Then, we compute the angle, and we attach the circular arc geometry and apply the transformation.

Tada!

The following video had been recorded on an iPhone XS Max. The angle is computed in real time. The tracking is not perfect, so don’t use this technology if you want very accurate data. Otherwise, it’s enough for a lot of use cases.