diff --git a/src/_posts/hands-on-i.md b/src/_posts/hands-on-i.md new file mode 100644 index 00000000..f87f3109 --- /dev/null +++ b/src/_posts/hands-on-i.md @@ -0,0 +1,149 @@ +--- +title: Hands-On! Building Interactive Training With A-Frame +author: twitter|salvadelapuente|Salvador de la Puente +date: 2017-08-09 +layout: blog +image: + src: hands-on-i.png +--- + +In an industry where manipulation of complex mechanisms involves tens of tools, expensive equipment and even safety risks, virtual reality helps to increase operator safety and reduce material costs. In addition, WebVR puts everything just one link away. + +This is the first of a series of articles providing an in-depth walkthrough of a hand-based interaction and manipulation model for training purposes ([check the demo online](https://delapuente.github.io/aframe-interactive-training/step3/)). The series is not a step-by-step tutorial but a postmortem breakdown covering one major feature at a time. At the same time you build the experience, you will also learn about core A-Frame concepts, its ecosystem, the entity-component-system architecture, how to use the [A-Frame physics system](https://hacks.mozilla.org/2017/05/having-fun-with-physics-and-a-frame/), common errors, workarounds and good practices. All you need to confront a real project. + + + +This is an advanced virtual reality experience and so it requires a head-mounted display supporting room-scale VR and [6DoF](https://en.wikipedia.org/wiki/Six_degrees_of_freedom) controllers such as those included in the [HTC VIVE](https://www.vive.com/us/product/#controller-intro) or the [Oculus Touch](https://www.oculus.com/rift). Personally, I used a VIVE. + +All articles in the series: + + 1. Hands-on! Building interactive training with A-Frame + 2. [Hands-on! Simple physical behaviours in A-Frame](/blog/hands-on-ii/) + 3. [Hands-on! Hand-based interactions in A-Frame](/blog/hands-on-iii/) + +## The project repository + +Use git to clone the [repository of the simulation](https://github.com/delapuente/aframe-interactive-training). Enter the repository and run `npm install` then `npm start` to install the dependencies and start the development server at port `8080`. + +At the root of the project, you'll find [`step1/`](https://github.com/delapuente/aframe-interactive-training/tree/master/step1), [`step2/`](https://github.com/delapuente/aframe-interactive-training/tree/master/step2) and [`step3/`](https://github.com/delapuente/aframe-interactive-training/tree/master/step3) with the **finished HTML** for each step. Look at the same folders inside the [`js/`](https://github.com/delapuente/aframe-interactive-training/tree/master/js) directory to browse the JavaScript code. + +Open the browser and enter the URL `localhost:8080` to find the index of steps. Append `step1` at the end of the URL path and click on the VR button on the bottom-right corner of the display to experience the project after step 1 is completed (or select step 1 in the list). + +The project is based on the [A-Frame SPA skeleton](https://github.com/delapuente/aframe-spa-skeleton) that quickly bootstraps a virtual reality single-page application with auto reload and publishing features. Read the [a-frame-skeleton README file](https://github.com/delapuente/aframe-spa-skeleton#a-frame-spa-skeleton) for further information. + +## Planning the demo + +The app’s initial aim was to provide a simulation of the process of backing up the contents of a chip from an in-car key reader device, which includes loosening the solder and removing the chip from a circuit board. As a reference, I used a YouTube video showing the [complete in-real-life procedure](https://youtu.be/qFGX3AHVLqA?t=90). The video comprises several steps involving around eight different tools. For this series, I'm going to focus on the specific step of separating the chip from the circuit board. See a detailed sequence here: + + + +The required steps are as follow: + + 1. With a hot air gun, soften the soldering around the chip. + 2. With the help of a suction pad, separate the chip from the circuit board. + 3. Carefully place the chip on the table, to avoid ruining the chip pins. + +I started prototyping to implement one step at a time without realistic models in the beginning. Once I am able to recreate the complete process, I can think about adding different levels of detail to the demo. + +But before providing specific simulations and behaviors, we need some basic hand-based interaction. + +## Setting the scene up + +The first thing I needed was a scene containing all the elements I wanted to work with: the air gun, the suction pad, the chip, the soldering around the chip, and the circuit board. Of course I needed a representation of my own hands and some environmental elements such as a sky, a floor and of course a table! + +This is the initial code for the scene: + +```html + + + + + + + + + + + + + + + + + + + + + + + +``` + +When working with A-Frame, all the scene objects are contained inside the `a-scene` element. A-Frame provides some special tags to achieve common tasks such as adding a sky dome or a camera into the virtual space and it also has primitives available such as boxes, planes, or cylinders. + +This is how I set up the environment with a sky dome, and a plane decorated with a couple of sad grey tones. + +```html + + + +``` + +Length units in A-Frame are expressed in meters and angles are described in sexagesimal degrees (0º to 360º). Thus, the floor has a surface of _15m x 15m_, the workbench is _2cm_ thick and has a surface of _2m x 1m_, and the circuit board is _3mm thick_ and measures _5cm x 5cm_. Real measures are important for two reasons: they **improve the feeling of immersion** and provide **realistic physics**. + +When placing elements, take into account that they are placed according to their central point in the coordinate system of the parent element. If they are direct children of the `a-scene` element, then they are positioned according to the world coordinates system. In the case of the chip, **it is positioned relative to the central point of the circuit board** because the circuit board is the parent node of the chip. + +```html + + + + +``` + +A-Frame provides a way of adding empty nodes to the scene or, in A-Frame nomenclature, an **entity**. [Entities](/docs/0.6.0/core/entity.html) are represented by the `a-entity` element and they have nothing associated with them: no geometry, model, or material. They do have implicit `rotation`, `position`, `scale` and `visible` attributes, which are automatically injected by the framework for all the elements within the scene. + +Adding the [`hand-controls`](/docs/0.6.0/components/hand-controls.html) attribute to an entity displays a hand model following the position and orientation of one of the tracked controllers of an HTC Vive or Oculus. + +```html + + +``` + +This is the resulting scene: + +[![Initial scene](/images/blog/hands-on/step1-initial-scene.png)](/images/blog/hands-on/step1-initial-scene.png) + +### One workspace to rule them all + +To increase the feeling of immersion, I placed the elements as if they were resting on my real desktop only 5cm higher up. This allowed me to avoid my hands colliding with my physical keyboard when I’m in VR mode. Adjusting the whole workspace at the same time is achieved by changing the `position` attribute of the `workspace` entity. Thanks to the real and virtual table heights matching, the demo has a great feeling of presence. + +A simpler alternative would have been to make the circuit board and tools children of the table. Unfortunately this configuration does not play well when adding physics during the next article. The physics engine would consider the nested elements to be compounding parts of the root element and I didn’t want my tools and circuit board to be considered part of the table but separated elements. + +Also notice that if you are experiencing room-scale VR, you will also need to manually position the workspace and camera relative to the center of your configured stage, making them match with your real-life table and yourself respectively. + +Finally, while developing I spend most of my the time sitting in front of my computer. So as a last touch I changed the height correction of the camera, the `user-height` attribute, to match the distance of my eyes from the floor when I'm sitting down: + +```html + +``` + +This correction is [automatically disabled when entering VR](/docs/0.6.0/components/camera.html#vr-behavior) mode since the position of the camera is provided by the VR head-mounted display (HMD). + +## Conclusion + +The very first thing you need to start implementing a simulation experience is a clear reference of the real procedure. Some clues? Watch videos, take descriptive notes and diagrams, and maintain a fluid conversation with an expert. + +The next step is to figure out a basic depiction of what the environment looks like. You don't need complex models to do it, just some basic shapes to provide placeholders for the realistic versions that you will add in the future. This is a common practice when prototyping game environments called [**greyboxing**](http://jackw-gamedesign.tumblr.com/post/139960850160/what-is-greyboxing). + +In the [next article](/blog/hands-on-ii/), I will add some physics in preparation for grabbing things, something imperative in almost any simulation experience. diff --git a/src/_posts/hands-on-ii.md b/src/_posts/hands-on-ii.md new file mode 100644 index 00000000..26fa1415 --- /dev/null +++ b/src/_posts/hands-on-ii.md @@ -0,0 +1,175 @@ +--- +title: Hands-On! Simple Physical Behaviours In A-Frame +author: twitter|salvadelapuente|Salvador de la Puente +date: 2017-08-09 +layout: blog +image: + src: hands-on-ii.png +--- + +This article is the second in a series showing how to build a hand-based, interactive training experience in VR for a circuit manipulation procedure. In the [first part](/blog/hands-on-i/), I chose a [specific step in the procedure](https://www.youtube.com/embed/tHP2kX6aAZM?rel=0&showinfo=0) and prototyped the workspace, tools and materials used in that step with simple geometrical primitives. This time around I will add basic physics to the elements of the scene and you will learn about A-Frame’s entity-component-system architecture. + +All articles in the series: + + 1. [Hands-on! Building interactive training with A-Frame](/blog/hands-on-i/) + 2. Hands-on! Simple physical behaviours in A-Frame + 3. [Hands-on! Hand-based interactions in A-Frame](/blog/hands-on-iii/) + + + +## The project repository + +If you’ve not done so already, clone the [repository of the simulation](https://github.com/delapuente/aframe-simulation-demo). Enter the repository and run `npm install` and `npm start` to install the dependencies and start the development server at port `8080`. + +Look at [`step2/index.html`](https://github.com/delapuente/aframe-interactive-training/blob/master/step2/index.html) and [`js/step2`](https://github.com/delapuente/aframe-interactive-training/tree/master/js/step2) to browse the finished HTML and JavaScript code of this step. Open a browser and enter the following URL `localhost:8080` to find the index of steps. Append `step2` at the end of the URL path to play the scene after step 2 is completed (or select step 2 in the list). + +## The holding problem + +An accurate correspondence between virtual and real objects is vital to increase the sense of presence. The problem with the hands models in the scene is that they are not oriented in the same way than your real hands hold the controllers. + +Take a look at the following video, focus on how the real fingertips and the virtual hand's don't match. The grab axis, around which the fingers close, is also misaligned. + + + +The simulation is more effective when using the 3D models of the VIVE controllers since the tools you see in VR will match the real tools hold in your hands. Using the `vive-controls` instead of `hand-controls` attribute also [allow us to change the model](/docs/0.6.0/components/vive-controls.html#value_model) in the future so we can provide a better representation of our hands. + +```html + + + + + +``` + +## The physics system +The [`aframe-physics-system`](https://github.com/donmccurdy/aframe-physics-system) extension is a wrapper around the [Cannonjs physics engine](http://www.cannonjs.org/) developed by [Don McCurdy](https://twitter.com/donrmccurdy). Mozilla Hacks blog has a good introduction to this extension in [Having fun with physics and A-Frame](https://hacks.mozilla.org/2017/05/having-fun-with-physics-and-a-frame/). + +Enabling physics in A-Frame consists on importing the proper module after importing A-Frame (see `/js/step2/index.js`): + +```js +import * as AFRAME from 'aframe'; +import * as physics from 'aframe-physics-system'; +``` + +Next, we need to set some attributes in the elements of the scene to define how they will be affected by the physics engine. I turned the floor, the workbench, and the operator's hands into _static bodies_ by adding the `static-body` attribute to these elements. A _static body_ is still a physical object but its properties are not controlled by the physics engine —they are not affected by gravity or other forces, which prevents the floor and table from falling indefinitely. + +```html + + + + + + + + + + + +``` + +On the contrary, a _dynamic body_ is an object fully controlled by the physics engine. I turned the circuit board containing the chip into a _dynamic body_ by setting the attribute `dynamic-body` on it. The value of the property establishes the mass of the object —I have set it to weigh 10g (which equates to 0.01, since Cannon.js uses [S.I.](https://en.wikipedia.org/wiki/International_System_of_Units) units, so kilograms for mass). + +```html + + + + +``` + +Dynamic bodies collide with other dynamic bodies **and with static bodies**. This is what the scene looks like after enabling physics: + +[![Red boxes around the floor and table. The circuit board is missing.](/images/blog/hands-on/step2-initial-scene.png)](/images/blog/hands-on/step2-initial-scene.png) + +Red boxes are a result of the physics engine debug mode being enabled. You can enable it by setting the `physics` attribute of the `scene` tag to `debug: true`. You'll see a red box around the workbench, floor and circuit board... wait a moment! Where the heck is the circuit board!? + +[![The circuit board is under the table. It fell through the table somehow.](/images/blog/hands-on/step2-where-is-the-chip.gif)](/images/blog/hands-on/step2-where-is-the-chip.gif) + +Oh! There it is. + +### Limits of the physics system + +It turns out that the chip is not resting on the table but on the floor. The chip fell through the table because of the thickness of their physical bodies. + +Indeed, Cannon.js [lacks from continuous collision detection](https://github.com/schteppe/cannon.js/issues/50#issuecomment-13730383): it only checks if an object is colliding another from frame to frame. With the Earth gravity (which is the default in Cannon.js), an [object falls around 3cm after the first 80ms](https://en.wikipedia.org/wiki/Free_fall#Uniform_gravitational_field_without_air_resistance) (~5 frames). Assuming I was probably missing some frames during the scene setup, a quick workardound was to increas the thickness (the `height` attribute of a box) of the workbench by a couple of centimetres to prevent the circuit board from passing through the table. This forced me to relocate the elements resting on the table, but this was preferable to changing the gravity. + +```html + + +``` + +Another problem regarding the physics engine is related to the collision bounding shapes used for rotated models such as the VIVE controllers. They are miscalculated in terms of dimensions and positioned out of place for certain initial configurations. + +[![Miscalculated and misplaced bounding box.](/images/blog/hands-on/step2-miscalculated-collision-box.png)](/images/blog/hands-on/step2-miscalculated-collision-box.png) + +Nevertheless, this is something I’ll fix during the next article. At this point, you can experiment with the scene using the bounding box to move the circuit board around. Since the controllers can pass through the table, a funny interaction is trying to raise the chip placing your controller below it: + +[![The VIVE controls interact with the circuit board realistically.](/images/blog/hands-on/step2-raising-the-circuit-board.gif)](/images/blog/hands-on/step2-raising-the-circuit-board.gif) + +## The entity-component-system architecture + +The [entity-component-system pattern](https://en.wikipedia.org/wiki/Entity%E2%80%93component%E2%80%93system) (ECS) is an architectural pattern in which objects or entities are composed of multiple aspects. A component captures an aspect of an entity and a system orchestrates the global interactions among the entities possessing a component of the same aspect. The A-Frame documentation includes a [complete chapter dedicated to ECS](/docs/0.6.0/introduction/entity-component-system.html). + +In A-Frame, the entities are the `a-entity` elements and all the primitives (`a-box`, `a-plane`, `a-cylinder`, `a-camera`...) included in the scene. Components are represented by the elements' attributes so, hereinafter, we will stop using the word _attribute_ and start using _component_ when applicable. For instance, consider the operator's hands: + +```html + + + + +``` + +Each hand is actually an entity which has two components `static-body` and `vive-controls`. The first is required by the physics engine to detect when it is colliding with other bodies. The second shows the specified controller making it automatically match the position and rotation of the tracked control. Components can be one-value such as the former `hand-controls` or multi-value like `static-body` or `vive-controls` (i.e., comprised of several named values like the _mass_ or the colliding _shape_). The different values of a multi-value component are called properties. + +Finally, systems are pure behaviour extensions and therefore, they are usually invisible. Take the physics engine for instance. It is a system and it orchestrates the interactions between those entities with `static-body` and `dynamic-body` components (among others). + +### A note on primitives + +[Primitives are entities under the hood](/docs/0.6.0/introduction/html-and-primitives.html#primitives). Primitives have specific attributes mapped to certain properties of components implied by the primitive. For instance: + +```html + + +``` + +This `a-cylinder` primitive implies [`geometry`](/docs/0.6.0/components/geometry.html) and [`material`](/docs/0.6.0/components/material.html) components. The `primitive` property of the `geometry` component is automatically set to `cylinder`. Then, `height`, `radius`, and `color` are attributes mapped to `geometry.height`, `geometry.radius`, and `material.color` respectively, while `position` and `rotation` are components on their own. + +The equivalent entity looks like: + +```html + + +``` + +## The A-Frame registry + +The [A-Frame registry](/aframe-registry/) is a centralized source of components. There you can find a collection of components, download the libraries and visit their home pages. Look at the [`aframe-auto-detect-controllers`](https://www.npmjs.com/package/aframe-auto-detect-controllers-component) component for instance. Given that I replaced the hand controls with the HTC VIVE controllers, I could have used this component to automatically detect which particular tracked controller to use, e.g. HTC Vive or Oculus Touch. + +## Conclusion + +All the bits we need are now in place to start implementing behaviours. Despite the limitations of the Cannon.js physics engine, [Don McCurdy’s](https://github.com/donmccurdy) A-Frame physics offers a simple and intuitive component-oriented API, and exposes enough features for covering our specific needs. Nevertheless, for simulation experiences to succeed in WebVR, we need more accurate and faster physics implementations. My hopes are with [Web Assembly](https://hacks.mozilla.org/2017/02/a-cartoon-intro-to-webassembly/), a new language for the Web that promises huge performance gains over JavaScript. + +In the [next chapter](/blog/hands-on-iii/) I will go deeper in the ECS pattern, implementing _grab and drop_ and fixing the collision bounding box problem by using some custom components and systems. diff --git a/src/images/blog/hands-on-i.png b/src/images/blog/hands-on-i.png new file mode 100644 index 00000000..89a90f66 Binary files /dev/null and b/src/images/blog/hands-on-i.png differ diff --git a/src/images/blog/hands-on-ii.png b/src/images/blog/hands-on-ii.png new file mode 100644 index 00000000..c578e83e Binary files /dev/null and b/src/images/blog/hands-on-ii.png differ diff --git a/src/images/blog/hands-on-iii.png b/src/images/blog/hands-on-iii.png new file mode 100644 index 00000000..1958264c Binary files /dev/null and b/src/images/blog/hands-on-iii.png differ diff --git a/src/images/blog/hands-on/step1-initial-scene.png b/src/images/blog/hands-on/step1-initial-scene.png new file mode 100644 index 00000000..912e797d Binary files /dev/null and b/src/images/blog/hands-on/step1-initial-scene.png differ diff --git a/src/images/blog/hands-on/step2-initial-scene.png b/src/images/blog/hands-on/step2-initial-scene.png new file mode 100644 index 00000000..0eaf41e2 Binary files /dev/null and b/src/images/blog/hands-on/step2-initial-scene.png differ diff --git a/src/images/blog/hands-on/step2-miscalculated-collision-box.png b/src/images/blog/hands-on/step2-miscalculated-collision-box.png new file mode 100644 index 00000000..4c24a6b0 Binary files /dev/null and b/src/images/blog/hands-on/step2-miscalculated-collision-box.png differ diff --git a/src/images/blog/hands-on/step2-raising-the-circuit-board.gif b/src/images/blog/hands-on/step2-raising-the-circuit-board.gif new file mode 100644 index 00000000..bdf08003 Binary files /dev/null and b/src/images/blog/hands-on/step2-raising-the-circuit-board.gif differ diff --git a/src/images/blog/hands-on/step2-the-holding-problem.mp4 b/src/images/blog/hands-on/step2-the-holding-problem.mp4 new file mode 100644 index 00000000..a248ba5c Binary files /dev/null and b/src/images/blog/hands-on/step2-the-holding-problem.mp4 differ diff --git a/src/images/blog/hands-on/step2-where-is-the-chip.gif b/src/images/blog/hands-on/step2-where-is-the-chip.gif new file mode 100644 index 00000000..611a4804 Binary files /dev/null and b/src/images/blog/hands-on/step2-where-is-the-chip.gif differ diff --git a/src/images/blog/hands-on/step3-grabbing-physical-bodies-thumb.png b/src/images/blog/hands-on/step3-grabbing-physical-bodies-thumb.png new file mode 100644 index 00000000..d7d9f8db Binary files /dev/null and b/src/images/blog/hands-on/step3-grabbing-physical-bodies-thumb.png differ diff --git a/src/images/blog/hands-on/step3-grabbing-physical-bodies.gif b/src/images/blog/hands-on/step3-grabbing-physical-bodies.gif new file mode 100644 index 00000000..88f6bff9 Binary files /dev/null and b/src/images/blog/hands-on/step3-grabbing-physical-bodies.gif differ diff --git a/src/images/blog/hands-on/step3-grabbing-tools-thumb.png b/src/images/blog/hands-on/step3-grabbing-tools-thumb.png new file mode 100644 index 00000000..a240a9dc Binary files /dev/null and b/src/images/blog/hands-on/step3-grabbing-tools-thumb.png differ diff --git a/src/images/blog/hands-on/step3-grabbing-tools.gif b/src/images/blog/hands-on/step3-grabbing-tools.gif new file mode 100644 index 00000000..95a3162c Binary files /dev/null and b/src/images/blog/hands-on/step3-grabbing-tools.gif differ