Recap2 User's Guide

COPYRIGHT This manual is copyright 2010, PhaseSpace, Inc., All Rights Reserved. This document may not, in whole or in part, be copied, photocopied, reproduced, translated, or reduced to electronic medium or machine-readable form without prior consent, in writing, from PhaseSpace, Inc. The distribution and sale of this product are intended for the use of the original purchaser only. Duplicating, selling, or otherwise distributing this product is a violation of the law.

DISCLAIMER
PHASESPACE INC. MAKES NO WARRANTIES, EITHER EXPRESSED OR IMPLIED, WITH RESPECT TO THE SYSTEM DESCRIBED HEREIN, ITS QUALITY, PERFORMANCE, MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE. THIS SYSTEM IS SOLD AS IS. THE ENTIRE RISK AS TO ITS PERFORMANCE IS WITH THE BUYER. SHOULD THE SYSTEM PROVE DEFECTIVE FOLLOWING ITS PURCHASE, THE BUYER (AND NOT PHASESPACE, INC. THEIR DISTRIBUTORS OR THEIR RETAILERS) ASSUMES THE ENTIRE COST OF ALL NECESSARY DAMAGES. IN NO EVENT WILL PHASESPACE, INC. BE LIABLE FOR DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL DAMAGES RESULTING FROM ANY DEFECT IN THE SYSTEM EVEN IF IT HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. SOME LAWS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF IMPLIED WARRANTIES OR LIABILITIES FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO THE ABOVE LIMITATION OR EXCLUSION MAY NOT APPLY.

PhaseSpace and the PhaseSpace logo are registered trademarks, and all PhaseSpace product names are trademarks of PhaseSpace Inc.

Preface
The purpose of this document is to describe the PhaseSpace Recap2 software that is included with the PhaseSpace Impulse system. It will give an overview of the features that will allow users to view, clean up, and generate skeletons for recorded and live data. PhaseSpace Recap2 is a motion capture data editing and skeleton fitting program. It allows users to stream motion capture data, edit motion capture data, and fit data to a skeleton entirely within PhaseSpace software. Data from Recap2 can be saved as 3D point or skeleton data to be used in other software.

Table of Contents
Preface................................................................................................................................................................... 2 Sessions................................................................................................................................................................ 4 Managing Takes................................................................................................................................................. 5 The Takes List.................................................................................................................................................... 5 Viewing C3D Data................................................................................................................................................. 7 Loading Data Into Recap2................................................................................................................................. 7 Viewing Window................................................................................................................................................ 8 Groups............................................................................................................................................................. 13 Playback Controls............................................................................................................................................ 13 Graph Window................................................................................................................................................. 14 Region Highlighting.................................................................................................................................... 15 View Options.................................................................................................................................................... 19 Exporting Data.................................................................................................................................................... 20 Marker Data Cleanup.......................................................................................................................................... 20 Track Tools....................................................................................................................................................... 20 Track Tool Types........................................................................................................................................ 22 Applying a Tool........................................................................................................................................... 22 Saving and Loading a Tool......................................................................................................................... 23 The Tool Queue............................................................................................................................................... 24 Track Tool History............................................................................................................................................ 25 Batch Processing Files...................................................................................................................................... 26 Rigid Bodies in Recap2...................................................................................................................................... 27 Creating Rigid Bodies...................................................................................................................................... 27 Managing Rigid Bodies.................................................................................................................................... 30 Interpolating With Rigid Bodies........................................................................................................................ 31 Skeletons in Recap2........................................................................................................................................... 33 Skeleton Training............................................................................................................................................. 33 The Training Data....................................................................................................................................... 33 Bone-Marker Mapping................................................................................................................................ 34 Joint Estimation.......................................................................................................................................... 35 Generating the Skeleton............................................................................................................................. 36 Occlusion Transition Smoothing................................................................................................................. 38 Generating Skeletons Without Training...................................................................................................... 39 Generating Multiple Skeletons.................................................................................................................... 39 Exporting Skeletons......................................................................................................................................... 39 Resetting and Removing Skeletons................................................................................................................. 40 Marker Cleanup With Skeleton Estimation...................................................................................................... 40 Viewing Data Streams........................................................................................................................................ 41 OWL Config..................................................................................................................................................... 41 Trackers........................................................................................................................................................... 42 Tracker Options............................................................................................................................................... 42 Cameras.......................................................................................................................................................... 43 Streaming Raw Pixel Data............................................................................................................................... 44 RPD List..................................................................................................................................................... 44 The Toolbar...................................................................................................................................................... 45 Viewing Options............................................................................................................................................... 45 The OWL Emulator.............................................................................................................................................. 47 Live Tools......................................................................................................................................................... 48 Live Skeletons................................................................................................................................................. 49 Appendix: Bone Marker Mapping Guide........................................................................................................... 51 Head And Limb Bones..................................................................................................................................... 51 Spine And Torso............................................................................................................................................... 52 Fingers............................................................................................................................................................. 56

Sessions
Multiple live takes and C3D files can be brought into Recap2 and grouped together as a session. Each file and recording is placed in a separate take, which can be accessed from the drop-down menu at the top or the Takes list (see The Takes List).

Sessions are saved and loaded as *.db files. To create a new session, select Session->New from the main Session menu or press Ctrl+N.

Look in: Sets the location of the session file. The file location also sets the default location Recap2 will have open when the user searches for recorded data. Auto-Record: The user can also set the auto-record properties of the session, automatically creating an RPD or C3D file when recording a take. Auto-recording of DB files means the data is immediately written to the file. Name: Sets the name of the session file. To save a session to file, click Session->Save... or Session->Save As.... Note that if many operations were performed before the save, it may take several minutes to write to the session file. To load a session file, click Session->Open... and select the session to open from the
4

file browser. Previously accessed sessions can be opened through Recent Session.

Managing Takes
When a session is loaded from a file, takes are initially closed (inactive). Select the take from the drop-down menu or open it from the Takes list to activate it. To save a modified take, select it and go to Takes->Save or press Ctrl+S. Takes->Save All Takes (Ctrl+Shift+S) saves all the takes in the session at once. Takes can be closed manually by going to Takes->Close Take or pressing Ctrl+W. Takes->Close All Takes deactivates all takes in a session. Takes can be removed from a session by opening the take and selecting Takes->Remove Take. Note that this permanently removes the take from the session file and cannot be undone.

The Takes List

The Take List tab shows all of the takes that have been recorded or imported into the session, as well as the status of each take. It also shows the marker count, frame count, sample frequency, and source file (if data came from a C3D).

The take currently opened in the viewer is highlighted in green. Black text represents open takes, gray text represents closed takes. The Open button appears on the bottom bar when a closed take is selected. Double-click on the entry or
5

select the entry and click the Open button to open a take. Blue text indicates a recently opened C3D (see Loading Data Into Recap2). The Import and Close buttons appear when the entry is selected. Click the Import button to add the data to the session file. Note that imported data becomes separate from the C3D file. Clicking the Close button gives the option of either importing the data or discarding the take. Red text indicates a take that has been modified but has not yet been saved. The Save and Close buttons appear when the entry is selected. Right-clicking an opened take in the Take List will open a pop-up menu of commands for that take, including Edit Info, Close, Save/Import, and Remove. Edit Info allows the user to rename a take and change the description text. The Close button or menu option can be used to close the currently selected take.

Viewing C3D Data

Loading Data Into Recap2
1) Go to Session->Import... or press Ctrl+I to load a C3D file into the session.

2) In the file chooser, select the C3D you want to open. 3) Once loaded, the marker data should be visible in the viewing window. Users can also use Session->Recent C3D to view a recently opened C3D.

Viewing Window
The user's view of the data can be manipulated using the middle mouse button, the scroll wheel, and the right mouse button: Clicking the middle mouse button in the grid moves the camera along the XZ plane. Holding Shift while pressing the middle or right mouse button moves the camera along the Y-axis to adjust height. The scroll wheel zooms the camera in and out. The right mouse button orbits the camera around the point of interest, indicated by a yellow circle with a line emerging from the center.

Pressing the space bar in the viewing window opens the context menu. From here, the user can access submenus for controlling selection, the display, the camera, marker coloration, and the skeleton's display. The Select submenu commands allow the user to select all markers or one at a time: Select All (Ctrl+A): Select all markers in the take Select Previous (Down Arrow): Select marker with previous ID Select Next (Up Arrow): Select marker with the next ID

The Display submenu commands toggle the visibility of elements in the viewer: Markers: Toggle display of markers. Rigids: Toggle display of rigid body markers. Cameras: Toggle display of cameras. Draw Trails: Recorded data only. Toggle the drawing of marker trails. Trails are displayed for selected markers at selected frames.

Draw Preview Trails: Recorded data only. Toggle the drawing of preview trails for markers when track tools will be used. Draw Trails must also be enabled before preview trails are displayed.

Draw Track Distance: Recorded data only. Toggle the display of distance traveled by a selected marker over selected frames. Distance is shown along with the time duration of the frames selected.

Draw Normals: Available for live and RPD streams. If enabled, selecting cameras shows which markers are visible to each camera as a line drawn from the visible markers to the camera. The color of the line determines which detector in the camera has seen the markeryellow for both detectors, red or green for one detector. Selecting markers along with cameras shows which of the selected markers are visible to the selected cameras.
9

The Camera submenu commands (and corresponding shortcut keys) offer several options for viewing the data: Follow Selection (Shift+F): When markers are selected and this is enabled, the camera will follow the markers' movements. Look Down (Y): View data from above Look Up (Shift+Y): View data from below Look Back (Z): View data from the back Look Forward (Shift+Z): View data from the front Look Right (X): View data from right side Look Left (Shift+X): View data from left side Reset Camera (Shift+R): Resets camera to initial position Toggle Ortho View (O): Toggle orthographic view Free Panning: When toggled, the middle mouse button pans the camera instead of traveling along the ground plane

The Colors submenu commands toggle the coloring of markers. By default, markers are displayed as white. Auto: Automatically assign markers colors based on the LED group they belong to. Enable Custom: Toggles user-defined marker coloring. Customize: Opens a dialog to set the color of selected markers.

10

The Skeleton submenu commands toggle the display of elements relating to the Recap2 skeleton. Reference Pose: Show the initial pose of the skeleton.

Estimated Markers: Show the positions of markers as estimated by bone position. Positions are shown in pink, with lines connecting the markers to their associated bones.

Joint Info: Show the labels for each joint. Bone Volumes: Show the bones as polygons. Otherwise, bones are displayed as lines.

11

Marker Weights: Color markers according to the amount of influence they have on the skeleton.

When markers are selected, additional options appear in the context menu: Hide Selected (Ctrl+H): Hide the selected marker(s). Hidden markers cannot be selected. Hide Inverse (Shift+H): Hide all but the selected marker(s). Create Rigid (Ctrl+B): See Rigid Bodies in Recap2. Unhide All (Ctrl+Shift+H): Make all hidden markers visible.

The left mouse button is used to select markers. Double clicking on a marker will center the camera on that marker, while double clicking in empty space deselects markers. The user can also center on a group of markers by selecting them and pressing F. To toggle the display of marker IDs (as well as camera IDs when streaming data), press N or click the IDs button next to the take selection. The size of displayed markers can be adjusted with the drop-down menu to the left of the IDs button. Markers can be shown as small (1x), medium (5x), or large (10x) cubes.

12

Groups
It is possible to isolate specific subsets of markers in the Viewer by using groups. The visibility of these groups can be toggled, allowing the user to hide some markers while inspecting other markers. Groups are managed using the pop-up menu commands, which are accessed by right-clicking in the Groups panel. Rename: When a group is created it is labeled group # by default. The user can change the group's name using this command. Hide/Unhide: Toggle the visibility of the selected group. When hidden, the group's name is parenthesized in the Groups panel, and the group's markers cannot be selected in the viewer and their trails are not visible. The markers' graphs can still be accessed by selecting the markers in the Trackers panel. Add Group: Create a group using the selected markers. Delete Group: Remove the selected group.

Playback Controls
The playback controls allow the user to review the captured data.

1) Playback Speed: Set the speed at which the data is played back. Note that downsampled data will not appear smooth at the lower playback speeds. 2) Loop Playback: If toggled, the playback repeats when the end is reached.
13

3) Go to Beginning: Jump to the beginning of the animation. 4) Play Reverse: Play back the data backwards. 5) Step Back One Frame: Go to the previous frame. 6) Frame Number: Indicates current frame number. This can also be used to jump to a specific frame. 7) Step Forward One Frame: Go to the next frame. 8) Play Forward: Play back the data forwards. 9) Go to End: Jump to the end of the animation. 10) Time: Indicates the current time position in the data. 11) Timeline: Allows the user to scrub through the animation. Click a location on the timeline to jump to a specific frame. 12) Start Frame: The first frame of the data. Usually, this will be 0. 13) Segment Start Frame: Starting frame number of data segment. If the user wants to look at a subset of the total frames, this can be used to set the starting point. 14) Set Segment: Drag either end of this bar to isolate a subset of the total frames. 15) Segment End Frame: Ending frame number of data segment. If the user wants to look at a subset of the total frames, this can be used to set the end. 16) End Frame: The total number of frames in the data.

Graph Window
The graph displays the curves of selected markers. This is also where the user can isolate a segment of marker-frames to examine and select segments to clean up.

The left mouse button allows the user to scrub through the animation. Holding Ctrl while using the left mouse button lets the user select a subset of frames. The middle mouse button pans the camera in the graph.
14

The scroll wheel zooms in and out on the graph, with the mouse position as the center of focus. The right mouse button adjusts the vertical and horizontal scales of the graph. For example, the user can stretch the horizontal scale to make gaps look wider and easier to see. The user can mouse over any data point on a curve to obtain information about the point. This information includes which marker the curve belongs to, which channel (X, Y, or Z) it describes, the frame number for the point (F), and the point's value (C). This information is displayed in the left-most box in the window's status bar. In the side bar are five buttons for manipulating the graph. Select All Frames: Select every frame in the sequence. Autoscale Viewport: Scale the selected region so that it fills the graph window. Toggle RGB: Turn coloring of graph lines on and off. Add Point: Add data point to the graph. Move Point: Toggle cursor to move selected data point.

Region Highlighting
When the user selects a subset of marker-frames, Recap2 uses different background colors to indicate the status of the data. The number of frames in the selected segment is also displayed at the top of the graph. Within the selected data segment, the visible data region is highlighted in light blue. Unselected data appears in dark gray.

Occluded regions appear as gray gaps.

15

Modified data regions are highlighted in tan (see Marker Data Cleanup).

If the Mark Noise tool is applied, regions containing noise peaks are highlighted in green.

When data of different types (such as occluded and visible) overlap, the region is highlighted in pink.

16

Pressing the space bar while the mouse is in the graph window opens the context menu. The Graph menu commands and shortcut keys provide viewing options useful for viewing data during the cleanup process.

Select All (Ctrl+A): Select all frames of selected marker(s) Autoscale (A): Zoom in on selected area Update track offsets (T): Set track offset to the current frame's values Next area of interest (Ctrl+.): Center on next gap or peak in graph Previous area of interest (Ctrl+,): Center on previous gap or peak in graph Select Region (R): Select all the frames in the region where the menu is opened Display: A submenu toggling the display of the graph elements

17

Show Tool Preview: See Applying a Tool in the Marker Data Cleanup section. Show Images: A diagnostic tool allowing the user to see the peaks that the cameras detect. To make camera images available in the graph window: 1. Camera images are not recorded unless their display is enabled. Enable images by selecting Display->Images in the graph context menu. When enabled, the box next to the entry will be checked. 2. Record a take from a live stream or RPD. 3. After recording, go to the take and select one or more cameras in the viewer. The peaks will appear as red and green points in the graph window. Dark red or dark green points represent unidentified peaks (rejected data) and bright red or green points represent identified peaks (visible data).

18

View Options

Under the View tab in Settings->Options, the user can change the display properties of the viewer and playback controls. ID display time limit causes displayed marker IDs to disappear after a certain length of time, to reduce CPU load. The length of time in which the IDs are visible is controlled by the adjacent text box. In the bottom half of the View tab, the user can toggle graph selection's effects on the viewer and toggle the playback control's effects on the frame cursor in the graph. Select Keymap allows the user to set the camera key controls so that they match those of Motionbuilder, Maya, 3dsMax, or Lightwave.

19

Exporting Data
In addition to importing C3D data, the user can also export to a C3D file containing a subset of the captured frames or markers. Click Session->Export... to open the Export dialog box.

The Selected and All buttons allow the user to export only the frames selected in the Graph or all frames in the take, respectively. Otherwise, the range of frames to be exported can be entered in the text boxes underneath. By default, this is the entire take. If Use Selected Markers is checked, only data from the markers selected in the viewer will be exported. Otherwise, the range of IDs for the marker data to be exported can be entered in the text boxes under the check box. If this is left blank, or Use Selected Markers isn't checked, all markers will be exported. Click Continue to open a file dialog and choose the file name and location of the exported file. This is also where the export type can be chosen. Recorded marker data can be exported as a .c3d or .owl (ASCII format) file.

Marker Data Cleanup

Track Tools
20

This is where the user can access the tools used to clean up motion capture data. Select Type: Choose the tool to use. Tool Settings: Depends on the tool selected. For example, when Interpolate is selected, the user can choose the type of interpolation to use, the maximum gap size to fill (0 means unlimited gap size), and the number of frames to extend the interpolation by on either side of the segment. Apply to Selection: If multiple markers were selected, the user can have the tool applied to only one of the markers in the group. By default, a tool is applied to every marker in the selection. For the Mimic Marker tool, use the selection box to choose which marker will be mimicking the other marker. Apply to Channel: The user can choose which channel(s) to apply the tool to. By default, a tool is applied to the X, Y, and Z channels. Apply to Region: When enabled, the tool will only be applied to the checked regions. Occluded: regions lacking data Visible: visible, non-modified data Peak: regions marked by the Mark Noise tool Modified: edited data

21

Track Tool Types

None: No tool selected. Use when the data will only be reviewed, not modified. Clear: Remove the data within the selected segment. Settings include Max Length (the maximum width of region to apply tool to, with 0 being no limit) and Extend (the number of frames beyond the region to apply the tool to). Smooth: Apply a smoothing filter to reduce marker noise. This tool has no effect on gaps. Settings include choice of filter (Butterworth or linear regression) and Frequency (Amount of smoothing; lower frequencies mean more smoothing). Mark Noise: Highlight data points that qualify as noise. Noise criteria can be adjusted in the tool's settings. Settings include choice of filter (Butterworth or linear regression), Frequency (the marker movement frequency that would qualify as noise), Threshold (the movement amplitude that would qualify as noise), and Merge (merge noise regions if the number of frames in between is no larger than the value specified). Interpolate: Apply interpolation to replace data segments or fill gaps. Settings include choice of interpolation (linear, quadratic L-slope, quadratic R-slope, cubic, rigid body, or mimic marker), Max Length, and Extend. in Recap2), bones can be used to approximate the positions of markers to account for noise and gaps. Settings include choice of action (generate or overwrite data), Max Length, and Extend. Skeleton estimation only works on markers that are part of the skeleton's bone-marker mapping.

Skeleton Interpolate: After a skeleton has been generated for the data (see Skeletons

Applying a Tool
After the tool has been selected and the settings have been adjusted as needed, click the Apply button underneath the Apply-to settings box. Tools can also be applied by the Apply Tool command in the Graph menu or by pressing Ctrl+Space.

22

If Display->Tool Preview in the context menu is enabled, a preview of the result is shown as dark gray curves before the tool is applied. One exception is when Clear or Mark Noise is used; the preview is shown as red curves instead.

After the tool is applied, the affected regions will be marked as Modified and highlighted in tan.

Saving and Loading a Tool

Frequently-used tool settings can be saved as tool files (*.tl) for future use. To create a tool file, click the Save ( ) button next to the Enqueue button. A file dialog will appear for choosing the location and file name in which to save the tool file. To load a preset tool into Recap2, click the Load ( ) button next to the Select Type drop

menu. A file dialog will appear for choosing the tool file to load.

23

The Tool Queue

Frequently-used sequences of tool operations can also be saved in Recap2. Sequences can be saved as tools list (*.tls) files for future use. Tool queues are independent of sessions, so the queue will persist when the user changes sessions. Creating Queues To add a tool in the queue, first create it in the Track Tools window. Then click the Enqueue button next to the Apply button. The Toolbar Applies the current tool sequence to the selected marker-frames. The drop-down box contains an option to apply the sequence to all markers and frames. Pushes the selection towards the start of the queue. Pushes the selection towards the end of the queue. Removes the selection from the queue. Tools in the queue can also be given custom names. Double click on an entry in the queue to rename it. Tool Queue->File Menu Save: Save the queue to a file. Load: Load a tools list file. Reset: Clear the queue Copy Tool: Duplicate the selected entry and append it to the queue. Set As Current Tool: Set the selected entry as the current tool in the Track Tools window.

Tool Queue->Options Menu

24

Track Tool History

Recap2 keeps track of the tool operations applied to a file during a session. Each file opened has its own history. Starting a new session clears the history. Undoing an Operation In the menu bar, click History->Undo or press Ctrl+Z to undo the most recent operation. Repeating Undo will reverse this. Stepping Back Multiple Operations Click History->Step Backward or press Ctrl+Shift+Z to undo multiple operations, one at a time. Note that if the user goes back several steps, then applies a new tool operation, the history is overwritten. Stepping Forward Click History->Step Forward or press Ctrl+Shift+Y to redo operations one at a time. Clearing Tool History The user can reset the track tool history by selecting History->Clear.

25

Batch Processing Files

If the user has multiple data files to clean up, Recap2 has batch processing capabilities to perform repetitive cleanup tasks, such as filling small gaps or removing peaks. To batch-process a set of files: 1) Go to Session->Batch Files to open the batch-processing dialog.

2) Add files to the list by clicking the Files button in the top left. Files can be removed from the list by selecting them and pressing the Delete button in the top right. 3) Select whether the tool sequence to apply to the files comes from a tool queue or a tool script file. If the source is a tool script, it must be first loaded with Script->Load. 4) Select the output location for the batch-processed files. 5) To differentiate the original data from the processed data, enter a suffix to be attached to the name of each output file. 6) Click Execute to start the batch processing.

26

Rigid Bodies in Recap2

Recap2 features the ability to perform rigid-body interpolation, for gaps that linear, quadratic, or cubic interpolation cannot accurately fill. Rigid bodies are created from groups of markers, and Recap2 predicts the position of a missing marker in the group based on the position of the rigid body (see Interpolating With Rigid Bodies).

Creating Rigid Bodies

1) Select a group of markers in the Viewer. Rigid bodies will perform best if all the markers in the group belong to an area with little deformation. For example, select the head markers as a rigid body group, and select the markers on the torso as a separate rigid body group.

2) In the Rigids window, right click and select Create from the pop-up menu. Alternatively, click the Create button at the bottom of the window, select Create Rigid from the viewing window's context menu, or press Ctrl+B.

27

28

3) Recap2 creates an entry in the rigid body list named rb#. The rigid body is displayed as a round blue marker with XYZ axes on it.

The rigid body's curves appear as yellow lines in the Graph window. Gaps appear in the curves when too many markers in the group are occluded.

29

Managing Rigid Bodies

Right-clicking on a rigid body opens up a pop-up menu to edit and remove the rigid body.

30

Update: Recomputes the rigid body. Use after editing the group's markers. Reset: Regenerates the rigid body. Use after changing to another take. Edit Info: Opens a window allowing the user to rename the rigid body. Delete: Removes the rigid body from the session. Rigids can also be removed with the menu command Rigids->Delete Rigid or Ctrl+Shift+B. This action cannot be undone.

Interpolating With Rigid Bodies

After generating the rigid body for a group of markers, the user can select Rigid Body interpolation in the Interpolate track tools to fill in marker gaps.

Note that if there are gaps in the rigid body curves, there will be gaps in the markers' interpolation. The rigid body's gaps can be filled using interpolation or by recomputing the rigid body after filling the gaps in its markers.

31

32

Skeletons in Recap2
Another feature of Recap2 is the ability to generate a full-body, partial (upper or lower body), and finger skeletons based on a set of training data. This allows the user to clean up data by constraining markers to bones or to export data files that can be brought into third-party animation software.

Skeleton Training
The Training Data
The training data required to generate the full-body skeleton consists of a T-pose and range of motion movements for the ankles, knees, hips, wrists, and elbows. For partial skeletons, upper body skeletons require motion from wrists and elbows, while lower body skeletons require motion from the ankles, knees, and hips. Finger skeletons need only wrist motion. Make sure to have this training data set loaded into the session before attempting to generate a skeleton for offline data.

Joints are estimated by analyzing the movements a set of landmarks, consisting of markers or other joints, which are spread across the two bones that are connected by the joint to be estimated. Recap2 requires at least three landmarks on one of the bones for the estimation to work. These three landmarks allow Recap2 to approximate the position and orientation of that bone and give a rigid body transform for it. The other bone attached to the joint then needs at least one landmark in order to give the position of the joint. This reference landmark sweeps approximately across the surface of a sphere that is centered around the joint's location. For the joint to be estimated accurately, the reference marker should sweep a large portion of the sphere's surface. A range of 30 degrees in both axes of the sphere is usually more than enough, though if the marker data is particularly noisy, more range of motion will be needed to
33

yield a better estimate of the joint. Ideally, the performer would make tight spirals with the joint so that the reference marker sweeps across a fair amount of the sphere's surface, within the range of motion for that joint.

Bone-Marker Mapping
Recap2 requires markers to be associated with each bone that will be generated. Which markers belong to which bone depends on the locations of the marker IDs on the performer. The information is visible in the Skeleton Training window when the skeleton's Bones entry is expanded.

Check the mapping by right-clicking each bone in the list and selecting Select Markers in the pop-up menu. This highlights the markers that will be used in generating the bone's animation in the viewer. If the performer's marker ID locations do not match the bone location, use the Bone Marker Mapping Guide in the Appendix to determine the markers to use for mapping each bone. Bone-marker maps can be saved to an *.sbc file for use in different sessions by right-clicking on the skeleton and selecting Save Bone-Marker Map. The files can then be brought into new skeletons using Load Bone-Marker Map.

34

Joint Estimation
The next step in generating a skeleton is estimating the positions of each joint. Most joint locations are implied by the locations of neighboring joints in the chain, but the locations of the wrists, elbows, hips, knees, and feet must be estimated using the training set data.

To estimate the position of a joint: 1) Import or open the take of the training set data. 2) In the Skeleton Training window, select the joint in the list. Make sure its status is Ready to Estimate. If the status is Not Ready, the joint is waiting for another joint to be estimated first. For example, the ankle joints must be estimated before the knee joints can be estimated. Joints with the Implicit status do not need to be estimated.

3) The markers that will be used to estimate the joint are automatically selected in the viewer and their curves displayed in the graph. 4) Select the segment of frames containing the range-of-motion for the selected joint. 5) Right-click on the joint and select Estimate Joint(s). A warning dialog may appear if the estimation error is too large.
35

6) Check the location of the estimated joint; it will appear as a colored dot. If it does not appear correct, it may be due to bad training data or an incorrect mapping. Sometimes, a misplaced joint occurs when frames outside the range of motion segment are selected; in that case, try selecting a subset within the segment and re-estimating.

After all joints have been estimated, a skeleton can be generated for the training set data as well as for any other takes involving the same performer, as long as the take contains a Tpose and the marker arrangement has not changed. Joint estimations can be saved to an *.skh file by right-clicking on the skeleton and selecting Save Estimated Joints. Saving a Recap2 session to file will also save bone-marker maps and joint estimations. Saved joint estimation files can be loaded into other sessions using the Load Estimated Joints command in the skeleton's pop-up menu.

Generating the Skeleton

After the performer's bone-marker map has been set and the joint estimations have been acquired, Recap2 can produce the bones and skeletal animation driven by captured data from the same performer. The skeleton can be generated for any take in the session, as long as the markers on the performer have not changed location and there is a T-pose in the take. 1) Go to a frame in the animation that contains the T-pose and where all the markers are visible.

36

2) Right-click on the skeleton in the list and select Generate Skeleton. If the data is not in a T-pose, a warning dialog will appear. Note that clicking Yes may yield unexpected results when the skeleton is generated.

37

3) Depending on the amount of data in the training set, it may take several minutes for Recap2 to generate the skeleton. The skeleton will then be displayed in the viewer and its animation can be examined with the playback controls. Once the skeleton is generated, the animation curves of the joints and bones can be viewed in the graph. Select a joint in the to display its data, shown as cyan points.

Select a bone to display its data, shown as yellow points, as well as the data of its associated markers.

Occlusion Transition Smoothing

To reduce unexpected distortions when occlusions occur, the user can set the number of frames over which a marker's influence on the skeleton decreases prior to being occluded. Fewer frames means a shorter interval between a bone's transition from using marker data to using joint interpolation. This value can be set in Settings->Options->Skeleton.

38

Generating Skeletons Without Training

If training data is not available for a capture, Recap2 can automatically generate a skeleton based on rough estimations. To generate a skeleton without a training set, first create or load a bone-marker mapping that matches the data and go to a frame in the take containing a Tpose. Then create a skeleton, right click on it, and select Generate Skeleton(s) Without Training. Note that the resulting skeleton may be more susceptible to noise and distortion than a skeleton produced from training data.

Generating Multiple Skeletons

Recap2 can generate skeletons for multiple performers in a session, provided there is training data for each performer. To create an empty skeleton for another performer, right-click in the Skeleton Training window and select New Skeleton. A dialog box will appear, allowing the user to name the new skeleton. If the performers in the session are not using the same marker IDs, such as when they are being captured all at once, be sure to create different bone-marker maps for each performer's skeleton.

Exporting Skeletons
After a skeleton has been generated, the resulting skeletal animation can be exported to a *.bvh file for use in a 3d animation program, such as MotionBuilder. To do this, right-click on the skeleton in the list and select Export BVH. Skeletons can also be exported to a *.fbx file, by selecting Export FBX from the skeleton's pop-up menu.

39

Resetting and Removing Skeletons

To clear all joint estimations that have been made on one or more skeletons, select the skeleton(s), right-click in the list, and select Reset Skeleton(s) from the pop-up menu. This will also remove the bones in the viewer.

To remove a skeleton from the list of skeletons, right-click on the skeleton and select Delete Skeleton(s).

Marker Cleanup With Skeleton Estimation

After generating the skeleton for a take, the user can choose the Skeleton Estimate tool in Track Tools for cleaning marker data. Skeleton Estimate allows the user to either regenerate skeletal data based on modified marker data (Generate) or overwrite marker data using data derived from the corresponding bone (Overwrite Data). The settings for the tool are similar to those for the Interpolate tool.

40

Viewing Data Streams

Similar to Master, Recap2 can obtain live data from an OWL server or start a stream from raw pixel data (RPD) files. The difference is Recap2 has functionality to pass the raw data through live filters and skeleton estimation for minor data cleanup before outputting processed frames to other client programs, such as MotionBuilder.

OWL Config

The OWL Config window is where the user sets the server configuration for streaming in live data or data from an RPD file. Server Address: Enter the IP address of the OWL server or select <find servers> from the drop-down menu. Slave Mode: If Recap2 will run as the primary client, leave the box unchecked. If the box is checked, the other server settings are disabled. Controller Mode: Set which tracking profile to use. The Default mode is equivalent to mode 1. Frequency: Set the frequency at which data should be streamed. Reduce the frequency if the data will be sent to a remote client. Postprocess: Turns on real-time post-processing if checked. Interpolate: Set the number of frames over which to interpolate. The number should be set to at least the number of LED groups in the profiles to be used.

If changes are made to the configuration while Recap2 is connected, they will not take effect until the user disconnects and reconnects. Server settings can be saved for future use by pressing the Save ( ) button in the upper right corner.

41

Trackers
The Trackers tab allows the user to see the markers that Recap2 is currently receiving data from in the live stream, as well as which markers are visible. It is also a place to select markers regardless of their visibility. Visible markers are displayed in green. Gray markers are those that Recap2 has not received data for, either due to occlusion or the marker being inactive, in the given take.

During live streaming, point trackers can be created to isolate groups of markers. In the Trackers window, select New->Point Tracker (Selection) or right-click on the Trackers menu bar to create a tracker using selected markers. The New->Point Tracker command creates a new tracker using all markers that have no associated tracker. Rigid body trackers can also be created. Unlike offline rigid bodies, the rigid bodies created by trackers cannot be used for marker cleanup, but their data can be acquired and recorded by client applications. Select a group of markers in the Viewer and select RigidBody Tracker (Selection) from the right-click pop-up menu, or click New->RigidBody Tracker (Selection) at the top of the panel. The rigid body will appear at the center of the selected markers. Both types of tracker can be created from a file, either of type *.pt (for point trackers) or *.rb (for rigid bodies). To load a tracker, click New->Load File and select the file to load from the dialog.

Tracker Options
Each tracker is separated by a bar indicating its index and the number of markers it contains. Right-clicking on the bar or left-clicking on the triangle in the bar opens the tracker's menu. Double-clicking on the bar hides the tracker's marker display. Optimize: Removes unused markers from the tracker and reassigns marker IDs as needed. The removal includes markers that were occluded at the time of optimization. Note that even if the removed markers are activated afterward, they will not appear in the tracker. Enabled: Toggles the data stream for the selected tracker. Markers belonging to deactivated trackers are not displayed in the viewer.
42

Fast Mode: If toggled, it will reduce the number of marker rejections for the selected tracker, at the cost of having more marker noise. Rigid Markers (Rigid body trackers only): Overwrite the markers in the tracker with markers constrained to the rigid body. These markers will be visible as long as the rigid body is visible. Save: Save the selected tracker as a *.pt file. Delete: Remove the tracker. Note that all markers belonging to that tracker will disappear from the viewer and remain invisible until the user creates a tracker using New->Point Tracker.

Cameras
During streaming, the status of the cameras is shown in the Cameras window. A blue box indicates that the camera is working normally, while orange and red indicate the camera is getting a significant amount of error, as would be the case if the performer stood too close to it. Gray boxes represent disabled cameras, such as those that are uncalibrated or shut off.

The user can disable and enable cameras through Recap2 by selecting a camera (either from the viewer or from Cameras), right-clicking in the Cameras display, and selecting Disable/ Enable Selected from the pop-up menu. The pop-up menu also contains Verbose Mode, which, when Settings->Camera Data is enabled, toggles the display of the number of markers detected by each camera, and error values when errors are detected (see Viewing Options for more details).
43

Streaming Raw Pixel Data

Go to Session->Stream RPD to choose an RPD file to play back. Once opened, the RPD will automatically start playing. Like a C3D, the RPD file contains marker data, but it also contains camera data so that the recorded take can be replayed as if it were live in the capture space. Data can be recorded from the RPD for editing in Recap2 or for data export. Only one RPD can be streaming at any given time in Recap2. Another RPD stream cannot be started until the current stream ends or the user disconnects from the server being used for streaming. Likewise, RPDs cannot be streamed from a server that is receiving live data until it has disconnected.

RPD List

The RPD List tab contains a table that lists the names and filepaths of the RPDs linked to the session. If the working directory for the current session is set to a directory containing RPD files, the RPD files will automatically appear in the RPD List window. Right clicking in the table opens a pop-up menu with the following commands: Stream RPD: Start streaming data from the selected file. Add RPD(s): Add files to the RPD List. Files added this way do not have to be in the session's working directory. Remove RPD: Remove the selected file. Clear Non-Session RPD(s): Remove all files that are not in the session's working directory. Copy Path to Clipboard: Copy the selected file's path for future input.
44

The Toolbar
After the server configuration is set, the user can connect to the OWL server to bring in realtime data using the toolbar. Connect to server and start streaming data.

Pause the data stream.

Start/Stop recording data. The data is recorded to a take, but data can also be recorded to C3D, RPD, or directly into the session file if specified under Settings->Options->Recording. The length of a recording can be set to a fixed size by enabling Recording Time-Limit and setting Max Frames in the Recording settings. Recap2 can automatically start recording a new take after the recording reaches its time limit if the option Automatically begin recording a new take is selected. Options can be enabled to specify the types of files (RPD, C3D, or DB) produced from recording. There is also an option to toggle the recording of data to RAM, so that the data is temporarily placed in memory before the user saves the file. Note that auto-recording to multiple files is resource-intensive, so Recap2 may slow down significantly if it is auto-recording to all three file types at once. The TTL-triggered recording options allow the user to enable starting recordings with hardware triggers. Optional data, such as camera planes and images, can also be recorded if enabled. Toggle between live stream and current recorded take. This can also be used to switch between the RPD stream and the current recorded take.

Viewing Options
The viewing window context menu provides options for toggling the display of additional camera and marker information in Recap2. This information can be used as a system diagnostic tool. Certain tools can only be used during streaming, while others work only on offline takes. Select All (Ctrl+A): Select all markers in the viewer. OWL: Toggle the display of cameras, markers, and rigid bodies.
45

Skeleton: This submenu toggles the display of skeleton elements. Colors: This submenu toggles the color-coding of markers according to the LED groups they belong to. Since C3Ds do not contain LED group information, Recap2 will assume that all markers belong to one LED group. There are two coloring options: Auto or Custom. If Auto is enabled, Recap2 automatically assigns colors to markers of each LED group. If Custom is enabled, the user can assign colors to selected markers and highlight marker groupings. Go to Colors->Customize to open the color chooser.

Display Draw Normals: Streams onlyshow which markers are currently detected by the selected camera(s) by drawing a line from the camera to each marker. If a marker is visible only to detector 0 on a camera, a red line is drawn. If a marker is visible only to detector 2 on a camera, a green line is drawn. If both detectors see the marker, a yellow line is drawn. Draw Trails: Recorded takes onlydraw the paths of selected marker(s) when frames are selected. If marker coloring is enabled, trails will be colored according to the marker's color. If marker coloring is not enabled, trails will be colored according to the region the marker data belongs to. Segments of visible data are white. Segments marked as peaks are green. Modified segments are tan. Draw Preview Trails: Recorded takes onlydraw a preview of the marker trail after the selected track tool is applied. The trail is drawn with a pink line.

46

The OWL Emulator

Recap2 features a server emulation system that can create a virtual server to output livefiltered frames when connected to an OWL server. Other client programs can then connect to the emulated server as if it were a real server. To set up and start the emulator, go to the OWL Emulator tab under Settings->Options. Press the Start button to begin the emulation, and press Stop button to end it. Set the port number from which to output the emulator's data using the input box below the buttons. Press Apply to have the change take effect. After the emulator is started, client programs can connect to the virtual server using the IP address of the machine running Recap2 and the port number specified in the Options window (<IP Address>:<port>).

47

Live Tools

In Recap2, marker cleanup tools can be applied in real-time to streamed data. The livefiltered data can then be recorded into Recap2 takes or sent to client programs through the OWL emulator. Go to the Auto Cleanup tab under Settings->Options. Enable Live Tools: If checked, marker cleanup will be applied to the data as it is streamed. Record Unmodified OWL Data: If checked, only the raw data will be recorded. Choose Tool Source: The set of tools to apply to the data can come from the current Tool Queue or from the currently loaded Tool Script ( *.qs) file Display in 3D Window: If checked, the filtered data will be displayed as yellow spherical markers in the viewer. These markers are available as visual reference only; they cannot be selected in the viewer. Largest Gap Length (frames): The maximum span of occlusion or noise that the tools can modify. Buffer Length (frames): Affects the number of frames to batch-process at any given time. Note that setting this value too high will increase calculation time and latency, but setting the value too low increases the number of batch-process operations.
48

Modification of this value should be done by advanced users only. Threads: Improve performance of Auto Cleanup by dividing it among multiple threads. Modification of this value cannot occur during streaming and should be done by advanced users only.

Live Skeletons
Live Joint Estimation

Skeletons can also be used to process streamed data in real time. Joint Estimations can be done beforehand in a recorded take of training data, loaded from a *.skh file, or performed live with the actor in the capture space. To do live joint estimation, start the live stream, create the skeleton and set up the bone-marker map as normal, then right-click on the skeleton and select Train Live in the pop-up menu. After the training wizard appears on screen: 1. The wizard starts at Ready. Click Next to go to the first joint to estimate. 2. Have the actor perform range of motion with the specified joint. The wizard automatically captures the data. 3. Click Next to stop recording and make an estimate for that joint and move onto the next joint.

49

4. When the wizard reaches Complete, click Finish. The wizard will automatically generate the skeleton after the window closes, and will appear in the viewer when Live Skeleton is enabled. Live Skeleton Options

Enable Live Skeleton: If checked, live marker data will drive the generated skeleton. Overwrite region types: Clean up marker data with skeleton estimation All: If checked, skeleton estimation will overwrite all marker data. Occlusions: If checked, the skeleton will be used to fill in gaps. Peaks: If checked, the skeleton will be used to remove noise peaks.

Occlusion Transition Smoothing (frames): Sets the number of frames used between the switch from marker-driven skeletal animation to computed skeletal animation. Used in both offline and real-time skeleton driving.

50

Appendix: Bone Marker Mapping Guide

The following images provide a visual guide to which markers should be selected for each bone during skeleton generation. Note that this is the mapping Recap2 creates by default, and your mapping may differ (either in marker choice or marker count) depending on marker placement.

Head And Limb Bones

Cranium: Head markers Upper Arms: Shoulder and upper arm markers for each arm Forearms : Forearm/elbow and wrist markers for each arm Hands : Markers on each hand Left Pelvis : Left hip markers Right Pelvis : Right hip markers Thighs : Uppermost leg markers on each leg Calves: Lower leg markers for each leg Feet: Markers on each foot
51

Spine And Torso

Vertebrae 0-1 Lower back marker and back hip markers

Vertebrae 1-2 Lower torso markers

52

Vertebrae 2-3 Upper back marker and lower front torso marker

Vertebrae 3-4 Upper torso markers

53

Neck Upper torso markers and right back head marker.

54

Left Chest Upper torso markers and left shoulder marker

Right Chest Upper torso markers and right shoulder marker

55

Clavicles Shoulder marker on each arm

Fingers
The bone-marker mapping guide for fingers is designed for the standard glove solution. The mapping may differ for gloves that have different LED configurations. Index 01, Middle 01, Ring 01, Pinky 01 Wrist and hand markers

56

Index 34 Tip of index finger

Middle 34 Tip of middle finger

Pinky 34 Tip of pinky finger

57

Ring 34 Tip of ring finger

Thumb 12 Marker at thumb base

Thumb 23 Tip of thumb

58

Fore-arm Forearm markers

59