1. Unreal Engine (UE) Plugin
Last updated
Last updated
Open the Data Casting Pane and enable "SDK" in the software.
The XINGYING software has completed calibration. Create a 53-point human model in real-time mode, or load data with a 53-point human model in post-processing, and play it in the software.
Based on the UE version you are using, download the corresponding NokovLiveLink plugin. The installation process remains consistent. For example, if you are using the UE_4.26 engine, unzip the "NokovLiveLink1.XXX" plugin version for UE_4.26. Copy the entire "NokovLiveLink" folder and paste it into the directory of the UE_4.26 engine, located at "Epic games\UE_4.26\Engine\Plugins".
Open the UE engine and create a project. In the UE interface, select "Edit -> Plugins", then check the boxes next to "Animation - Live Link" and "Motion Capture - NokovLiveLink" (17.1.2, 17.1.3). Restart UE to apply the plugins.
Select "Window -> Live Link", click on "Source - Nokov LiveLink", make sure the ServerIP matches the network card address set in the XingYing software data broadcast panel, which is typically set to 10.1.1.198. Keep the axis consistent with XingYing by choosing "Up Axis" accordingly. Click OK. Play XingYing to receive motion capture data, and the green indicator light should illuminate (17.1.4).
Create an Animation Blueprint. In the lower content browser menu, locate the skeleton with "Skeletal Mesh" (17.1.5). Right-click on this skeletal mesh and select Create -> Animation Blueprint (17.1.6). Double-click to open the created Animation Blueprint. Right-click within the Animation Blueprint, search for "Live Link Pose," and double-click to open it (17.1.7).
In the "Live Link Pose, " select the name of the target MarkerSet in "Live Link Subject Name" to drive the motion of the model in UE4 with that MarkerSet. Connect "Live Link Pose" to "Output Pose" by dragging (17.1.8). On the right side, select Retarget, then Retarget Asset, and choose NOKOVLiveLinkRetarget Asset (17.1.9).
Click the compile button, and the model in UE will be driven, enabling synchronized motion.
Data from XingYing can be directly integrated into UE to drive the human model through a plugin. However, when there is a significant difference between the model and the body's motion, driving the model directly may lead to issues like sliding steps. In such cases, you can bind and redirect through MotionBuilder, then drive the human model in UE via MotionBuilder's UE plugin (MotionBuilder LiveLink), which can be downloaded from the UE official website or obtained by consulting NOKOV engineers. Alternatively, you can import the human model into XingYing and enable the redirection function. After the data is transformed, the output can bypass MotionBuilder and directly drive the model in UE. For specific operation steps on driving the model using XingYing's redirection function, refer to section "Fifteen, Data Redirection".
First, create a human model in Live mode in XINGYING software or load human data in Edit Mode. Use the model asset import feature to import an FBX or HTR model file and perform skeletal redirection. For detailed operation steps, please refer to the section "Fifteen, Data Redirection" above;
In XINGYING, after completing the human skeleton redirection, click the solve button. Once the skeleton solving is complete, enable "Enable Redirected Data" in the settings. Open UE, and import the model file with skeleton redirection from XINGYING into UE. Right-click in the blank area, select Blueprint Class (17.1.12), and type "Nokov" in the input box of the pop-up window. Choose "NokovLiveLinkRetargetAsset" and click select (17.1.13). Name the Blueprint Class; if not renamed, the default name will be "NewBlueprint." At this point, you will see that we have successfully created a Blueprint Class with a cube icon in the content sidebar menu.
Double-click the Blueprint Class you just created. You will see that the pop-up window has two options at the bottom: Self Adaption and Use Translation. The Use Translation option is unchecked by default, while Self Adaption is checked by default (17.1.14). You can keep the default settings. For some externally imported FBX models, you may need to check both the Self Adaption and Use Translation options, depending on the model.
In the dropdown menu on the right side of the window under Retarget — Retarget Asset, select the Blueprint Class you created (17.1.15). Click Compile. When the XINGYING software plays and the Markerset starts to move, the model in UE will be driven and synchronized accordingly.
When driving the UE model with redirected human data, ensure that the "Use Redirected Data" feature in XINGYING is enabled. If it is not, enable it, then disconnect and re-establish the LiveLink connection between UE and XINGYING. Failure to do so may cause inconsistencies between the human data and the SDK's information, leading to abnormalities in the UE model's skeleton.
Double-click the created Blueprint Class to open it (17.1.16).
The NOKOV Skeleton Mapping table displays XINGYING's human skeleton names on the left and the UE model skeleton names on the right. To edit the list, select the "Enable skeletonMapping" checkbox in the NOKOV Skeleton Mapping option. Otherwise, the list will be uneditable. Each joint skeleton name has a checkbox on its left; if checked, the joint skeleton will participate in data driving. If unchecked, it will not.
Click the Select button to the right of an individual joint (row), then enter the skeleton name in the search box to perform a fuzzy match for the target joint. For example, if you select the "Spine1" skeleton and click the Select button, you can enter "Spine" in the search box to perform a fuzzy match for the target joint (17.1.18).
Click "Select a skeleton for mapping" at the bottom of the window to display all the skeleton resources (17.1.19). Select a skeleton resource to apply it and check if its skeleton names correspond correctly with those of XINGYING's human model. If there's a mismatch, use the Select button to match the target joints via fuzzy search.
If you are using motion capture data to drive the UE model and the model's skeleton displays abnormalities or incorrect positions, this may be due to some skeleton names being inconsistent with those in the XINGYING software, leading to abnormal skeletons in UE. In this case, check the mapping of the skeleton names. Double-click on the created Blueprint Class, and check the skeleton joints on both sides in the NOKOV skeleton mapping list to ensure they correspond correctly. If there's a mismatch, make the necessary changes, save, and click Compile. Then, select the name of the modified Blueprint Class in the Retarget—Retarget Asset drop-down menu and click Compile to successfully drive the model's movement.
To see the skeleton names and hierarchy of the model file imported into UE, double-click on the model, click on the skeleton icon in the model window, and you will see a tree diagram on the right side showing the model's skeleton names (17.1.20).
When selecting a part of the model's skeleton from the skeleton name tree diagram, the model will display the specific location of that skeleton on its body. This method is used to verify that the UE skeleton names correspond correctly with the XINGYING human model skeleton names. For example, select the UE model's forearm skeleton and check in the NOKOV skeleton Mapping list if the UE forearm skeleton name corresponds correctly with the XINGYING human body forearm skeleton. If you find that the UE forearm skeleton is incorrectly mapped to the XINGYING body upper arm skeleton, copy the name of the forearm skeleton from the UE model and map it to the forearm skeleton of the XINGYING body in skeleton Mapping. Do this for each skeleton to ensure all names match correctly. After completing the checks, click "Save." You can view the XINGYING human model skeleton positions and their names by clicking the "Joints" list in the XINGYING assets panel.
If the skeletons are correctly mapped between XINGYING and the UE model but the model still shows abnormalities in UE, please contact our technical engineers for assistance.
To use the UE rigid body application, first place an object with reflective markers in the motion capture area. In Live mode, use the one-click create rigid body function to create a Markerset. After successfully creating the rigid body, open the Soft Data Broadcast panel and enable "Enable SDK."
Open the UE project file. If this is your first time creating a UE project, please contact our technical engineers to obtain the appropriate plugin version, and refer to the section above titled "Plugin Installation and UE Settings" to properly configure our plugin.
Click Window — Virtual Production — LiveLink in UE. In the LiveLink window, click to add a source and select "NOKOV LiveLink." Ensure the IP address matches that of the XINGYING software, which defaults to 10.1.1.198, and that the axis directions are consistent, which defaults to Y-up (17.1.21).
In the dropdown list for NOKOV Server IP, all the network card addresses of the local machine will be displayed. You can also click "Remote Server IP" and enter the IP address in the "NOKOV Remote IP" input field.
After completing the settings, click OK and play the XINGYING software. You will see a green light indicating that it is active and displaying the names of all connected Markersets. A character name of "Transform" means that the MarkerSet is a rigid body, while a name of "Animation" indicates that the MarkerSet is a human (17.1.22).
Next, quickly create a static mesh in UE and add it to your project by selecting a geometric shape (17.1.23).
After creation, you can see the instance you just created in the "Details" property bar on the right, and the geometry sphere has been added to the view (17.1.24). Click the add button, enter "LiveLink, " and select the "Live Link Controller" component (17.1.25).
After adding the LiveLinkComponentController component, select it. In the dropdown menu on the right side of "Subject Representation, " choose the name of the rigid body connected earlier in the LiveLink tab. Click on "Tracker0" (17.1.26) to use the rigid body named "Tracker0" to drive the geometry sphere added in UE.\
Select the root component of the default scene and change its mobility to movable (17.1.27).
Next, play the XINGYING software. As the "Tracker0" rigid body moves, the sphere in UE will move in synchronization with it.
When using the XINGYING plugin to drive a model in UE, if XINGYING loses the human body and re-recognition does not display the model in the UE scene, this issue is caused by a refresh problem in the UE engine. To resolve this refresh issue, after linking the Live Link Pose to the Output Pose to drive the model in the Animation Blueprint, you also need to add nodes in the event graph of the Animation Blueprint and connect them (17.1.28). Doing this can solve the problem of the UE engine not refreshing.
