SimAiHub Humanoid Robot G1+Dex5 Single and Bimanual Grasping Dataset

Data Product Features

• Hardware Configuration: Unitree G1 humanoid robot + Dex5 three-fingered dexterous hands. Single-hand tasks use one dexterous hand; bimanual tasks use two dexterous hands cooperatively.
• Grasp Types: Each data episode covers one grasp type:
  • Pinch-Side: Two vertical sides of the object are clamped by three fingers with a top-down enclosing grasp.
  • Power-Envelop: The dexterous hand wraps around the mid-section of the object. Grasp position is 30–40 mm below the lower edge of the object opening; grasping on object edges is prohibited.
  • Hook-Grasp: Fingertips insert into the inner cavity of the object and hook against the inner surface, with the thumb providing outer-side constraint.
  • Bimanual-Envelop Grasp: Both hands wrap around and envelop the object from opposite sides, with palm and finger surfaces conforming cooperatively to form a stable enveloping grasp. Suitable for stable transport and placement of medium-to-large objects.
  • Bimanual-Oppose Pinch: Both hands press inward from opposite directions using fingertips or finger pads to form an opposed pinch. Clamping force is controlled. Suitable for stable grasping and placement of smaller objects.
  • Bimanual-Cooperative Manipulation: Both hands take different roles during the task, e.g., one hand stabilizes or fixes the object while the other adjusts, rotates, aligns, or assembles it.
• Recording: Primarily records wrist poses and dexterous-hand states, including wrist Cartesian target poses, wrist poses, dexterous finger joint commands/states, etc.
• Rewards: Decomposed for offline reinforcement learning: terms for approach, lifting, target tracking, task success, motion smoothness, object drop, and other reward components.
• Labels: 12-stage expert finite-state-machine tags (Perception, Base Alignment, Pre-Grasp, Approach, Closure, Locking, Lifting, Transport, Placement Alignment, Lowering, Release, Retraction), imitation-learning segmentation markers, and success / failure / truncation status flags.
• Domain Randomization: Object XY-plane jitter; 10–20% of episodes are initialized with small tilt angles. Friction coefficients are randomized independently per grasp type. Data array structures are identical across successful and failed sample sets.
• Sensor Noise Randomization: Baseline configuration adopts reference sensor noise values for G1 and Inspire dexterous hand from official Isaac Lab examples.
• Synchronization and Storage: Each data episode contains clock-aligned observations, rewards, and termination flags synchronized via env.step() in Isaac Lab. Standard HDF5 storage frame rate is 50 Hz.
• Data Source: Each episode data is collected from real-world simulation runs, not generated from existing data.
• Sampling and Timing: HDF5: 50 Hz; MP4 encoding: 10 fps.

Distribution

• Format: HDF5 trajectories under /data/demo_*; each episode is also saved as MP4 video. Shared Lab primitives only: RecorderManager + HDF5 /data/demo_*.
• Structure: Each episode contains clock-aligned observations, rewards, and termination flags. Each episode covers only one grasp type. Successful and failed sample sets have identical array structures.
• Data Volume: Includes normal operation, failure, and recovery data.
• Video: Each episode is saved as MP4. With vision_hz = 10.0 and vision_stride = 5, one frame is saved every 5 env steps. Encode at 10 fps for real-time simulation playback. If one frame per env step is saved, encode at 50 fps.

Usage

• Application: Training robot grasping and manipulation policies.
• Application: Sim-to-real transfer research.
• Application: Offline reinforcement learning and imitation learning.
• Application: Research on single-hand and bimanual cooperative manipulation.
• Application: Benchmarking robot manipulation and grasping tasks.

Coverage

• Geographic Coverage: Globally usable.
• Time Range: The original specification does not provide a specific start date or end date.
• Demographics (if applicable): Not applicable.

Custom Development

License

AI Training Rights

• Use the Data Product to train, fine-tune, and evaluate machine learning models, including large language models.
• Incorporate Data Product content into models and commercialize resulting model outputs.
• Create derivative works (model weights, embeddings, etc.) for any lawful purpose.

• The Data Product itself may not be sold, redistributed, or shared outside of licensed usage.
• Licensee must comply with all applicable laws, including data protection and privacy regulations.

Who Can Use It

• Data Scientists: For training machine learning models.
• Researchers: For academic or scientific studies.
• Businesses: For analysis, insights, or AI development.
• Robotics Engineers: For policy development, sim-to-real transfer, and manipulation benchmarking.

Data Dictionary

| Column Name / Key | Data Type | Description | Possible Values/Notes |
|-------------|-----------|-------------|-----------------------|
| episode_id | string/integer | Episode identifier | Format not specified in the original description |
| observations.wrist_pose | float array | Wrist pose | Single/dual wrist |
| observations.wrist_cartesian_target_pose | float array | Wrist Cartesian target pose | Expert-generated |
| observations.dex_hand_joint_commands | float array | Dexterous finger joint commands | Single/dual hand |
| observations.dex_hand_joint_states | float array | Dexterous finger joint states | Single/dual hand |
| observations.dex_hand_contact_forces | float array | Dexterous-hand contact force data | Single/dual hand |
| observations.vision_step_index | integer array | Proprio step index corresponding to each stored image frame | Used for vision alignment |
| rewards.* | float | Reward components | Approach, lifting, target tracking, task success, motion smoothness, object drop, etc. |
| labels.fsm_stage | categorical/integer | 12-stage expert finite-state-machine tags | Perception, Base Alignment, Pre-Grasp, Approach, Closure, Locking, Lifting, Transport, Placement Alignment, Lowering, Release, Retraction |
| labels.imitation_segmentation | categorical/marker | Imitation-learning segmentation markers | - |
| labels.status_flags | categorical | Success/failure/truncation status flags | success/failure/truncation |
| domain_randomization.* | float/config | Domain randomization settings | XY jitter, tilt angles, friction coefficients |
| sensor_noise.* | config | Sensor noise randomization | Baseline uses official Isaac Lab G1 and Inspire dexterous-hand reference noise values |
| timing.physics_dt | float | Physics time step | 0.005 s |
| timing.decimation | integer | Physics substeps per env step | 4 |
| timing.physics_hz | integer | Physics simulation frequency | 200 Hz |
| timing.env_step_hz | integer | Environment step frequency | 50 Hz |
| timing.hdf5_sample_hz | integer | HDF5 storage sample rate | 50 Hz |
| timing.vision_hz | float | Vision capture rate in simulation time | 10.0 Hz |
| timing.vision_stride | integer | Number of env steps between saved image frames | 5 |
| timing.vision_align | string | Vision-to-proprio alignment rule | obs/rgb_*[i] matches proprio step obs/vision_step_index[i]; train: rgb[t // vision_stride] |
| video_encoding_fps | integer | Recommended MP4 encoding frame rate | 10 fps |