push the yellow hexagon to the top right corner
Interactive Language: Talking to Robots in Real Time
- Corey Lynch
- Ayzaan Wahid
- Jonathan Tompson
- Tianli Ding
- James Betker
- Robert Baruch
- Travis Armstrong
- Pete Florence
Abstract
We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuolinguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. "make a smiley face out of blocks". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots.
Video Summary (with narrated audio, 3 minutes)
Examples: Diverse, Open-Vocabulary Skills
We train an open-vocabulary, language-conditioned visuomotor policy on over 87,000 unique short-horizon strings, and estimate a 93.5% success rate. These videos are at 1x speed:
(1 training example)
push the green circle and
blue triangle towards
the group of blocks
(0 training examples)
move the green star
between the yellow blocks
(2 training examples)
move blue triangle to the
right of the greencircle
Examples: Interactive Language Guidance
A human can iteratively guide the robot to accomplish complex long horizon goals. These videos are at 4x speed:
Goal: sort the blocks by colors into corners
move the yellow heart to the yellow hexagon
push the red circle to the bottom right corner
move the red star to the red circle
push the green star to the bottom left corner
move the green circle to the bottom left corner
nudge the green star down and left a bit
nudge the green circle closer to the green star
move the blue cube to the top left corner
push the blue triangle slowly to the blue cube
done
move the green star to the bottom right corner
push the green circle to the green star
move the red circle to the bottom left corner
push the red star to the red circle
move the blue triangle to the top left corner
push the blue cube to the blue triangle
move the yellow hexagon to the top right
move the heart to the hexagon
nudge the green star closer to the green circle
nudge the green blocks to the bottom right corner
move the green star right
done
move the red circle to the bottom left corner
move the red star to the red circle
nudge the red star closer to the red circle
push the green circle to the top left corner
move the green star to the green circle
push the yellow heart to the top right corner
move the yellow hexagon to the yellow heart
push the blue cube to the bottom right corner
move the blue triangle to the blue cube
touch the right side of the yellow hexagon
push the blue triangle closer to the blue cube
done
Goal: arrange the blocks into a vertical line in the center of the table.
push the red star to the bottom center
move the blue triangle to the top side of the red star
nudge the blue triangle left a bit
nudge the yellow hexagon down a bit
push the green circle to the top side of the yellow hexagon
nudge the green circle down a bit
nudge the green circle right a bit
put the red circle to the top side of the green circle
nudge the red circle towards the top
nudge the green circle right a bit
nudge the red circle down a bit
push the yellow heart to the top side of the red circle
move the yellow heart to the top side of the red circle
nudge the yellow heart down a bit
push the green star to the top side of the yellow heart
nudge the green star left a bit
nudge the green star up a bit
move the green star up and left
touch the left side of the green star
nudge the green star right a bit
push the blue cube to the top side of the green star
nudge the blue cube down a bit
nudge the blue cube to the right
nudge the blue cube right a bit
nudge the blue cube down and right
nudge the blue cube down
nudge the blue cube right
slide the blue cube right
nudge the blue cube towards the right side
touch the left side of the blue cube
nudge the blue cube right
nudge the blue cube up
done
move the red circle to the bottom center
move the red circle right a bit
nudge the red circle down
push the red circle to the bottom side of the board
push the red circle to the bottom center
push the green circle to the top side of the red circle
nudge the green circle left
nudge the green circle right
push the green circle right a bit
move the blue triangle to the top side of the green circle
nudge the blue triangle down a bit
nudge the green circle left a bit
move the red circle closer to the bottom side of the green circle
touch the blue cube
nudge the blue triangle left
push the blue cube to the top side of the blue triangle
nudge the blue cube left
touch the top side of the red star
push the yellow heart to the top side of the red star
nudge the yellow heart left
touch the left side of the yellow heart
nudge the yellow heart right a bit
put the yellow hexagon to the top side of the yellow heart
nudge the yellow hexagon right a bit
done
push the green star to the bottom center
move the green circle to the top side of the green star
move the red circle to the top side of the green circle
nudge the red circle right a bit
move the red star to the top side of the red circle
move the yellow heart up and left away from the red circle
push the yellow heart up
move the yellow heart up
nudge the red star right a bit
move the yellow heart to the top side of the red star
push the yellow hexagon to the top side of the yellow heart
nudge the yellow hexagon closer to the yellow heart
nudge the yellow hexagon left a bit
nudge the yellow hexagon to the left
move the blue cube to the top side of the yellow hexagon
nudge the blue cube left a bit
move the blue triangle to the top side of the blue cube
done
Goal: make a smiley face with green eyes.
move the red circle to the bottom center
push the green star to the center
move the heart to the right side of the red star
move the heart to the right side of the red circle
push the blue cube to the right side of the heart
move the hexagon between the triangle and red circle
move the green star to the center
push the red star to the right side of the hexagon
move the yellow hexagon left a bit
nudge the hexagon left
move the red circle down just a bit
push the yellow hexagon between the blue triangle and red circle
nudge the hexagon down a bit
move the hexagon towards the bottom
touch the green star
move the yellow hexagon closer to the blue triangle
move the red star closer to the hexagon
nudge the red star down
put the yellow hexagon between the triangle and red star
nudge the hexagon left a bit
move the blue cube closer to the heart
push the hexagon to the top right side of the blue cube
nudge the hexagon to the right
move the red star to the top left side of the triangle
move the red star up a bit
nudge the hexagon up a bit
nudge the blue cube up a bit
nudge the red star up a bit
point at the green star
point at the green circle
nudge the blue triangle up a bit
nudge the green star right a bit
move the green star up a bit
nudge the green star to the right
touch the green circle
move the green star to the center
move the green star to the right side of the board
nudge the green circle right
nudge the green star left
nudge the green circle left a bit
move the green circle to the left side
done
push the red star to the bottom center
move the yellow hexagon to the right side of the red star
push the yellow heart between the red and green stars
move the red circle to the right of the yellow hexagon
push the green star to the left of the green circle
move the blue triangle to the left of the yellow heart
push the green star to the top center
push the green circle to the top center
move the green star left a bit
push the red star between the yellow heart and yellow hexagon
move the blue cube between the red star and yellow heart
push the green circle to the right of the green star
move the green blocks to the top center
nudge the green circle left a bit
nudge the blue cube down and left a bit
nudge the blue cube down a bit
move the yellow heart between the triangle and cube
nudge the blue triangle up a bit
move the red star closer to the blue cube
move the red circle to the right side of the yellow hexagon
nuge the hexagon down and right a bit
nudge the yellow hexagon down and right a bit
slightly nudge the hexagon right
push the hexagon slightly to the right
touch the green circle
nudge the hexagon between the red star and red circle
put the yellow hexagon to the bottom side of the red circle
touch the green circle
touch the red circle
nudge the red star up and left a bit
push the yellow hexagon between the red blocks
nudge the yellow hexagon between the blue cube and red star
nudge the hexagon left and down a bit
nudge the red star up and right a bit
push the yellow hexagon closer to the yellow heart
move the blue cube to the right side of the hexagon
push the blue cube to the yellow hexagon
nudge the blue cube up
put the red circle on the right side of the blue cube
nudge the red circle down a bit
put the red star on the top right side of the red circle
nudge the green star up and right a bit
nudge the green star down
nudge the green star down a bit
nudge the green star right a bit
nudge the green star down a bit
done
move the heart to the right side of the red circle
separate the green star from the heart
move the heart to the right side of the red circle
nudge the heart to the right
slide the heart right
slide the blue cube to the left side of the red circle
slide the blue cube left
move the green star to the top center
push the red star to the right side of the yellow heart
move the blue cube to the left side of the red circle
slide the red star up
nudge the red star up and right
push the blue cube to the left side of the red circle
bring the blue triangle between the blue cube and red circle
nudge the blue triangle right
slide the blue triangle right
separate the blue triangle from
nudge the red circle closer to the blue triangle
touch the right side of the red star
move the green star to the top left corner
touch the right side of the green circle
push the green star towards the top left
push the green star towards the top left
nudge the blue triangle left
touch the right side of the green circle
nudge the blue triangle down
slide the blue triangle down
move the blue triangle towards the bottom
touch the right side of the green circle
nudge the red star up a bit
nudge the red star towards the top right
nudge the red star up and right a bit
nudge the red star up and left a bit
nudge the red star left
nudge the green circle up slightly
done
Additional Interactive Examples
2x speed
4x speed
Interactive Language: Method
In short, our system uses a mix of: high-throughput data collection, a new event-selectable variation of hindsight relabeling, language-conditioned behavioral cloning, and a large Transformer-based policy.
See the paper for more details and analysis.
Event-selectable Hindsight
LAVA Transformer
Interactive Guidance
Language-Table: Dataset and Benchmark
We will be releasing the Language-Table dataset and associated simulated benchmark. This is the largest dataset of its kind by over an order of magnitude.
Dataset release is expected soon, we are working on it.
Language-Table
