CVPR 2024 Embodied AI Workshop PRS Challenge: Human-centered In-building Embodied Delivery
by Apple & Peking University & PRGS
What is Polar Research General Services Organization?
Polar Research General Services (PRGS) is a start-up organization that provides academic services and technical support to individuals and groups. Its service scope includes, but is not limited to, the construction of embodied intelligence simulation environments, human-robot interaction scene simulations, business scenario solutions (feasibility assessments, dynamic management, data collection and generation, decision support, and equipment prototype manufacturing), metaverse gaming solutions, and virtual solutions for the film industry.
What is Polar Research Station Platform?
The Polar Research Station (PRS) Platform is simulator project provided by PRGS Organization.
We aim to address the critical challenge of gradually integrating embodied agents into human society.
To further explore the scenarios of human-agent interaction, PRS Platform simulates a small-scale human community (ecosystem)
for the gradual trial of comprehensive issues related to the integration of embodied robots into human society.
PRS Platform boasts the following features:
- A 3D interactive continuous environment supported by the Nvidia PhysX physics engine.
- The three-story building containing activity rooms, meeting rooms, kitchens, bedrooms, laboratories, greenhouses, warehouses, supermarkets, gyms, and other communal scenes.
- Human characters (created using Generative Agent Tech) with their own meaningful and human-like needs (basic physiological, work, entertainment, social, etc.). When a character is hungry, it must seek food; otherwise, hunger can result in low blood sugar and a range of negative consequences.
- Characters autonomously move through the building based on their needs, preferences, habits, states, and social relationships. We have developed an LLM based system to generate reasonable behavior for these characters, ensuring that their behavior resembles complex and unpredictable behaviors in the real world, rather than simply fixed patterns.
- Characters have their own emotional and facial expression system, which is influenced by events and the characters' own traits.
- Characters accomplish their goals by interacting with massive objects and diverse specific devices within the environment.
- The community is designed to operate in a long-term and stable operation, facilitating simulation and calculation with the ability to start, accelerate time, and even pause in any given situation.
- In the simulator, robots can gather perceptual signals in the scene using visual, tactile, auditory, olfactory, temperature, and humidity sensors. They interact with a majority of objects in the environment through actions such as movement, grasping, and manipulation.
- To accommodate users with varying technical backgrounds, we offer a range of APIs, for motor control at the basic level, from verb-based semantic APIs to high-level phrase APIs for robot control. Furthermore, we have opened up numerous APIs for controlling human characters, items, environments, and devices within the scene—allowing for comprehensive control over nearly everything.
- We have implemented a lightweight framework that makes our environment user-friendly, easily extendable, modifiable, and developable using only Python!
Rooms examples in the PRS Platform follow real-world building.
Characters live in the building and perform social behavior.
The embodied robot can move anywhere and manipulate devices with its arm.
Diverse items and furniture in the PRS environment.
To Everyone:
Before the full version is released (2024.5.1), there will be a CVPR EAI workshop challenge version available for PRS Challenge and demo .
We have a daily maintenance community, and building a simulated human social ecosystem for generalist robot development is a highly challenging and interesting task.
We are eager to hear everyone's feedback and opinions. Especially considering our team's current limited resources, we are keen to communicate, work, and collaborate with volunteers who share the same interests.
Additionally, we warmly welcome professional research teams to explore the potential of the PRS platform together with us.
Our Purpose
We have developed a universal Large Language Model (LLM) platform and testing benchmark on the PRS subject of AI integration into human society, focusing on discussions surrounding AI agents, Embodied AI research, and the role of large language models. To create a realistic simulation environment, we have replicated a polar scientific research station with a three-story building to scale. Within this simulated environment, we meticulously furnished the building with various items, equipment, and furniture, all supported by a PhysX physical engine that enables physical interactions and dynamic state changes. Implementing autonomous non-player characters (NPCs) within the simulation is central to our project. These NPCs are designed to meet individual survival needs such as healthy eating, drinking, and sleeping, as well as social needs, work goals, and emotional responses. This setup allows us to explore the complexities of integrating AI systems into societal frameworks and observing their interactions in a controlled yet dynamic environment.
Within the PRS community, human NPCs and other NPCs fulfill their work tasks and social needs. These NPCs are equipped with emotions, undergoing state changes and emotional fluctuations (such as joy, anger, sorrow, and happiness) as activities progress. This system mirrors actual human behavior, providing a platform to examine the integration of large language models (operating at a hierarchical API semantic level for human-robot interaction systems) into human society. Through this environment, we aim to explore the capabilities of LLMs in understanding human behavior, solving complex problems through summarization, and adapting to evolving environments by observing surroundings. Additionally, we seek to investigate how embodied robots (functioning at a skill-learning semantic level API) can offer comprehensive services to humans, particularly considering the dynamic nature of human interactions. Discussions will also revolve around practical universal robot forms for diverse applications.
PRS Challenge
More details in challenge.
Author
Academic Advisors of Challenge
There is a demo .
The complete API documentation and Human-centered Delivery Benchmark .
Various projects, such as social expressions, more interactive devices, and more environmental features (e.g., mirrors), are to be announced.
(1) Projector simulation. (2) Interactive devices for various purposes. (3) Monitoring and display screen simulations. (4) Mirrors that can reflect visual content.
NPC's social emotional system