Friday, September 16, 2005
Robotics and gaming - a new kind of "convergence"
At first glance, it seems there couldn't be more difference between the robotics and gaming worlds. Most robotics groups work with custom hardware, heavily based on slow microprocessors, while game developers work with the fastest consumer PCs. Games create an artificial reality with no connection to the world except a mouse or joystick, while Robots that Jump have large amounts of sensors. Game developers experiment with "classic" artificial intelligence for their virtual characters, while robotics increasingly has abandoned AI for behavior-based computing. Games are huge consumer products while robots have yet to be fully accepted commercially. The game industry is showing signs of maturity (nostalgia for the 8-bit world) while the robotics industry has barely gotten off the ground.
In spite of this, the gaming world has something to offere robotics developers. I say the benefit comes from the gaming world because at present robotics has little to offer games beyond a "reality check" for the impossible capabilities of the killer robots which haunt their virtual landscapes. But robotics developers looking for a way to advance their art should consider the following things about games:
1. Games are fast - even running on a multitasking PC, games manage to display a real-time, 3D world in full color.
2. Games are fast because they can exploit the specialized parallel processing being built into the GPUs (Graphics Processing Units) of today's high-performance video cards. These cards don't only draw - they can do translations, rotations, resizing of extremely complex models hundreds or thousands of times faster than is possible in software.
3. Many of the hardware-based computations used in games are readily adapted for robots. If your robot attempts to build a map of its environment, you could use the rapid 3D "world building" capabilities of game-related SDKs like Microsoft's Direct3D to accomplish it in rapid time. One could build a model of their robot plus its environment, transform its location in a virtual world as the robot moves, and use standard game "collision detection" methods to predict when you get too close to an object. This would be ideal for a robot using a large number of sensors to define its environment.
4. The gaming industry has a tradition of releasing support for "video mods" - software hooks that allow the gaming community to start with the generic "gaming engine" (a treasure trove of software for moving in 3D space) and modify it using external code in C++ or other languages. In fact, many games are based off the same engine, e.g. Unreal or Source. Thriving communities exist of people who've completely transformed a game engine like Source into a different game from the "Half Life" series it was developed for.
5. Finally, the "first person shooter" aspect of many games is suited to a robotic design placing the robot as the central agent moving in its environment. Substitute your robot's wishes for a joystick, and off you go...
With this in mind, I've been exploring game development, particularly Direct3D, as a vehicle for robotics. The Direct3D SDK is extremely complex, but there are lots of worked examples out there - the advantage of "hijacking" the technology from a highly successful industry. If you've got a high-performance graphics card (e.g. geforce) the speed of vector calculations, point movement/translation, and terrain modeling (using the gaming mesh paradigm) is breathtaking.
Even using the object-oriented version of visual basic, I've found that I can feed in data from multiple microprocessors (each in turn with several sensors) and display the resulting sensory grid in real time on a standard PC. Distance calculations are also incredibly fast, applying collision detection and ray-tracing techniques used in games. It also seems likely that the Direct3D anti-aliasing methods (used to process edge information for smoothness) and "pixel shaders" could be used in reverse - to process computer vision.
In fact, it isn't that hard in DirectX to grab video at a high frame rate and do quick sensory fusion with data from 3D "point" sensors like infra-red and ultrasound. In effect, one uses the point sensor data to make a 3D mesh surrounding the vehicle. The video data is then rapidly "wrapped" over this 3D wireframe. This converts traditional edge detection from 2D to 3D space - one looks for "bulges" and then checks the "bulges" for colored "blobs" and shapes. Since most robots don't need to know about distant objects, it provides a spectacular way to do sensory fusion on the cheap.
Now, the question is whether one can use an actual gaming engine for the "mind" of the robot. Depending on the engine's design, it is easy or hard to drop in new "terrains". Ideally, one would like to load an existing 3D topographic map of the physical world and "drape" what the robot's sensors see over it. However, in the interests of speeds, large terrain models are typically loaded at game startup into the video card memory. Objects detected by sensors might have to be modeled using non-terrain methods - as weapons lying around, for instance.
A related unanswered question is whether the game engine AIs are useful. In many engines, the game AIs hang around even when the user is not looking at them and continue to execute behavior. It is possible that using this method anticipated behaviors could be assigned to objected detected by sensors and brought into the "game world". The engine would then check for deviation of the real object data overlaid on the virtual object (e.g. rocks should not move).
It may even be that the game AIs are smart enough to drive the robot. Advanced game engines do have path-planning systems. Unfortunately, games also make extensive use of "teleporting" instead of actually moving characters through the 3D world, so this may not be as useful as it seems.
I encourage roboticists to explore the world of game development hardware, software, modding and algorithms. In the worst case, you've got yourself a new marketable skill!
At first glance, it seems there couldn't be more difference between the robotics and gaming worlds. Most robotics groups work with custom hardware, heavily based on slow microprocessors, while game developers work with the fastest consumer PCs. Games create an artificial reality with no connection to the world except a mouse or joystick, while Robots that Jump have large amounts of sensors. Game developers experiment with "classic" artificial intelligence for their virtual characters, while robotics increasingly has abandoned AI for behavior-based computing. Games are huge consumer products while robots have yet to be fully accepted commercially. The game industry is showing signs of maturity (nostalgia for the 8-bit world) while the robotics industry has barely gotten off the ground.
In spite of this, the gaming world has something to offere robotics developers. I say the benefit comes from the gaming world because at present robotics has little to offer games beyond a "reality check" for the impossible capabilities of the killer robots which haunt their virtual landscapes. But robotics developers looking for a way to advance their art should consider the following things about games:
1. Games are fast - even running on a multitasking PC, games manage to display a real-time, 3D world in full color.
2. Games are fast because they can exploit the specialized parallel processing being built into the GPUs (Graphics Processing Units) of today's high-performance video cards. These cards don't only draw - they can do translations, rotations, resizing of extremely complex models hundreds or thousands of times faster than is possible in software.
3. Many of the hardware-based computations used in games are readily adapted for robots. If your robot attempts to build a map of its environment, you could use the rapid 3D "world building" capabilities of game-related SDKs like Microsoft's Direct3D to accomplish it in rapid time. One could build a model of their robot plus its environment, transform its location in a virtual world as the robot moves, and use standard game "collision detection" methods to predict when you get too close to an object. This would be ideal for a robot using a large number of sensors to define its environment.
4. The gaming industry has a tradition of releasing support for "video mods" - software hooks that allow the gaming community to start with the generic "gaming engine" (a treasure trove of software for moving in 3D space) and modify it using external code in C++ or other languages. In fact, many games are based off the same engine, e.g. Unreal or Source. Thriving communities exist of people who've completely transformed a game engine like Source into a different game from the "Half Life" series it was developed for.
5. Finally, the "first person shooter" aspect of many games is suited to a robotic design placing the robot as the central agent moving in its environment. Substitute your robot's wishes for a joystick, and off you go...
With this in mind, I've been exploring game development, particularly Direct3D, as a vehicle for robotics. The Direct3D SDK is extremely complex, but there are lots of worked examples out there - the advantage of "hijacking" the technology from a highly successful industry. If you've got a high-performance graphics card (e.g. geforce) the speed of vector calculations, point movement/translation, and terrain modeling (using the gaming mesh paradigm) is breathtaking.
Even using the object-oriented version of visual basic, I've found that I can feed in data from multiple microprocessors (each in turn with several sensors) and display the resulting sensory grid in real time on a standard PC. Distance calculations are also incredibly fast, applying collision detection and ray-tracing techniques used in games. It also seems likely that the Direct3D anti-aliasing methods (used to process edge information for smoothness) and "pixel shaders" could be used in reverse - to process computer vision.
In fact, it isn't that hard in DirectX to grab video at a high frame rate and do quick sensory fusion with data from 3D "point" sensors like infra-red and ultrasound. In effect, one uses the point sensor data to make a 3D mesh surrounding the vehicle. The video data is then rapidly "wrapped" over this 3D wireframe. This converts traditional edge detection from 2D to 3D space - one looks for "bulges" and then checks the "bulges" for colored "blobs" and shapes. Since most robots don't need to know about distant objects, it provides a spectacular way to do sensory fusion on the cheap.
Now, the question is whether one can use an actual gaming engine for the "mind" of the robot. Depending on the engine's design, it is easy or hard to drop in new "terrains". Ideally, one would like to load an existing 3D topographic map of the physical world and "drape" what the robot's sensors see over it. However, in the interests of speeds, large terrain models are typically loaded at game startup into the video card memory. Objects detected by sensors might have to be modeled using non-terrain methods - as weapons lying around, for instance.
A related unanswered question is whether the game engine AIs are useful. In many engines, the game AIs hang around even when the user is not looking at them and continue to execute behavior. It is possible that using this method anticipated behaviors could be assigned to objected detected by sensors and brought into the "game world". The engine would then check for deviation of the real object data overlaid on the virtual object (e.g. rocks should not move).
It may even be that the game AIs are smart enough to drive the robot. Advanced game engines do have path-planning systems. Unfortunately, games also make extensive use of "teleporting" instead of actually moving characters through the 3D world, so this may not be as useful as it seems.
I encourage roboticists to explore the world of game development hardware, software, modding and algorithms. In the worst case, you've got yourself a new marketable skill!
