Projects in VR
Editor: Lawrence Rosenblum and Simon Julier
Dynamic Terrain for Multiuser Real-Time Environments Christopher Ellis, Pavel Babenko, Brian Goldiez, Jason Daly, and Glenn A. Martin, University of Central Florida
D
espite the pursuit for increased realism in VR training simulations and massively multiplayer online games, we find little deployed capability to modify terrain. Usually, terrain is stored in a static database and then preprocessed to support multiple levels of detail and to rectify features with the ground. Also, dynamic-terrain research has concentrated on graphics simulations for single users; we’ve found no research on systems that can support 500 or more user systems. Many obstacles to implementing large-scale dynamic-terrain simulations are hardware based— such simulations need large amounts of memory and computing power. However, recent advances in computer hardware can help ameliorate these problems. In particular, high-performance computing clusters (HPCCs) offer an expandable set of computing elements, large amounts of memory and storage, and high-speed connections that remove many of the hardware limitations. Additionally, the industry’s movement toward multi- or many-core machines indicates that research done today on high-performance computers (HPCs) could find its way into future commodity systems. A single terrain database represents a significant bottleneck when it supports many users. This situation has led us to develop a distributed terrain database architecture. In such an architecture, many details must be addressed. We’re addressing some of the more significant issues, such as how to avoid conflict when multiple terrain modifications occur simultaneously in the same area, how to detect and recover from faults, and how and when to distribute load among servers. Our goals in prototyping dynamic terrain are to ■
■ ■ ■
80
support many users interacting in the same virtual environment, resolve conflicting terrain queries, support vast regions of modifiable terrain, and deliver simulation imagery to a variety of client devices.
January/February 2010
Existing dynamic-terrain systems tend to focus on how best to represent the terrain data. Our research emphasizes creating modular, abstract terrain representations so that heterogeneous clients can interact with the simulation. We also heavily consider fault tolerance to give simulations resilience against hardware failures, timing inconsistencies between machines and processes, or software faults. In previous iterations, the dynamic-terrain systems were designed to work well, despite hardware limitations. An explicit assumption is that intercommunication between nodes or processors is both slow and unreliable. The next big question then is, how do you distribute the dynamic-terrain data? Here, we describe how we solve this problem for a VR training system by exploiting an HPCC environment.
Gemini—the Dynamic-Terrain Engine As the baseline system for our research, we chose Gemini1 (see Figure 1). Its modular interface lets it use different database formats and simulation protocols as well as different communication schemes. Gemini allows communication between multiple servers and multiple clients. A server processes messages from clients at a fi xed frame rate to prevent issues when servers become desynchronized. Gemini translates existing simulator protocols into its own internal protocol, which facilitates the integration of heterogeneous servers and clients. Terrain is represented on disk as a 2D matrix of values for each terrain attribute. Each attribute represents one terrain property, such as elevation, density, cohesion, or temperature. This system is extensible; to add new terrain properties, you merely add a new attribute layer to the existing terrain representation. Gemini represents the terrain data as a nonparametric surface—specifically, a uniform nonrational bicubic B-spline. To implement our design, we created dynamicterrain resource (DTR) clients (see Figure 2). These clients listen for a specific type of terrain modification event, query the terrain database, compute
Published by the IEEE Computer Society
0272-1716/10/$26.00 © 2010 IEEE
The Partitioned-Data Model One approach divides the database into equal-sized terrain patches and allocates each patch to a separate database on a different compute node. Such partitioning is naïve, however, and can result in a drastic load imbalance. Some regions of the simulation might be heavily accessed, whereas others could remain unused. This isn’t an unreasonable assumption owing to the spatial and temporal locality of entities and events in a simulation. So, we must be able to change the size and assignment of terrain patches to compute nodes dynamically during the simulation. One method to do this is by implementing a partitioned-data model. This allocation should be based on load, which we can measure as the number and size of the queries a particular compute node is processing in a given time period. Every time we reallocate a patch of terrain to a different compute node, we must send the entire patch from the old node to the new one. However, owing to the compact representation of terrain in Gemini and the high-speed InfiniBand connection, this operation should minimally affect real-time performance. This model has some drawbacks, however. For example, whenever the client wants to perform a read or write on terrain data, its request must be routed to the node containing that data. Additionally, if a client request affects data on multiple
nodes, it must be distributed to those nodes, and the resulting responses must be combined before being sent back to the client. This model also requires some metric to determine when to reallocate terrain. If we reallocate too frequently, internode communication dominates the available bandwidth, slowing the rate at Compute node Client processes Track DTR
Terrain database
Crater DTR User client
Other processes
Compute node
Databases
Entity database
Other database
Shared memory
Gemini server
InfiniBand (TCP or remote direct memory access)
We can distribute the database in one of two ways.2 We can partition it into mutually exclusive pieces (called patches) and distribute them across multiple compute nodes, or we can fully replicate the entire database on each compute node.
Figure 1. An artistic example of dynamic terrain. This simulation uses the Gemini dynamic-terrain engine (DTR). A dynamically generated crater is visible in the foreground
…
Distributing the Database
DTR-generated crater
…
the necessary changes, and submit the updated terrain back to the database. Examples of DTR clients include the track client, which leaves tracks in the ground behind vehicles, and the crater client, which generates craters in the soil as the result of mines or other explosives. We’ve implemented these two clients on an HPCC at the University of Central Florida (for more on HPCCs, see the sidebar). Each compute node runs duplicate sets of DTR clients to handle terrain events local to it. With the computationally intensive pieces of terrain simulation offloaded to the DTR clients, the terrain database has the sole responsibility of responding to queries and updates the user clients send. User client processes handle keyboard or joystick input and 3D graphics output. Remote users connect to these processes via a secure Internet connection (see Figure 3).
Gemini server
Clients Data
Compute node Gemini server
Clients Data
Compute node Gemini server
Clients Data
Compute node Gemini server
Clients Data
Figure 2. A compute node’s structure. The Gemini server acts as an interface between client processes and data and sends and receives messages from servers on other compute nodes via dynamic-terrain resources (DTRs). IEEE Computer Graphics and Applications
81
Projects in VR
High-Performance Computing Clusters
H
igh-performance computing clusters (HPCCs) are capable of computational power far exceeding that of desktop or workstation PCs. HPCC compute nodes contain many fast multicore processors and large quantities of memory and are considered computational workhorses. All nodes are connected to each other via a low-latency, high-bandwidth switch and communication link, such as InfiniBand or Gigabit Ethernet. HPCCs can compute trillions of floating-point operations per second (teraflops). InfiniBand can transmit billions of bits per second (gigabits/sec.). Such powerful computation and communication brings the goal of large, many-client, dynamic-terrain-enabled VR systems within our reach. The question is how best to harness this power.
HPCC
Internet Remote client
Compute node Compute node Compute node
TCP/IP interconnect
InfiniBand interconnect
Compute node
Remote client
Compute node
Remote client Remote client
Remote client Remote client
Remote client Remote client Remote client
Remote client
Figure 3. Remote clients connecting to the high-performance computing cluster (HPCC). Each compute note can handle multiple remote clients. So, a load-balancing scheme is useful to distribute the clients among the nodes.
which client requests can be serviced. If we don’t reallocate frequently enough, clients spend too much time on congested nodes, and other nodes remain idle. Any allocation scheme must consider when the reallocation overhead outweighs the load-balancing benefits.
The Replicated-Data Model Rather than partitioning the terrain, it can be replicated across each compute node. This approach lets clients connect directly to any compute node, thus supporting a fine-grained load-balancing mechanism without terrain partitioning (see Figure 4). However, duplicating the terrain data in this fashion requires a mechanism to prevent concurrent terrain updates from creating database inconsistencies. A naïve approach would be to 82
January/February 2010
broadcast all terrain updates to all compute nodes, but this would defeat the purpose of distributing the terrain databases because all clients are sending their update requests to all compute nodes. Instead, we use a lazy synchronization scheme that synchronizes terrain data between nodes only when necessary. For a distributed dynamicterrain database, this means that when a client on one node requests data that’s out of date owing to an update from another node, the outdated node must copy the data from the more recent node before servicing the client’s request. If clients from one node modify a region, other nodes needn’t know all the intermediate updates to that region. They only need to know the region’s final state if their own clients move into that modified region. If clients are clustered by location in different nodes, this can drastically reduce the synchronization required. Additionally, nodes can actively synchronize data during times of low load in anticipation of future requests. Because each computing node contains the entire terrain database and handles its own dynamicterrain functionality, we allocate users on the basis of their location on the terrain. Our approach assesses each compute node so that, if the load is too high, the clients are reallocated to produce a more equal distribution (for example, using mean-shift clustering to group the clients by spatial locality). To support this scheme, nodes must know whether their data copies are invalid and which node has the latest version. In the event of a terrain update, a lightweight message is broadcast to all nodes notifying them of the update. This message contains the node where the update occurred, the extent of the updated data, the attribute that was modified, and a time stamp. When a node receives this message, the node stores this data in a synchronization table. This table tracks whether a region has been modified and, if so, which node has the most recent data and how recent it is. If a node receives a request for modified terrain, that node queries the node with the most recent copy of the data and updates its own before servicing the request. The update notifications in the synchronization table can be safely overwritten because any node must synchronize before it modifies a region, preventing updates to inconsistent terrain data. Synchronization can also occur when the node is idle.
Concurrency and Conflict Resolution In any distributed database that lets a client concurrently modify its contents, precautions must be taken to prevent these updates from overwrit-
User clients
ing one another. Consider the following scenario. In a dynamic-terrain simulation, a small-wheeled vehicle hits a land mine as it’s driving across a patch of terrain. This destroys the vehicle, leaving a crater in its place Before the detonation, a track client and a crater client are generating the jeep’s tire tracks and craters for mines. After the detonation, those clients each query the terrain database and receive the same data. Each calculates how the terrain will be shaped after applying some transformation to a local copy of the terrain data and sends an update message back to the terrain database. Consequently, whichever update arrives later will completely overwrite the previous update, possibly leading to strange visual artifacts such as a plateau of tire tracks jutting into an otherwise deep crater. To prevent such conflicting updates, some form of concurrency control is necessary. We opt for a fine-grained locking scheme that assumes queries are either read-only or read-write. A read-only query requires a read lock, which blocks write queries while allowing processing of other read queries. With a read-write query, a client first queries the data so that it can modify the data in a local buffer before sending the data back to the database as an update. This means that all additional read-write queries in the same area that are answered before the corresponding update arrives could yield invalid results (the terrain data could change in the interim). So, read-write queries effectively function as exclusive write locks on the corresponding area of terrain. The lock is released when the corresponding terrain update arrives. The system automatically removes locks that persist past a given time-out period to prevent any process from permanently locking an area of terrain. When a compute node initiates a lock, it first broadcasts a lock request to the other compute nodes. These nodes respond with an acknowledgment message if they can lock the region or with a negative acknowledgment if they can’t. The Paxos algorithm is one way to ensure enforcement of proper distributed locking.3
W
e’re in the process of deploying our system across multiple processing nodes, after which we’ll evaluate its performance under heavy loads as the number and density of users increase. Additionally, we’ll evaluate delivery of imagery to a variety of client devices, approaches for and impacts of adding natural and manmade features to the terrain, and the introduction of multiple independent viewpoints to the same database.
User clients
Server 1
Server 2
Server 3
Server 4
User clients
User clients
Figure 4. The replicated data model. Each server maintains a copy of the entire terrain database. Dark blue dots represent entities. Green terrain represents data that’s up-to-date for the given server. Yellow terrain represents out-of-date data, which must be synchronized with the servers containing the most current data. Arrows represent synchronization messages.
Acknowledgments This work was funded partly by US Army RDECOM Contract W91CRB08D0015/0006 and Troy Dere, the contracting officer’s representative. The views and opinions expressed are the authors’ and don’t reflect the position of the US government.
References 1. C. Lisle et al., “Architectures for Dynamic Terrain and Dynamic Environments in Distributed Inter active Simulation,” Proc. 10th Distributed Interactive Simulation Workshop on Standards for the Interoperability of Defense Simulations, vol. 2, Institute for Simulation and Training, 1994, pp. 88–106. 2. C. Ellis, P. Babenko, and B. Goldiez, “Distributed Dynamic Terrain in a High-Performance Computing Cluster Environment,” 2009; http://webstokes.ist.ucf. edu. 3. L. Lamport, Generalized Consensus and Paxos, tech. report MSR-TR-2005-33, Microsoft Research, 2005.
Christopher Ellis is a graduate assistant at the University of Central Florida’s Institute for Simulation and Training. Contact him at [email protected]. IEEE Computer Graphics and Applications
83
Projects in VR
Pavel Babenko is a graduate assistant at the University of Central Florida’s Institute for Simulation and Training. Contact him at [email protected]. Brian Goldiez is the deputy director and principal investigator at the University of Central Florida’s Institute for Simulation and Training. Contact him at [email protected]. Jason Daly is an associate in simulation at the University of Central Florida’s Institute for Simulation and Training. Contact him at [email protected].
COMPUTING LIVES www.computer.org/annals/computing-lives The “Computing Lives” Podcast series of selected articles from the IEEE Annals of the History of Computing cover the breadth of computer history. This series features scholarly accounts by leading computer scientists and historians, as well as firsthand stories by computer pioneers.
Glenn A. Martin is the senior research scientist and lab director at the University of Central Florida’s Institute for Simulation and Training. Contact him at [email protected]. Contact the department editors at cga-vr@computer. org. Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.
ADVERTISER/PRODUCT INDEX
January/February 2010 • IEEE Computer Graphics and Applications Advertising Personnel Marion Delaney IEEE Media, Advertising Dir. Phone: +1 415 863 4717 Email: [email protected]
Advertising Sales Representatives Recruitment:
Marian Anderson Sr. Advertising Coordinator Phone: +1 714 821 8380 Fax: +1 714 821 4010 Email: [email protected]
Mid Atlantic Lisa Rinaldo Phone: +1 732 772 0160 Fax: +1 732 772 0164 Email: lr.ieeemedia@ ieee.org
Sandy Brown Sr. Business Development Mgr. Phone: +1 714 821 8380 Fax: +1 714 821 4010 Email: [email protected]
New England John Restchack Phone: +1 212 419 7578 Fax: +1 212 419 7589 Email: j.restchack@ ieee.org Southeast Thomas M. Flynn Phone: +1 770 645 2944 Fax: +1 770 993 4423 Email: flynntom@ mindspring.com
84
January/February 2010
Midwest/Southwest Darcy Giovingo Phone: +1 847 498 4520 Fax: +1 847 498 5911 Email: dg.ieeemedia@ ieee.org Northwest/Southern CA Tim Matteson Phone: +1 310 836 4064 Fax: +1 310 836 4067 Email: tm.ieeemedia@ ieee.org Japan Tim Matteson Phone: +1 310 836 4064 Fax: +1 310 836 4067 Email: tm.ieeemedia@ ieee.org Europe Hilary Turnbull Phone: +44 1875 825700 Fax: +44 1875 825701 Email: impress@ impressmedia.comava
Product: US East Dawn Becker Phone: +1 732 772 0160 Fax: +1 732 772 0164 Email: [email protected] US Central Darcy Giovingo Phone: +1 847 498 4520 Fax: +1 847 498 5911 Email: [email protected] US West Lynne Stickrod Phone: +1 415 931 9782 Fax: +1 415 931 9782 Email: [email protected] Europe Sven Anacker Phone: +49 202 27169 11 Fax: +49 202 27169 20 Email: sanacker@ intermediapartners.de
Editor: Lawrence Rosenblum and Simon Julier
Dynamic Terrain for Multiuser Real-Time Environments Christopher Ellis, Pavel Babenko, Brian Goldiez, Jason Daly, and Glenn A. Martin, University of Central Florida
D
espite the pursuit for increased realism in VR training simulations and massively multiplayer online games, we find little deployed capability to modify terrain. Usually, terrain is stored in a static database and then preprocessed to support multiple levels of detail and to rectify features with the ground. Also, dynamic-terrain research has concentrated on graphics simulations for single users; we’ve found no research on systems that can support 500 or more user systems. Many obstacles to implementing large-scale dynamic-terrain simulations are hardware based— such simulations need large amounts of memory and computing power. However, recent advances in computer hardware can help ameliorate these problems. In particular, high-performance computing clusters (HPCCs) offer an expandable set of computing elements, large amounts of memory and storage, and high-speed connections that remove many of the hardware limitations. Additionally, the industry’s movement toward multi- or many-core machines indicates that research done today on high-performance computers (HPCs) could find its way into future commodity systems. A single terrain database represents a significant bottleneck when it supports many users. This situation has led us to develop a distributed terrain database architecture. In such an architecture, many details must be addressed. We’re addressing some of the more significant issues, such as how to avoid conflict when multiple terrain modifications occur simultaneously in the same area, how to detect and recover from faults, and how and when to distribute load among servers. Our goals in prototyping dynamic terrain are to ■
■ ■ ■
80
support many users interacting in the same virtual environment, resolve conflicting terrain queries, support vast regions of modifiable terrain, and deliver simulation imagery to a variety of client devices.
January/February 2010
Existing dynamic-terrain systems tend to focus on how best to represent the terrain data. Our research emphasizes creating modular, abstract terrain representations so that heterogeneous clients can interact with the simulation. We also heavily consider fault tolerance to give simulations resilience against hardware failures, timing inconsistencies between machines and processes, or software faults. In previous iterations, the dynamic-terrain systems were designed to work well, despite hardware limitations. An explicit assumption is that intercommunication between nodes or processors is both slow and unreliable. The next big question then is, how do you distribute the dynamic-terrain data? Here, we describe how we solve this problem for a VR training system by exploiting an HPCC environment.
Gemini—the Dynamic-Terrain Engine As the baseline system for our research, we chose Gemini1 (see Figure 1). Its modular interface lets it use different database formats and simulation protocols as well as different communication schemes. Gemini allows communication between multiple servers and multiple clients. A server processes messages from clients at a fi xed frame rate to prevent issues when servers become desynchronized. Gemini translates existing simulator protocols into its own internal protocol, which facilitates the integration of heterogeneous servers and clients. Terrain is represented on disk as a 2D matrix of values for each terrain attribute. Each attribute represents one terrain property, such as elevation, density, cohesion, or temperature. This system is extensible; to add new terrain properties, you merely add a new attribute layer to the existing terrain representation. Gemini represents the terrain data as a nonparametric surface—specifically, a uniform nonrational bicubic B-spline. To implement our design, we created dynamicterrain resource (DTR) clients (see Figure 2). These clients listen for a specific type of terrain modification event, query the terrain database, compute
Published by the IEEE Computer Society
0272-1716/10/$26.00 © 2010 IEEE
The Partitioned-Data Model One approach divides the database into equal-sized terrain patches and allocates each patch to a separate database on a different compute node. Such partitioning is naïve, however, and can result in a drastic load imbalance. Some regions of the simulation might be heavily accessed, whereas others could remain unused. This isn’t an unreasonable assumption owing to the spatial and temporal locality of entities and events in a simulation. So, we must be able to change the size and assignment of terrain patches to compute nodes dynamically during the simulation. One method to do this is by implementing a partitioned-data model. This allocation should be based on load, which we can measure as the number and size of the queries a particular compute node is processing in a given time period. Every time we reallocate a patch of terrain to a different compute node, we must send the entire patch from the old node to the new one. However, owing to the compact representation of terrain in Gemini and the high-speed InfiniBand connection, this operation should minimally affect real-time performance. This model has some drawbacks, however. For example, whenever the client wants to perform a read or write on terrain data, its request must be routed to the node containing that data. Additionally, if a client request affects data on multiple
nodes, it must be distributed to those nodes, and the resulting responses must be combined before being sent back to the client. This model also requires some metric to determine when to reallocate terrain. If we reallocate too frequently, internode communication dominates the available bandwidth, slowing the rate at Compute node Client processes Track DTR
Terrain database
Crater DTR User client
Other processes
Compute node
Databases
Entity database
Other database
Shared memory
Gemini server
InfiniBand (TCP or remote direct memory access)
We can distribute the database in one of two ways.2 We can partition it into mutually exclusive pieces (called patches) and distribute them across multiple compute nodes, or we can fully replicate the entire database on each compute node.
Figure 1. An artistic example of dynamic terrain. This simulation uses the Gemini dynamic-terrain engine (DTR). A dynamically generated crater is visible in the foreground
…
Distributing the Database
DTR-generated crater
…
the necessary changes, and submit the updated terrain back to the database. Examples of DTR clients include the track client, which leaves tracks in the ground behind vehicles, and the crater client, which generates craters in the soil as the result of mines or other explosives. We’ve implemented these two clients on an HPCC at the University of Central Florida (for more on HPCCs, see the sidebar). Each compute node runs duplicate sets of DTR clients to handle terrain events local to it. With the computationally intensive pieces of terrain simulation offloaded to the DTR clients, the terrain database has the sole responsibility of responding to queries and updates the user clients send. User client processes handle keyboard or joystick input and 3D graphics output. Remote users connect to these processes via a secure Internet connection (see Figure 3).
Gemini server
Clients Data
Compute node Gemini server
Clients Data
Compute node Gemini server
Clients Data
Compute node Gemini server
Clients Data
Figure 2. A compute node’s structure. The Gemini server acts as an interface between client processes and data and sends and receives messages from servers on other compute nodes via dynamic-terrain resources (DTRs). IEEE Computer Graphics and Applications
81
Projects in VR
High-Performance Computing Clusters
H
igh-performance computing clusters (HPCCs) are capable of computational power far exceeding that of desktop or workstation PCs. HPCC compute nodes contain many fast multicore processors and large quantities of memory and are considered computational workhorses. All nodes are connected to each other via a low-latency, high-bandwidth switch and communication link, such as InfiniBand or Gigabit Ethernet. HPCCs can compute trillions of floating-point operations per second (teraflops). InfiniBand can transmit billions of bits per second (gigabits/sec.). Such powerful computation and communication brings the goal of large, many-client, dynamic-terrain-enabled VR systems within our reach. The question is how best to harness this power.
HPCC
Internet Remote client
Compute node Compute node Compute node
TCP/IP interconnect
InfiniBand interconnect
Compute node
Remote client
Compute node
Remote client Remote client
Remote client Remote client
Remote client Remote client Remote client
Remote client
Figure 3. Remote clients connecting to the high-performance computing cluster (HPCC). Each compute note can handle multiple remote clients. So, a load-balancing scheme is useful to distribute the clients among the nodes.
which client requests can be serviced. If we don’t reallocate frequently enough, clients spend too much time on congested nodes, and other nodes remain idle. Any allocation scheme must consider when the reallocation overhead outweighs the load-balancing benefits.
The Replicated-Data Model Rather than partitioning the terrain, it can be replicated across each compute node. This approach lets clients connect directly to any compute node, thus supporting a fine-grained load-balancing mechanism without terrain partitioning (see Figure 4). However, duplicating the terrain data in this fashion requires a mechanism to prevent concurrent terrain updates from creating database inconsistencies. A naïve approach would be to 82
January/February 2010
broadcast all terrain updates to all compute nodes, but this would defeat the purpose of distributing the terrain databases because all clients are sending their update requests to all compute nodes. Instead, we use a lazy synchronization scheme that synchronizes terrain data between nodes only when necessary. For a distributed dynamicterrain database, this means that when a client on one node requests data that’s out of date owing to an update from another node, the outdated node must copy the data from the more recent node before servicing the client’s request. If clients from one node modify a region, other nodes needn’t know all the intermediate updates to that region. They only need to know the region’s final state if their own clients move into that modified region. If clients are clustered by location in different nodes, this can drastically reduce the synchronization required. Additionally, nodes can actively synchronize data during times of low load in anticipation of future requests. Because each computing node contains the entire terrain database and handles its own dynamicterrain functionality, we allocate users on the basis of their location on the terrain. Our approach assesses each compute node so that, if the load is too high, the clients are reallocated to produce a more equal distribution (for example, using mean-shift clustering to group the clients by spatial locality). To support this scheme, nodes must know whether their data copies are invalid and which node has the latest version. In the event of a terrain update, a lightweight message is broadcast to all nodes notifying them of the update. This message contains the node where the update occurred, the extent of the updated data, the attribute that was modified, and a time stamp. When a node receives this message, the node stores this data in a synchronization table. This table tracks whether a region has been modified and, if so, which node has the most recent data and how recent it is. If a node receives a request for modified terrain, that node queries the node with the most recent copy of the data and updates its own before servicing the request. The update notifications in the synchronization table can be safely overwritten because any node must synchronize before it modifies a region, preventing updates to inconsistent terrain data. Synchronization can also occur when the node is idle.
Concurrency and Conflict Resolution In any distributed database that lets a client concurrently modify its contents, precautions must be taken to prevent these updates from overwrit-
User clients
ing one another. Consider the following scenario. In a dynamic-terrain simulation, a small-wheeled vehicle hits a land mine as it’s driving across a patch of terrain. This destroys the vehicle, leaving a crater in its place Before the detonation, a track client and a crater client are generating the jeep’s tire tracks and craters for mines. After the detonation, those clients each query the terrain database and receive the same data. Each calculates how the terrain will be shaped after applying some transformation to a local copy of the terrain data and sends an update message back to the terrain database. Consequently, whichever update arrives later will completely overwrite the previous update, possibly leading to strange visual artifacts such as a plateau of tire tracks jutting into an otherwise deep crater. To prevent such conflicting updates, some form of concurrency control is necessary. We opt for a fine-grained locking scheme that assumes queries are either read-only or read-write. A read-only query requires a read lock, which blocks write queries while allowing processing of other read queries. With a read-write query, a client first queries the data so that it can modify the data in a local buffer before sending the data back to the database as an update. This means that all additional read-write queries in the same area that are answered before the corresponding update arrives could yield invalid results (the terrain data could change in the interim). So, read-write queries effectively function as exclusive write locks on the corresponding area of terrain. The lock is released when the corresponding terrain update arrives. The system automatically removes locks that persist past a given time-out period to prevent any process from permanently locking an area of terrain. When a compute node initiates a lock, it first broadcasts a lock request to the other compute nodes. These nodes respond with an acknowledgment message if they can lock the region or with a negative acknowledgment if they can’t. The Paxos algorithm is one way to ensure enforcement of proper distributed locking.3
W
e’re in the process of deploying our system across multiple processing nodes, after which we’ll evaluate its performance under heavy loads as the number and density of users increase. Additionally, we’ll evaluate delivery of imagery to a variety of client devices, approaches for and impacts of adding natural and manmade features to the terrain, and the introduction of multiple independent viewpoints to the same database.
User clients
Server 1
Server 2
Server 3
Server 4
User clients
User clients
Figure 4. The replicated data model. Each server maintains a copy of the entire terrain database. Dark blue dots represent entities. Green terrain represents data that’s up-to-date for the given server. Yellow terrain represents out-of-date data, which must be synchronized with the servers containing the most current data. Arrows represent synchronization messages.
Acknowledgments This work was funded partly by US Army RDECOM Contract W91CRB08D0015/0006 and Troy Dere, the contracting officer’s representative. The views and opinions expressed are the authors’ and don’t reflect the position of the US government.
References 1. C. Lisle et al., “Architectures for Dynamic Terrain and Dynamic Environments in Distributed Inter active Simulation,” Proc. 10th Distributed Interactive Simulation Workshop on Standards for the Interoperability of Defense Simulations, vol. 2, Institute for Simulation and Training, 1994, pp. 88–106. 2. C. Ellis, P. Babenko, and B. Goldiez, “Distributed Dynamic Terrain in a High-Performance Computing Cluster Environment,” 2009; http://webstokes.ist.ucf. edu. 3. L. Lamport, Generalized Consensus and Paxos, tech. report MSR-TR-2005-33, Microsoft Research, 2005.
Christopher Ellis is a graduate assistant at the University of Central Florida’s Institute for Simulation and Training. Contact him at [email protected]. IEEE Computer Graphics and Applications
83
Projects in VR
Pavel Babenko is a graduate assistant at the University of Central Florida’s Institute for Simulation and Training. Contact him at [email protected]. Brian Goldiez is the deputy director and principal investigator at the University of Central Florida’s Institute for Simulation and Training. Contact him at [email protected]. Jason Daly is an associate in simulation at the University of Central Florida’s Institute for Simulation and Training. Contact him at [email protected].
COMPUTING LIVES www.computer.org/annals/computing-lives The “Computing Lives” Podcast series of selected articles from the IEEE Annals of the History of Computing cover the breadth of computer history. This series features scholarly accounts by leading computer scientists and historians, as well as firsthand stories by computer pioneers.
Glenn A. Martin is the senior research scientist and lab director at the University of Central Florida’s Institute for Simulation and Training. Contact him at [email protected]. Contact the department editors at cga-vr@computer. org. Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.
ADVERTISER/PRODUCT INDEX
January/February 2010 • IEEE Computer Graphics and Applications Advertising Personnel Marion Delaney IEEE Media, Advertising Dir. Phone: +1 415 863 4717 Email: [email protected]
Advertising Sales Representatives Recruitment:
Marian Anderson Sr. Advertising Coordinator Phone: +1 714 821 8380 Fax: +1 714 821 4010 Email: [email protected]
Mid Atlantic Lisa Rinaldo Phone: +1 732 772 0160 Fax: +1 732 772 0164 Email: lr.ieeemedia@ ieee.org
Sandy Brown Sr. Business Development Mgr. Phone: +1 714 821 8380 Fax: +1 714 821 4010 Email: [email protected]
New England John Restchack Phone: +1 212 419 7578 Fax: +1 212 419 7589 Email: j.restchack@ ieee.org Southeast Thomas M. Flynn Phone: +1 770 645 2944 Fax: +1 770 993 4423 Email: flynntom@ mindspring.com
84
January/February 2010
Midwest/Southwest Darcy Giovingo Phone: +1 847 498 4520 Fax: +1 847 498 5911 Email: dg.ieeemedia@ ieee.org Northwest/Southern CA Tim Matteson Phone: +1 310 836 4064 Fax: +1 310 836 4067 Email: tm.ieeemedia@ ieee.org Japan Tim Matteson Phone: +1 310 836 4064 Fax: +1 310 836 4067 Email: tm.ieeemedia@ ieee.org Europe Hilary Turnbull Phone: +44 1875 825700 Fax: +44 1875 825701 Email: impress@ impressmedia.comava
Product: US East Dawn Becker Phone: +1 732 772 0160 Fax: +1 732 772 0164 Email: [email protected] US Central Darcy Giovingo Phone: +1 847 498 4520 Fax: +1 847 498 5911 Email: [email protected] US West Lynne Stickrod Phone: +1 415 931 9782 Fax: +1 415 931 9782 Email: [email protected] Europe Sven Anacker Phone: +49 202 27169 11 Fax: +49 202 27169 20 Email: sanacker@ intermediapartners.de
