Pastry and Chord - Semantic Scholar
Pastry and Chord: A scalable peer to peer approaches for decentralized control, self organization for internet applications Christian Radke [email protected] Institute of Computer Science Freie Universität Berlin Takustraße 9 14195 Berlin
Abstract. Peer-to-peer systems were getting popular in the past by applications that have built over them, e.g. file sharing programs. They also are interesting in the actual research because of they could be widely used to build lots of network applications. This paper focus on two protocols used to build peer-to-peer systems and evaluates their performance: Pastry and Chord. They are classified as a second generation of peer-to-peer systems. Recently, there are some efforts to use the advantage of peer-topeer in Wireless Sensor Networks (WSN) and Mobile Ad-hoc Networks (MANETs). Keywords: Pastry, Chord, peer-to-peer systems.
1 Introduction The first generation of peer-to-peer systems where getting popular through the widely use of file sharing programs like Napster [18]. So the issue peer-to-peer systems were getting also focused by actual research. The first infirmity of these first generation peer-to-peer protocols where their central servers which become a bottleneck and inhibit good scalability. Second generation of peer-to-peer protocols removed the centralized structures, so that real peer-to-peer systems with only equal peers could be established e.g. the Gnutella [14] or BitTorrent [19] networks. But some of these also had the lack of good scalability because of widely use of broadcasts which created lots of inefficient traffic or network congestion if a huge number of peers joins these systems. In actual research there are a lot of protocols introduced which should avoid the weaknesses of these protocols. Tapestry [9], CAN [10], Chord [5], [6], [7] and Pastry [1], [2], [4] are some of these structured peer-to-peer overlay network protocols which achieve good performance. There are also a lot of applications build on top of these protocols which shows that they can be used for application building e.g. PAST[16], Squirrel[15], SCRIBE[20], etc. which are name lookup services, file storage or web caching systems. Recently peer-to-peer systems are used in embedded systems, especially those that have limited resources like WSNs. In this report we explain
2
Christian Radke
Pastry and Chord as representatives for these next generation peer-to-peer overlay network protocols. Pastry and Chord provide self organization, scalability with a low management overhead, reliability and are theoretically able to find data in Ο(log n) steps. This report is organized as following. Section 2 and 3 show how Pastry and Chord works. Afterwards a discussion of their performance and a few related issues like churn, packet loss, proximity routing, number of hops, latency and stabilization is shown in section 4 by taking a literary survey of actual papers to Pastry and Chord. There are also papers about the caching needs of these algorithms [8], but in this paper we won’t care about caching strategies because these protocols should be evaluated for use on mobile clients with embedded systems which mostly have very limited resources which prohibit use of extensive caching. In Section 5 this paper gives a short conclusion to the discussed protocols and their performance.
2 Pastry Fig. 1. Sample routing table of a Node with NodeID 10233102, where b = 2 and l = 8 and digits in base 4 = (2b). The NodeID Space ranges in this case from 0 to 216-1. The grey cells in the nth rows shows entry where the prefix is equal to this node up to the nth digit. At this point the entry is null because it could not be used efficiently for routing. In this case the Leaf set is used for routing. In the Neighborhood set the nodes localized next to the sample node are kept. The associated IP-Addresses (or underlying network addresses) are not shown in this figure.
Pastry is an overlay routing protocol which uses mostly a 128 bit NodeID. Every Pastry node signs a random NodeID which could be calculated by a hash algorithm from the nodes public key or its underlying network address each time a node joins. There are some parameters which mostly have the same values at the most Pastry implementations. The nodeIDs are kept as a string with digits in base 2b notation and every Node contains 3 tables for routing information, R, L and N. The routing table R with log 2b N rows and 2b-1 entries in each row contains NodeIDs and their
underlying physical network adress with a prefix equal to the nodeID of the node containing these tables. The parameter L indicates the number of leaf nodes which are kept in the Leaf Set. A typically value of these configuration parameters are b=4 and b b L=16 or 32 (2 or 2 x 2 ). The leaf set of node i contains the L/2 nodes numerically greater closest to node i and the L/2 nodes numerically smaller closest to node i. Also the Neigborhood Set N of M neighbors in sense of locality are kept in the nodes. M is mostly equal to L.
Pastry and Chord: A scalable peer to peer approaches for decentralized control, self organization for internet applications 3
Fig. 2. A sample routing set R of a Pastry node with NodeID 65a1x. b = 4, l = 16 where x means an arbitrary suffix
Fig. 3. Shows the routing of a message to the node closest to d46a1c
2.1 Routing a message Routing is a simple algorithm which routes a message with key D to the node closest to the value D. Therefore if a message arrives the node checks if the key D is in Range of his Leaf set (Lsmallest ≤ D ≤ Lgreatest). If so, the message is forwarded to the node which has the smallest difference to the key, which could be the actual node itself. If the key is not covert by the Leaf set, the routing Table R is used by the actual node which checks if there is a node that shares a prefix with 1 more digit equal to the given key than itself. If so, the message is forwarded to this node. If there is no such entry in the routing table the message is forwarded to a node that shares a prefix with the key as long as the actual node but is numerically closer to the key. A message is therefore delivered to the numerically closest node in log 2b N steps.
Routing messages works if the routing tables of all nodes in the Pastry network are accurate. Therefore the algorithm to let a node join a Pastry network has to get correct values for entries in the new nodes routing tables. 2.2 Node Join Whenever a new node X wants to join an existing Pastry network it has to initializes its routing tables and has to inform existing nodes of its arrival. Therefore the node gets an address of a nearby Pastry node A by the system administrator or by using “expanding ring” IP multicast or something. The new node X then sends a special join message to node A with a key equal to the nodeID X which is calculated by some hashing algorithm (mostly SHA-1) as mentioned above. Node A delivers the message by routing it to a node Y numerically closest to X. All nodes encountered by the join message send their state tables to node X. Node X inspects all state tables and uses the values to set up his own routing sets. The Neigbhorhood set is taken from node A
4
Christian Radke
because it is very likely that this is close-by node X. The Leaf set is taken from node Y because it is numerically the nearest node to X in Pastry network and so most of the entries of Y’s Leaf set can be taken. X just have to change one of the entries closest to its own ID and shift the other entries of Y’s Leaf set in Y’s direction (numerically smaller or greater than X). The rows for the routing set R are taken from the nodes on the path from X to Y, because node A’s first row is usable for X since the entries shares no prefix with X. The second row of node B which is the node which receives the node join message after A is applicable to the second row of X because it is very likely that A sends the join message to a node which shares the first digit with X. Furthermore X can request the state tables from additional nodes. After X initialized its routing sets successfully it informs nodes which need to update their tables by sending its Leaf set, its Neighborhood set and its routing table, so that they can update their routing sets to correct values. 2.3 Nodes leaving If a node of the Leaf set is not reachable or left the Pastry network silently the effected node with the incorrect Leaf set asks the next leaf nodes for their leaf sets to update its Leaf set. This works if less then L/2 nodes of the Leaf set fail simultaneously, so this case is very unlikely. To avoid the loss of a key/value pair Pastry also copies its key/value pairs to some nodes numerically close to the key. Section 4 shows also performance evaluation of a special Pastry implementation called MSPastry [3] which additionally uses some constrained gossiping and heuristics which implements Proximity neighbor selection (PNS) [2].
3
Chord Fig. 4. Chord network (m = 3) with 3 nodes (1, 3, 0) containing 3 keys (1, 2, 6), each successor of a key contains the key and and the associated value.
Chord uses an overlay for routing like Pastry with an m bit (m ist mostly 128) nodeID space with 2m possible nodeIDs. It uses Consistent Hashing and not prefix routing to store and find key/value pairs in the nodes. A node that has a key k or has a numerically equal identifier is called successor(k). Each node has a routing table called finger table with m entries. The ith entry of the finger table of node n contains i-1 the ID of the first node s that succeeds n by at least 2i-1 (s > (n + 2 ) mod m).
Pastry and Chord: A scalable peer to peer approaches for decentralized control, self organization for internet applications 5
This node s is called the ith finger of node n called n.finger(i) and so its clear that n.finger(1) is always the successor of n on the Chord circle. The finger table contains therefore the nodeIDs of n.finger(i) and their underlying network address which is mostly their IP address. Also the nodeID and IP address of the direct predecessor of node n is kept in the state tables of n. 3.1 Routing a message So if a key k should be found, a node n returns his successor n.finger(1) in case the key is between (n, n.finger(1)], otherwise the lookup for key k is forwarded to the highest predecessor of k, therefore the node n checks whether n.finger(i) where i=m,m-1,…,1 is in the range (n,k). If this is the case, the lookup request is forwarded to n’ = n.finger(i). Node n’ has then better local information in the region around k, so it is more likely that successor(k) which holds the key/value pair is found. It is shown in [5] that this lookup is done in O(log N) steps where N is the number of nodes in this Chord network. 3.2 Node Join We assume as before at Pastry that a node joins a network where each node has a correct state of his routing information. Then a node X gets the address of an already in the Chord network existing node A. Node X asks Node A for the successor(X 0 + 2 ) which has to be the X.finger(1) node (the direct successor of X on the Chord circle). Node X fills his finger table by asking node A for successor(X + 2i) for i=1,2,…,m-1. X then checks whether X.finger(i) = X.finger(i+1) which means that interval [X.finger(i), X.finger(i+1)] contains no node in this Chord network. The predecessor of X is found similarly. After node X has filled its finger table it has to introduce itself to nodes where he could have become successor or ith finger. Therefore it finds all nodes which precedes i-1 it by at least 2i-1 (X.predecessor(n-2 ) for i=1,2,…,m) and calls an RPC with his ID X and i as arguments an that node P. P checks if X is between (P, P.finger(i)). If this is the case, Ps ith finger is set to X. This assures that every finger table stay in a correct state. After that X asks his direct successor X.finger(1) for all key/value pairs which X is successor(k). Then the Chord network is in a correct state and all key/value pairs can be found. [5] shows that joining results also in O(log N) steps. 3.3 Nodes leaving There are also some improvements for stabilization of a Chord network which are mostly gossiping with the successors or predecessors, so that a node arrival or node which silently left the network will not be missed. For fault tolerance every node has
6
Christian Radke
a list of his r following successors to which it propagates its key/value pairs on an update of these. So if a node F fails the key/value pairs are also available in his r successors, so r is a parameter that influences robustness of Chord in trade-off to more communication overhead.
Fig. 5. New node joins in Chord network
4 Performance evaluation Pastry and Chord are introduced as self organizing, robust and effective structured peer-to-peer overlay networks. As mentioned above Pastry and Chord have theoretically lookup times in O(log N) and node joins also take O (log N) time. 4.1 Churn rate At this point the influence of the churn rate (number of nodes leaving and joining the network) should be discussed. Fig. 6. Shows the quality of the routing tables if 500 nodes simultaneously fail in a 5000 node Pastry network where b = 4, L =16 and M =32.
Figure 6 shows the impact of 500 nodes simultaneously leaving a 5000 node Pastry network. The average hops are slightly increased before the routing tables are repaired. In [1] is shown that with 57 messages per failed node the routing tables of all Pastry nodes are again in a correct state. It seems that Pastry is able to handle
Pastry and Chord: A scalable peer to peer approaches for decentralized control, self organization for internet applications 7
normal churn rates of common peer-to-peer applications. [3] gives there also different simulations which shows that Pastry scales good in terms of the churn rate. Fig. 7. Chord Network: Shows the failed lookups in percentage to all lookups if up to 20% of all nodes fail.
Chord shows robustness against high churn rates. Figure 7 shows the amount of failed lookups if a fraction of nodes fail simultaneously. This scales linear which is clear because there where probably as much keys than nodes involved. 4.2 Packet loss Packet loss could result in severe nuisance to structured peer-to-peer overlay networks. So the behavior of Pastry and Chord in presence of high packet loss rates is very interesting. Not as much in wired network topologies but for instance in WSNs which are actual issue to research.
Fig. 8. Impact of Network packet loss rate on Pastry (with the MSPastry implementation [3]
RDP is the relative delay penalty ratio between delay achieved by Pastry and the network delay between these to nodes. The control traffic shows the control messages per node per second and the 3rd diagram in figure 8 shows the lookup loss rate and routing inconsistencies measured in the rate of incorrect routed messages. The stated data of these 3 diagrams show that Pastry scales very well with an increasing rate of packet loss. Just the graph of routing inconsistencies seems to grow more then strait proportional. In [12] the Chord protocol was implemented in a simulator for mobile ad hoc networks (MANETs) and there Chord showed bad results in environments with faulty connections, but Cramer and Fuhrmann said there that Chord probably is good for environments where packet loss results from network congestion because of
8
Christian Radke
Chord’s pessimistic failover strategy whereas in mobile ad hoc networks replicate transmission seems more useful.
4.3 Proximity routing Pastry can be extended with Proximity Neighbor Selection (PNS) to achieve better proximity awareness with a little overhead which is shown and discussed in [2]. PNS is not applicable to Chord because or at least till now no one tried to implement PNS in Chord. But there are different approaches like proximity routing that extend Chord with such routing information in trade off for greater routing tables which have to be maintained in each node. 4.4 Number of hops
Fig. 9. Shows the average number of hops in a Pastry Network with N = 1000,…,100000 nodes. 2 nodes where chosen randomly and a message was delivered from one to another node. This was repeated 200000 times for each measurement.
Pastry shows in the simulation the predicted maximum number of hops when routing a message between to randomly chosen nodes. The upper limitation of log 2b N
wasn’t reached by the experimental results. Fig. 10. Chord number of hops on different Chord network dimensions (nodes) where m=32.
Pastry and Chord: A scalable peer to peer approaches for decentralized control, self organization for internet applications 9
Chords number of hops also was in his theoretical limit of O(log2 N). Both protocols met the predicted limits for the number of hops as network size grows in terms of number of clients. That’s an important factor when reviewing peer-to-peer protocols in terms of performance. 4.5 Latency
Fig. 11. Chord lookup delay in growing Chord network.
The experimental results described in [6] show that latency is also good scaled with Chord when more and more clients approach in the Chord network. Pastry minimizes latency by its network topology aware approach. It uses the round trip time (RTT) between to nodes as proximity metric, so the latency times should be also acceptable if someone uses Pastry with PNS like introduced in [2]. 4.6 Stabilization The Pastry’s and Chord’s stabilization algorithms introduced in the cited papers scale also theoretically like the join and lookup in O (log N). The experimental results seem to confirm this theoretical upper bound.
5. Conclusion Pastry and Chord are structured peer-to-peer overlay network protocols which give huge potential for building self organizing applications to today’s programmers. They cover a lot of needed services which are need to build a peer-to-peer application. The lack of security management now is a great issue to actual research and can surely be solved in trade off for some performance of these protocols. But the question is whether these protocols are also this top-performing in a mobile or wireless environment. Chord shows here some negative impact on high packet loss rates in MANETs. At least the Pastry implementation MSPastry uses an
10
Christian Radke
retransmission strategy on occurrence of packet loss in the underlying network. So it seems to be more applicable to this case of use. Future research is about improvements of these protocols and applications build on top of Pastry and Chord. There seems to be more research in wired environments which is especially pushed through the Microsoft corporation on Pastry, but until now only few effort was done on building applications on Pastry and Chord in wireless environments like WSNs or MANETs.
References 1. Druschel, P., Rowstron, A.: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems. 18th IFIP/ACM International Conference on Distributed Systems Platforms (Middleware 2001). Heidelberg, Germany. (November 2001) 2. Castro, M., Druschel, P., Hu, Y. Ch., Rowstron, A.: Proximity neighbor selection in treebased structured peer-to-peer overlays. Technical Report MSR-TR-2003-52 (2003) 3. Castro, M., Costa, M., Rowstron, A.: Performance and dependability of structured peer-topeer overlays. Technical report MSR-TR2003 -94. (2003) http://citeseer.ist.psu.edu/castro04performance.html 4. Castro, M., Druschel, P., Rowstron, A., Ganesh, A., Wallach, D. S.:Secure routing for structured peer-to-peer overlay networks. Proceedings of the 5th Usenix Symposium on Operating Systems Design and Implementation, Boston, MA. (December 2002) 5. Stoica, I., Morris, R., Karger, D., Kaashoek, F., Balakrishnan, H.: Chord: {A} Scalable {Peer-To-Peer} Lookup Service for Internet Applications. Proceedings of the 2001 ACM SIGCOMM Conference (2001) 6. Stoica, I., Morris, R., Karger, D., Kaashoek, F., Balakrishnan, H., Dabek, F., Liben-Nowell, D.: Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications. (January 10, 2002) 7. Dabek, F., Brunskill, E., Kaashoek, M. F., Karger, D., Morris R., Stoica, I., Balakrishnan, H.: Building Peer-to-Peer Systems with Chord, a Distributed Lookup Service. (2001) 8. Timo Tanner, Distributed Hash Tables in P2P Systems - A literary survey, Seminar on Internetworking, Helsinki University of Technology (2005-04-26) 9. K. Hildrum, J. D. Kubiatowicz, S. Rao, and B. Y. Zhao. Distribted data location in a dynamic network. In SPAA’02, Aug. 2002. 10. S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker. A scalable contentaddressable network. In SIGCOMM’01, San Diego, CA, Aug. 2001 11. Keong Lua, E., Crowcroft, J., Pias, M., Sharma, R. and Lim, S., A Survey and Comparison of Peer-to-Peer Overlay Network Schemes, IEEE COMMUNICATIONS SURVEY AND TUTORIAL, MARCH 2004 12. Cramer, C. and Fuhrmann, T., Performance Evaluation of Chord in Mobile Ad Hoc Networks, MobiShare’06, September 25, 2006, Los Angeles, California, USA. 13. Clarke, I., Sandberg, O., Wiley, B., and Hong, T. W.. Freenet: A distributed anonymous information storage and retrieval system. In Workshop on Design Issues in Anonymity and Unobservability, July 2000. ICSI, Berkeley, California. 14. The Gnutella protocol specification, 2000. http://dss.clip2.com/GnutellaProtocol04.pdf. 15. Iyer, S., Rowstron, A. and Druschel, P.. Squirrel: A decentralized peer-to-peer web cache. In PODC, July 2002. 16. Rowstron, A. and Druschel, P.. Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility. In SOSP’01, Banff, Canada, Oct. 2001
Pastry and Chord: A scalable peer to peer approaches for decentralized control, self organization for internet applications 11 17. Dabek, F., Kaashoek, M. F., Karger, D., Morris, R., and Stoica, I. Wide-area cooperative storage with CFS. SOSP ’01 (To appear; Banff, Canada, Oct. 2001). 18. Napster. http://www.napster.com/. 19. BitTorrent. http://www.bittorrent.com/ 20. Castro, M., Druschel, P., Kermarrec, A.-M. and Rowstron, A.. Scribe: A large-scale and decentralized application-level multicast infrastructure. IEEE JSAC, 20(8), October 2002.
Abstract. Peer-to-peer systems were getting popular in the past by applications that have built over them, e.g. file sharing programs. They also are interesting in the actual research because of they could be widely used to build lots of network applications. This paper focus on two protocols used to build peer-to-peer systems and evaluates their performance: Pastry and Chord. They are classified as a second generation of peer-to-peer systems. Recently, there are some efforts to use the advantage of peer-topeer in Wireless Sensor Networks (WSN) and Mobile Ad-hoc Networks (MANETs). Keywords: Pastry, Chord, peer-to-peer systems.
1 Introduction The first generation of peer-to-peer systems where getting popular through the widely use of file sharing programs like Napster [18]. So the issue peer-to-peer systems were getting also focused by actual research. The first infirmity of these first generation peer-to-peer protocols where their central servers which become a bottleneck and inhibit good scalability. Second generation of peer-to-peer protocols removed the centralized structures, so that real peer-to-peer systems with only equal peers could be established e.g. the Gnutella [14] or BitTorrent [19] networks. But some of these also had the lack of good scalability because of widely use of broadcasts which created lots of inefficient traffic or network congestion if a huge number of peers joins these systems. In actual research there are a lot of protocols introduced which should avoid the weaknesses of these protocols. Tapestry [9], CAN [10], Chord [5], [6], [7] and Pastry [1], [2], [4] are some of these structured peer-to-peer overlay network protocols which achieve good performance. There are also a lot of applications build on top of these protocols which shows that they can be used for application building e.g. PAST[16], Squirrel[15], SCRIBE[20], etc. which are name lookup services, file storage or web caching systems. Recently peer-to-peer systems are used in embedded systems, especially those that have limited resources like WSNs. In this report we explain
2
Christian Radke
Pastry and Chord as representatives for these next generation peer-to-peer overlay network protocols. Pastry and Chord provide self organization, scalability with a low management overhead, reliability and are theoretically able to find data in Ο(log n) steps. This report is organized as following. Section 2 and 3 show how Pastry and Chord works. Afterwards a discussion of their performance and a few related issues like churn, packet loss, proximity routing, number of hops, latency and stabilization is shown in section 4 by taking a literary survey of actual papers to Pastry and Chord. There are also papers about the caching needs of these algorithms [8], but in this paper we won’t care about caching strategies because these protocols should be evaluated for use on mobile clients with embedded systems which mostly have very limited resources which prohibit use of extensive caching. In Section 5 this paper gives a short conclusion to the discussed protocols and their performance.
2 Pastry Fig. 1. Sample routing table of a Node with NodeID 10233102, where b = 2 and l = 8 and digits in base 4 = (2b). The NodeID Space ranges in this case from 0 to 216-1. The grey cells in the nth rows shows entry where the prefix is equal to this node up to the nth digit. At this point the entry is null because it could not be used efficiently for routing. In this case the Leaf set is used for routing. In the Neighborhood set the nodes localized next to the sample node are kept. The associated IP-Addresses (or underlying network addresses) are not shown in this figure.
Pastry is an overlay routing protocol which uses mostly a 128 bit NodeID. Every Pastry node signs a random NodeID which could be calculated by a hash algorithm from the nodes public key or its underlying network address each time a node joins. There are some parameters which mostly have the same values at the most Pastry implementations. The nodeIDs are kept as a string with digits in base 2b notation and every Node contains 3 tables for routing information, R, L and N. The routing table R with log 2b N rows and 2b-1 entries in each row contains NodeIDs and their
underlying physical network adress with a prefix equal to the nodeID of the node containing these tables. The parameter L indicates the number of leaf nodes which are kept in the Leaf Set. A typically value of these configuration parameters are b=4 and b b L=16 or 32 (2 or 2 x 2 ). The leaf set of node i contains the L/2 nodes numerically greater closest to node i and the L/2 nodes numerically smaller closest to node i. Also the Neigborhood Set N of M neighbors in sense of locality are kept in the nodes. M is mostly equal to L.
Pastry and Chord: A scalable peer to peer approaches for decentralized control, self organization for internet applications 3
Fig. 2. A sample routing set R of a Pastry node with NodeID 65a1x. b = 4, l = 16 where x means an arbitrary suffix
Fig. 3. Shows the routing of a message to the node closest to d46a1c
2.1 Routing a message Routing is a simple algorithm which routes a message with key D to the node closest to the value D. Therefore if a message arrives the node checks if the key D is in Range of his Leaf set (Lsmallest ≤ D ≤ Lgreatest). If so, the message is forwarded to the node which has the smallest difference to the key, which could be the actual node itself. If the key is not covert by the Leaf set, the routing Table R is used by the actual node which checks if there is a node that shares a prefix with 1 more digit equal to the given key than itself. If so, the message is forwarded to this node. If there is no such entry in the routing table the message is forwarded to a node that shares a prefix with the key as long as the actual node but is numerically closer to the key. A message is therefore delivered to the numerically closest node in log 2b N steps.
Routing messages works if the routing tables of all nodes in the Pastry network are accurate. Therefore the algorithm to let a node join a Pastry network has to get correct values for entries in the new nodes routing tables. 2.2 Node Join Whenever a new node X wants to join an existing Pastry network it has to initializes its routing tables and has to inform existing nodes of its arrival. Therefore the node gets an address of a nearby Pastry node A by the system administrator or by using “expanding ring” IP multicast or something. The new node X then sends a special join message to node A with a key equal to the nodeID X which is calculated by some hashing algorithm (mostly SHA-1) as mentioned above. Node A delivers the message by routing it to a node Y numerically closest to X. All nodes encountered by the join message send their state tables to node X. Node X inspects all state tables and uses the values to set up his own routing sets. The Neigbhorhood set is taken from node A
4
Christian Radke
because it is very likely that this is close-by node X. The Leaf set is taken from node Y because it is numerically the nearest node to X in Pastry network and so most of the entries of Y’s Leaf set can be taken. X just have to change one of the entries closest to its own ID and shift the other entries of Y’s Leaf set in Y’s direction (numerically smaller or greater than X). The rows for the routing set R are taken from the nodes on the path from X to Y, because node A’s first row is usable for X since the entries shares no prefix with X. The second row of node B which is the node which receives the node join message after A is applicable to the second row of X because it is very likely that A sends the join message to a node which shares the first digit with X. Furthermore X can request the state tables from additional nodes. After X initialized its routing sets successfully it informs nodes which need to update their tables by sending its Leaf set, its Neighborhood set and its routing table, so that they can update their routing sets to correct values. 2.3 Nodes leaving If a node of the Leaf set is not reachable or left the Pastry network silently the effected node with the incorrect Leaf set asks the next leaf nodes for their leaf sets to update its Leaf set. This works if less then L/2 nodes of the Leaf set fail simultaneously, so this case is very unlikely. To avoid the loss of a key/value pair Pastry also copies its key/value pairs to some nodes numerically close to the key. Section 4 shows also performance evaluation of a special Pastry implementation called MSPastry [3] which additionally uses some constrained gossiping and heuristics which implements Proximity neighbor selection (PNS) [2].
3
Chord Fig. 4. Chord network (m = 3) with 3 nodes (1, 3, 0) containing 3 keys (1, 2, 6), each successor of a key contains the key and and the associated value.
Chord uses an overlay for routing like Pastry with an m bit (m ist mostly 128) nodeID space with 2m possible nodeIDs. It uses Consistent Hashing and not prefix routing to store and find key/value pairs in the nodes. A node that has a key k or has a numerically equal identifier is called successor(k). Each node has a routing table called finger table with m entries. The ith entry of the finger table of node n contains i-1 the ID of the first node s that succeeds n by at least 2i-1 (s > (n + 2 ) mod m).
Pastry and Chord: A scalable peer to peer approaches for decentralized control, self organization for internet applications 5
This node s is called the ith finger of node n called n.finger(i) and so its clear that n.finger(1) is always the successor of n on the Chord circle. The finger table contains therefore the nodeIDs of n.finger(i) and their underlying network address which is mostly their IP address. Also the nodeID and IP address of the direct predecessor of node n is kept in the state tables of n. 3.1 Routing a message So if a key k should be found, a node n returns his successor n.finger(1) in case the key is between (n, n.finger(1)], otherwise the lookup for key k is forwarded to the highest predecessor of k, therefore the node n checks whether n.finger(i) where i=m,m-1,…,1 is in the range (n,k). If this is the case, the lookup request is forwarded to n’ = n.finger(i). Node n’ has then better local information in the region around k, so it is more likely that successor(k) which holds the key/value pair is found. It is shown in [5] that this lookup is done in O(log N) steps where N is the number of nodes in this Chord network. 3.2 Node Join We assume as before at Pastry that a node joins a network where each node has a correct state of his routing information. Then a node X gets the address of an already in the Chord network existing node A. Node X asks Node A for the successor(X 0 + 2 ) which has to be the X.finger(1) node (the direct successor of X on the Chord circle). Node X fills his finger table by asking node A for successor(X + 2i) for i=1,2,…,m-1. X then checks whether X.finger(i) = X.finger(i+1) which means that interval [X.finger(i), X.finger(i+1)] contains no node in this Chord network. The predecessor of X is found similarly. After node X has filled its finger table it has to introduce itself to nodes where he could have become successor or ith finger. Therefore it finds all nodes which precedes i-1 it by at least 2i-1 (X.predecessor(n-2 ) for i=1,2,…,m) and calls an RPC with his ID X and i as arguments an that node P. P checks if X is between (P, P.finger(i)). If this is the case, Ps ith finger is set to X. This assures that every finger table stay in a correct state. After that X asks his direct successor X.finger(1) for all key/value pairs which X is successor(k). Then the Chord network is in a correct state and all key/value pairs can be found. [5] shows that joining results also in O(log N) steps. 3.3 Nodes leaving There are also some improvements for stabilization of a Chord network which are mostly gossiping with the successors or predecessors, so that a node arrival or node which silently left the network will not be missed. For fault tolerance every node has
6
Christian Radke
a list of his r following successors to which it propagates its key/value pairs on an update of these. So if a node F fails the key/value pairs are also available in his r successors, so r is a parameter that influences robustness of Chord in trade-off to more communication overhead.
Fig. 5. New node joins in Chord network
4 Performance evaluation Pastry and Chord are introduced as self organizing, robust and effective structured peer-to-peer overlay networks. As mentioned above Pastry and Chord have theoretically lookup times in O(log N) and node joins also take O (log N) time. 4.1 Churn rate At this point the influence of the churn rate (number of nodes leaving and joining the network) should be discussed. Fig. 6. Shows the quality of the routing tables if 500 nodes simultaneously fail in a 5000 node Pastry network where b = 4, L =16 and M =32.
Figure 6 shows the impact of 500 nodes simultaneously leaving a 5000 node Pastry network. The average hops are slightly increased before the routing tables are repaired. In [1] is shown that with 57 messages per failed node the routing tables of all Pastry nodes are again in a correct state. It seems that Pastry is able to handle
Pastry and Chord: A scalable peer to peer approaches for decentralized control, self organization for internet applications 7
normal churn rates of common peer-to-peer applications. [3] gives there also different simulations which shows that Pastry scales good in terms of the churn rate. Fig. 7. Chord Network: Shows the failed lookups in percentage to all lookups if up to 20% of all nodes fail.
Chord shows robustness against high churn rates. Figure 7 shows the amount of failed lookups if a fraction of nodes fail simultaneously. This scales linear which is clear because there where probably as much keys than nodes involved. 4.2 Packet loss Packet loss could result in severe nuisance to structured peer-to-peer overlay networks. So the behavior of Pastry and Chord in presence of high packet loss rates is very interesting. Not as much in wired network topologies but for instance in WSNs which are actual issue to research.
Fig. 8. Impact of Network packet loss rate on Pastry (with the MSPastry implementation [3]
RDP is the relative delay penalty ratio between delay achieved by Pastry and the network delay between these to nodes. The control traffic shows the control messages per node per second and the 3rd diagram in figure 8 shows the lookup loss rate and routing inconsistencies measured in the rate of incorrect routed messages. The stated data of these 3 diagrams show that Pastry scales very well with an increasing rate of packet loss. Just the graph of routing inconsistencies seems to grow more then strait proportional. In [12] the Chord protocol was implemented in a simulator for mobile ad hoc networks (MANETs) and there Chord showed bad results in environments with faulty connections, but Cramer and Fuhrmann said there that Chord probably is good for environments where packet loss results from network congestion because of
8
Christian Radke
Chord’s pessimistic failover strategy whereas in mobile ad hoc networks replicate transmission seems more useful.
4.3 Proximity routing Pastry can be extended with Proximity Neighbor Selection (PNS) to achieve better proximity awareness with a little overhead which is shown and discussed in [2]. PNS is not applicable to Chord because or at least till now no one tried to implement PNS in Chord. But there are different approaches like proximity routing that extend Chord with such routing information in trade off for greater routing tables which have to be maintained in each node. 4.4 Number of hops
Fig. 9. Shows the average number of hops in a Pastry Network with N = 1000,…,100000 nodes. 2 nodes where chosen randomly and a message was delivered from one to another node. This was repeated 200000 times for each measurement.
Pastry shows in the simulation the predicted maximum number of hops when routing a message between to randomly chosen nodes. The upper limitation of log 2b N
wasn’t reached by the experimental results. Fig. 10. Chord number of hops on different Chord network dimensions (nodes) where m=32.
Pastry and Chord: A scalable peer to peer approaches for decentralized control, self organization for internet applications 9
Chords number of hops also was in his theoretical limit of O(log2 N). Both protocols met the predicted limits for the number of hops as network size grows in terms of number of clients. That’s an important factor when reviewing peer-to-peer protocols in terms of performance. 4.5 Latency
Fig. 11. Chord lookup delay in growing Chord network.
The experimental results described in [6] show that latency is also good scaled with Chord when more and more clients approach in the Chord network. Pastry minimizes latency by its network topology aware approach. It uses the round trip time (RTT) between to nodes as proximity metric, so the latency times should be also acceptable if someone uses Pastry with PNS like introduced in [2]. 4.6 Stabilization The Pastry’s and Chord’s stabilization algorithms introduced in the cited papers scale also theoretically like the join and lookup in O (log N). The experimental results seem to confirm this theoretical upper bound.
5. Conclusion Pastry and Chord are structured peer-to-peer overlay network protocols which give huge potential for building self organizing applications to today’s programmers. They cover a lot of needed services which are need to build a peer-to-peer application. The lack of security management now is a great issue to actual research and can surely be solved in trade off for some performance of these protocols. But the question is whether these protocols are also this top-performing in a mobile or wireless environment. Chord shows here some negative impact on high packet loss rates in MANETs. At least the Pastry implementation MSPastry uses an
10
Christian Radke
retransmission strategy on occurrence of packet loss in the underlying network. So it seems to be more applicable to this case of use. Future research is about improvements of these protocols and applications build on top of Pastry and Chord. There seems to be more research in wired environments which is especially pushed through the Microsoft corporation on Pastry, but until now only few effort was done on building applications on Pastry and Chord in wireless environments like WSNs or MANETs.
References 1. Druschel, P., Rowstron, A.: Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems. 18th IFIP/ACM International Conference on Distributed Systems Platforms (Middleware 2001). Heidelberg, Germany. (November 2001) 2. Castro, M., Druschel, P., Hu, Y. Ch., Rowstron, A.: Proximity neighbor selection in treebased structured peer-to-peer overlays. Technical Report MSR-TR-2003-52 (2003) 3. Castro, M., Costa, M., Rowstron, A.: Performance and dependability of structured peer-topeer overlays. Technical report MSR-TR2003 -94. (2003) http://citeseer.ist.psu.edu/castro04performance.html 4. Castro, M., Druschel, P., Rowstron, A., Ganesh, A., Wallach, D. S.:Secure routing for structured peer-to-peer overlay networks. Proceedings of the 5th Usenix Symposium on Operating Systems Design and Implementation, Boston, MA. (December 2002) 5. Stoica, I., Morris, R., Karger, D., Kaashoek, F., Balakrishnan, H.: Chord: {A} Scalable {Peer-To-Peer} Lookup Service for Internet Applications. Proceedings of the 2001 ACM SIGCOMM Conference (2001) 6. Stoica, I., Morris, R., Karger, D., Kaashoek, F., Balakrishnan, H., Dabek, F., Liben-Nowell, D.: Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications. (January 10, 2002) 7. Dabek, F., Brunskill, E., Kaashoek, M. F., Karger, D., Morris R., Stoica, I., Balakrishnan, H.: Building Peer-to-Peer Systems with Chord, a Distributed Lookup Service. (2001) 8. Timo Tanner, Distributed Hash Tables in P2P Systems - A literary survey, Seminar on Internetworking, Helsinki University of Technology (2005-04-26) 9. K. Hildrum, J. D. Kubiatowicz, S. Rao, and B. Y. Zhao. Distribted data location in a dynamic network. In SPAA’02, Aug. 2002. 10. S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker. A scalable contentaddressable network. In SIGCOMM’01, San Diego, CA, Aug. 2001 11. Keong Lua, E., Crowcroft, J., Pias, M., Sharma, R. and Lim, S., A Survey and Comparison of Peer-to-Peer Overlay Network Schemes, IEEE COMMUNICATIONS SURVEY AND TUTORIAL, MARCH 2004 12. Cramer, C. and Fuhrmann, T., Performance Evaluation of Chord in Mobile Ad Hoc Networks, MobiShare’06, September 25, 2006, Los Angeles, California, USA. 13. Clarke, I., Sandberg, O., Wiley, B., and Hong, T. W.. Freenet: A distributed anonymous information storage and retrieval system. In Workshop on Design Issues in Anonymity and Unobservability, July 2000. ICSI, Berkeley, California. 14. The Gnutella protocol specification, 2000. http://dss.clip2.com/GnutellaProtocol04.pdf. 15. Iyer, S., Rowstron, A. and Druschel, P.. Squirrel: A decentralized peer-to-peer web cache. In PODC, July 2002. 16. Rowstron, A. and Druschel, P.. Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility. In SOSP’01, Banff, Canada, Oct. 2001
Pastry and Chord: A scalable peer to peer approaches for decentralized control, self organization for internet applications 11 17. Dabek, F., Kaashoek, M. F., Karger, D., Morris, R., and Stoica, I. Wide-area cooperative storage with CFS. SOSP ’01 (To appear; Banff, Canada, Oct. 2001). 18. Napster. http://www.napster.com/. 19. BitTorrent. http://www.bittorrent.com/ 20. Castro, M., Druschel, P., Kermarrec, A.-M. and Rowstron, A.. Scribe: A large-scale and decentralized application-level multicast infrastructure. IEEE JSAC, 20(8), October 2002.
