I'm on my way to Taipei, so this post will be the last one until I recover from the 24 hour trip and 13 hour time difference.
I wanted to mention Tom Offenbach's very interesting blog post about peering at the edge. Tom says:
1. Internet access is becoming increasingly ubiquitous.
2. P2P can improve download efficiency and save costs, but it works best when sharing happens locally. The closer your peers, the fewer hops the data has to travel.
3. It's crazy that last mile service providers aren't peering with one another. If Tom has Comcast and his next door neighbor has Pac Bell DSL, a P2P request from the neighbor would have to travel through Pac Bell's Central Office, Pac Bell's regional hub, Pac Bell's upstream provider, some public peering point, Level 3, Comcast's regional data center, Comcast's headend and Comcast's neighborhood node before it reaches Tom.
(BTW, this is the kind of inefficiency that CoralCDN is trying to quantify through its Illuminati network measurement project.)
Instead of this madness, Tom wants to know why Comcast and Pac Bell can't peer at the Central Office or headend level?
Let's say this does happen down the line. With increasing P2P efficiency, more and more of your customers' content will be shared at the edge, rather than downloaded from your data center. The good news is, you'd be under less pressure to deploy additional GigEs (or 10GigEs - and sooner or later, 100GigEs). On the other hand, the high capacity network that you've invested heavily in may become less of a value proposition.
A few months ago, Jon Udell (who recently became a Microsoft evangelist) wrote in an InfoWorld article that:
"We've already seen how open source software projects harness collective effort to produce quality results. We're now seeing how open content projects such as Wikipedia do the same. Can open infrastructure be far behind?"
I was intrigued by the idea and dug up a bit of info on other open infrastructure projects besides CoralCDN. All are academic, and none have been commercialized. Lots of cool technologies are out there though. Makes me wonder what the web hosting business will look like 5 years from now...
CUXLDEN For Lite Server makes it possible for very large files like online game files, HD movies, original UCC movies and zipped back up files that required high cost in distributing to be easily distributed to thousands of users with little cost incurred.
"CUXLDEN For Lite Server", has a great capability of distribution with one single computer with Internet connection through peer-to-peer network. Compared to existing client-server based file distribution method that require dozens of ftp/http servers and incurs high cost, this product is economically and functionally far advanced than client-server based.
http://www.download.com/CUXLDEN-For-Lite-Server/30...
Because the bandwidth over there (in US) is just so damn cheap.
Where I am from (Australia), where ingress and egress bandwidth is expensive, and GigE to the Internet is almost unthinkable -- local peering becomes very popular. ISP here frequently peer with each other from ISP's regional routers, and some ISP don't charge users peered traffic (yes in Australia broadband connections usually come with download limits).
It all depends on the demand I guess.
Routing policy is tricky enough when we peer at 20 or so points - and yes, with International networks, its at least that many. Equinix Ashburn, Chicago, Dallas, San Jose, PAIX, 56 Marietta, 111 8th Ave, LINX, AMSIX, ESPANIX, NOTA, JPIX, HKIX, TIX, DECIX, and more. IP follows a least cost routing paradigm. Peering in local communities would be a disaster of congestion and poor performance, because its simply not possible to perform finely grained enough routing policy to control ingress and egress traffic. In theory one could do this programatically, but a successful solution that can handle complex routing policy at a large scale hasn't been found.
In addition to the technical issues, there are lots of economic ones. Edge capacity is pricey, while core capacity is dirt dirt cheap. Cross connects in Equinix cost me $300 MRC and trenching between my local cable company head end and the local telco CO cost me tens of thousands (times thousands of CO/head-end pairs). Its a dude with a reel of orange fiber in a ladder rack vs men with picks and shovels. Also, remember the power of dense wave division multiplexing.
Sorry to go on so long, but this is a complex issue, one that folks study for years to be an expert in. When folks like Cringely or Offenbach sort of assume that the engineers designing this stuff haven't considered local peering - it boggles the mind. For those interested in this stuff, come to the NANOG conference in Toronto next week and you can talk to the experts.
For a nice deconstruction of this sort of nonsense from a smart engineer, see: http://www.renesys.com/blog/2007/01/cringley_amsix...
It seems you are using my example of PacBell, UUnet, Wcom, Etc, literally. My intent was to use them as placeholders as they can be replaced with various network providers that sell to other network providers or buy from other network providers. The point of my post was that it costs these companies money to carry that p2p traffic per the example to EquinixSJ or 11 Hudson or LINX, etc. and it appears to be inefficient from a network resource perspective...why carry the traffic if you don't have to? In my original example, I used two neighbors who were on the same p2p network but on seperate service providers. One was using PB and had a DSL connection. If you've been around long enough you would know that PacBellInternet was not a tier one isp and purchased transit from an upstream provider. I happened to call that upstream uunet for simplicity. The other neighbor was on a cable modem. Today the MSOs have grown their networks to the point where they do have backbones but for illustration purposes I said they were also buying transit from an upstream provider.
This example illustrates the path the packets would take to get from neighbor A to neighbor B. There is something that stands out as inefficient in this example and that is that the packets destination is 100 feet away yet it has to travel X miles over fiber to get there and then do the same thing to get back.
If CLECs(for lack of a better term - referring to MSOs, Wireless, Copper, etc) interconnected this additional 'travel' path would be significantly shortened. My question/thought/suggestion to investigate/discussion starter was whether or not a p2p could act as the intermediary of such a transaction. Just like Equinix GigExchange does but on the edge.
FWIW - I do have a fair amount of background experience relating to peering and interconnecting ISPs so I'm not assuming that back in 1994 the peering coordinator at UUNet was talking to the peering coordinator at NETCOM and thinking that one day video would be transmitted over their networks because they weren't. Infact, I wouldn't exactly describe peering relationships as being cooperative until about five years ago and by that time the bets were placed, investments made by the parties and there wasn't any turning back. Today it's a different landscape, many of the backbone providers own the local transport in addition to their backbones, content companies are just as powerful from a peering perspective as the backbones, people download videos to their phones, and so on and so on. Stuff that was designed 15+ years ago isn't necessarily whats best for stuff that is happening today. Maybe p2p could help drive efficiency, maybe not. A combination of p2p over wifi, wimax, or some other wireless protocol would negate the need to trench which seems to be the roadblock you throw out as to why edge peering won't work. Sooner or later you won't need to dig up dirt build a network, it'll come over the air.
Unfortunately a lot of folks who understand peering aren't writing about this stuff; when are you going to start a blog?? :)
I've definitely "written well beyond my knowledge-base" (as Todd Underwood puts it) with this post. But even after reading your arguments against local peering (which make a lot of sense; thank you!!), I'm still wondering what the solution is to inefficient P2P sharing.
It sounds like we have an incentive misalignment problem. The high price of edge capacity doesn't prevent Tom's neighbor from installing Kontiki/Red Swoosh/BitTorrent clients. It also doesn't stop content owners from wanting end users to download content from each other. If P2P traffic will use up edge capacity anyhow, why not at least limit consumption of upstream bandwidth?
Given current usage patterns, this may not an important enough issue to break out picks and shovels over. But considering Rich Miller's recent post that the Venice Project (or Joost, rather) could eat up 2GB of bandwidth per user per day, might the economics change at some point down the line? Especially if - as Tom points out - wireless access becomes more prevalent?
BTW, you are evaluating the situation from a here-and-now point of view, based on decisions made and knowledge accumulated within the peering community over time, while Tom's post was written with increasing P2P data transfer in mind. I didn't get the sense that he's assuming lack of expertise on anyone's part.