Distributed, multi-operator exchange "points"* (DEP) and policy

(WEJohnston, )

DEP motivation: R&E nets must gain control of the rising tide of large-scale science** data flows and manage them:

- to provide sufficient capacity and capability to enable the science

- to preserve a credible best-effort capability for the rest of the community

What makes it a distributed exchange point?

- multiple switches at multiple geographic locations

- transit from any ingress to any egress anywhere in the DEP

- a consistent set of services and policy at all ingress points of the DEP

Speaking now from the experience of the design of the LHCONE DEP, but by no means for the LHCONE DEP team, as there is not agreement on the policy issues among all of the participants, consider this summary. (See lhcone.net for the motivational and current architecture docs.)

Very briefly, the services offered are to be

- L3 connections (current thinking is via a broadcast VLAN and route server)

- L2, guaranteed bandwidth, point to point virtual circuits (selectable bandwidth lightpaths, if you wish) such as provided by ESnet's OSCARS, Ineternet2's ION, GEANT's AutoBHAN, etc.)

Architecturally, the infrastructure is envisioned as islands of switches operated by a single operator and internally interconnected by “policy-free” links. Sites gain access in the usual way by connecting to a DEP switch. The islands are interconnected with policy free links, potentially provided by a third-party (e.g. for transoceanic inter-island connectivity). Taken together, this is the distributed exchange point of LHCONE.

Policy involves who may connect and what capacity is guaranteed to the connecting community.

This leads to the interesting policy issues.

In Europe, it may be that some of the islands of the DEP, by policy, are dedicated to LHC traffic. The LHC community wants a dedicated infrastructure. In the US parts of the infrastructure may be opened up to other science communities. So, is there a policy conflict here? I don't think so, as least not in a “soft” sense, using the following reasoning.

BTW, if this sort of DEP constitutes building a “parallel Internet,” as one person complained, then so be it: It is justified. For example, when we designed and built ESnet4 some six years ago, we very deliberately built two, parallel,nation-wide networks with a very specific goal in mind: Separate large-scale science flows from general traffic, and we have never regretted that decision. In addition to separating the two very different types of flows (data-intensive science traffic is between a small number of end points, is high bandwidth, and flows are sustained for long periods of time vs. the very short, low bandwidth traffic to almost anywhere of the general Internet of the commodity traffic) so that they do not interfere with each other, this infrastructure lets us push the data-intensive traffic to the lowest possible layer (à la Cees de Latt’s admonition).

This brings us to the policy issues: Can a common infrastructure provide “dedicated” service to multiple communities? The LHC community (referred to henceforth as a Virtual Organization – VO) wants guaranteed capacity and restricted access to capacity, then an ingress policy that permits LHC site access together with a guarantee of a certain capacity can easily be provided. L2 VC requests are accepted from any LHC site and the circuit constructed (assuming resources are available). L3 connections are accepted, and the contents of the route server restrict the connections that can be made.

What if some DEP islands allow other VOs – other science communities - and other islands do not? It seems to me that with no further mechanism, that usage by the second VO will naturally be restricted to the islands that accommodate that community. In otherwords, L2 connections can only be made across islands whose ingress policy allows the second VO. Similarly, route server entries are restricted to those that match the second VO policy of certain islands. That is, the traffic of the second (non-LHC VO) only flows to the parts of the DEP whose policies permit that traffic. In those that don’t, L2 connection requests are refused (so you have only one end of the request being validated) and route server entries for sites of the second VO in islands that only support the LHC are also refused.

As long as the islands what allow the second VO provides the guaranteed capacity to the LHC (and I think that we have sufficient tools available to manage this), then this (supporting the second VO) is a non-interfering (traffic-wise) arrangement from the point of view of the LHC.

I call this “soft” policy compliance (with respect to the LHC) because there may be multiple VOs associated with, e.g., a large university. That is, my version of soft policy compliance described here only works if sites and VOs are synonymous. If there are multiple VOs per site, then a finer grained discrimination is required.

The point of this rambling is that in the largely like-minded community of the R&E world (even allowing for specific VOs putting their own resources into building “dedicated” DEPs) I think that there is the technical ability, and I would hope the political accommodation, to permit multiple policies within the same physical infrastructure, and this, I think, should allow for exchange points to accommodate multiple communities with differing policy desires. That is, I am arguing both for accommodating the (apparently restrictive) policies of various R&E communities, while at the same time “sharing” the underlying infrastructure.

* Despite the semantic ambiguity of a "distributed point," I think the term is useful since many people know what an Internet exchange point is and will probably understand the meaning of "distributed exchange point."

** The term "science" here is a placeholder for any R&E community that generates large "non-best effort" flows. The CERN / LHC is the first science case to bring this issue to the fore, but climate simulation, geonomics, and various large-scale instruments are not far behind. (see Further, I certainly do not intend to exclude other R&E disciplines that involve, e.g. various sorts of high data rate multimedia traffic, which will exhibit many of the same characteristics as large science flows.

PS – please excuse the somewhat telegraphic and somewhat rambling nature of this text: It was mostly composed on a smallish tablet with no keyboard, while sipping beer in a Left Bank café.