TECHNOLOGY, NETWORK & TELECOMMUNICATIONS
1. KEY STATISTICS
Please outline the key numbers that defined the scope of your program. If this information cannot be included I this report you can provide relevant attachments. If you decide to provide attachments with numbers please list below these files.
Network and Telecommunications Department was responsible for:
a. Planning, implementation and monitoring/operation of the Games Network and the Free Internet Access Network.
b. Planning, implementation and monitoring/operation of the Voice over IP (VoIP) Telecommunications Platform.
c. Distribution and management of the Mobile Phones.
d. Distribution and management of the Wireless Communication Devices (Private Mobile Radios - PMRs).
2. HIGHLIGHTS
Please outline the major elements that defined the performance of your FA during the Games, taking into account the service level defined prior to the event. Furthermore, please state what (if applicable extra non-planned services were delivered after last minutes requests. At this section, it is also important to state what functioned well and what not.
The Network & Telecommunications department implemented a 35 Venues IP MPLS VPN Network (“Games Network”) with speeds between 2 – 800 Mbps, where data & VoIP services were provided. A different network, the 54 Venues “Free Internet Access Network” was also implemented with speeds between 2 – 60Mbps.
All planning was made according to the needs of other FAs but, always, taking under consideration the budget available. Many requests were not approved since they were not critical for the proper conduct of the games.
The GOC Headquarters infrastructure was implemented in time according to all planning and the received needs from all other FAs. All equipment (both bought and sponsored) was, mostly, received in time and no problems were faced during its preparation. All GOC personnel were trained to be able to use all telecommunications equipment using guidelines created by the Network & Telecommunications staff.
Yet, some changes had to be done due to last minute requests.
- Due to lack of extensive and proper testing/planning of the Games Software (GMS/VMS/VSYS etc. ) our GOC HQ Games Network connection had to be upgraded, one week prior to the Games Opening Ceremony, from 50Mbps to 800Mbps to be able to cover the bandwidth needs of the software mentioned above.
- The creation of several IVRs (Intelligent Voice Routing groups) for FAs needs to have multiple call receivers that were not included in their initial planning (e.g. Transportation).
- Additional network equipment (3G Wireless Routers) had to be bought to cover last minute Internet Access needs in some Venues (Healthy Athletes, Ceremonies).
Besides those last minute changes the Network and Telecommunications equipment functioned flawlessly providing top of the line Voice and Data services.
- Thanks to the proper implementation of QoS (Quality of Service) in all Venues, no interruptions were faced in the Voice Services even in Venues with low speed connections.
- Several monitoring systems were installed providing the ability, to all Technology departments, to act proactively on a 24x7 basis.
- Any network equipment configuration failures faced due to power interruptions, were easily restored within a few minutes thanks to the daily network configuration backup. Note that no such failures were faced during games time in any Competition Venues.
3. FAs MAJOR ISSUES & RECOMMENDATIONS
List all major issues encountered by your FA during the Games and your recommendations on how to resolve/avoid these issues
No / Issue Description / Recommendation1 / Games Software, huge bandwidth needs. / Redesign all Games Software (GMS/VMS/VSYS etc.) to be able to operate smoothly without the need of either Terminal Services or huge bandwidth circuits. It should be simple, user friendly and, of course, to be able to work at low speed connections.
2 / Level of knowledge of some technicians / All personnel should be hired at least 1 month prior to the games so that they will all be well trained and able to provide troubleshooting to user issues. An alternative to that is to be able to have a well-organized GOC HQ technical support team, and have all escalated problems resolved easily. The 2nd option suggested, was the one we also used.
3 / Technical Support requests escalation / The use of emails and/or telephones for the escalation of technical support requests is not considered as ideal. The use of a ticketing system is suggested. This must be done upon the initial planning so that there will be time for implementation and fine tuning.
4 / Delayed Access to Technology Rooms and spaces that will be used during “Games Time” of all Venues / Even though this does not concern only the Technology FA, access to the Technology Rooms and in all Venues spaces should be arranged in time and not to be delayed. This way many last-minute changes can be avoided.
4. MAJOR ISSUES WITH OTHER FAs
List all major issues encountered in your cooperation with other FAs during the Games and/or other issues observed during the ATHENS 2011 Special Olympics World Summer Games
No / Functional Area / Issue Description1 / All Other FAs / Many needs (hot desks, additional personnel etc.) were not taken in account in time and many technological overlays had to be done, in several Venues, a few days prior the Games Opening Ceremony.
2 / All Other FAs / Prompt identification of the rooms & spaces that will be used during Games Time in all Venues so that all necessary overlays, network & telecommunication equipment will be taken into account and avoid several last minute changes.
3 / All Other FAs / Better Planning regarding physical access to Venues. GOC HQ Staff managed to get in the F.O.P. in some Venues without being checked by anyone.
4 / All Other FAs / Better Planning regarding physical security to storage rooms. Technology should have a dedicated storage room (for spare equipment) and no one else besides the Technology Staff should have access to it. At the Ceremonies Venue, Technological Equipment was stolen from the storage room.
5 / All Other FAs / Better Planning regarding Bump In & Bump Out processes, Technology should be informed in time of the personnel that will move in & out of a Venue.
5. SUPPLEMENTARY MATERIAL
Please list of all supplementary material that you would like to include in the transfer of knowledge process (i.e. publications, staffing numbers, presentations, etc.)
Network & Telecommunications department consisted of 46 persons in total. These were divided in:
- The department’s Director
- The Network & Telecommunications Administrators (2 persons)
- The GOC HQ Central Technical Support Team (7 persons)
- The GOC HQ Central Technical Support Team Supervisor (1 person)
- The Venues Support Technicians (35 persons)
- The Venues Network Support Volunteers (130 persons)
For the better administration and management of user access, several VLANs were implemented in the GOC HQ Local Network, while only two separate VLANs were used at the Venues (Voice & Data). Due to the small number of users that were located in each Venue, the need for more VLANs was not considered as necessary (see images below).
Image #1 General Network Topology
Image #2 GOC HQ (General) Network Topology
Image #3 Venues (common) Network Topology
Access to all active network equipment (firewalls, routers, switches) was limited to specific users only and only via the Management VLAN, where SSH servers were located providing terminal access. This was implemented using Tacacs+ AAA (authentication/accounting/authorization) & access-lists to limit user access and privileges on all network equipment.
All network equipment was automatically backed up on a daily basis. Each backup file was stored for 99 days giving us the ability to revert to older configurations if necessary.
For all active network equipment the following open source (Linux based) monitoring software was used to provide usage statistics, monitoring and alarming:
- Xymon Monitoring System (previously known as “Hobbit” / “Big Brother”). A utility that can monitor Servers and with the use of “Devmon” addon package, it can monitor via SNMP all network equipment and provide alarms when necessary. Xymon was the main network & servers monitoring tool used in SOWSG – Athens 2011.
- Smokeping Network Latency Grapher to provide Round Trip Times (RTT) based on ICMP. It can also provide alarms when necessary.
- Cricket Graph, a SNMP based tool that provides statistics regarding bandwidth utilization, temperature, connections, etc. in network equipment. It can also provide statistics regarding total active calls, registered IP Phones for the VoIP Infrastructure. Cricket Graph was used to provide us information regarding the temperature of our network equipment located in Venues and to monitor our low bandwidth Venues for bottleneck problems.
- Nagios, an ICMP & SNMP based monitoring tool that can monitor network equipment, VoIP Infrastructure and Network Printers. It was used mostly to monitor Network Printers and IP Phones status in all Venues.
We suggest the use of the same monitoring software in future Games in order to be proactive and be able to resolve any issues in time.
3