Upstream Connectivity Issue

There are currently routing issues near our data center in the US which are causing speed issues and timeouts for some customers. ProWorkflow itself is running normally and all data is safe and secure, but the routing issues are severely affecting speed at times. The engineers at the data center are working with Internap’s network engineers and we will update you as soon as they update us.

Further updates from our service provider can be found at http://ltstatus.com/?p=10821 but we have provided them below:

EVENT UPDATE 22-MAR-10 16:45 CDT: Network Services continues to monitor the overall status of the Chicago and Cedar Falls networks.  The issue that impacted customers with route latency is, as previously report, largely resolved.  There are continued reported issues with customers with services which reside in the 74.200.0.0 network when traversing Internap.  We are working with Internap to address these issues.  Thank you for your continued patience as we work through these remaining issues.  An update will be posted as we have additional information.

EVENT UPDATE 22-MAR-10 18:55 CDT: Network Services continues to monitor the overall status of the Chicago and Cedar Falls networks.  The Network Services Team is aware of routing issues reported by customers in the 74.200.0.0 network.  We are working to identify any common elements of these networks and to identify a solution to these issues.  We appreciate your patience as we work to identify the source of the routing anomalies.  An update will be posted as we have additional information

EVENT UPDATE 22-MAR-10 22:15 CDT:  In an attempt to stabilize the networks in Cedar Falls and Chicago and to restore full connectivity to all IP subnets, the Network Services Team will perform some emergency intrusive hardware testing of one of the devices in Chicago on 23-MAR-2010 at 00:00.  During this time, we expect customer connectivity to experience brief periods of unavailability as the testing progresses, with a total disruption of 20 minutes or less. We appreciate your patience while we complete this emergency work tonight.  An update will be posted as we have additional information

EVENT UPDATE 23-MAR-10 05:30 CDT:  In an attempt to stabilize the networks in Cedar Falls and Chicago and to restore full connectivity to all IP subnets, the Network Services Team performed several intrusive hardware testing procedures on devices in the Chicago network.  The results of these maintenance events suggest that most of the routing anomalies have been identified and resolved.  However, there is a component of variability that exists as a result of our interaction with Internap.  We have escalated these residual issues to Internap management and will pursue them to final resolution. We appreciate your patience during this entire process.  An update will be posted as additional information is available.

EVENT UPDATE 23-MAR-10 10:45 CDT:  As a result of our investigation with Internap, they have discovered and repaired our primary routing connection.  Network Services believes that this is and has been a primary contributor to the routing anomalies we have experienced over the past 36 hours.  Because of our strong desire to recover the service, we will be enabling BGP routing on that repaired circuit in an emergency window in the next 15 minutes.  While this may cause some momentary latency as routes re-converge, this will be a transitory experience and should result in significantly better connectivity for LT customers in Cedar Falls and Chicago.  We appreciate your patience as we work to resolve this issue permanently.

We sincerely apologize for any inconvenience this is causing  and we would like to assure our customers that we are doing our utmost to resolve this issue at soon as possible.