DATE:	August 7, 2008
TO:	Office of Commission Clerk (Cole)
FROM:	Division of Regulatory Compliance (Harvey, Hallenstein) Office of the General Counsel (Teitzman)
RE:	Docket No. 000121A-TP – Investigation into the establishment of operations support systems permanent performance measures for incumbent local exchange telecommunications companies. (AT&T FLORIDA TRACK)
AGENDA:	08/19/08 – Regular Agenda – Proposed Agency Action  – Interested Persons May Participate
COMMISSIONERS ASSIGNED:			All Commissioners
PREHEARING OFFICER:			Carter
CRITICAL DATES:			None
SPECIAL INSTRUCTIONS:			None
FILE NAME AND LOCATION:			S:\PSC\RCP\WP\000121A.RCM.DOC

 

 Case Background

By Order No. PSC-01-1819-FOF-TP, issued September 10, 2001, in Docket No. 000121A-TP, the Commission adopted a Performance Assessment Plan for the purpose of monitoring performance levels of Operations Support Systems (OSS) provided to CLECs. The performance measurement plan provides a standard against which CLECs and the Commission can measure performance over time to detect and correct any degradation of service provided to CLECs.  AT&T’s performance measurement plan also includes a Self-Effectuating Enforcement Mechanism (SEEM) remedy plan. The Order also recognizes the Commission’s vested authority, per section 364.01(3), Florida Statutes, to provide regulatory oversight necessary to ensure effective competition in the telecommunications industry. This docket has remained open since that time to address issues and concerns arising from OSS performance.

 

Following the BellSouth and AT&T merger, AT&T began plans to consolidate the operations support systems (OSS) of the two companies.  AT&T started the process of providing official notification to CLECs of its OSS consolidation plans (known as the 22-State OSS Release) over a year ago.   The 22-State OSS Release plan involves a phased-in approach.  The first phase commenced with the April 19, 2008 release (“April Release”).

 

On May 12, 2008,  Cbeyond Communications, LLC (Cbeyond), Time Warner Telecom, LP (TWTC), and DeltaCom, Inc. (Deltacom), jointly referred to as the petitioners, filed a petition with the Commission requesting an audit of the April Release.  The petition also requested a stay of CLEC-impacting OSS Releases and that the Commission show cause AT&T to explain in detail the circumstances surrounding the April Release and explain why AT&T should not be penalized for its failure to appropriately implement the April Release.

 

On May 15, 2008, Commission staff held two informal workshops with AT&T, the petitioners,  and other participating CLECs to discuss  issues surrounding the current and future OSS releases.  During the meetings, AT&T acknowledged that a variety of CLEC-impacting issues arose in connection with the April Release. Furthermore, AT&T made several commitments to address and resolve the issues and to prevent similar occurrences in future OSS releases.  AT&T memorialized the list of commitments in a filing with the Commission on May 27, 2008.

 

On June 2, 2008,  AT&T filed a response to the CLECs’ May 12, 2008 Complaint.  In the response, AT&T attached the list of commitments to resolve the remaining issues associated with the April Release.  Included in the list is the temporary suspension of future 22-State OSS releases, expanded customer support and communications, expanded testing for future OSS releases, and providing proactive billing adjustments related to the April Release. AT&T stated that it has used, and continues to use, best efforts to resolve such issues, and it expects to resolve any known, remaining April Release issues by the end of June 2008.  AT&T requested that the Commission issue an order dismissing the CLECs’ petition.[1]

 

On June 5, 2008 the petitioners filed a response requesting that the Commission deny AT&T’s request to dismiss.  The petitioners argue that AT&T’s request to dismiss fails to state a cause of action for which relief can be granted.

 

This recommendation addresses whether the Commission should accept the stipulation between AT&T and the petitioners.

 

 

 

Jurisdiction

 

The Commission is vested with jurisdiction over this matter pursuant to Sections 364.01(3) and (4)(g), Florida Statutes.  Pursuant to Section 364.01(3), Florida Statutes, the Florida legislature has found that regulatory oversight is necessary for the development of fair and effective competition in the telecommunications industry.  To that end, Section 364.01(4) (g), Florida Statutes, provides, in part, that the Commission shall exercise its exclusive jurisdiction in order to ensure that all providers of telecommunications service are treated fairly by preventing anticompetitive behavior.  Furthermore, the FCC has encouraged the states to implement performance metrics and oversight for purposes of evaluating the status of competition under the Telecommunications Act of 1996.

 

Discussion of Issues

 

 

Issue 1: 

 Should the Commission accept the stipulation “Agreement Regarding Audit of AT&T Florida’s April OSS Release”?

Recommendation: 

 Yes. Upon review of the parties’ stipulation, staff recommends the Commission accept the stipulation regarding the audit of AT&T Florida’s April OSS Release as set forth in Attachment 1 (Harvey, Hallenstein)

Staff Analysis: 

 On April 19, 2008, AT&T implemented a software release titled Release 27.1 (April Release) in the nine state former BellSouth region as part of AT&T’s 22-State OSS consolidation plan.  Since April 19, 2008 the petitioners and numerous other CLECs, have experienced a severe impact in their ability to interface with AT&T’s OSS.  Specifically, the CLECs reported that they had not received order confirmations, requests for order clarifications, disconnection notices, rejection notices, and communications related to meetings at the customer premises for installations for a period of time following the release. CLEC orders submitted to AT&T during this timeframe were adversely affected by this release.

At a May 7, 2008, AT&T Change Control Process meeting between AT&T and participating CLECs, AT&T  admitted that problems occurred with the April Release. At the meeting, AT&T provided a detailed chronology of events that occurred the first two weeks after the April release. AT&T acknowledged that outgoing transactions, clarifications and rejections were not being delivered to CLECs.  AT&T also acknowledged that a backlog of orders in the manual processing center occurred due to outages and instability of the system used by the AT&T representatives in the company’s manual processing center. In subsequent Change Control Process meetings, AT&T identified additional issues associated with the release.  Most notably over 200,000 billing completion notices and 12,000 completion notices had not been properly delivered to CLECs in the region.

On May 15, 2008, AT&T participated in two informal workshops with the Commission staff to discuss issues surrounding the April Release and AT&T’s plans for future OSS Releases in the Southeast region. At the workshop, and in response to the petition, AT&T acknowledges that a variety of CLEC-impacting issues arose in connection with the April Release in the Southeast region and that such issues primarily impacted CLECs’ ability to timely receive outbound transmissions from AT&T in certain situations. AT&T asserts that it has used, and continues to use best efforts to resolve such issues, and expected to resolve any known, remaining April Release issues in June.  AT&T also asserted that it would continue to communicate the status of such issues to all CLECs in a timely and informative manner.

At the workshops, AT&T estimated that 71,000 orders in the nine state region were negatively affected by the April Release.  Of these orders,  AT&T stated that 59,000 were electronic orders, and 11,000 were backlogged manual orders.  The CLEC petitioners describe this April Release as the most significant competitively damaging OSS failure in the State of Florida since enactment of the Telecommunication Act of 1996. 

In a June 30, 2008  response to a staff data request, AT&T reported that there have been 229 identified software defects open from the April Release, 125 of which have had a direct impact on CLECs.  Below is a chart detailing the severity of the April Release defects.

April Release Defect Status As of  June 30, 2008

	1-Critical	2-Major	3-Average	Total
CLEC Impacting Defects	58	41	26	125
Non-CLEC Impacting Defects	31	53	20	104
Total CLEC/Non-CLEC Impacting Defects	89	94	46	229

 

In response to the May 15, 2008  workshop, AT&T provided a list of commitments to resolve the open issues associated with the April Release. At staff’s request, AT&T memorialized the commitments made during the May 15th meeting.  A copy of the AT&T “commitment” letter is shown in Attachment 2.  Staff acknowledges AT&T for their willingness to step forward with these commitments. AT&T states that the disruption to the normal processing of CLEC ordering was the result of a series of unexpected events including hardware, software, and network issues. AT&T contends that there was no total breakdown of AT&T’s OSS. AT&T intends to address and resolve the April Release issues and implement process improvements that should reduce the likelihood of issues of this magnitude arising in connection with future 22-State OSS Releases in the Southeast. AT&T also committed to keeping the Commission Staff and interested CLECs informed of its efforts.

In the current petition, the CLECs requested an independent audit be conducted that focuses on the cause of the OSS failures.  In its June 2, 2008 Response, AT&T denies that an independent audit of the April Release is necessary. AT&T states that it has provided, and will continue to provide, information and explanations regarding the April Release, and has fully supported CLEC requests for status, escalation, and assistance. AT&T also states that it will continue to respond to CLEC inquiries through individual customer support and weekly conference calls opened to all CLECs.[2] AT&T asserts that its internal review and expanded test plan will include any necessary root cause analysis of the April Release issues.

A conference call was held between staff and the parties on July 31, 2008 exploring the possibility that the audit be conducted by Commission staff.  After further discussions, on August 5, 2008, the parties agreed to the audit being conducted by Commission staff and entered into the attached stipulation (Attachment 1).   The scope of the audit would be to:

·       Document and assess AT&T’s root cause analysis of the April Release.

·       Document and assess AT&T’s defect resolution associated with the April Release.

·       Document and assess pre-release and post-release communications.

The stipulation also states the petitioners will file a Motion to Dismiss the request for an independent audit and request the Commission to hold the remaining portions of the complaint in abeyance pending the Commission’s acceptance of staff’s recommendation addressing the final audit report.[3]  Following the filing of this Motion, AT&T  agrees to refrain from implementing future 22-State OSS releases until the earlier of the Commission vote on the final audit report and staff’s recommendation or a mutually agreeable timeframe.[4] Additionally, the stipulation states that AT&T shall still be accountable for its commitments made in Attachment 2.

Conclusion:  Upon review of the parties’ stipulation, staff recommends the Commission accept the stipulation regarding the audit of AT&T Florida’s April OSS Release as set forth in Attachment 1.

Issue 2: 

 Should this docket be closed?

Recommendation: 

 No. If the Commission approves staff’s recommendation in Issue 1, the resulting Order will be issued as Proposed Agency Action.  The Order will become final upon issuance of a Consummating Order if no person whose substantial interests are affected timely files a protest within 21 days of the issuance of the Order. This docket should remain open pending the conclusion of the audit and for purpose of future performance measure monitoring. (Teitzman)

Staff Analysis: 

 If the Commission approves staff’s recommendation in Issue 1, the resulting Order will be issued as Proposed Agency Action.  The Order will become final upon issuance of a Consummating Order if no person whose substantial interests are affected timely files a protest within 21 days of the issuance of the Order. This docket should remain open pending the conclusion of the audit and for purpose of future performance measure monitoring.

 

 

 

 

 

 

 

 

 

 

FLORIDA COMMISSION DOCKET NO. 000121A-TP

FOLLOW UP TO MAY 15, 2008 MEETINGS

 

As discussed during the meetings held on May 15, 2008, with the Commission Staff and interested competitive local exchange carriers (“CLECs”), AT&T remains committed to resolving the remaining issues associated with the April OSS Release.  To address future 22 State Releases in the Southeast “(SE”) region, AT&T will provide the following:

 

I.                 Temporarily suspend next three planned releases for implementing 22 State Pre-Order and Ordering capabilities into the SE region (“scheduled or future 22 State Releases”) until:

o      Resolve Severity 1 and Severity 2 defects in the SE region resulting from the April 2008 OSS release.

o      Expand Pre Release, customer training and Post Release communications as outlined below.

o      Develop an expanded Test Plan to address defect issues resulting from the April Release and include in future release testing.

 

II.               April Release Defect Resolution

 

o      Severity 1 defects identified as of our 5/15/08, with the exception of defect #183638 were fixed by 5/19/08 as committed. Defect #183638 was fixed on 5/21/08.

o      Fixes for Severity 1 and 2 defects in the SE region resulting from the April 2008 OSS release and in existence as of 5/19/08 will be in production by 6/21/08.  If there is any change in this schedule, AT&T will notify Staff and CLECs.

o      The following status relates to the transmittal of Line Loss Notifications (LLN) and Billing Completion Notices (BCN):

·       LLNs – All outstanding notifications were transmitted successfully on 5/16/08. Normal transmission resumed on 5/17/08.

·       BCNs – Issues with the daily transmission of notifications were resolved on 5/16/08. Transmittal of outstanding notifications began on 5/20/08 using a metered approach;  expect completion of the backlog by 6/2/08.

 

III.             Expanded Communications

 

o      Maintain recurring status calls with customers until the earlier of the resolution of Severity 1 and 2 defects resulting from the April OSS release or consensus that calls are no longer necessary.

 

o      Continue to status plans for currently scheduled 22 State releases within existing monthly CMP/CCP Meetings.  Enhance clarity of pre-release communications by providing a review of all systems and customer interface changes included in future 22 State releases in advance of Accessible Letter communications.  Use this input to improve the clarity of Accessible Letter information.

 

o      Outline CLEC training plans and materials for future 22 State releases.  Take into consideration customer input prior to finalization of such training.  Release CLEC training materials in accordance with CMP/CCP timeframes.

 

o      In addition to standard communications, Post Release communications for scheduled 22 State releases will be expanded as follows:

 

§       Recorded messages will be made available during Release Implementation Weekends reporting on current status and “go/no go” readout.

§       A virtual “War Room” will be established during the initial three days after scheduled releases to update customers of any Post Release issues.  Daily calls can be expanded/extended as necessary.

 

o      Provide a single document that clearly describes roles/responsibilities/titles for the following AT&T personnel to assist with more effective customer contact and escalation points:

§       Wholesale Customer Support Managers

§       Information Services Call Center

§       Mechanized Customer Production Support Center  (MCPSC)

 

o      Perform internal documentation review to enhance clarity of the following communications:

§       Web-based Defect Reporting (EDR Report) – Updated beginning 5/15 and ongoing.

§       XML Documentation – Corrected for identified issue with reject reason field to restore to pre-release length of 5 characters.

§       Systems Outages Notifications

§       Will take into consideration comments received from customers to date and cover results with CLECs once completed.

 

IV.            Expanded Testing/Backout/Failure Management for scheduled 22 State releases

o      Utilize root cause analysis of release defects to expand Testing Plans with special focus in the area of delivering outbound transactions. Specific tests will be established for validating that outbound transactions such as FOCs, Clarifications, Completion Notices and Billing Completion Notices are delivered to their destination point in a form compatible with existing standards and interface agreements.

o      Encourage and support greater CLEC participation in cooperative testing for all releases, utilizing existing CLEC test environments.

o      Evaluate manual process to determine what steps can be taken to test the process and allow CLECs the opportunity to practice for new forms/templates.

o      Going forward, scheduled 22 State releases for the SE region will provide overlap between the existing and new Customer Interfaces (EDI, XML and Verigate/LEX/LENS), in order to allow customers to plan/test/develop individual migration strategies to the new interfaces.

o      Testing will include appropriate back out plans for the implementation weekend.

o      Prior to next scheduled 22-State release outline post-release action plan to provide expedited response in the event of critical release failures (Emergency Plan), focused on:

·       Customer Notification and Support

·       Defect Resolution

·       Expanded AT&T Staffing requirements

 

 

V.              Email/Manual Forms Process

 

Support/Education

o      Continue proactive, individualized customer support on Email/Manual Forms process for next 60 days to facilitate user introduction.

·       Including customer working sessions to assist in successful submission of the Manual LSR Forms through use of the email process. 

o      Lead a monthly Email/Manual Forms User Forum to provide common support and address current manual process issues until all 22 State releases are completed.

o      Continue to work with all CLEC customers who request further assistance or education on the manual LSR ordering process.

 

 

Review of Manual Processes and CLEC Change Requests

o      Assessed CLEC concerns where all pages of a form are required whether or not all pages contain data.

·       The manual ordering process only requires the customer to populate fields necessary for the product being ordered. 

·       To address customer concerns regarding the requirement of additional data when submitting forms:

o      Accessible Letter CLECSE08-077 dated 05/21/08 provided additional information within the Manual Ordering Guidelines and the Frequently Asked Questions (FAQ) documents.

o      Will review and assess the prioritized list of customer change requests for enhancements to the 22 State Email LSR Process.  In addition, will re-evaluate merger related OSS Change Requests previously submitted through the Change Management process.

 

Documentation

o      Completed updates to the LOH to reflect all changes noticed through Accessible Letter issued as a result of the April 19, 2008 release. Will continue to review to ensure clarity, currency and completeness.

o      Proactive review and update of the 22 State Manual LSR Forms and Email Process documentation was done to address customer feedback. Documentation posted to CLEC Website and available 05/21/08:

§       Enhanced 22 State Non-Mechanized Ordering Guide, now renamed as “Manual Ordering Guidelines”.

§       Published Frequently Asked Questions (FAQ) document

o      Expand documentation quality control processes to ensure multiple layers of review prior to release of documents to the customer.

 

VI.            Proactive Billing adjustments

 

A call has been scheduled for May 28, 2008 with CLEC customers to discuss the plan for providing generic bill adjustments and addressing billing issues related to the release.  Plan entails three key steps: (1) Proactive Billing Adjustments, (2) Claims Clean-up Process for addressing and resolving any exceptions that are not addressed in the Proactive Approach and (3) Communication Plan for the CLECs. Key points regarding the Proactive Billing Adjustments part of the plan include:

 

·       Billing Solution has been developed to identify areas were CLEC billing has been impacted by the Release issues and to proactively provide billing credits, where possible.

·       Proactive billing credits will be triggered by the Billing Completion Notices that are currently being sent in batches starting on 5/20/08.

·       Billing Solution is designed to address the most common CLEC concerns on both NRC and MRC billing resulting from the April Release.

 

o      Target timeline will be to process these proactive adjustments within 45 business days after the sending of the Billing Completion Notice are sent, with a goal to process as many of these proactive adjustments as possible within the first 30 days.

 

 

VII.          Center /Support Team Responsiveness

 

o      All April Release related calls temporarily handled by other service centers outside of the SE Region, were moved back into the Birmingham LSC by May 19, 2008.

o      As of the Close of Business on May 24, 2008, the  Birmingham LSC is current with all order processing.  Center is now processing orders within standard performance commitments.

o      Continue to manage April Release related expedites for the processing of acknowledgements for any individual situations if not addressed by the mechanized transmittal efforts.

o      Continue to review staffing levels to meet anticipated demand, including:

·       Wholesale Customer Support Managers

·       Centers