DATE:	April 8, 2010
TO:	Office of Commission Clerk (Cole)
FROM:	Office of Auditing and Performance Analysis (Harvey, Hallenstein) Office of the General Counsel (Brooks)
RE:	Docket No. 090430-TP – Amended petition for verified emergency injunctive relief and request to restrict or prohibit AT&T from implementing its CLEC OSS-related releases, by Saturn Telecommunication Services, Inc.
AGENDA:	04/20/10 – Regular Agenda – Proposed Agency Action – Interested Persons May Participate
COMMISSIONERS ASSIGNED:			All Commissioners
PREHEARING OFFICER:			Argenziano
CRITICAL DATES:			None
SPECIAL INSTRUCTIONS:			None
FILE NAME AND LOCATION:			S:\PSC\APA\WP\090430.RCM.DOC

 

 Case Background

Following the BellSouth and AT&T merger, AT&T began plans to migrate and consolidate the former BellSouth nine-state southeast Operations Support Systems (OSS) platform into a single pre-ordering and ordering OSS platform for use across AT&T’s new 22-state region.[1]  AT&T believes the OSS system used in the pre-merger AT&T 13-state region would produce greater efficiencies for the benefit of both AT&T and its customers throughout the new 22-state region. [2]

As part of the OSS consolidation process, AT&T announced plans to phase-out the Local Exchange Navigation System (LENS) ordering interface currently available for use by CLECs in AT&T’s nine-state region (the former BellSouth region).  The LENS interface will be replaced with another front-end CLEC ordering interface, known as the Local Service Request Exchange System (LEX), currently used in the pre-merger AT&T 13-state region.  The CLEC community was notified of AT&T’s plans to implement LEX in May 2007.   As part of the phase-in process to give the nine-state CLECs time to adequately learn the new LEX interface without disrupting operations, AT&T intended to run the existing LENS ordering interface in parallel until March 20, 2010.

On September 3, 2009, Saturn Telecommunication Services, Inc. (“STS”), a CLEC, filed an Emergency Petition for Injunctive Relief and a Request for Stay of AT&T’s CLEC November 2009 OSS release.  According to STS, the LEX ordering interface does not allow for the same edit-checking capabilities that are currently being provided and made available to CLECs via the LENS ordering interface.  As a result, STS claims that it “will be irreparably harmed by erosion of customer confidence, inability to efficiently add, convert and service its customers on Petitioner’s network, and loss of customers to Respondent [AT&T].”

On September 22, 2009, AT&T filed a Partial Motion to Dismiss and Answer and Affirmative Defenses to STS’ petition.  AT&T asserts that the LEX interface includes additional features that are comparable to, and in some instances exactly like, the LENS interface.  AT&T further denies that STS should be entitled to any relief in its petition.

On October 13, 2009, STS filed an Amended Petition for Injunctive Relief and Request to Restrict or Prohibit AT&T from Implementing its CLEC OSS-Related Releases.  STS claimed that LEX continues to lack adequate edit-checking capabilities.  In its amended petition, STS specifically requested:

a.   A Commission order to restrain or prohibit AT&T from implementing the AT&T 22-State OSS Alignment in November 2009, and/or file an action in circuit court for an injunction, until such time as AT&T can demonstrate through an independent third-party testing that it has provided edit-checking capability substantially equal to what it provide to itself in its retail order system “RNS”;

b.   A Commission order requiring that AT&T Florida cannot retire LENS without this Commission’s approval;

c.   A Commission order requiring LEX to have the same edit-checking, quality, and capabilities as LENS, prior to retiring LENS;

d.   A Commission order requiring AT&T to correct any further deficiencies in LEX and Verigate which may be discovered during the course of these proceedings and as determined by this Commission through testing and otherwise, prior to the Commission’s recommendation for the retirement of LENS;

e.   A Commission order prohibiting AT&T from retiring LENS until this Commission completes an audit of LEX and AT&T corrects all deficiencies found by this Commission;

f.    A Commission order assessing penalties against AT&T pursuant to s. 364.03, Florida Statutes;

g.   A Commission order requiring that AT&T make its LENS OSS with edit-checking capabilities available to STS and other CLECs until any new OSS replacement system contains the same capabilities;

h.   A Commission order requiring that AT&T continue to provide its LENS OSS with all of its current capabilities available until such time as the Commission has verified that AT&T has complied with paragraph (e), and;

i.    A Commission order for costs and for such further relief as the Commission deems just and appropriate.

On October 23, 2009, AT&T filed a Partial Motion to Dismiss and Answer and Affirmative Defenses to STS’ Amended Petition. In its Partial Motion, AT&T asserts the following:

a.   STS’ Petition fails to state a cause of action upon which relief     may be granted.

b.   STS lacks standing to allege violation of a Commission Order to which it was not a party.

c.   The provisions of s. 364.14, Florida Statutes, are not applicable to AT&T Florida and the Commission has no authority to find AT&T Florida in violation of this statute.

d.   Section 364.15, Florida Statutes, is limited solely to the provision of “basic local telecommunications services” and is not applicable to the systems at issue in the Petition.

On October 29, 2009, STS filed its response to AT&T’s Partial Motion to Dismiss STS’ Amended Petition.  In its response in opposition, STS requested that the Commission deny AT&T’s Partial Motion.

The Commission dismissed STS’ requests for penalties and costs to be assessed against AT&T by Order` No. PSC-09-0799-PAA-TP issued December 2, 2009.[3] However, the Commission did grant STS’ Amended Petition in part by ordering AT&T to run the LENS OSS interface in parallel to the LEX interface until completion of a staff audit and a decision by the Commission on this matter.  Furthermore, the Commission found that the remaining requests in STS’ Amended Petition [paragraphs (c.), (d.), (e.), (g.), and (h.)] shall be held in abeyance until staff brings a recommendation back to the Commission after completion of the audit.

This recommendation addresses staff’s audit and whether the Commission should allow AT&T to move forward with the retirement of the LENS OSS interface.  A copy of  staff’s audit report was filed in Docket No. 090430-TP, on May 5, 2010, Document No. 02479.

Jurisdiction

 

The Commission is vested with jurisdiction over this matter pursuant to ss. 364.01(3) and (4)(g), Florida Statutes.  Pursuant to s. 364.01(3), Florida Statutes, the Florida Legislature has found that regulatory oversight is necessary for the development of fair and effective competition in the telecommunications industry.  To that end, s. 364.01(4)(g), Florida Statutes, provides, in part, that the Commission shall exercise its exclusive jurisdiction in order to ensure that all providers of telecommunications service are treated fairly by preventing anticompetitive behavior.  Furthermore, the FCC has encouraged the states to implement performance metrics and oversight for purposes of evaluating the status of competition under the Telecommunications Act of 1996.

 

 

 

 

Discussion of Issues

Issue 1:  Should the Commission allow AT&T to move forward with the retirement of the LENS OSS ordering interface currently used by CLECs in AT&T’s southeast region?

Recommendation:  Yes.  Staff believes that AT&T should be allowed to move forward with the retirement of the LENS interface under the following conditions:

 

1.      AT&T should conduct LEX volume testing in the production environment, or otherwise prove that capacity in the production environment is adequate in the Southeast back-end systems;

 

2.      AT&T should perform a root cause analysis on the reason for the poor flow-through results for the LEX interface and take appropriate corrective action;

 

3.      AT&T should update all appropriate CLEC documentation for commingled orders via the LEX interface; and

 

4.      AT&T, STS, and staff should work together to resolve the specific LEX ordering issues raised by STS in the March 1, 2010 matrix.

 

Staff Analysis:

            Upon implementation of the LEX ordering interface in November 2009, AT&T planned to run the LENS ordering interface in parallel until March 20, 2010.  According to AT&T, this four-month transitional period would provide CLECs with the opportunity to process orders via the new LEX interface.  The transitional period would also attempt to ease any CLEC concerns by keeping the LENS interface operational should any problems occur with the processing of orders through LEX.  However, pursuant to STS’ concerns, the Commission ordered AT&T to continue running LENS in parallel until staff completed an audit to determine the adequacy of  LEX.

            STS’ Position

STS claims that the LEX ordering interface does not have the same edit-checking capabilities as the LENS ordering interface currently in place for use by CLECs in the AT&T nine-state region.  The edit-checking capabilities available in LENS notify the CLEC of errors on a “real-time” basis and do not allow the CLEC to continue processing an order until the error is corrected.  STS further claims that many of these edit checks programmed into the LENS interface are omitted from the LEX interface.  As a result, the omission of these prompts or edit-checks in LEX would, in effect, cause orders with errors to be rejected or returned for clarification by AT&T after the order is submitted.  STS asserts that the delay in processing the order erodes customer confidence and may ultimately result in the loss of the customer.

According to STS, there are as many as 25 edits within LENS that will no longer be provided  in LEX.  Because of the lack of these edit checks in LEX, a CLEC customer service representative must work an order back-and-forth in LEX.  In other words, the LEX system allows for a CLEC order to be submitted to AT&T with errors, rejected by AT&T, reworked by the CLEC, resubmitted by the CLEC, and possibly rejected by AT&T again, over and over.  STS claims that it is highly unlikely for a CLEC to process an order through the system in a timely manner without errors.  The end result is that the CLEC is delayed in submitting a completed order to AT&T which delays the customers’ service from being changed to the CLECs’ network. The delays and multiple customer contacts can potentially be great enough to cause the customer to cancel their order with STS.  As a result, the CLECs' ability to satisfy and retain the end-user will ultimately be affected.

STS further contends that AT&T is in violation of a Commission decision issued in Order No. PSC-98-1001-FOF-TP, issued on July 22, 1998, in Docket No. 980119-TP.  In the Order, the Commission required BellSouth (now AT&T) to modify the LENS ordering interface to provide the same online edit-checking capabilities to Supra Telecommunications (a former CLEC) that BellSouth’s retail ordering systems provide.  STS argues that via implementation of the LEX ordering interface, AT&T is reverting to an ordering process for CLECs that is not in parity with AT&T’s own retail system.

AT&T’s Response

AT&T contends that the implementation of the LEX interface provides the same and/or like functionality as LENS currently provides to the CLECs.  AT&T claims that the new LEX interface is every bit as efficient as the existing LENS interface and will provide for all necessary functionality to create, manage, track, maintain, change, and/or supplement orders.  According to AT&T, the new LEX interface will also provide for a number of enhancements that are not currently available for use by CLECs via the LENS ordering interface.

AT&T did not perform a cost estimate for enhancing LEX to include up-front edit-checking similar to the edit-checking process performed in LENS.  According to AT&T, the costs associated with implementing the upfront edits would include significant changes to the interface architecture, as well as the significant coding effort that would be required to implement many of AT&T’s back-end or downstream OSS interfaces. Additionally, these changes would need to be completed for each of the other regions in AT&T’s operating territory and would take at least three years to fully implement.

In response to STS’ allegations that AT&T is in violation of Order No. PSC-98-1001-FOF-TP, AT&T contends that the Order has no relevance to this proceeding as neither STS nor any other CLEC (besides Supra) was a party to Docket No. 980119-TP.   AT&T further noted that other CLECs may have second-handedly received some benefit from the Supra Order, yet there is nothing in the Order that suggests it applies to CLECs that were not parties to the 1998 proceeding.

 

 

Staff’s Audit

Pursuant to the first Order addressing STS’ Amended Petition, Order No. PSC-09-0799-PAA-TP, the Commission ordered AT&T to run LENS OSS interface in parallel to the LEX interface until completion of a staff audit.  The primary objectives of staff’s audit were to:

1.   Review and document the history of the edit-checking capability as a requirement in Florida;

2.  Review the nondiscriminatory access and operational readiness decisions regarding LEX;

 

3.  Document AT&T’s pre-ordering, ordering, and editing processes and assess the LEX and LENS functionality;

 

4.  Document and assess any additional LEX issues or deficiencies discovered during the audit; and

 

5.  Conduct a survey of Florida CLECs that are currently using LEX to determine user satisfaction.

 

Below are the summaries and the conclusions for each objective.

 

1.  Review and document the history of the edit-checking capability as a requirement in Florida.

 

In 1997, the Commission identified several BellSouth (now AT&T) OSS related problems.  One specific problem was that LENS interface did not have edit-checking capabilities at parity with BellSouth’s retail systems.  This problem, among several others, resulted in the Commission’s denial of BellSouth’s application for Section 271 approval of the 1996 Telecommunications Act.[4] In 1998, the edit-checking issue was raised by Supra Telecommunications.  The Commission found that without edit-checking, again the LENS interface was not at parity with the BellSouth retail system and ordered BellSouth to implement edit-checking capabilities.[5] 

 

In 2000, Supra Telecommunications raised the edit-checking issue another time and believed that BellSouth still had not implemented the edit-checking capabilities.[6] The Commission ordered that the determination on whether BellSouth had implemented the edit-checking capability be delayed until completion of independent third-party testing of BellSouth’s OSS for purposes of Section 271 approval.[7]^,[8]  Upon completion of the third-party testing, the Commission found that BellSouth was providing nondiscriminatory access to its OSS, including the LENS interface.[9]  Nondiscriminatory access means that BellSouth provides CLECs access to ordering functionalities in substantially the same time and manner as BellSouth’s retail systems. Additionally, upon approving Bellsouth’s 271 application, the Federal Communications Commission (FCC) agreed that BellSouth provides nondiscriminatory access to its OSS.[10]

 

2.  Review the nondiscriminatory access and operational readiness decisions regarding LEX.

 

Since the LEX interface is replacing LENS, it is important to assure Florida CLECs, as well as the Commission, that LEX also provides nondiscriminatory access to AT&T’s OSS.  To provide assurance, staff examined the following areas:

 

·        Independent third-party testing conducted in other states on the LEX interface for 271 approval;

 

·        AT&T and CLEC preproduction testing of LEX for the November 2009 release;

 

·        LEX CLEC impacting defects occurring since the November 2009 release; and

 

·        Florida aggregate performance measurement data for ordering metrics for LEX.

 

Staff reviewed the OSS independent third-party testing done on the LEX interface in three other states to determine if LEX had historically been deemed as providing nondiscriminatory access to AT&T’s OSS. [11]  Staff’s review revealed that, at the conclusion of testing, the Michigan, California, and Texas State Commissions and the FCC found that nondiscriminatory access was being provided to the OSS, which included the LEX interface.

 

In order to make a determination on operational readiness, staff reviewed the November 2009 LEX pre-production test plans and results, post-production defects, and commercial data for the month of January 2010.  Staff was satisfied with pre-production testing, with the exception of volume testing.  Staff believes that the lack of volume testing in the production environment represents a possible risk for AT&T.  Staff is concerned that once all CLECs have migrated to LEX the back-end system may not be able to effectively respond to CLEC inquiries and orders.  Previous OSS volume testing for Florida, as well that done in other states for Section 271 approval was done in the production environment.  Staff believes AT&T should conduct LEX volume testing in the production environment or otherwise prove that capacity in the production environment is adequate in the Southeast back-end systems.

 

Staff’s review of the post-production defects associated with the implementation of the LEX interface revealed 33 defects.  As of March 15, 2010, 20 of the defects were considered resolved and closed-out by AT&T and 13 remained open for resolution.  AT&T indicated to staff that all open defects will be resolved by April 17, 2010.

 

Finally, staff’s review of LEX aggregate performance measurement data shows that for January 2010, AT&T appears to be providing service at parity with the LEX interface, with the exception of flow-through results.  Flow-through is an indicator of electronic orders that are designed to “flow-through” AT&T’s OSS for processing without manual intervention. Performance results indicate that a large percentage of LEX orders that were designed to flow-through did fall-out for manual handling.  Staff believes AT&T  should  perform a root cause analysis on the reason for the poor flow-through results for the LEX interface and take appropriate corrective action.

 

3.  Document AT&T’s pre-ordering, ordering, and editing processes and assess the LEX and LENS functionality.

 

Staff’s audit of the LEX and LENS interfaces revealed two key differences in the systems; the operational structure and the edit-checking process. The LENS operational structure is linear and requires a user to enter data in certain fields before being allowed to move onto the next page or screen.  In contrast, The LEX interface was developed and  structured as a navigation tree that allows the user to move freely around the ordering process through the use of icons and directory files. This process-flow allows the user to determine the sequence of completing an order. 

With regards to the edit-checking process, LENS generates error messages while a user is populating an LSR and will further require the error to be corrected before the user can move forward in processing an order.  In other words, if a CLEC sales representative incorrectly inputs data for a field in the LSR, LENS will immediately alert the representative (while still on the phone with the customer) of the error.  In LEX, the user is not informed of errors on the LSR until after the LSR has been issued to AT&T.  The CLEC sales representative is required to return to the applicable page or screen to correct the error and then resubmit the corrected LSR. 

Staff believes the operational structure of the LEX interface is not as user-friendly when compared to the LENS linear process and also believes the edit-checking process performed in LEX may cause some delays in the overall time to complete an order when compared to LENS. However, staff believes that AT&T has provided CLECs with adequate time to learn and train their personnel on the new LEX interface without disrupting CLECs ongoing operations.  CLECs in AT&T’s southeast region have had the opportunity to transition from the LENS interface and to the LEX interface since November 2009 (over four months).  Additionally, AT&T’s Local Service Center representatives are also available to help CLECs on a variety of issues when transitioning from the LENS to the LEX interface.

Staff further notes that a primary purpose of the implementation of the LEX interface in the Southeast region is to provide uniformity to the OSS systems across the AT&T 22-state region. Staff believes, the 22-state LEX interface does include a number of significant enhancements that provide for more functionality to create, manage, or change LSRs.  While the methods used to execute a function differ between the LENS and LEX applications, staff believes that LEX provides the same desired end-result as LENS regardless of how executed.  Staff believes that LEX and LENS generally provide like functionality.

4.  Document and assess any additional LEX issues or deficiencies discovered during the audit.

 

Staff discovered an additional concern regarding STS’ ability to order commingled arrangements. Commingled arrangements are the ordering of Unbundled Network Elements (UNE) and UNE combinations commingled with special access services.[12]  STS provided a detailed matrix to staff that documents the various  types of service requests that STS believes it will be ordering via LEX.  STS further provided comments or concerns relevant to each type of service order.  Given the complexity and quantity of information provided by STS, staff believes that it is necessary for AT&T, STS, and staff to work together and conduct regularly scheduled conference calls to resolve the issues raised in the matrix.

5.  Conduct a survey of Florida CLECs that are currently using LEX to determine user satisfaction.

 

Staff surveyed Florida CLECs that are currently placing orders over the LEX interface. There are currently 24 Florida CLECs that have been authorized by AT&T to use LEX, yet only 14 reported actually using LEX.  Of the 14 CLECs currently using LEX, staff asked each to compare LEX and LENS with regards to ease of use, timeliness, and functionality.  In sum, the survey results indicated that LEX is harder to use, yet the timeliness and functionality are about the same as LENS.  Staff notes that with only 14 Florida CLECs currently using the LEX interface, staff believes that Florida CLECs, as a whole, have not taken advantage of the four-month transitional period to fully implement the LEX interface.  Staff believes that delaying the retirement of LENS would further encourage CLECs to delay the transitioning from LENS to LEX.

      Conclusion

 

            Staff believes that AT&T should be allowed to move forward with the retirement of the LENS interface under the following conditions:

 

1.   AT&T should conduct LEX volume testing in the production environment or otherwise prove that capacity in the production environment is adequate in the Southeast back-end systems;

 

2.   AT&T should perform a root cause analysis on the reason for the poor flow-through results for the LEX interface and take appropriate corrective action;

 

3.   AT&T should update all appropriate CLEC documentation for commingled orders via the LEX interface; and

 

4.   AT&T, STS, and staff should work together to resolve the specific LEX ordering issues raised by STS in the March 1, 2010 matrix.

 

 

 

 

 

Issue 2:  Should this docket be closed?

Recommendation:  Yes. If the Commission approves staff’s recommendation in Issue 1, staff recommends that the docket be closed administratively once AT&T has provided documentation that the conditions identified in issue 1 have been met.  Staff further recommends that once staff has verified that AT&T has met the conditions, AT&T may retire LENS after a 14 day notice period. Additionally, the remaining requests in STS’ amended petition that were placed in abeyance have either been resolved or rendered moot, upon the Commission’s approval of staff’s recommendation in Issue 1.

 

            The resulting decision to approve issue 1 and close the docket will be issued as a Proposed Agency Action.  The decision will become final upon issuance of a Consummating Order, if no person whose substantial interests are affected timely files a protest within 21 days of the issuance of the Order. (Brooks)

 

Staff Analysis: In Order No. PSC-09-0799-PAA-TP, the Commission placed all remaining requests in STS’ Amended Petition in abeyance until staff completed its audit and brought forth a recommendation addressing the findings.  Order No. PSC-09-0799-PAA -TP also set forth that the close of this docket is contingent upon resolution of the remaining STS requests. If the Commission approves staff’s recommendation in Issue 1, staff believes that STS’ remaining requests are either resolved or rendered moot.

 

            The following requests are resolved, upon approval of staff’s recommendation in Issue 1, because the Commission will have found that LEX’s editing, quality and capabilities are the functional equivalent of the LENS interface:

             

·     A Commission order requiring that LEX has the same pre-order edits and that is has the same quality and capabilities as LENS, prior to retiring LENS,

 

·      A Commission order requiring AT&T to correct any further deficiencies in LEX and Verigate which may be discovered during the course of these proceedings and as determined by this Commission through testing and otherwise, prior to the Commission’s recommendation for the retirement of LENS,

                                                                                       

            The following requests will be rendered moot, upon approval of staff’s recommendation in Issue 1, because the Commission will have found that AT&T has continued to make LENS available to the CLECs throughout the time period staff conducted its audit.

 

·       A Commission order prohibiting AT&T from retiring LENS until this Commission completes an audit of LEX and Verigate and AT&T corrects all deficiencies found by this Commission,

 

·      A Commission order requiring that AT&T make its LENS OSS with its edit checking capabilities available to STS and other CLECs until any new OSS replacement system contains the same capabilities, and;

 

·      A Commission order requiring that AT&T continue to provide its LENS OSS with all of its current capabilities available until such time as the Commission has verified that AT&T has complied with paragraph (e) [of STS’ amended petition].

 

Conclusion

            If the Commission approves staff’s recommendation in Issue 1, staff recommends that the docket be closed administratively once AT&T has provided documentation that the conditions identified in issue 1 have been met.  Staff further recommends that once staff has verified that AT&T has met the conditions, AT&T may retire LENS after a 14 day notice period.

 

            The resulting decision to approve issue 1 and close the docket will be issued as a Proposed Agency Action.  The decision will become final upon issuance of a Consummating Order, if no person whose substantial interests are affected timely files a protest within 21 days of the issuance of the Order.