DATE:	February 2, 2012
TO:	Office of Commission Clerk (Cole)
FROM:	Office of Auditing and Performance Analysis (Harvey, Hallenstein) Office of the General Counsel (Robinson)
RE:	Docket No. 090430-TP – Amended petition for verified emergency injunctive relief and request to restrict or prohibit AT&T from implementing its CLEC OSS-related releases, by Saturn Telecommunication Services, Inc. d/b/a Earthlink Business.
AGENDA:	02/14/12 – Regular Agenda – Proposed Agency Action – Interested Persons May Participate
COMMISSIONERS ASSIGNED:			All Commissioners
PREHEARING OFFICER:			Edgar
CRITICAL DATES:			None
SPECIAL INSTRUCTIONS:			None
FILE NAME AND LOCATION:			S:\PSC\APA\WP\090430.RCM.DOC

 

 Case Background

The Telecommunications Act of 1996 requires AT&T to provide Competitive Local Exchange Companies (CLECs) with non-discriminatory access to its Operations Support Systems (OSS) on appropriate terms and conditions.  Operations Support Systems are the computer systems used by AT&T that support the ordering, provisioning, maintenance, and billing of services for CLECs.  As part of the BellSouth and AT&T merger in 2006, AT&T began migrating and consolidating the former BellSouth nine-state southeast Operations Support Systems platform into a single pre-ordering and ordering OSS platform for use across AT&T’s new 22-state region.

 Pursuant to Order No. PSC-10-0253-PAA-TP, issued April 26, 2010, the Commission, allowed AT&T, under certain conditions, to move forward and retire the Local Exchange Navigation System (LENS) ordering interface and replace it with the Local Service Request Exchange System (LEX) ordering interface. The LENS interface was made available for use by CLECs in AT&T’s pre-merger nine-state region (the former BellSouth region).  AT&T replaced LENS with LEX to be used across the post-merger 22-state region.

            One condition the Commission imposed in allowing the replacement of the LENS ordering interface was that AT&T, staff, and Saturn Telecommunication Services, Inc. (“STS”) should collaborate to resolve specific LEX ordering issues raised by STS during the course of an audit conducted by staff.[1]  STS provided a detailed matrix to staff that documented 61 specific issues concerning service requests that STS will be ordering through the new LEX interface. The matrix identified concerns with the applicable AT&T ordering requirement guides or business rules that STS must follow to correctly place orders.  One issue within the matrix remains unresolved between the parties. This one remaining issue is a matter of organization and presentation of instructional information for CLEC LEX users attempting to provide service to their end-users in an expeditious manner.

 

Prior to the consolidation of AT&T and BellSouth’s business operations, BellSouth had maintained a set of tables, known as the Required/Conditional/Optional (“R/C/O”) tables, within a set of business rules, known as the Local Ordering Handbook.  The tables provided a “snapshot” view of the necessary steps CLECs must take to accurately populate information fields when placing orders. In November 2009, AT&T began integrating the Local Ordering Handbook into another set of business rules, called the Local Service Order Requirement guide. The new Local Service Order Requirements would be available for use by all CLECs that operate within AT&T’s 22-state region.  In doing so, the R/C/O tables were replaced with “Product Activity tables.” STS contends that the organization and use of the Product Activity tables are inefficient, time consuming, and places STS at a competitive disadvantage.

STS’ concerns are supported by DeltaCom, Inc., XO Communications, Cbeyond, LLC, and TW Telecom.  Each of these CLECs filed letters in this docket on June 1, 2011 that echoes the concerns expressed by STS regarding the need for R/C/O tables and supports STS's position concerning the R/C/O tables.

After the parties failed to resolve the remaining issue, on July 28, 2011, staff filed a recommendation for the August 9, 2011 Commission Agenda Conference.  However, just prior to the scheduled Commission Agenda Conference, on August 5, 2011, STS filed an Unopposed Emergency Request to defer staff’s recommendation for the Commission’s Agenda Conference scheduled for August 9, 2011.  In its request, STS stated that both STS and AT&T would confer in good faith on the matter in order to attempt an amicable resolution without further Commission action, other than administrative procedures to close the case. 

In November 2011, STS notified staff that both STS and AT&T were unable to reach a settlement in good faith and requested that the Commission address this issue at the Commission’s Conference Agenda.

On December 13, 2011, staff conducted an informal conference call with the parties to discuss the status of this issue and current processes used by STS to accurately populate information fields when placing orders.

This recommendation addresses whether AT&T should continue to maintain and update the R/C/O tables.

Jurisdiction

 

            In 2009, when this docket was opened, the Commission was vested with jurisdiction over this matter pursuant to Sections 364.01(3) and 364.01(4)(g), Florida Statutes (1997) (F.S.).  Pursuant to Sections 364.01(3) and 364.01(4)(g), F.S. (1997), the Florida Legislature found that regulatory oversight was necessary for the development of fair and effective competition in the telecommunications industry.

 

            Effective July 1, 2011, pursuant to Legislative reforms of Chapter 364, F.S. (1997), the Commission’s jurisdiction as delineated in 2009 changed.  However, pursuant to Section 364.16, F.S. (2011), the Legislature reaffirmed its intent for this Commission to continue its regulatory oversight of carrier-to-carrier relationships to provide for the development of fair and effective competition.  Accordingly, despite the Legislative changes to Section 364.01, F.S. (2011), staff believes Section 364.16, F.S. (2011), still grants the Commission jurisdiction over this matter.

 

Discussion of Issues

Issue 1: 

  Should AT&T be required to update the Required/Conditional/Optional (R/C/O) tables?

Recommendation: 

  Yes. Staff recommends that AT&T should update the “required” information fields contained within the Required/Conditional/Optional (R/C/O) instructional tables to reflect any changes AT&T has made to the LEX ordering interface between November 2009 and March 2012.  This update should be performed one time and should be made available to STS within 60 days of the issuance of the Consummating Order (when PAA becomes final).  The update should be in a Microsoft Word file and formatted in the same manner as the R/C/O tables contained within AT&T’s retired Local Ordering Handbook.  Additionally, AT&T should work with STS over the next six months to reduce the number of order rejections STS experiences. (Harvey, Hallenstein)

Staff Analysis: 

 The Commission, in its continuing oversight role of AT&T’s operations support systems (OSS), has authority to prevent anticompetitive behavior among telecommunications providers. Pursuant to Order No. PSC-10-0253-PAA-TP, issued April 26, 2010,[2] the Commission allowed AT&T to move forward with the implementation of a new 22-state LEX OSS interface under certain conditions detailed in a Commission staff audit filed in this docket on May 5, 2010.  All the conditions in the audit report, with the exception of one, were satisfied. 

The one remaining condition pertains to AT&T, STS, and staff collaborating to resolve 61 specific LEX ordering issues raised by STS in a matrix provided to Commission staff on March 1, 2010.  Given the complexity and quantity of issues provided by STS in the matrix, AT&T, STS, and staff initiated a series of weekly calls to discuss the specific LEX ordering issues raised.  By November 2010, AT&T, STS, and staff were able to work together to resolve all issues within the matrix with the exception of the following:

·         The R/C/O tables contained within AT&T’s Local Ordering Handbook will not be retained in the same format when converted to AT&T’s Local Service Order Requirements (LSOR).

 

            AT&T’s Business Rules

 

When ordering products and services from AT&T through the LEX interface, STS and other CLECs that operated in the pre-merger nine-state region relied on an electronic set of instruction manuals or business rules to guide them through the ordering process.  These business rules were known as the Local Ordering Handbook.  The handbook was divided into three sections and the specific instructions STS uses to place orders was contained within section three.  Section three alone is approximately 600 pages. The ordering handbook was arranged and navigated through a series of links based on the product the CLEC is ordering.  For example, if a CLEC was ordering a non-designed loop for a new customer through the LEX ordering interface, the CLEC user was able to click on a “non-designed loop” link and be directed to a set of ordering instructions. The CLEC user was then able to determine the specific forms (e.g., Local Service Request form, Directory Listing form) and information fields that are required to be completed based on the product being ordered.

Once the CLEC service representative determines the product that was being ordered and the required forms to be used, the CLEC service representative then accesses a series of R/C/O tables also contained within the Local Ordering Handbook.  The R/C/O tables are a “snap-shot” view of the necessary data inputs for the information fields that have to be populated depending on the product and activity type being ordered. [3] There are hundreds of R/C/O table permutations contained within the Local Ordering Handbook and each table varies in accordance with the product being ordered.

            As part of AT&T’s post-merger consolidation process, STS and other CLECs were informed of AT&T’s plans to phase-out the Local Ordering Handbook at the July 8, 2009 Change Management/Change Control meeting.  The meeting is held monthly and allows for all participating CLECs and AT&T to discuss any interface or documentation changes to the Operation Support Systems.  During the July meeting, AT&T informed the CLECs that section three of the Local Ordering Handbook would be moved into the Local Service Order Requirements.[4]

 

            The structure of the Local Service Order Requirements is different when compared to the now retired Local Ordering Handbook.  The Local Service Order Requirements is arranged and navigated based on the forms used by CLECs to place an order.  If a CLEC is ordering a non-designed loop for a new customer through the LEX ordering interface, the CLEC user must first determine which forms need to be completed, such as a Local Service Request form.  Next, the CLEC user is directed to a series of links for each information field to be completed within the ordering form.  Each link provides the CLEC user with a “Product Activity table” showing the various products that can be ordered and the necessary data entry for an information field. 

 

            STS Position

 

            Since the retirement of LENS and implementation of the LEX ordering interface in July 2010, STS argues that it is critical for AT&T to maintain and update the R/C/O tables to assist in the placement of orders using the LEX interface.  The R/C/O tables were implicitly built-in to the retired LENS ordering interface, whereas LENS provided “real-time” up-front edit-checking capability.  The LEX system does not contain the up-front edit-checking capability and, in effect, may cause orders to be rejected or returned for clarification by AT&T after the order is submitted.   In other words, the LEX system allows for a CLEC order to be submitted to AT&T with errors, rejected by AT&T, reworked by the CLEC, resubmitted by the CLEC, and possibly rejected by AT&T again.  

 

            STS states that it relies on the R/C/O tables as a valuable tool to support the creation of error free orders before submitting to AT&T.  According to STS, using the new Product Activity tables contained within the Local Service Order Requirements is time consuming, delays implementation of a customer’s service, and may ultimately result in the loss of the customer if the delay is too long.

 

            When using the Local Service Order Requirements as a guide to placing orders, STS states that it must click on every information field link to determine the necessary data to be entered into each information field.[5]  STS typically completes four separate forms when submitting an order to AT&T through LEX:  the Local Service Request form, End User form, Directory Listing form, and Loop Service Form. When completing a Local Service Request form, STS states that a STS service representative must click on 142 separate information field links in the Local Service Order Requirements as opposed to observing a “snap-shot” view of all the information fields as provided in the R/C/O tables within the now retired Local Ordering Handbook.

 

            During the July 8, 2009 Change Management/Change Control meeting when AT&T informed the CLECs that section 3 of the Local Ordering Handbook, including the R/C/O tables, would be moved into the Local Service Order Requirements, STS specifically inquired as to whether the formatting of the documents within the Local Ordering Handbook would change.  AT&T stated that the change would not affect formatting.[6]  Despite AT&T’s response, the format changed.

 

            After learning that the R/C/O tables were replaced with Product Activity tables in the Local Service Order Requirements, STS submitted a Change Request to AT&T’s Change Management/Change Control team on March 26, 2010. STS requested that AT&T’s Local Service Order Requirements contain the R/C/O tables originally within the Local Ordering Handbook.  AT&T denied STS’ request and emailed the following response to STS on April 12, 2010:

The attached change request is shown as not approved due to cost.  The LSOR [Local Service Order Requirements] is developed by an external software application.  This application would require modification to create the R/C/O tables which only duplicates information that is already included within a field’s Notes, Conditions, and Data Entry Conditions. The R/C/O tables do not eliminate the need to view the individual fields because the rules are within the aforementioned Notes, Conditions, and Data Entry Conditions. 

 

            STS’ position is supported further by additional CLECs (DeltaCom, Inc., XO Communications, Cbeyond, LLC, and TW Telecom) who filed letters in the docket on June 1, 2011.  DeltaCom and XO Communications, who jointly filed a letter, specifically stated, “. . . navigating through the AT&T documentation is cumbersome and inefficient and represents a significant step backwards by AT&T compared to what was available to CLECs before AT&T decided to abolish the R/C/O tables.”  Furthermore, “[t]he lack of current R/C/O tables impacts Joint CLECs production centers by adding time and expense to locate ordering information to submit clear, error-free orders and places an undue burden on CLECs.”

           

            AT&T’s position

AT&T agrees that the format and use of the Local Service Order Requirements, including the Product Activity tables, differs substantially from the R/C/O tables within the Local Ordering Handbook. AT&T contends that the design of the Local Service Order Requirements complies with the Ordering and Billing Forum industry standards that dictate the composition of the information fields.  According to AT&T, the format of the Local Service Order Requirements is particularly useful for CLECs that have built their own front-end ordering interface, as opposed to CLECs that use LEX.  AT&T agrees that first-time users of LEX would find the Local Ordering Handbook more useful; however, as CLEC users become experienced, AT&T believes the Product Activity tables within the Local Service Order Requirements will sufficiently serve as a replacement for the R/C/O tables.  AT&T also believes that CLECs can create their own tables and guides to facilitate their data entry issues with LEX.

AT&T did deny STS’ request to maintain the R/C/O tables in response to STS Change Request submitted to AT&T’s Change Management/Change Control team on March 26, 2010; however, in response to STS’ concerns, AT&T offered to give STS and other CLECs a copy of the existing R/C/O tables in a Microsoft Word format. AT&T stated that STS could update the tables themselves as changes are introduced to the Local Service Order Requirements. Updates to the Word document would be communicated by AT&T’s Accessible Letters available at AT&T’s on-line website for CLECs and documented in AT&T’s Local Service Revision History.  The updates are typically tri-annual and coincide with new OSS releases.

 

Discussion

 

            On July 28, 2011, staff filed a recommendation in this docket for the August 9, 2011 Commission Agenda Conference. At that time, staff recommended that AT&T should update the R/C/O tables for a one year period.  Staff limited the updates to one year to give STS adequate time to consider transitioning to another ordering interface or contract with a third-party to place orders for them. Prior to the Agenda Conference, STS requested, and AT&T agreed, to defer staff’s recommendation in order for the parties to attempt resolution of this issue. The parties have since failed to reach a resolution, and six months later, the issue remains unresolved.

 

            At this point, the Local Ordering Handbook    has now been retired for over two years. However, staff believes this issue is still important because ordering wholesale services from an ILEC is extremely complex.  For example, to order a Digital Data Design Loop, a CLEC LEX user must potentially complete three different forms with 35 required fields, 59 conditional fields and 14 optional fields.  Not populating a field, or not populating it correctly will cause a CLEC order to be rejected, which results in a delay to an end-user getting service initiated. The instructional manuals are the key to what forms and fields are required for each product. Therefore, it is imperative that the instructional material provided by AT&T to CLECs should be clear, concise, user-friendly, and accurate.

 

AT&T has not provided updates to the R/C/O tables since its November 2009 OSS release.  In response to AT&T’s offer to give STS a one-time copy of the existing R/C/O tables in Word format, STS contends that it would be overly burdensome and does not have the available resources to keep the Word file up-to-date.  However, STS has since developed some templates as a work-around substitute for the R/C/O table updates. The templates are used to assist STS service representatives in placing orders with AT&T.  STS created the templates by attempting to update R/C/O tables for specific various products and services STS orders.  According to STS, creating the templates is resource intensive and does not resolve this issue because the templates are only applicable to some of the combinations of products and services STS orders with AT&T.  Despite STS’ efforts to improve its ordering process, STS ordering rejection rate continues to be high. 

From January through November 2011, STS placed 2,758 orders with AT&T via the LEX interface for service in Florida.  STS experienced 1,277 rejections on these orders, with some orders receiving multiple rejections. STS’ ratio of rejects to orders placed is 46 percent.  When compared to the analogous data for the top three LEX users in Florida, the ratio of rejects to orders placed experienced by these CLECs averaged 36 percent. 

 Two of the most common rejection error codes experienced by STS from January through November 2011 was “field information invalid/incomplete” and “required field not populated”. This information may further substantiate the need for the R/C/O tables as requested by STS.  STS believes that historically its rejection rate under LENS was lower than it is now because of the availability of the R/C/O tables and the up-front edit checking capability inherent in LENS.  ATT could not provide historical data on LENS rejects which would indicate whether this is true. In light of the fact that even the top three users of LEX are experiencing an average 36 percent rejection rate, staff believes that ATT should proactively explore whether upfront edit checking in LEX is a priority for the CLECs.

STS cannot explain why its LEX rejection rate is higher than other CLECs.  They believe that other CLECs may contract with third-party vendors to create and submit orders on their behalf, or other CLECs have created their own front-end ordering interface that includes edit-checking capabilities.  Staff believes that STS may want to consider pursuing either of these options in an effort to reduce its rejection rate.  In light of the recent acquisition of STS by EarthLink in March 2011, staff encourages STS to pursue these options. It should be noted that DeltaCom, which filed a letter in this docket in support of STS’ position, is also a subsidiary of Earthink.  It may be possible for DeltaCom to aid STS in reducing its rate of rejections.

In the meantime, it is important to assure that the LEX ordering interface and its documentation continues to provide non-discriminatory access to AT&T’s OSS.   Staff would further note that AT&T states that the Local Service Order Requirements comply with the Ordering and Billing Forum industry standards; however, the Ordering and Billing Forum does not dictate the organization and presentation of the instructional manuals.  Staff believes it would be easier for AT&T to update the R/C/O tables than STS and every other CLEC who is in need of this information, but staff does not believe continuous updating of the R/C/O tables is a long-term solution.

            Additionally, AT&T is responsible for making changes to the Operations Support Systems through the Change Management/Change Control process. CLECs who request changes to the Operation Support Systems must submit a Change Request to AT&T.  In 2002, this Commission ordered AT&T to implement a “50/50 Capacity Plan” to allow for CLECs to have a voice in the changes to the Operation Support Systems.  Release capacity is the total number of hours implement a Change Request.  Per the Plan, AT&T’s release capacity would be equally split (50/50) between AT&T and CLECs.[7]  Staff’s review of the release capacity reports revealed that AT&T may not be allocating 50 percent of its release capacity to CLECs.  Given this, staff fails to understand the denial of the change request STS submitted in March 2010.  It appears that AT&T has sufficient capacity within its releases to make appropriate updates to the R/C/O tables which was requested by STS using the appropriate process.

 

            To assure that AT&T is providing non-discriminatory access to its OSS,  staff believes that AT&T should perform a one-time update to the “required” information fields contained within the Required/Conditional/Optional (R/C/O) instructional tables.  Knowing the information fields that are “required” to be completed will assist STS in reducing the number of orders being initially rejected by AT&T.  AT&T should work with STS over the next six-months to assist with reducing STS’ rejection rate and determine a way to expedite the ordering process given the retirement of the Local Ordering Handbook.

 

Conclusion

 

Staff recommends that AT&T should update the “required” information fields contained within the Required/Conditional/Optional (R/C/O) instructional tables to reflect any changes AT&T has made to the LEX ordering interface between November 2009 and March 2012.   This update should be performed one time and should be made available to STS within 60 days of the issuance of the Consummating Order (when PAA becomes final).  The update should be in a Microsoft Word file and formatted in the same manner as the R/C/O tables contained within AT&T’s retired Local Ordering Handbook.  Additionally, AT&T should work with STS over the next six months to reduce the number of order rejections STS experiences.

 

Issue 2:  Should this docket be closed?

 

Recommendation: 

 Yes.  If the Commission approves staff recommendation on Issue 1, then STS has indicated that there will be no remaining issues and staff recommends that the docket be closed.   The resulting decision to approve issue 1 and close the docket will be issued as a Proposed Agency Action.  The decision will become final upon issuance of a Consummating Order, if no one person whose substantial interests are affected timely files a protest within 21 days of the issuance of the Order.  (Robinson)

Staff Analysis: 

 If the Commission approves staff recommendation on Issue 1, staff recommends that the docket be closed.   The resulting decision to approve issue 1 and close the docket will be issued as a Proposed Agency Action.  The decision will become final upon issuance of a Consummating Order, if no one person whose substantial interests are affected timely files a protest within 21 days of the issuance of the Order.  (Robinson)