Who:
What:

BT built an innovative Digital Cloud Services Platform (DCSP) to enable new, agile squads to support customer-facing colleagues.

How:

BT’s PaaS used cloud-native, open-source technologies to create a scalable, distributed and flexible platform that allows a high degree of autonomy and innovation.

Results:
  • Overall time to market for cloud-based services fell 40%
  • Onboarding of BT squad members reduced from two weeks to three hours
  • Cut API deviation by 90% thereby saving multiple millions on reworking
  • 704 developers were onboarded to use BT’s DCSP within six months
  • A production version of GitLab was delivered in five days instead of 60-90 days
  • Automation and GitOps eliminated 100% of human errors
How automation helped BT’s agile delivery squads meet new business demands

During Covid-19 lockdowns, BT experienced a surge in customer demand for cloud-based services to support newly remote workforces. At the same time, BT’s IT transformation was moving the company away from monolithic IT development models and towards lightweight, autonomous, self-contained microservices solutions which depended on new, agile delivery squads. This combination of rapid digitalization and increased customer demand, however, placed heavy demands on the consumer and enterprise division’s delivery teams which BT needed to alleviate.

BT’s IT transformation aimed to create economies of scale and accelerate delivery to businesses by migrating away from legacy and traditional waterfall methodologies and towards a federated model driven by business-focused teams. To this end it replaced its previous, heavily siloed architecture with an agile, factory-style, re-use approach.

One of the operator’s first steps was to move to a cloud native IT architecture, breaking applications down into a collection of loosely coupled, containerized services encapsulated in software - microservices. Correctly deployed, microservices bring big business and operational benefits – such as faster delivery and self-maintaining services – but their use is not an automatic guarantee of success.

However, as software delivery within BT became more complicated against the backdrop of rapid digitalization and the adoption of multi-cloud ecosystems, the squad teams’ workloads expanded rapidly. They also found they were hampered by dependencies on other platform teams. This led to slower decision making, reduced innovation and a lack of autonomy. The combination of a fast-growing workload and operational dependencies on other teams, meant the delivery squads became a bottleneck as they struggled to scale the model.

Platform-as-a-service benefits

Keen to support the demands of its business customers, BT therefore turned to a platform-as-a-service (PaaS) solution – the DCSP –to give its service development squad teams the autonomy and self-serve features they need to deliver services quickly, accurately and cost effectively.

BT built the DCSP using cloud native, open-source technologies to be scalable, distributed and flexible enough to incorporate big and small changes, quickly and as required. Crucially, the platform was also designed to address the complexity of producing and managing software for and on the cloud; automating the entire workflow; providing delivery teams with greater autonomy, cutting the cost of delivery, and building a culture of innovation.

Some of the specific functions BT needed included self-service onboarding of new squad team members using a single sign-on tool to grant smooth, rapid access to all the tools on the DCSP. It also wanted to make it simpler for squad members to get started on the platform and increase their productivity from day one. To this end, the platform gives squad team members standardized templates and an out-of-the-box, ready to use ‘build pack’ that included all the tools, deployment strategies and best practices needed to deliver microservices and deploy them. As a result, the average time for a new squad member to become productive is now one hour compared with six or seven days previously for the simplest services.

The new platform also allows BT’s delivery squads to go from development to production in one hour by automating the promotion of built-up code to the production environment faster than ever before. This relies on a high degree of automation allowed by DevOps and GitOps models, which ensure each ‘commit for feature’ is tested and the developer has the option of pushing it into production immediately if the test is satisfactory.

Cost-effectiveness was another key requirement of the platform and the DCSP is designed to monitor actual resource consumption. In addition to reducing cost, the resource consumption function enhances service stability by ensuring the correct allocation of resources (compute, memory and storage) for a given service.

When it comes to enhancing squad members’ autonomy, the platform gives them a kit that that makes it simple to improvise when assembling complex services, thereby reducing their reliance on other platform and infrastructure teams. The platform also encourages a community approach to enhancing its features by allowing users to contribute to templates and best practices.

BT is building further on this platform to evolve this into their strategic CI/CD platform - Mobius, creating an accelerated path to delivery with consistent and predictable developer experience across all of their business.

Making the most of a cloud-native architecture

Several elements of a cloud native architecture helped the DCSP achieve its goals, from the use of self-contained software components which are organized around BT’s business domains and capabilities, and owned by small, expert squad teams to bring greater business and operational speed and agility, through to the ability to scale segments of a service to match business requirements instead of having to scale the whole application. And because the DCSP makes use of microservices, any development errors that arise do not affect the entire application. In addition, the use of Chaos Engineering practices to help leverage automation by injecting faults into the network, file and operating systems, to monitor how the microservices and other components handle errors and ensure the response is efficient and effective.

Results

BT achieved several quantifiable results from DCSP including:

 

The average onboarding time for new members to a squad fell from two weeks to just three hours, drastically improving delivery times to the business and helping to reduce times to market by 40% overall.

 

No capacity issues were reported over a span of 10 days owing to the scalable, cloud native approach to the platform, replacing the use of statistics at a monthly review to address recurring capacity challenges.

 

704 developers were onboarded to use the DCSP within six months. The platform has been viewed as a big success with more teams across BT’s business spectrum looking to adopt it.

 

Production incidents fell to two a month due to the platform’s continuous integration/continuous delivery (CI/CD) capabilities, saving the business significant opportunity costs.

 

More features can be released more often, as releases no longer have to be in out-of-office hours. Since launch, there have been no out-of-office releases.

 

Automation and adopting a GitOps approach to the platform ensured there have been zero human errors to date.

 

The delivery time of new features and services to customers has reduced massively. For example, a self-hosted production version of GitLab was delivered in five days instead of 60 to 90 days previously. A production version of Apache Kafka was delivered within an hour instead of 15 to 20 days as was the case before the DCSP and tools such as Dynatrace, SonarQube, Pact Broker and Kubecost were provisioned for use within an hour versus anything between three and seven days for each tool previously.

Facilitating integration with TM forum’s Open APIs and OD

Working with Torry Harris Integration Solutions (THIS), BT visualized the transformation journey using resources from TM Forum’s Cloud Native IT & Agility theme to ease migration of components to the cloud and opted for an Open Digital Architecture, based on TM Forum’s principals.

THIS helped with the uniform adoption of a globally recognized, industry-standard Open APIs from TM Forum, deployed in conjunction with TM Forum’s blueprint for a Hybrid Infrastructure Platform or HIP (and specifically the Component Accelerator). The benefits from using these assets include:

  • Internal integration was accelerated and simplified as the APIs provide a common language for group and local markets allowing BT to share its capabilities to construct global products and services, regardless of the underlying vendors. They include product catalogs, orders, service usage, trouble tickets and more.
  • External integration was integral to the success of the DCSP as the APIs allow BT to provide a consistent technical ‘face’ to partners and co-dependent operators worldwide.

This enabled BT to create new revenue channels through a partner ecosystem and drive operational efficiency, especially by simplifying the customer journey. Today, there is an ecosystem of 200+ partners, using 900+ APIs, creating new revenue channels through co-creation and bundling.