AEM Cloud Service Operations and Deployment Features

This is Part Two of a four-part series on Adobe Experience Manager as a Cloud Service.  Part One provided an overview of AEM Cloud Service as compared with AEM 6.5.  This part focuses on the unique operations and deployment features of AEM Cloud Service.  Part Three will describe how AEM development changes under AEM Cloud Service, along with migration options and lessons learned using AEM Cloud Service to date.  Part Four completes the series with an overview of the tools that Adobe provides to facilitate migrating to AEM Cloud Service.

Operations and deployment for AEM Cloud Service have several things in common with AEM Managed Services but also a few differences.  The main overlap is in the use of Cloud Manager for deployment.  Cloud Manager’s use was always strongly encouraged by the AEM Managed Services (AMS) team and so most customers would have migrated from manual or Jenkins-driven deployments to the use of Cloud Manager (and its associated Git repository).  In comparison, Cloud Manager is mandated for AEM Cloud Service as it ensures that each AEM instance is identical – something that is critical when instances may be spun up or torn down in an automated manner.

Unique AEM Cloud Service Deployment Features

Cloud Manager also adds several new features specifically for AEM Cloud Service:

• Self-service provisioning – Gone are the days of waiting for the AMS team to provision all the AEM instances; instead customers and partners such as Sirius/FiveOut are provided with access to Cloud Manager and the ability to create the Program (essentially a container for a set of AEM environments) and the AEM environments within the Program.  Environments can be spun up in as little as 15 minutes.
• Pre-built integrations with Adobe I/O – As part of instance provisioning, Cloud Manager will also create an Adobe Launch property, an Adobe I/O project for accessing the Launch service, an Adobe I/O project for accessing Smart Content Services such as smart tagging, and Adobe IMS configurations in AEM to link to the Adobe I/O services.
• Certificate management – This is another self-service feature, whereby users with the Deployment Manager role can add SSL certificates to a Program, associate domains with environments and tiers(Author, Publish, Preview), and map SSL certificates to domains. This eliminates the need to raise a support ticket to ask the AMS team to add or update the SSL certificate (such as when rolling out a new website or microsite).
• IP allow lists – this feature enables Deployment Managers to lock down specific environments/tiers so that only a whitelisted set of IP addresses can access them.  This is a good option for ensuring that search engine crawlers and unauthorized users cannot access Preview tiers and Publish tiers in non-production environments.

It is also worth mentioning that the Fastly CDN used by AEM Cloud Service has a fixed set of IP addresses regardless of customer/brand.  From a DNS perspective, this has the added benefit that CNAME and A records can be used to map both the sub-domain (e.g. www.customer-website.com) and the naked domain (customer-website.com) to AEM.  Previously AEM Managed Services Basic customers hosted on AWS were limited to CNAME records only and had to stand up separate web server infrastructure to redirect requests to the naked domain to the sub-domain (and hence AEM).  This restriction did not exist for AEM Managed Services customers hosted on Azure.

Restrictions on AEM Cloud Service Operations

From an operations perspective, there are a number of important restrictions, largely driven by the requirement that every instance be identical and the fact that a given instance can be transitory – it may be torn down and replaced at any moment (e.g. as part of a deployment or a patch release):

• No SSH access – There is no ability to access the underlying operating system for AEM or Dispatcher instances via SSH. Such access is typically used for accessing log files and checking or updating Dispatcher configurations.  AEM Cloud Service provides alternative means of doing this:
  • Log files can be downloaded directly from the Cloud Manager user interface.  Alternatively they can be accessed via a command line interface that leverages Adobe I/O to access the log files, including the ability to tail a log file. This requires a small amount of setup before it can be used.
  • Dispatcher configurations are maintained as code in the Git repository and are deployed as part of the Cloud Manager CI/CD deployment pipeline, alongside the components, templates and/or content that are typically deployed. This option exists for AMS as well but is mandatory for AEM Cloud Service.
• No CRXDE – CRXDE Lite is a tool that enables the Java Content Repository that underpins AEM to be browsed and searched. For AEM Cloud Service, CRXDE Lite is only available within the Author tier in the Development environment and within local instances. This compares with AEM Managed Services, where it was possible to request access to CRXDE Lite from a whitelisted IP address.
• No System Console – The System Console provides access to a number of important configuration tools such as the OSGI configuration manager, the resource resolver and the Sling servlet resolver. AEM Cloud Service removes access to the System Console so that the configuration for all instances remains consistent. Instead, the following options are available:
  • OSGI configurations are managed as code and are deployed as part of Cloud Manager’s CI/CD deployment pipelines. 
  • Cloud Manager provides access to a Developer Console for each environment that enables the following to be reviewed in read only manner:
    • OSGI configuration settings
    • Sling Servlet resolutions
    • Status of OSGI bundles, components, Oak indexes and Sling jobs
• No Maintenance Task Configurations – The AEM Admin user interface no longer includes options for configuring maintenance tasks such as workflow purge. Instead these must be configured in code and deployed via the Cloud Manager CI/CD pipeline. The version purge and audit log purge tasks are currently owned by Adobe, although Adobe is planning to make limited configuration options for these tasks available to customers in the future.

AEM Cloud Service Deployments

The deployment itself has changed significantly.  In AEM 6.5, deployments are to existing AEM instances (which may be restarted during deployment). In AEM Cloud Service, a deployment is never applied to an existing AEM instance; instead, a completely new set of AEM instances is spun up by combining a baseline image for the targeted version of AEM Cloud Service with the components, templates, configurations, etc. that are part of the deployment packages. This is performed as part of a blue/green deployment model, where new AEM and Dispatcher instances are stood up alongside the existing instances, the CDN pointed to the new instances, and then the old instances are torn down. This results in zero downtime for both the Author and Publish tier as shown below.

---

---

The complete Production deployment pipeline consists of the following activities:

• Build and unit testing – This is the regular Maven build set up by the development team.
• Code scanning – Running a set of code quality checks against the code base.
• Image building – Where the Maven build is applied to a baseline AEM Cloud Service image to create a cloud-deployable image as described above.
• Stage deployment – Leveraging the cloud-deployable images to create new Dispatcher, Author and Publish instances in the Stage environment using the blue/green approach described above.
• Security testing – Running a set of security checks against the Stage environment.
• Regression testing – Running additional server-side tests (integration tests and front-end tests if created in the git repository) in the Stage environment.
• Performance testing – Running a set of performance tests against key pages in the Stage environment.
• Production deployment – Leveraging the cloud-deployable images to create new Dispatcher, Author and Publish instances in the Production environment using the blue/green approach described above.

Disadvantages of the AEM Cloud Service Deployment Approach

The number of steps is reflected in the length of time to make deployments.  We have seen deployments to the Dev environment take anywhere from 30-60 minutes and deployments to Stage/Production take anywhere from 60-130 minutes.  This can be frustrating when you are trying to push out an urgent Production fix.  It also means that Dispatcher changes must be tested very carefully as an error in a filter or rewrite rule could take the website offline for a couple of hours before a fix can be deployed.

Fortunately, Adobe provides Dispatcher tools as part of the AEM Cloud Service SDK (see Part 3 of this series for more discussion about the SDK) that can be used to test the majority of Dispatcher configurations; however any that require CDN headers (e.g. geoblocking rewrite rules) will need to be tested in the Dev environment.  Adobe has promised to provide a Dispatcher-only CI/CD pipeline in the near future that will improve turnaround time on Dispatcher changes; in the meantime, test, test, test!

Important Recommendations for AEM Cloud Service Operations

Another important factor to consider from an operations perspective is that Adobe will push version updates to the environments several times per month. These consist of critical updates (bug fixes, security patches, and performance enhancements) and monthly feature releases. It is possible (albeit rare) for a release to deprecate or even remove APIs. This means that it is possible that an Adobe update could break your website. To mitigate against this happening, we recommend the following:

1. Follow AEM development best practices, particularly with respect to extending OOTB AEM functionality.  This is discussed further in Part 3 of this series.
2. Regularly review the AEM Cloud Service release notes to ensure that you are aware of upcoming changes.  This also means that you will become aware of any new features that are about to come available as these do not necessarily get communicated to you otherwise.
3. Review the APIs that you are using against Adobe’s list of deprecated APIs and make sure that you are up to date.  For example, Adobe will remove access to Apache Commons Lang v2 and Apache Commons Collections v3 at the end of 2021.  More recent versions of these APIs have been available for years now; however, if reusable code has been copied from project to project without review, then it is possible that usage of older versions of APIs remains undetected.
4. Keep on top of any code quality issues reported by the CI/CD pipeline.  Fix even minor issues early and ensure that code quality issues are communicated back to the entire development team so that they do not crop up again.
5. Expand your automated testing suite.  Most developers are used to creating unit tests of Java code using JUnit and mocking frameworks.  Adobe also provides support for server-side testing (“integration tests”) and testing of front-end code using Selenium and will roll back any deployments (including Adobe-initiated push updates) that fail any automated test.  Bear in mind however that each automated test requires effort to set up and essentially becomes “code” that needs to be maintained going forward and updated as the underlying code or feature being tested changes.  There is a happy medium to be had in terms of automated testing.
6. Define a set of manual regression tests that validate that the key parts of your website are working and run them in a non-production environment against the upcoming monthly feature release.
7. Subscribe to the Pre-release Channel in a non-production environment in order to preview (and test) feature releases a month in advance.  This gives you more time to replace deprecated APIs and fix anything that will be inadvertently broken by a new release.

Although these constraints and additional tests will add overhead to your project, bear in mind that they largely eliminate the need for expensive version upgrade projects down the line. This allows you to focus most of your budget going forward on deliverables that provide more direct business value, such as functional and user experience enhancements. These involve AEM development, which is the focus of Part 3 of this series.