Maintaining a ‘project initialize’ in SAP Commerce

Koen van de Leur, 26 juli 2023

In this blog post, I advocate for the regular execution of the ‘initialize’ command on test and local systems throughout a SAP Commerce project. To ensure a smooth process, this initialization can be customized with a limited set of curated test data. My experience has shown that this approach can significantly enhance team efficiency, improve test system performance, expedite bug resolution, and reduce developer onboarding time. I’ll provide insights into establishing a maintainable initialization process, highlighting some key do’s and don’ts.

What is initialization?

In SAP Commerce projects, the ‘initialize’ command is a rarely used but significant operation. The initialization command/process drops all existing type definitions from the database, generates the database schema and the type systems, and imports any essential system data such as admin users. It also imports sample data if configured. Typically, the initialization process is executed only once on each server, whether it’s an on-premise server, a cloud environment, or a local development setup for a developer. Subsequent updates to the data model are applied to the database using ‘system update’ commands.

Initializing in Commerce Cloud Portal

The importance of test data

In SAP Commerce, being a Java Spring project, addressing bugs and implementing new features as a backend developer often requires setting up the project on a local environment. The ability to replicate scenarios from the live environment is crucial, and this necessitates loading relevant data into the local environment. For instance, without any products in the catalog, it’s practically impossible to test a ‘product lister page’ on a local system. The absence of essential test data can severely impede developers, as even testing minor changes or reproducing simple bugs might demand extensive effort to create a specific set of test data. Given that data is frequently interconnected and certain models have mandatory values, this situation can quickly snowball, turning a ten-minute fix into a half-day ordeal.

Common Solutions

One quick solution I’ve observed in some projects is using a copy of the production database for testing purposes. While this approach ensures scenarios from the live environment can be accurately reproduced, it also comes with significant downsides.

GDPR issues: You can imagine the potential problems arising from developers having production database backups on their laptops.
The issue of size: Transferring a 200GB database to each new developer on the team can be a hassle. It consumes substantial space on an SSD, and keeping the original file accessible for data corruption scenarios during local development is essential.
Performance: Executing processes like scripts on the full dataset may take a considerable amount of time, either matching or even exceeding the duration on the production environment due to hardware differences.
Unexpected side-effects: Imagine that you have a job that sends emails to customers, or resets account after a certain time of not visiting the site. Those jobs can be triggered when you start the system.

In my opinion, having the complete dataset on a local or test environment is unnecessary. Instead, utilizing a staging/acceptance environment, appropriately provisioned with resources, is the ideal approach to test functionality on a complete dataset. For SAP Commerce Cloud, achieving this is straightforward by creating database snapshots and transferring them across environments.

Our team’s Solution

As both having no data and having all the data present their own challenges, adopting a limited, curated set of test data imported during ‘system initialize’ offers numerous advantages:

Quick Setup for Local Development: Onboarding new team members becomes smoother when they can set up their system in just a few hours.
A Reset Option: In case a local environment becomes messy, the option to re-initialize without losing all test data and starting fresh is valuable.
Consistent Test Data: Including the initialization script in the repository ensures all team members have access to the same set of test data. Eliminating the need for everyone to create and manage their own test data prevents unnecessary duplication of efforts.
Automated Testing: A static test data set allows running automatic (integration) tests without the need to update them after every new import of live data. It also prevents tests from breaking due to constantly changing data when connected to a live test system.
Faster Processes: Activities like catalog synchronizations, search index creation, and migration scripts complete faster, saving valuable development time.
Accelerated Bug Resolution and Local Testing: The ability to quickly reproduce bugs and test new features can lead to significant time savings.

What’s the Downside?

As with many valuable endeavors, this approach requires an initial investment of time and effort, and some continuous discipline:

Curating the data: Acquiring test data from the client may prove challenging, and creating a consistent and cohesive set of data representative of live data can be tedious.
Initialization can take long: the relatively long cycle time of ‘system initialize’ on a local system, ranging from 20 minutes to over an hour, can make development of the initialization scripts cumbersome.
Requires discipline: Changes in the data model can potentially ‘break’ the initialization process, leading to issues where certain imports fail due to missing mandatory properties or other errors. Neglecting to add test data supporting new features to the initialization script will also necessitate developers to create their own test data for local debugging purposes.

In conclusion, there’s some downsides, but in the next paragraph I will describe how those can be mitigated by implementing a process to keep the initialization script up-to-date.

Keeping the Initialization Up-to-Date: Do’s and Don’ts

To ensure the initialization script remains relevant and effective, consider the following strategies:

Integrate into the Development Process: Include the initialization process in the team’s Definition of Done (DoD) to ensure that test data is created and organized as part of each new feature development.
Initialize often: It can help to run your local install without a license (even though you should have a license), as this prompts an initialization every 30 days. Any issues with the initialization will be quickly detected and addressed.
Leverage SAP Commerce Patch Framework: Utilize the SAP Commerce patch framework for data loads and scripts that need to run on all environments. This ensures that the system remains up-to-date with the latest production version after each initialization. Make sure they run after the test data is initialized, and that they don’t conflict with the test data!
Avoid ZIP Files as Import Scripts: Refrain from using ZIP files for ImpEx files as they may not be fully searchable in your IDE, potentially leading to missing dependencies and ‘breaking’ the initialization process.
Regularly Initialize the Test Environment: For the test environment, consider regular re-initialization on a set interval, such as monthly, to maintain a more predictable and aligned system with local development environments.

Choosing the Right Amount of Test Products

In our case, the production database contains 600k products, with 100k of those indexed for display on the storefront. However, our test set comprises only 3,000 products, with 1,000 indexed. This restricted number is manageable for local and test systems, and various scripts or indexers can execute efficiently. Although this set represents only 1% of the total products in production, it covers nearly all scenarios necessary for testing categorization, classification, pagination, and rudimentary facet search and filtering functionalities.

An Example

We created a dedicated test extension to house all our initialization test data. If you’re only using this data on local system, you can simply opt not to include it into the cloud manifest (in a cloud-based setup). Including it in the production version should be harmless though, as the ‘initialize’ should not be done on production and is a blocked operation for that environment in the Cloud Portal. With system properties you can further control what particular test data is loaded, as developers may work on differents part on the codebase.

package com.mycompany.testextension.setup;

import de.hybris.platform.commerceservices.setup.AbstractSystemSetup;
import de.hybris.platform.commerceservices.setup.SetupImpexService;
import de.hybris.platform.core.initialization.SystemSetup;
import de.hybris.platform.core.initialization.SystemSetup.Process;
import de.hybris.platform.core.initialization.SystemSetup.Type;
import de.hybris.platform.core.initialization.SystemSetupContext;
import de.hybris.platform.core.initialization.SystemSetupParameter;
import de.hybris.platform.core.initialization.SystemSetupParameterMethod;
import de.hybris.platform.scripting.engine.ScriptingLanguagesService;
import de.hybris.platform.util.Config;
import org.apache.log4j.Logger;

import java.util.*;


/**
 * This class provides hooks into the system's initialization and update processes.
 */
@SystemSetup(extension = "testextension")
public class TestDataSystemSetup extends AbstractSystemSetup {

    private static final Logger LOG = Logger.getLogger(TestDataSystemSetup.class);
    
    private static final String IMPORT_DEVELOPMENT_DATA_KEY = "testextension.import.development.data";
    private static final String CREATE_TEST_DATA = "createTestData";

    private ScriptingLanguagesService scriptingLanguagesService;
    private SetupImpexService setupImpexService;

    public TestDataSystemSetup(SetupImpexService setupImpexService, ScriptingLanguagesService scriptingLanguagesService) {
        this.setupImpexService = setupImpexService;
        this.scriptingLanguagesService = scriptingLanguagesService;
    }

    /**
     * Generates the Dropdown and Multi-select boxes for the projectdata import
     */
    @Override
    @SystemSetupParameterMethod
    public List&lt;SystemSetupParameter&gt; getInitializationOptions() {
        final List&lt;SystemSetupParameter&gt; params = new ArrayList&lt;SystemSetupParameter&gt;();
        params.add(createBooleanSystemSetupParameter(CREATE_TEST_DATA, "Create Test Data", true));
        return params;
    }

    /**
     * Implement this method to create initial objects. This method will be called by system creator during
     * initialization and system update. Be sure that this method can be called repeatedly.
     *
     * @param context the context provides the selected parameters and values
     */
    @SystemSetup(type = Type.ESSENTIAL, process = Process.ALL)
    public void createEssentialData(final SystemSetupContext context) {
        // no op
    }

    /**
     * Implement this method to create data that is used in your project. This method will be called during the system
     * initialization.
     *
     * @param context the context provides the selected parameters and values
     */
    @SuppressWarnings("all")
    @SystemSetup(type = Type.PROJECT, process = Process.ALL)
    public void createProjectData(final SystemSetupContext context) {
        if (getBooleanSystemSetupParameter(context, CREATE_TEST_DATA) &amp;&amp; Config.getBoolean(IMPORT_DEVELOPMENT_DATA_KEY, false)) {
            importTestProductCatalog();
            importTestPromotions();
            triggerSolrFullIndexJobs();
            importTestContentCatalog();
        }
    }

    private void importTestProductCatalog() {
        LOG.info("Creating all objects related to products (not catalog aware, like Brand data)...");
        getSetupImpexService().importImpexFile("/testextension/import/product/product-related/product-related-objects.impex", Boolean.FALSE, Boolean.FALSE);

        LOG.info("Creating Test Categories...");
        getSetupImpexService().importImpexFile("/testextension/import/product/test-categories.impex", Boolean.FALSE, Boolean.TRUE);

        LOG.info("Creating Test Products...");
        getSetupImpexService().importImpexFile("/testextension/import/product/test-products.impex", Boolean.FALSE, Boolean.FALSE);

        LOG.info("Importing classification data");
        getSetupImpexService().importImpexFile("/testextension/import/product/import-classification-data.impex", Boolean.FALSE, Boolean.TRUE);

        LOG.info("Assigning Products to Classifying categories");
        getSetupImpexService().importImpexFile("/testextension/import/product/assign-products-to-classifying-categories.impex", Boolean.FALSE, Boolean.TRUE);

        LOG.info("Assigning Products to navigation categories");
        getSetupImpexService().importImpexFile("/testextension/import/product/assign-products-to-nav-categories.impex", Boolean.FALSE, Boolean.TRUE);

        LOG.info("Sync the product catalog to Online");
        getSetupSyncJobService().executeCatalogSyncJob("myProductCatalog");
    }

    private void importTestPromotions() {
        LOG.info("Import &amp; publish the promotions after product catalog sync");
        getSetupImpexService().importImpexFile("/testextension/import/product/promotions.impex", Boolean.FALSE, Boolean.TRUE);
    }

    private void triggerSolrFullIndexJobs() {
        LOG.info("Executing SOLR full indexing job for backoffice (config was already loaded in essential data)");
        getSetupSolrIndexerService().executeSolrIndexerCronJob("backofficeSolrConfiguration", Boolean.TRUE);

        LOG.info("Executing SOLR full indexing job for storefront");
        getSetupSolrIndexerService().executeSolrIndexerCronJob("storefrontSolrConfiguration", Boolean.TRUE);
    }

    private void importTestContentCatalog() {
        LOG.info("Setting preview category and preview product (needed for Category Landing Page in SmartEdit)");
        getSetupImpexService().importImpexFile("/testextension/import/cms/cms-site-previews.impex", Boolean.FALSE, Boolean.TRUE);

        LOG.info("Importing CMS Content (loaded as last because of references the navigation makes to categories)");
        getSetupImpexService().importImpexFile("/testextension/import/cms/example-content.impex", Boolean.FALSE, Boolean.TRUE);

        LOG.info("Synchronizing CMS catalog");
        getSetupSyncJobService().executeCatalogSyncJob("myContentCatalog");
    }
}

Conclusion

In conclusion, I sincerely hope that this blog post has been of great help to fellow SAP Commerce developers, shedding light on the immense benefits of embracing the ‘system initialize’ process with a limited set of curated test data. By recognizing the advantages it offers – from faster onboarding and bug resolution to streamlined testing and performance improvements – we can optimize our development workflows and enhance overall project efficiency. While the initial investment and maintenance may pose challenges, the rewards in terms of productivity and teamwork are undeniable.