Glen Mazza's Weblog

Salesforce CRM's REST API allows for inserting up to 200 records into Salesforce using just a single API call, a considerable time- and cost-savings over inserting records one-by-one. I've updated my Java Salesforce client to be able to make such calls. Some notes about this process:

• This multiple insertion process is distinct from Salesforce's Bulk API, an asynchronous method that relies on CSV files that is suggested for data sets of over 2000 records where immediate responses are not necessary.

• This process is just for record insertions. For other CRUD actions, Salesforce's Composite API can be used, albeit with lower limits (25 requests per query).

Salesforce provides a multi-insert example showing the request and successful (201 Created) response JSON, but is missing the format of error messages specific to these types of calls. I'm providing below the format of the error responses determined while updating the Java client. Given a snippet of the SF example's request body:

{
"records" :[{
    "attributes" : {"type" : "Account", "referenceId" : "ref1"},
    "name" : "SampleAccount1",
    "phone" : "1111111111",
    "website" : "www.salesforce.com",
    "numberOfEmployees" : "100",
    "industry" : "Banking"   
    },{
    "attributes" : {"type" : "Account", "referenceId" : "ref2"},
    "name" : "SampleAccount2",
    "phone" : "2222222222",
    "website" : "www.salesforce2.com",
    "numberOfEmployees" : "250",
    "industry" : "Banking"
    },...
]
}

Each item to be inserted needs an attributes metadata property, specifying the type and a referenceId, the latter of which can be anything but must be unique for each item in the insertion. The reference IDs are used in the response for either providing the Salesforce IDs for successful insertions, or in referring to any errors with that particular record. The attribute type field seems redundant, as the API call made already specifies the type being inserted, but it is nonetheless required for these types of calls.

Generic error response (403) if any reference IDs are missing:

  [
      {
          "message": "Include a reference ID for each record in the request.",
          "errorCode": "INVALID_INPUT"
      }
  ]

Above message will be uncommon so long as the request has referenceIds for every record provided. The more common 400 Bad Request response can occur due to missing attribute types, duplicate reference IDs, as well as validation failures, missing required fields, etc. For 400s, the response body will list the problem records by the referenceId provided in the request, example:

  {
      "hasErrors": true,
      "results": [
          {
              "referenceId": "ref3",
              "errors": [
                  {
                      "statusCode": "INVALID_INPUT",
                      "message": "Duplicate ReferenceId provided in the request.",
                      "fields": []
                  }
              ]
          },
          {
              "referenceId": "ref8",
              "errors": [
                  {
                      "statusCode": "INVALID_INPUT",
                      "message": "Include an entity type for each record in the request.",
                      "fields": []
                  }
              ]
          }
      ]
  }

What is important to note with multiple record insertion is that, if there are any reported problems with any of the items being inserted, none of the records in the request will be inserted (all-or-nothing). One way to handle failures is to make a second request of the same records minus those reported as failures in the prior response (matching on referenceId), to at least get those records inserted. The failed records can instead be logged and analyzed to see what to do with them.

As for making these calls using the Salesforce Client, an included integration test shows the process for inserting multiple rows with one call, and also how to trap and read any 400 exceptions that may occur. The code is fairly the same regardless of which objects are being inserted, however each type of object being inserted will need a MultipleEntityRecord subclass (similar to here for the integration test). The MultipleEntityRecord base class stores the required type and referenceId attributes, while the subclass is to store the fields specific to the object being inserted.

This post provides SOAP and Java examples of activating list sends in Marketing Cloud (see my earlier post for the alternative of triggered sends). In this situation, we provide the email contents along with the MC-stored subscriber list(s) to send the email to.

For SOAP, one creates a Send object which wraps (among other values) an Email, EmailSendDefinition and one more more subscriber List objects. Some of the more important values stored at each level:

Object	Information to provide within object
Send	Wrapper for below three objects, also stores the email from-address and from-name.
Email	HTML and text versions of the email, subject line, and character set. See some examples.
List	MC List IDs to send the email to. List IDs are available from MC Email Studio, menu item Subscribers | Lists, selecting the list and viewing its Properties tab.
EmailSendDefinition	Whether or not to use multipart emails, to send de-duplicate (not to send multiple copies to the same email address if the address is on multiple lists that the email is being sent to). The default values, need to provide them, and whether MC actually does anything with certain properties aren't always clear, you will probably need to experiment a bit.

Here's a SOAP example using the MC Postman workspace:

<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
    <s:Header>
        <a:Action s:mustUnderstand="1">Create</a:Action>
        <a:To s:mustUnderstand="1">https://{{et_subdomain}}.soap.marketingcloudapis.com/Service.asmx</a:To>
        <fueloauth xmlns="http://exacttarget.com">{{dne_etAccessToken}}</fueloauth>
    </s:Header>
    <s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
        <CreateRequest xmlns="http://exacttarget.com/wsdl/partnerAPI">
            <Objects xsi:type="Send">
                <Client>
                    <ID>{{et_mid}}</ID>
                </Client>
                <Email>
                    <Name>Sample Email Send</Name>
                    <IsHTMLPaste>true</IsHTMLPaste>
                    <Subject>Sample Test Message</Subject>
                    <CharacterSet>UTF-8</CharacterSet>
                    <HTMLBody>Testing message: %%[ if listid != 1234567 then ]%% Welcome reader!  %%[else]%% Greetings reader! %%[endif]%%</HTMLBody>
                    <TextBody>Welcome Reader! (text only)</TextBody>
                </Email>
                <List>
                    <ID>1234567</ID>
                </List>
                <List>
                    <ID>2345678</ID>
                </List>
                <EmailSendDefinition>
                </EmailSendDefinition>
                <FromAddress>bobsemail@yopmail.com</FromAddress>
                <FromName>Bob Sender</FromName>
            </Objects>
        </CreateRequest>
    </s:Body>
</s:Envelope>

The "Name" field with value "Sample Email Send" does not appear in the email but is used to help identify a specific email send. It is what is displayed in the Sends section of the Email Studio home page and the also the Tracking section of the list details. It does not have to be unique (sends are identified by a unique Job ID) but making it so helps make sends easier to tell apart from each other.

Sometimes you may wish to adjust the email a bit depending on the list being sent to. In the HTMLBody element of the above example, I've added AMPScript tags showing how this can be done.

For Java, an example using the Fuel SDK is below.

public void sendEmail() {
    CreateRequest createRequest = new CreateRequest();

    CreateOptions createOptions = new CreateOptions();
    createOptions.setRequestType(RequestType.SYNCHRONOUS);
    createOptions.setQueuePriority(Priority.HIGH);
    createRequest.setOptions(createOptions);

    Send send = new Send();

    com.exacttarget.fuelsdk.internal.Email email = new com.exacttarget.fuelsdk.internal.Email();
    email.setName("Sample Email Send via Java");
    email.setEmailType(EmailType.HTML.value());
    email.setIsActive(Boolean.TRUE);
    email.setIsApproved(Boolean.TRUE);
    email.setIsHTMLPaste(Boolean.TRUE);
    email.setSubject("Sample test message subject");
    email.setCharacterSet("UTF-8");
    email.setHtmlBody("<p>Email Body</p>");
    email.setTextBody("Text version of email body");
    send.setEmail(email);

    // add as many lists as needed
    List listToSendTo = new List();
    listToSendTo.setId(1234567);
    send.getList().add(listToSendTo);

    // More on EmailSendDefinition:
    // https://developer.salesforce.com/docs/marketing/marketing-cloud/guide/creating_an_email_send_definition_using_the_web_service_api.html
    EmailSendDefinition emailSendDefinition = new EmailSendDefinition();
    emailSendDefinition.setIsMultipart(isMultipart);
    emailSendDefinition.setDeduplicateByEmail(true);
    send.setEmailSendDefinition(emailSendDefinition);

    send.setFromAddress("bobsemail@yopmail.com");
    send.setFromName("Bob Sender");

    createRequest.getObjects().add(send);

    // configure ETClient similar to here: https://salesforce.stackexchange.com/a/312178
    // ETClient etClient = ....
    CreateResponse response = etClient.getSoapConnection().getSoap().create(createRequest);
    if (response != null && "OK".equalsIgnoreCase(response.getOverallStatus())) {
        // success! Check email inbox...
        LOGGER.info("Success sending email w/Request ID {}", response.getRequestID());
    } else {
        Optional.ofNullable(response)
                .ifPresent(cr -> Optional.ofNullable(cr.getResults())
                        .filter(errorList -> !errorList.isEmpty())
                        .map(errorList -> errorList.get(0))
                        .ifPresent(createResult -> {
                                    LOGGER.error("{}: {}", createResult.getErrorCode(), createResult.getStatusMessage());
                                }
                        )
                );
    }
}

Further Reading

• Send Definitions and Email Message Interactions - Marketing Cloud documentation

Within Marketing Cloud's Automation Studio, SQL Query activities on the ListSubscribers view can be run to determine subscription information for users and lists. For SOAP calls, and programming languages using them, we work with ListSubscriber and SubscriberList types defined in the MC SOAP WSDL for querying and updating, respectively. This entry provides some examples of querying and updating, both via direct SOAP calls using MC's Postman workspace and programmatically with the FuelSDK-Java (see here for initialization code not covered in the code snippets below.)

How to query a user's subscriptions: The below SOAP request provides options on querying user subscription information by subscriberKey and optionally on subscription status (Active or Unsubscribed). The filters equals and IN are shown but there are several others, see the WSDL above (search on SimpleOperator) to see the full list.

<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
    <s:Header>
        <a:Action s:mustUnderstand="1">Retrieve</a:Action>
        <a:To s:mustUnderstand="1">https://{{et_subdomain}}.soap.marketingcloudapis.com/Service.asmx</a:To>
        <fueloauth xmlns="http://exacttarget.com">{{dne_etAccessToken}}</fueloauth>
    </s:Header>
    <s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
        <RetrieveRequestMsg xmlns="http://exacttarget.com/wsdl/partnerAPI">
            <RetrieveRequest>
                <ObjectType>ListSubscriber</ObjectType>
                <Properties>SubscriberKey</Properties>
                <Properties>ListID</Properties>
                <Properties>Status</Properties>
                <Filter xsi:type="SimpleFilterPart">
                    <Property>SubscriberKey</Property>
                    <SimpleOperator>IN</SimpleOperator>
                    <Value>...subscriberKey1...</Value>
                    <Value>...subscriberKey2...</Value>
                    <Value>...subscriberKey3...</Value>
                </Filter>
                <!--Filter xsi:type="ComplexFilterPart">
                    <LeftOperand xsi:type="SimpleFilterPart">
                        <Property>SubscriberKey</Property>
                        <SimpleOperator>equals</SimpleOperator>
                        <Value>...subscriberKey1...</Value>
                    </LeftOperand>
                    <LogicalOperator>AND</LogicalOperator>
                    <RightOperand xsi:type="SimpleFilterPart">
                        <Property>Status</Property>
                        <SimpleOperator>equals</SimpleOperator>
                        <Value>Active</Value>
                    </RightOperand>
                </Filter-->
            </RetrieveRequest>
        </RetrieveRequestMsg>
    </s:Body>
</s:Envelope>

Sample output:

<Results xsi:type="ListSubscriber">
    <PartnerKey xsi:nil="true" />
    <ObjectID xsi:nil="true" />
    <Status>Unsubscribed</Status>
    <ListID>2552723</ListID>
    <SubscriberKey>...subscriberKey1...</SubscriberKey>
</Results>
<Results xsi:type="ListSubscriber">
    <PartnerKey xsi:nil="true" />
    <ObjectID xsi:nil="true" />
    <Status>Active</Status>
    <ListID>2591367</ListID>
    <SubscriberKey>...subscriberKey2...</SubscriberKey>
</Results>

If querying by subscriber it is much faster to query many at once using the IN clause as shown above instead of making separate API calls for each subscriber. (I've found 50 at a time works well, I did not test larger amounts.) Most of the overhead involves making a request and receiving the response, with each extra user queried adding only a minor amount to the total response time.

A similar query, this time using Java and the Fuel SDK:

public RetrieveRequestMsg buildActiveListRequestMsg(String subscriberKey) {
    RetrieveRequest retrieveRequest = new RetrieveRequest();

    retrieveRequest.setObjectType("ListSubscriber");
    retrieveRequest.getProperties().add("SubscriberKey");
    retrieveRequest.getProperties().add("ListID");
    retrieveRequest.getProperties().add("Status");

    retrieveRequest.setFilter(ETUtils.composeFilter(
            ETUtils.composeFilter("SubscriberKey", SimpleOperators.EQUALS, subscriberKey),
            LogicalOperators.AND,
            ETUtils.composeFilter("Status", SimpleOperators.EQUALS, "Active")
    ));

    RetrieveRequestMsg retrieveRequestMsg = new RetrieveRequestMsg();
    retrieveRequestMsg.setRetrieveRequest(retrieveRequest);

    return retrieveRequestMsg;
}

private static FilterPart composeFilter(FilterPart leftPredicate, LogicalOperators operator, FilterPart rightPredicate) {
    ComplexFilterPart filter = new ComplexFilterPart();
    filter.setLeftOperand(leftPredicate);
    filter.setLogicalOperator(operator);
    filter.setRightOperand(rightPredicate);
    return filter;
}

Subsequent processing to get the subscriber's active subscriptions:

RetrieveResponseMsg response = etClient.getSoapConnection().getSoap().retrieve(buildActiveListRequestMsg("subscriberKey1"));
if ("OK".equalsIgnoreCase(retrieveResponseMsg.getOverallStatus())) {
    List results = retrieveResponseMsg.getResults();
    List listIDs = results.stream()
            .map(apiObject -> ((ListSubscriber) apiObject).getListID())
            .collect(Collectors.toList());
}

How to update a user's subscriptions: The MC documentation provides several examples of working with the SubscriberList object for subscribing and unsubscribing users. Note there are two ways to unsubscribe a user from a list: marking the user Unsubscribed, which keeps an entry for the user on the subscription list with an unsubscribed date, and just deleting the user from the list. Marketing Cloud generally recommends the former approach. For list maintenance, we specify a user by subscriber key, and a series of lists. For each list, the List ID, an action (create, update, upsert, delete) and sometimes a status (Active or Unsubscribed) is provided. The delete action does not take a status, and if status is otherwise omitted, it is treated as Active if the user is not already on the list or keeps the status as-is if the user is. Sample Postman query:

<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
    <s:Header>
        <a:Action s:mustUnderstand="1">Create</a:Action>
        <a:To s:mustUnderstand="1">https://{{et_subdomain}}.soap.marketingcloudapis.com/Service.asmx</a:To>
        <fueloauth xmlns="http://exacttarget.com">{{dne_etAccessToken}}</fueloauth>
    </s:Header>
    <s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
        <UpdateRequest xmlns="http://exacttarget.com/wsdl/partnerAPI">
            <Objects xsi:type="Subscriber">
                <SubscriberKey>...subscriberkey1...</SubscriberKey>
                <Lists>
                    <ID>2716501</ID>
                    <Action>upsert</Action>
                </Lists>
            </Objects>
            <Objects xsi:type="Subscriber">
                <SubscriberKey>...subscriberkey1...</SubscriberKey>
                <Lists>
                    <ID>2712867</ID>
                    <Action>delete</Action>
                </Lists>
            </Objects>
            <Objects xsi:type="Subscriber">
                <SubscriberKey>...subscriberkey2...</SubscriberKey>
                <Lists>
                    <ID>2712867</ID>
                    <Status>Active</Status>
                    <Action>upsert</Action>
                </Lists>
                <Lists>
                    <ID>2716962</ID>
                    <Status>Active</Status>
                    <Action>upsert</Action>
                </Lists>
            </Objects>
        </UpdateRequest>
    </s:Body>
</s:Envelope>

Some notes for the above request:

• Multiple lists can be updated for a single user by either creating one Objects block per list, as with the SubscriberKey1 user, or by having multiple Lists under a single Objects block as done with the SubscriberKey2 user. For the latter case, if an error would occur with any single list, testing is showing that none of the updates will occur under that Objects block, and furthermore the error response will specify the problem but not which of the subscription changes is causing it.
• Multiple users cannot be specified under a single Objects block, each user needs its own Objects block(s).
• Using an action of upsert for subscribes (status = Active) and unsubscribes (status = Unsubscribed) are very unlikely to throw exceptions, providing the list ID and the user's subscriber key exists. This makes upserts a better option for many-lists-in-one-Objects-block changes. (The upsert creates a record if it does not exist, but updates it otherwise.)
• If you'd like exceptions if you're adding someone to a list who is already there, or making someone unsubscribed who is not subscribed presently, instead of upsert use create or update.
• Common error codes: SubscriberNotFound (12001), ListNotFound (13000), OnListAlready (13006) (for a create of a user of whatever status who is already on the list), NotOnList (13007) (for an update of a user not on the list).
• If for an upsert you don't provide a status for the user, if the user is not on the list he will be added with status of Active. Otherwise, no change for the user (will remain Active or Unsubscribed as before.) Providing you don't delete users from lists but just mark them as Unsubscribed, this will allow you to add users to a list while respecting past unsubscribes.

In Java using the FuelSDK, subscription changes are handled via an UpdateRequest. Here, we take Subscriber object(s) in which we identify each user by his SubscriberKey, and then populate each Subscriber object's lists attribute with SubscriberList objects, with each object containing a List ID, action (create, update, delete, upsert, with the default apparently being upsert) and optionally status (Active, Unsubscribed), subject to the same rules given above for the direct SOAP calls. Sample adding and unsubscribing lists from a given Subscriber:

import com.exacttarget.fuelsdk.internal.Subscriber;
import com.exacttarget.fuelsdk.internal.SubscriberList;
import com.exacttarget.fuelsdk.internal.SubscriberStatus;
import com.exacttarget.fuelsdk.internal.UpdateOptions;
import com.exacttarget.fuelsdk.internal.UpdateRequest;
import com.exacttarget.fuelsdk.internal.UpdateResponse;
import com.exacttarget.fuelsdk.ETSdkException;

class BulkSubscriptionUpdate {
    private Subscriber subscriber;
    private final Set subscribes;
    private final Set unsubscribes;
    
    ....
}

public UpdateResponse updateSubscriptions(BulkSubscriptionUpdate bulkSubscriptionUpdate) throws ETSubscriberServiceException {

    UpdateRequest updateRequest = new UpdateRequest();
    UpdateOptions options = new UpdateOptions();
    options.setQueuePriority(Priority.HIGH);
    options.setRequestType(RequestType.SYNCHRONOUS);
    updateRequest.setOptions(options);

    Subscriber subscriber = new Subscriber();
    subscriber.setSubscriberKey(generateSubscriberKey(user.getId()));

    List subscriptions = new ArrayList<>();
    if (!CollectionUtils.isEmpty(bulkSubscriptionUpdate.getSubscribes())) {
        for (Integer etid : bulkSubscriptionUpdate.getSubscribes()) {
            subscriptions.add(buildSubscriptionUpdate(etid, SubscriberStatus.ACTIVE));
        }
    }
    if (!CollectionUtils.isEmpty(bulkSubscriptionUpdate.getUnsubscribes())) {
        for (Integer etid : bulkSubscriptionUpdate.getUnsubscribes()) {
            subscriptions.add(buildSubscriptionUpdate(etid, SubscriberStatus.UNSUBSCRIBED));
        }
    }

    if (!CollectionUtils.isEmpty(subscriptions)) {
        subscriber.getLists().addAll(subscriptions);
        updateRequest.getObjects().add(subscriber);

        try {
            return update(updateRequest);
        } catch (ETSdkException e) {
            LOGGER.error("Failed to update subscriptions for subscriber {}", bulkSubscriptionUpdate.getSubscriber(), e);
            throw e;
        }
    }
    
    // nothing to update
    return null;
}

private static SubscriberList buildSubscriptionUpdate(Integer etid, SubscriberStatus status) {
    SubscriberList subscription = new SubscriberList();
    subscription.setId(etid);
    subscription.setStatus(status);
    subscription.setAction("upsert");
    return subscription;
}

For my FuelSDK fork I found another component that would benefit from updating, namely the Partner API WSDL defining the SOAP requests and responses, used by Apache CXF to generate Java classes representing those objects. The main Salesforce branch is still relying on a copy of the WSDL downloaded in 2017. A comparison of the two WSDLs is showing quite a few new and modified request and response objects since then:

ClientID: CustomerID
TaskResult: TblAsyncID
SaveOption: TrackChanges
AccountUser: IsSendable
APIObject: __AdditionalEmailAttribute 1-5.
Attribute: no longer inheriting APIObject
SubscriberResult: ErrorCodeID
TriggeredSendClassEnum
TriggeredSendSubClassEnum
SenderProfile: FallbackFromAddress
deleted: MessagingConfiguration
ChatMessagingEventType
SalesforceSendActivity
ImportDefinition: HasMultipleFiles
ImportResultsSummary: NumberRestricted
JsonWebKey
DirectoryTenant
AuditLogUserContext
AutomationActivity: SerializedObject
AttributeEntityV1
Thumbnail
NameIdReference
CategorynameIdReference
UserBasicsEntity
AssetAnyProperty
Asset
Category
ScheduledRequest
ScheduledConversation

I did need to make two adjustments to the WSDL copied into my fork, as detailed on the project README and commented in the WSDL. For one of them--a mismatch between what the WSDL claims the SOAP response will be for a particular call and what it actually is, causing a validation exception with CXF--I've sent a help ticket to Marketing Cloud requesting they update their WSDL. This particular issue has been around a long time, apparently, as even the 2017 WSDL needed this manual adjustment.

I updated my FuelSDK fork to JDK 11 and switched its build from Maven to Gradle. For those running CXF wsdl2java tasks in Maven and wishing to use Gradle instead, comparing the former pom and current build.gradle for this project should help in showing the process involved.

For the CXF wsdl2java process, I used Francisco Mateo's cxf-codegen-gradle plugin which offers a very nice user's guide. The FuelSDK also uses JavaCC to generate some model classes, which thankfully has its own Gradle plugin, allowing me to complete the build script conversion.

For change data capture and push topic events, Salesforce itself generates event messages whenever specified changes to specified objects occur. Salesforce also offers platform and generic events to allow for users to send messages of a specified structure at times of the developer's choosing. For coverage of Platform Events, Salesforce offers a Developer Guide as well as a Trailhead Module for hands-on learning.

Salesforce's generic events have their own guide but generally provide less developer support compared to platform events. For one, it is only with the CometD client that generic events can be listened to, and these messages apparently have a message size limit of 3K compared to the 1 MB provided for platform events. On the other hand, some advantages for generic events include its ability to send any arbitrary JSON as well as specify the users to receive each message. While Platform event messages will stream as JSON, they don't appear to support JSON objects within its payload. Instead, JSON objects need to be streamed as JSON strings and manually unpacked from the client receiving the message.

In the steps below I'll be featuring additional functionality beyond that covered in the Trailhead tutorial, providing examples of including data from multiple object types (here, both Accounts and Contacts in one message) as well as publishing platform events via triggers attached to specified objects and via an Apex REST controller (the latter allowing external systems to request that an event be generated.). My Salesforce Java client from a previous tutorial includes support for calling Apex REST controllers and the separate Change Data Capture (CDC) listener can be modified to read the platform events generated, as CDC events are a specialized case of platform events.

1. Define the custom platform event. The Trailhead tutorial shows how platform events are defined. Here, I'm creating an AccountAndContacts__e event below, to show how we can create a message consisting of multiple sObject types (here, Account and Contact), as well as generate multiple messages to report on all contacts (there is a 128K limit for contacts per message, but for demonstration purposes below I'll be limiting each message to five contacts). Note the Contact field will exist as a JSON string, containing an array of objects (for this example, we'll have the User's Salesforce ID, name, and email address). From the perspective of the platform event, however, we can define this only as a string without specifying further definition for its contents.

   

2. Create an Apex method (and test class) for generating platform events.. This generator will create, for a given account, one platform event message for every five contacts at that account. I keep a "calls" static variable to indicate a platform event getting published, which helps immensely when writing tests. Note as given in the comments this variable's lifetime is just within- and per-transaction and not global as it would be in Java, further helping its usage in testing.

   public class AccountAndContactsEventGenerator {
   	
       // Counter to check that this generator is being called (useful for testing objects calling the generator)
       // https://salesforce.stackexchange.com/questions/204805/test-that-a-platform-event-was-published/204806
       // Note, lifetime of this object per transaction, not system-wide as in Java, see:
       // https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_classes_static.htm
       public static Integer calls = 0;
   
       public static void publishAccountAndContactsEvents(Account account) {
           List<AccountAndContacts__e> events = getEventMessages(account);        
           List<Database.SaveResult> results = EventBus.publish(events);
               
           for (Database.SaveResult sr : results) {
               if (sr.isSuccess()) {
                   System.debug('Successfully published event: '+ sr);
                   calls++;
               } else {
                   System.debug('Error when publishing event: ' + sr);
               }
           } 
       }
       
       @testVisible 
       private static List<AccountAndContacts__e> getEventMessages(Account account) {
           List<Contact> contacts = [SELECT FirstName, LastName, Email FROM Contact WHERE AccountId = :account.id]; 
   
           // Sending one event message for every five contacts at the company
           List<AccountAndContacts__e> eventList = new List<AccountAndContacts__e>();
           
           AccountAndContacts__e event = initializeNewEvent(account);
           List<ContactData> contactList = new List<ContactData>();
           for (Integer i = 0; i < contacts.size(); i++) {
               if (i > 0 && math.mod(i, 5) == 0) {
                   event.Contacts__c = JSON.serialize(contactList);
                   contactList.clear();
                   eventList.add(event);
                   event = initializeNewEvent(account);
               }
               ContactData cd = new ContactData();
               Contact con = contacts.get(i);
               cd.userId = con.Id;
               cd.emailAddress = con.Email;
               cd.lastName = con.LastName;
               cd.firstName = con.FirstName;
               contactList.add(cd);
           }
           event.Contacts__c = JSON.serialize(contactList);
           eventList.add(event);
           
           return eventList;
       }
       
       @testVisible
       private static AccountAndContacts__e initializeNewEvent(Account account) {
           AccountAndContacts__e event = new AccountAndContacts__e();
           event.AccountName__c = account.name;
           event.AccountSFID__c = account.id;
           return event;
       }
       
       @testVisible private class ContactData {
           public String userId { get; set; }
           public String emailAddress { get; set ; }
           public String lastName { get; set; }
           public String firstName { get; set; }
       }
   }
   

   Test class: The test creates an account with six contacts (so therefore, two messages will be generated) and parses a sample of the AccountAndContacts__e messages generated to confirm they contain the expected contents. Note within Apex test cases it is not necessary to manually delete objects created as they will be rolled back automatically once the test completes.

   @IsTest
   public class AccountAndContactsEventGeneratorTest {
   
       @IsTest
       private static void testEventGeneration() {
           Account account = new Account();
           account.Name = 'Acme Corp';
           insert account;
   
           List<Contact> contacts = new List<Contact>();
           for (Integer i = 0; i < 6; i++) {
               contacts.add(createContact('First' + i, 'Last' + i, account.id));
           }
           
           List<AccountAndContacts__e> messages = AccountAndContactsEventGenerator.getEventMessages(account);
           System.assertEquals(messages.size(), 2);
           AccountAndContacts__e message = messages.get(0);
           System.assertEquals(message.AccountName__c, 'Acme Corp');
           System.assertEquals(message.AccountSFID__c, account.Id);
           List<AccountAndContactsEventGenerator.ContactData> firstFiveContacts 
               = (List<AccountAndContactsEventGenerator.ContactData>) 
               JSON.deserialize(messages.get(0).Contacts__c, List<AccountAndContactsEventGenerator.ContactData>.class);
   
           System.assertEquals(firstFiveContacts.size(), 5);
           System.assertEquals(firstFiveContacts.get(1).userId, contacts.get(1).Id);
           System.assertEquals(firstFiveContacts.get(2).firstName, 'First2');
           System.assertEquals(firstFiveContacts.get(3).LastName, 'Last3');
           System.assertEquals(firstFiveContacts.get(4).emailAddress, 'last4@yopmail.com');
   
           List<AccountAndContactsEventGenerator.ContactData> sixthContact = 
               (List<AccountAndContactsEventGenerator.ContactData>) 
               JSON.deserialize(messages.get(1).Contacts__c, List<AccountAndContactsEventGenerator.ContactData>.class);
           
           System.assertEquals(sixthContact.size(), 1);
       }
       
       private static Contact createContact(String firstName, String lastName, Id accountId) {
           Contact contact = new Contact();
           contact.FirstName = firstName;
           contact.LastName = lastName;
           contact.AccountId = accountId;
           contact.Email = lastName + '@yopmail.com';
           insert contact;
           return contact;
       }
   }
   

3. Create an Apex trigger (and test class) to activate generator based on desired criteria.. Here I'm placing an after-update trigger on the Account object that will trigger the generator if the Account's name has changed. This demonstrates how the generator can be selectively activated based on the nature of changes to an object.

   trigger trigger_AccountUpdate on Account (after update) {
       
       for (Account a : Trigger.New) {
           if (!a.Name.equals(Trigger.OldMap.get(a.Id).Name)) {
               AccountAndContactsEventGenerator.publishAccountAndContactsEvents(a);
           }
       }
   }
   

   Test class: The trigger test checks that a platform event message is generated if and only if the Account name changes. As discussed earlier, I rely on the calls static variable in the generator for this.

   @isTest
   public class TestAccountUpdate {
       @isTest
       static void testUpdateWithNameChangeGeneratesEvent() {
          Account account = new Account(Name = 'Acme Corp', BillingCity = 'Philadelphia');
          insert account;
           
          account.name = 'New Acme Corp';
          update account;
           
          System.assertEquals(1, AccountAndContactsEventGenerator.calls);   
       }
       
       @isTest
       static void testUpdateWithNoNameChangeGeneratesNoEvent() {
          Account account = new Account(Name = 'Acme Corp', BillingCity = 'Philadelphia');
          insert account;
           
          account.BillingCity = 'Baltimore';
          update account;
   
          System.assertEquals(0, AccountAndContactsEventGenerator.calls);           
       }
   }
   

4. Create an Apex REST Controller (and test class) to activate generator for a specific Account.. If you're new to Salesforce REST Controllers, recommend its Trailhead tutorial.. It shows convenient ways of making REST calls via cURL and Salesforce's Developer Workbench. Additionally, my Salesforce client has an ApexRestCaller and integrated tests showing a way to call Apex endpoints from Java.

   @RestResource(urlMapping='/AccountAndContactsPE/*')
   global with sharing class AccountAndContactsEventRESTEndpoint {
       
       @HttpPost
       global static void generatePlatformEvent() {
           RestRequest req = RestContext.request;
           String accountId = req.requestURI.substring(req.requestURI.lastIndexOf('/')+1);
           // using List<Account> instead of Account below for safer handling of no matching records
           // see: https://developer.salesforce.com/forums/?id=906F000000094QZIAY
           List<Account> accountList = [SELECT Id, Name FROM Account WHERE Id = :accountID];
           Account account = (accountList != null && accountList.size() > 0) ? accountList[0] : null;
           if (account != null) {
               AccountAndContactsEventGenerator.publishAccountAndContactsEvents(account);
           } else {
               System.debug('Could not find Account w/ID = ' + accountId);
           }
       }
   }
   

   @isTest
   public class AccountAndContactsEventEndpointTest {
   
       @isTest
       static void testGeneratePlatformEventMessage() {
           // Testing method: https://trailhead.salesforce.com/content/learn/modules/apex_integration_services/apex_integration_webservices
           Account account = new Account(Name = 'RestTestAccount');
           insert account;
       	RestRequest request = new RestRequest();
           request.httpMethod = 'POST';
           request.requestUri = 'https://doesntmatterignored/services/apexrest/AccountAndContactsPE/' + account.id;
           RestContext.request = request;
           
           // call method
           AccountAndContactsEventRESTEndpoint.generatePlatformEvent();
           System.assertEquals(1, AccountAndContactsEventGenerator.calls);
       }
   
       @isTest
       static void testNoAccountNoPlatformEventMessage() {
           // creating a contact just to get a SF ID not tied to an account.
           Contact contact = new Contact(firstname='First', lastname='Last');
           
           // Testing method: https://trailhead.salesforce.com/content/learn/modules/apex_integration_services/apex_integration_webservices
           Account account = new Account(Name = 'RestTestAccount');
           insert account;
       	RestRequest request = new RestRequest();
           request.httpMethod = 'POST';
           request.requestUri = 'https://doesntmatterignored/services/apexrest/AccountAndContactsPE/' + contact.id;
           RestContext.request = request;
           
           // call method, no PE because no Account with Contact's SF ID
           AccountAndContactsEventRESTEndpoint.generatePlatformEvent();
           System.assertEquals(0, AccountAndContactsEventGenerator.calls);
       }
   }
   

5. Subscribe to the event messages. I added a JSON supporting class and processor for this particular platform event to my Salesforce Event Listener covered in an earlier tutorial. The sample logs the contents of any messages received. Due to the trigger created in this tutorial, all that is needed is to change the name of an account in Salesforce and messages, one for every five contacts at the company, will be sent out and picked up by this listener.

Salesforce Trailhead's Create a Custom Channel and Enrich Change Events tutorial shows how to create new Change Data Capture (CDC) channels providing "enriched" messages, i.e., those having certain fields always present. While CREATE and UNDELETE event messages always provide all non-null fields, UPDATE and DELETE events do not, but event enriching can be used for the latter two cases. For example, in my previous tutorial, it would be more readable to always return the Account Name in update and delete events so it can be logged alongside the Account Salesforce ID. And the tutorial gives an example of creating a custom External Account ID field in Salesforce that can be provided in every message and used to identify an account record in the external system for synchronization.

Some drawbacks/limitations of event enriching:

• It is no longer possible to send multiple changes in one event message (e.g., receiving one message that 100 Salesforce IDs of a given entity had the same field changed to a specific value), as each record will have a separate enriched field value.
• The enriched fields cannot contain elements from other entities beyond their Salesforce ID, so for a Contact message one can get the Contact's Account ID but not properties of the Account such as Name.
• It is more cumbersome to create and modify custom channels having enriched fields than relying on the default behavior from the standard CDC streams. The Salesforce Setup UI lacks this functionality, one instead can work with the Salesforce Postman project (GitHub) making REST calls against either Tooling or Metadata APIs. The Salesforce tutorial effectively shows the process using the former.

I expanded my CDC listener sample from the previous tutorial to allow for reading from a custom channel "CDCSample__chn" instead of the standard AccountChangeEvent. The channels can be switched just by adjusting the configuration file and restarting the application. Unfortunately, I was unable to get the Account "Name" property to always be provided in the message as an enriched field. I tried five or six other Account fields and they all worked fine, I'm not sure what the problem with Name is, whether my error or a Salesforce bug.

Using Salesforce's Postman project and the Trailhead tutorial, below are the commands I ran to create and configure this custom channel. Main thing to keep in mind is that a channel is a holder for message event streams, but not a message stream itself. A channel contains one or more channel members, with each member providing messages on a particular entity. It is also with the channel member that any desired enriched fields are defined.

• Creating the custom Platform Event Channel:

  POST to: {{_endpoint}}/services/data/v{{version}}/tooling/sobjects/PlatformEventChannel
  
  with body:
  {  
    "FullName": "CDCSample__chn",
    "Metadata": {
      "channelType": "data",
      "label": "Custom Channel for Change Data Capture Sample"
    }
  }
  

  There should be a success message giving the Salesforce ID of the new channel. Metadata on the PlatformEventChannel can be queried using the following GET call:

  {{_endpoint}}/services/data/v{{version}}/tooling/query/?q=SELECT Id, FullName FROM PlatformEventChannel WHERE DeveloperName='CDCSample'
  

• Creating a PlatformEventChannelMember in the PlatformEventChannel. Here I'm choosing three fields to always appear.

  POST to {{_endpoint}}/services/data/v{{version}}/tooling/sobjects/PlatformEventChannelMember
  
  {
    "FullName": "CDCSample_chn_AccountChangeEvent",
    "Metadata": {
      "enrichedFields": [
        {
          "name": "Industry"
        },
        {
          "name": "Name"
        },
        {
          "name": "TickerSymbol"
        }
      ],
      "eventChannel": "CDCSample__chn",
      "selectedEntity": "AccountChangeEvent"
    }
  }
  

  The success message provides the ID for the Channel Member, whose details can be later queried with a GET similar to the following:

  {{_endpoint}}/services/data/v{{version}}/tooling/sobjects/PlatformEventChannelMember/0v85e0000004Cl7AAE
  

  Sample output:

  {
      "attributes": {
          "type": "PlatformEventChannelMember",
          "url": "/services/data/v51.0/tooling/sobjects/PlatformEventChannelMember/0v85e0000004Cl7AAE"
      },
      "Id": "0v85e0000004Cl7AAE",
      "IsDeleted": false,
      "DeveloperName": "CDCSample_chn_AccountChangeEvent",
      "Language": "en_US",
      "MasterLabel": "AccountChangeEvent",
      "NamespacePrefix": null,
      "ManageableState": "unmanaged",
      "CreatedDate": "2021-05-15T16:01:21.000+0000",
      "CreatedById": "0055e000000nMxcAAE",
      "LastModifiedDate": "2021-05-19T10:27:59.000+0000",
      "LastModifiedById": "0055e000000nMxcAAE",
      "SystemModstamp": "2021-05-19T10:27:59.000+0000",
      "FullName": "CDCSample_chn_AccountChangeEvent",
      "Metadata": {
          "enrichedFields": [
              {
                  "name": "Industry"
              },
              {
                  "name": "Name"
              },
              {
                  "name": "TickerSymbol"
              }
          ],
          "eventChannel": "CDCSample__chn",
          "selectedEntity": "AccountChangeEvent",
          "urls": null
      },
      "EventChannel": "0YL5e0000008OIAGA2",
      "SelectedEntity": "AccountChangeEvent"
  }
  

  Above query provides the enriched fields, but for links to them specifically, this query can be used:

  {{_endpoint}}/services/data/v{{version}}/tooling/query/?q=SELECT Id,ChannelMemberId,Field FROM EnrichedField ORDER BY ChannelMemberId
  

• Keep this tutorial page handy for whenever it is needed to adjust the enriched fields of a platform event channel member. It involves PATCH requests to the endpoint originally used to create the stream.

I posted to Github a Spring Boot-based client library for making OAuth2-enabled REST calls to Salesforce's API. Supported are Salesforce's JWT Bearer Token and username/password flows discussed in my earlier blog post. The library supports use of Salesforce's REST API, SOQL Query, and Apex REST functionality. It uses Spring Security's OAuth 2 client to obtain access tokens necessary for making these calls.

The integrated test cases give examples of the client in action. As they involve creating, updating, and deleting Salesforce Accounts they should be run against a non-production instance. Salesforce offers free developer instances you can sign up for. Note the test case for the Apex REST functionality will require installing this Apex REST endpoint from the Salesforce documentation. To run the tests, first create an application-test.properties file in the itest resources folder with the configuration necessary for the flow you are using. There is a template file in that folder specifying what is needed for each OAuth2 flow type. For usage of this library by other applications, this configuration would be placed in the importing application's properties file. The library's SalesforceOAuth2Config class reads that configuration, and will halt on startup with informational messages if anything needed is missing. Once done, the integrated tests can be run from IntelliJ or command-line via ./gradlew integratedTest.

The Username/Password flow is supported out of the box by Spring, but the JWT bearer token flow requires some extra classes that I implemented following Spring's source code for their standard password and client grant flows:

• SalesforceJwtBearerOAuth2AuthorizedClientProvider - For each outgoing request to a resource server, checks if the JWT Bearer Token method is being used for access tokens and also if an authorized client (i.e., one with a valid access token) is already available. If yes and no, respectively, initiates the process to obtain a new access token to use with the resource API call.
• OAuth2SalesforceJwtBearerGrantRequest - created by above class, it holds the information necessary to create the JWT bearer token for making an access token request.
• OAuth2SalesforceJwtBearerGrantRequestEntityConverter - Constructs the JWT bearer token
• SalesforceJwtBearerTokenResponseClient - Uses the JWT Bearer token created by the above class to obtain a new access token from the authorization server.

Within the app, SOQL queries are handled by the SOQLQueryRunner which provides two options for responses: a JSON-formatted string or a developer-defined parameterized implementation of SOQLQueryResponse (example in the integrated test). The latter takes advantage of the fact that SOQL queries share much common structure and need only a relatively small portion to be overridden to hold fields specific to the SOQL query.

What happens when access tokens expire? The WebClient calls have a retry(1) setting that allows for one additional call to the resource server in case of an error such as using an expired access token. In such cases, for the first call, the failure handler in SalesforceOAuth2Config removes the authorized client instance (which has the invalid access token) from memory. For the retry call, SalesforceJwtBearerOAuth2AuthorizedClientProvider notes that there is not an authorized client instance anymore so proceeds to obtain a new access token to allow the second call to proceed. This functionality can be verified by revoking the access token from either Salesforce Setup's Session Management screen or from Connected Apps usage, and confirming that a subsequent resource API call still provides the data. Code breakpoints can also be used to confirm another access token was requested.

Additional Resources

• OAuth 2.0 Client - official Spring documentation
• Carey Development - Blog contains many useful tutorials, including on logging exceptions and reporting them to clients helpful for the creation of this client library.
• Spring WebClient – GET, PUT, POST, DELETE examples
• Spring 5 WebClient and WebTestClient Tutorial with Examples
• Salesforce Username/Password Flow Example - Very helpful in creating the Username-Password flow option in this library.