Glen Mazza's Weblog
Tuesday September 06, 2022
Activating Triggered Sends with Marketing Cloud
Marketing Cloud has two primary ways of sending an email. One method involves specifying the email content and a Marketing Cloud list. For these list sends, everyone marked as subscribed on the list will receive the email. A second method, demonstrated here, involves Triggered Sends, in which a triggered email interaction ("TEI") defined in Email Studio is specified along with subscriber(s) to receive an email. While the TEI defines the Content Builder template that will be used in sending the email, attributes defined for each subscriber with the triggered send request can be used by the template to personalize emails. Indeed, if the template itself is largely blank, with the contents defined in those attributes, then each email sent can differ radically for each subscriber.
For both SOAP calls and using the FuelSDK wrapper, the Triggered Send wraps a TriggeredSendDefinition primarily used to identify the TEI that will be sending the emails, along with the Subscribers to be receiving an email. TEIs are created and viewable from Email Studio, menu item Interactions -> Triggered Emails. Links at the bottom provide information on creating TEIs as well as the Content Builder templates that they use.
Briefly, for the below SOAP and Java samples, in Content Builder I created first a template and then a template-based email, with the latter importing the former. For the template, this is what I used:
<html><head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"><title></title></head> <body> <div style="font-size:0; line-height:0;"><custom name="opencounter" type="tracking"><custom name="usermatch" type="tracking" /></div> <div data-type="slot" data-key="t81q1twaz2p" data-label="Drop blocks or content here"></div> <custom type="footer" /> </body> </html>
The data-key above (apparently) refers to a specific "enter HTML" widget visible when editing the Template-based email, as shown in the screenshot above. When editing the latter, click on that widget to be able to enter the email-specific content that you see on the left side of the screenshot:
<ctrl:field name="contentHtml"/> <hr> This email was sent to %%emailaddr%% by:<br> %%Member_Busname%%<br> %%Member_Addr%%<br> %%Member_City%%, %%Member_State%% %%Member_PostalCode%%<br> %%Member_Country%%
The plain text version (for subscribers specifying a preference for it) can be entered by clicking on the "Plain Text" area in the screenshot above. Here is what I used:
<ctrl:field name=contentText /> This email was sent to %%emailaddr%% by: %%Member_Busname%% %%Member_Addr%% %%Member_City%%, %%Member_State%%, %%Member_PostalCode%% %%Member_Country%%
Using the Marketing Cloud Postman workspace, a Sample SOAP request to send an email to two subscribers using a specified TEI would be as follows:
<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
<s:Header>
<a:Action s:mustUnderstand="1">Create</a:Action>
<a:To s:mustUnderstand="1">https://{{et_subdomain}}.soap.marketingcloudapis.com/Service.asmx</a:To>
<fueloauth xmlns="http://exacttarget.com">{{dne_etAccessToken}}</fueloauth>
</s:Header>
<s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<CreateRequest xmlns="http://exacttarget.com/wsdl/partnerAPI">
<Objects xsi:type="TriggeredSend">
<Client>
<ID>{{et_mid}}</ID>
</Client>
<TriggeredSendDefinition>
<CustomerKey>external-id-of-TEI</CustomerKey>
</TriggeredSendDefinition>
<Subscribers>
<SubscriberKey>...one subscriber key...</SubscriberKey>
<Attributes>
<Name>subject</Name>
<Value>Greetings 4!</Value>
</Attributes>
<Attributes>
<Name>contentHtml</Name>
<Value>This is <strong>HTML<strong>!</Value>
</Attributes>
<Attributes>
<Name>contentText</Name>
<Value>This is plaintext.</Value>
</Attributes>
</Subscribers>
<Subscribers>
<SubscriberKey>...another subscriber key...</SubscriberKey>
<Attributes>
<Name>subject</Name>
<Value>Another email 4</Value>
</Attributes>
<Attributes>
<Name>contentHtml</Name>
<Value>This is <em>HTML<em>!</Value>
</Attributes>
<Attributes>
<Name>contentText</Name>
<Value>This is in plaintext.</Value>
</Attributes>
</Subscribers>
</Objects>
</CreateRequest>
</s:Body>
</s:Envelope>
Note the attributes attached to the subscriber ("contentHtml", "contentText", etc.) will need to match the attributes used by the TEI's configured email template. For an email to be sent, all attributes referenced by the template must be present for each subscriber in the triggered send.
If there are obvious errors with the SOAP request (for example, not providing any subscribers), the SOAP response coming back normally will identify the problem for you. But if the request appears solid, you'll get the following response that the triggered send was enqueued:
<soap:Body>
<CreateResponse xmlns="http://exacttarget.com/wsdl/partnerAPI">
<Results xsi:type="TriggeredSendCreateResult">
<StatusCode>OK</StatusCode>
<StatusMessage>Created TriggeredSend</StatusMessage>
<OrdinalID>0</OrdinalID>
<NewID>0</NewID>
</Results>
<RequestID>...some UUID...</RequestID>
<OverallStatus>OK</OverallStatus>
</CreateResponse>
</soap:Body>
This does not guarantee a successful sending of an email, however. From Email Studio, menu item Interactions | Triggered Emails shows completed, queued, and errored counts for each triggered email interaction. If the intended recipient is not getting an email while the error counts for the triggered email interaction keep increasing with each attempt, opening a support ticket with Marketing Cloud, providing them the RequestID above for one of the responses will allow them to search their logs to see what the error is and report back to you.
In Java using the FuelSDK: Java programming involves putting TriggeredSend object(s) into a CreateRequest. The triggered send objects are populated in a similar manner as direct SOAP requests. Sample snippet:
TriggeredSend triggeredSend = new TriggeredSend();
TriggeredSendDefinition tsDef = new TriggeredSendDefinition();
tsDef.setCustomerKey("...external ID of TEI...");
triggeredSend.setTriggeredSendDefinition(tsDef);
Subscriber subscriber = new Subscriber();
subscriber.setSubscriberKey("...subscriber key...");
addAttribute(subscriber, "subject", "email subject line");
addAttribute(subscriber, "contentHtml", "<b>html</b> here");
addAttribute(subscriber, "contentText", "plain text here");
triggeredSend.getSubscribers().add(subscriber);
// Add more subscribers to triggeredSend as desired
CreateOptions options = new CreateOptions();
options.setRequestType(asyncSendsEnabled ? RequestType.ASYNCHRONOUS : RequestType.SYNCHRONOUS);
options.setQueuePriority(Priority.MEDIUM);
CreateRequest request = new CreateRequest();
request.setOptions(options);
request.getObjects().add(triggeredSend);
// configure ETClient similar to here: https://salesforce.stackexchange.com/a/312178
// ETClient etClient = ....
CreateResponse response = etClient.getSoapConnection().getSoap().create(request);
if (!response.getOverallStatus().equalsIgnoreCase("OK")) {
// success
}
The addAttribute(...) above is just a convenience method:
private void addAttribute(Subscriber s, String name, String value) {
Attribute a = new Attribute();
a.setName(name);
a.setValue(value);
s.getAttributes().add(a);
}
Further reading:
Official Marketing Cloud documentation on the non-API portions of this process: Create a Triggered Email Message Interaction and Create a Content Builder Email.
Introduction to Triggered Sends - Nice overview of entire process by Zuzanna Jarczynska, which also includes another option of sending a Triggered Email, here using AmpScript on a Cloud Page.
Out of scope for this tutorial, but see here for adding a custom tag to your email template to enable tracking (How many subscribers opened the email, etc.)
Posted by Glen Mazza in Salesforce at 07:00AM Sep 06, 2022 | Tags: emails marketingcloud | Comments[0]
Sunday August 28, 2022
Using Marketing Cloud's SOAP endpoint to manage subscriptions
Within Marketing Cloud's Automation Studio, SQL Query activities on the ListSubscribers view can be run to determine subscription information for users and lists. For SOAP calls, and programming languages using them, we work with ListSubscriber and SubscriberList types defined in the MC SOAP WSDL for querying and updating, respectively. This entry provides some examples of querying and updating, both via direct SOAP calls using MC's Postman workspace and programmatically with the FuelSDK-Java (see here for initialization code not covered in the code snippets below.)
How to query a user's subscriptions: The below SOAP request provides options on querying user subscription information by subscriberKey and optionally on subscription status (Active or Unsubscribed). The filters equals and IN are shown but there are several others, see the WSDL above (search on SimpleOperator) to see the full list.
<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
<s:Header>
<a:Action s:mustUnderstand="1">Retrieve</a:Action>
<a:To s:mustUnderstand="1">https://{{et_subdomain}}.soap.marketingcloudapis.com/Service.asmx</a:To>
<fueloauth xmlns="http://exacttarget.com">{{dne_etAccessToken}}</fueloauth>
</s:Header>
<s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<RetrieveRequestMsg xmlns="http://exacttarget.com/wsdl/partnerAPI">
<RetrieveRequest>
<ObjectType>ListSubscriber</ObjectType>
<Properties>SubscriberKey</Properties>
<Properties>ListID</Properties>
<Properties>Status</Properties>
<Filter xsi:type="SimpleFilterPart">
<Property>SubscriberKey</Property>
<SimpleOperator>IN</SimpleOperator>
<Value>...subscriberKey1...</Value>
<Value>...subscriberKey2...</Value>
<Value>...subscriberKey3...</Value>
</Filter>
<!--Filter xsi:type="ComplexFilterPart">
<LeftOperand xsi:type="SimpleFilterPart">
<Property>SubscriberKey</Property>
<SimpleOperator>equals</SimpleOperator>
<Value>...subscriberKey1...</Value>
</LeftOperand>
<LogicalOperator>AND</LogicalOperator>
<RightOperand xsi:type="SimpleFilterPart">
<Property>Status</Property>
<SimpleOperator>equals</SimpleOperator>
<Value>Active</Value>
</RightOperand>
</Filter-->
</RetrieveRequest>
</RetrieveRequestMsg>
</s:Body>
</s:Envelope>
Sample output:
<Results xsi:type="ListSubscriber">
<PartnerKey xsi:nil="true" />
<ObjectID xsi:nil="true" />
<Status>Unsubscribed</Status>
<ListID>2552723</ListID>
<SubscriberKey>...subscriberKey1...</SubscriberKey>
</Results>
<Results xsi:type="ListSubscriber">
<PartnerKey xsi:nil="true" />
<ObjectID xsi:nil="true" />
<Status>Active</Status>
<ListID>2591367</ListID>
<SubscriberKey>...subscriberKey2...</SubscriberKey>
</Results>
If querying by subscriber it is much faster to query many at once using the IN clause as shown above instead of making separate API calls for each subscriber. (I've found 50 at a time works well, I did not test larger amounts.) Most of the overhead involves making a request and receiving the response, with each extra user queried adding only a minor amount to the total response time.
A similar query, this time using Java and the Fuel SDK:
public RetrieveRequestMsg buildActiveListRequestMsg(String subscriberKey) {
RetrieveRequest retrieveRequest = new RetrieveRequest();
retrieveRequest.setObjectType("ListSubscriber");
retrieveRequest.getProperties().add("SubscriberKey");
retrieveRequest.getProperties().add("ListID");
retrieveRequest.getProperties().add("Status");
retrieveRequest.setFilter(ETUtils.composeFilter(
ETUtils.composeFilter("SubscriberKey", SimpleOperators.EQUALS, subscriberKey),
LogicalOperators.AND,
ETUtils.composeFilter("Status", SimpleOperators.EQUALS, "Active")
));
RetrieveRequestMsg retrieveRequestMsg = new RetrieveRequestMsg();
retrieveRequestMsg.setRetrieveRequest(retrieveRequest);
return retrieveRequestMsg;
}
private static FilterPart composeFilter(FilterPart leftPredicate, LogicalOperators operator, FilterPart rightPredicate) {
ComplexFilterPart filter = new ComplexFilterPart();
filter.setLeftOperand(leftPredicate);
filter.setLogicalOperator(operator);
filter.setRightOperand(rightPredicate);
return filter;
}
Subsequent processing to get the subscriber's active subscriptions:
RetrieveResponseMsg response = etClient.getSoapConnection().getSoap().retrieve(buildActiveListRequestMsg("subscriberKey1"));
if ("OK".equalsIgnoreCase(retrieveResponseMsg.getOverallStatus())) {
List results = retrieveResponseMsg.getResults();
List listIDs = results.stream()
.map(apiObject -> ((ListSubscriber) apiObject).getListID())
.collect(Collectors.toList());
}
How to update a user's subscriptions: The MC documentation provides several examples of working with the SubscriberList object for subscribing and unsubscribing users. Note there are two ways to unsubscribe a user from a list: marking the user Unsubscribed, which keeps an entry for the user on the subscription list with an unsubscribed date, and just deleting the user from the list. Marketing Cloud generally recommends the former approach. For list maintenance, we specify a user by subscriber key, and a series of lists. For each list, the List ID, an action (create, update, upsert, delete) and sometimes a status (Active or Unsubscribed) is provided. The delete action does not take a status, and if status is otherwise omitted, it is treated as Active if the user is not already on the list or keeps the status as-is if the user is. Sample Postman query:
<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
<s:Header>
<a:Action s:mustUnderstand="1">Create</a:Action>
<a:To s:mustUnderstand="1">https://{{et_subdomain}}.soap.marketingcloudapis.com/Service.asmx</a:To>
<fueloauth xmlns="http://exacttarget.com">{{dne_etAccessToken}}</fueloauth>
</s:Header>
<s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<UpdateRequest xmlns="http://exacttarget.com/wsdl/partnerAPI">
<Objects xsi:type="Subscriber">
<SubscriberKey>...subscriberkey1...</SubscriberKey>
<Lists>
<ID>2716501</ID>
<Action>upsert</Action>
</Lists>
</Objects>
<Objects xsi:type="Subscriber">
<SubscriberKey>...subscriberkey1...</SubscriberKey>
<Lists>
<ID>2712867</ID>
<Action>delete</Action>
</Lists>
</Objects>
<Objects xsi:type="Subscriber">
<SubscriberKey>...subscriberkey2...</SubscriberKey>
<Lists>
<ID>2712867</ID>
<Status>Active</Status>
<Action>upsert</Action>
</Lists>
<Lists>
<ID>2716962</ID>
<Status>Active</Status>
<Action>upsert</Action>
</Lists>
</Objects>
</UpdateRequest>
</s:Body>
</s:Envelope>
Some notes for the above request:
- Multiple lists can be updated for a single user by either creating one
Objectsblock per list, as with the SubscriberKey1 user, or by having multipleListsunder a singleObjectsblock as done with the SubscriberKey2 user. For the latter case, if an error would occur with any single list, testing is showing that none of the updates will occur under thatObjectsblock, and furthermore the error response will specify the problem but not which of the subscription changes is causing it. - Multiple users cannot be specified under a single
Objectsblock, each user needs its ownObjectsblock(s). - Using an action of
upsertfor subscribes (status = Active) and unsubscribes (status = Unsubscribed) are very unlikely to throw exceptions, providing the list ID and the user's subscriber key exists. This makes upserts a better option for many-lists-in-one-Objects-block changes. (Theupsertcreates a record if it does not exist, but updates it otherwise.) - If you'd like exceptions if you're adding someone to a list who is already there, or making someone unsubscribed who is not subscribed presently, instead of
upsertusecreateorupdate. - Common error codes: SubscriberNotFound (12001), ListNotFound (13000), OnListAlready (13006) (for a
createof a user of whatever status who is already on the list), NotOnList (13007) (for anupdateof a user not on the list). - If for an
upsertyou don't provide a status for the user, if the user is not on the list he will be added with status of Active. Otherwise, no change for the user (will remain Active or Unsubscribed as before.) Providing you don't delete users from lists but just mark them as Unsubscribed, this will allow you to add users to a list while respecting past unsubscribes.
In Java using the FuelSDK, subscription changes are handled via an UpdateRequest. Here, we take Subscriber object(s) in which we identify each user by his SubscriberKey, and then populate each Subscriber object's lists attribute with SubscriberList objects, with each object containing a List ID, action (create, update, delete, upsert, with the default apparently being upsert) and optionally status (Active, Unsubscribed), subject to the same rules given above for the direct SOAP calls. Sample adding and unsubscribing lists from a given Subscriber:
import com.exacttarget.fuelsdk.internal.Subscriber;
import com.exacttarget.fuelsdk.internal.SubscriberList;
import com.exacttarget.fuelsdk.internal.SubscriberStatus;
import com.exacttarget.fuelsdk.internal.UpdateOptions;
import com.exacttarget.fuelsdk.internal.UpdateRequest;
import com.exacttarget.fuelsdk.internal.UpdateResponse;
import com.exacttarget.fuelsdk.ETSdkException;
class BulkSubscriptionUpdate {
private Subscriber subscriber;
private final Set subscribes;
private final Set unsubscribes;
....
}
public UpdateResponse updateSubscriptions(BulkSubscriptionUpdate bulkSubscriptionUpdate) throws ETSubscriberServiceException {
UpdateRequest updateRequest = new UpdateRequest();
UpdateOptions options = new UpdateOptions();
options.setQueuePriority(Priority.HIGH);
options.setRequestType(RequestType.SYNCHRONOUS);
updateRequest.setOptions(options);
Subscriber subscriber = new Subscriber();
subscriber.setSubscriberKey(generateSubscriberKey(user.getId()));
List subscriptions = new ArrayList<>();
if (!CollectionUtils.isEmpty(bulkSubscriptionUpdate.getSubscribes())) {
for (Integer etid : bulkSubscriptionUpdate.getSubscribes()) {
subscriptions.add(buildSubscriptionUpdate(etid, SubscriberStatus.ACTIVE));
}
}
if (!CollectionUtils.isEmpty(bulkSubscriptionUpdate.getUnsubscribes())) {
for (Integer etid : bulkSubscriptionUpdate.getUnsubscribes()) {
subscriptions.add(buildSubscriptionUpdate(etid, SubscriberStatus.UNSUBSCRIBED));
}
}
if (!CollectionUtils.isEmpty(subscriptions)) {
subscriber.getLists().addAll(subscriptions);
updateRequest.getObjects().add(subscriber);
try {
return update(updateRequest);
} catch (ETSdkException e) {
LOGGER.error("Failed to update subscriptions for subscriber {}", bulkSubscriptionUpdate.getSubscriber(), e);
throw e;
}
}
// nothing to update
return null;
}
private static SubscriberList buildSubscriptionUpdate(Integer etid, SubscriberStatus status) {
SubscriberList subscription = new SubscriberList();
subscription.setId(etid);
subscription.setStatus(status);
subscription.setAction("upsert");
return subscription;
}
Posted by Glen Mazza in Salesforce at 07:00AM Aug 28, 2022 | Tags: marketingcloud salesforce | Comments[0]
Monday August 22, 2022
Deleting subscriber lists from Marketing Cloud
At work we used to create mailing lists on various search categories of interest to our readers, resulting in thousands of lists being created over the years. Since then we've switched to triggered sends, making these lists no longer of use, and hence we were looking for an automated way of deleting them.
Note it's very easy to delete a list, and in any automated process to delete a large number of lists it's important to make sure you're not accidentally deleting a list you need. For my work I checked any List ID to be deleted against a list of IDs that I knew our systems still used, disallowing the deletion were that the case. Such a process saved me from accidentally deleting two or three needed lists.
Some further points to note:
- Looking at the list of data views available, for some reason Marketing Cloud does not offer a "List" view to see all available lists in our account, similar to the Subscribers view to see all subscribers. Instead a ListSubscribers view is provided, which perhaps can be thought of as a link table between Subscribers and Lists. I can query this view to obtain lists that at least one person is (or was) subscribed to, probably getting me close to 100% of all lists if perhaps missing some that never had subscribers.
- We had far more lists than were displayed in the Marketing Cloud UI Email Studio (perhaps ten times more). I don't know why some are hidden from the UI. It could be lists created via Marketing Cloud SOAP calls will not be viewable, or those without certain parameters, etc.
- If you have comparatively few lists to delete, and they are visible in the Email Studio UI, they can simply be deleted there.
- Removing our unused lists made the Email Studio UI dramatically faster.
My first step was to get a list of lists in the system that I could examine to see which could be removed. To generate the list, I first created a data extension to store the results of a SQL query on ListSubscribers. Once I ran the query from Automation Studio, filling the extension in the process, I then viewed the extension in Email Studio, which also allows us to download the list from the browser. Here are the columns of the Data Extension that I used. What is needed to delete a list is its list ID, the other columns were used to help me determine whether each list is still in use.
I ran this SQL query on our main (production) Business Unit as it provides the most complete information about lists and their subscribers:
select ListID, ListName, ListType, count(*) as NumSubscribers, max(CreatedDate) as MaxCreateDate, max(DateUnsubscribed) as MaxUnsubscribeDate from _ListSubscribers group by ListID, ListName, ListType
I primarily needed the ListNames to determine which lists I could dispose of. In some cases, the MaxCreateDate and MaxUnsubscribe dates, which indicates the last time someone subscribed and unsubscribed to each list, was helpful in making that determination.
While I eventually needed a Java program to automate the deletion of our large number of unneeded lists, to get an idea of what would be needed, I first practiced with Postman to query and delete individual lists. SFMC offers an unofficial Postman workspace for making SOAP and REST MC calls, with documentation for using same on GitHub.
While the SFMC Postman collection doesn't provide out-of-the-box SOAP calls for Lists, it does have one for data extensions that I leveraged in creating commands for lists. The below request returns lists with three filter options: by list ID, list name (exact match), and list name (substring match), with no filter providing all lists up to a SFMC-determined maximum. The list ID that is to be subsequently used for deleting the list is the "id" value (see here for available values to query.) For a list query, it will be returned under the PartnerProperties element.
List query SOAP call:
<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
<s:Header>
<a:Action s:mustUnderstand="1">Retrieve</a:Action>
<a:To s:mustUnderstand="1">https://{{et_subdomain}}.soap.marketingcloudapis.com/Service.asmx</a:To>
<fueloauth xmlns="http://exacttarget.com">{{dne_etAccessToken}}</fueloauth>
</s:Header>
<s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<RetrieveRequestMsg xmlns="http://exacttarget.com/wsdl/partnerAPI">
<RetrieveRequest>
<ObjectType>List</ObjectType>
<Properties>ObjectID</Properties>
<Properties>id</Properties>
<Properties>CustomerKey</Properties>
<Properties>ListName</Properties>
<Properties>Category</Properties>
<!--Filter xsi:type="SimpleFilterPart">
<Property>id</Property>
<SimpleOperator>IN</SimpleOperator>
<Value>123456</Value>
<Value>234567</Value>
</Filter-->
<!--Filter xsi:type="SimpleFilterPart">
<Property>ListName</Property>
<SimpleOperator>like</SimpleOperator>
<Value>%News%</Value>
</Filter-->
<!--Filter xsi:type="SimpleFilterPart">
<Property>ListName</Property>
<SimpleOperator>equals</SimpleOperator>
<Value>My Sample Newsletter</Value>
</Filter-->
</RetrieveRequest>
</RetrieveRequestMsg>
</s:Body>
</s:Envelope>
Response fragment:
<Results xsi:type="List">
<PartnerKey xsi:nil="true" />
<PartnerProperties>
<Name>id</Name>
<Value>123456</Value>
</PartnerProperties>
<ObjectID>....</ObjectID>
<CustomerKey>....</CustomerKey>
<ListName>My Newsletter</ListName>
<Category>...folder ID...</Category>
</Results>
A SOAP call for deleting list(s) by their List IDs are as follows:
<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
<s:Header>
<a:Action s:mustUnderstand="1">Delete</a:Action>
<a:To s:mustUnderstand="1">https://{{et_subdomain}}.soap.marketingcloudapis.com/Service.asmx</a:To>
<fueloauth xmlns="http://exacttarget.com">{{dne_etAccessToken}}</fueloauth>
</s:Header>
<s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<DeleteRequest xmlns="http://exacttarget.com/wsdl/partnerAPI">
<Options></Options>
<Objects xsi:type="List">
<ID xsi:nil="true">123456</ID>
</Objects>
<!-- Add as many additional List IDs as desired to delete all in one call-->
<!--Objects xsi:type="List">
<ID xsi:nil="true">234567</ID>
</Objects-->
</DeleteRequest>
</s:Body>
</s:Envelope>
Switching over to Java, we created a utility that would read a list of ET List IDs that we no longer needed, and proceeded to delete them after confirming they weren't on our "needed" list. As shown in the above SOAP call, multiple lists can be deleted in just one call -- for the thousands we needed to delete, deleting 50 at a time seemed to work fastest, and our code managed about 3000 lists per minute.
Using the FuelSDK, pseudocode to delete lists would be as follows:
public DeleteResponse deleteListsInMC(java.util.List<Integer> etListIds) throws ETSdkException {
// etListIds checked earlier to confirm they are not needed (e.g., query against a table of in-use lists)
DeleteRequest deleteRequest = new DeleteRequest();
DeleteOptions options = new DeleteOptions();
options.setRequestType(RequestType.SYNCHRONOUS);
options.setQueuePriority(priority);
deleteRequest.setOptions(options);
for (Integer i : etListIds) {
List list = new List();
list.setId(i);
deleteRequest.getObjects().add(list);
}
// configure ETClient similar to here: https://salesforce.stackexchange.com/a/312178
// ETClient etClient = ....
return etClient.getSoapConnection().getSoap().delete(deleteRequest);
}
The returned DeleteResponse object returns a list of DeleteResult objects, one object per list deleted. The result objects have an ordinal ID which map 1-to-1 to the order of lists given in the delete request. Sample result processing code:
DeleteResponse dr = deleteListsInMC(listToDelete);
List<DeleteResult> delResult = dr.getResults();
for (DeleteResult drTemp : delResult) {
Integer drTempListID = listToDelete.get(drTemp.getOrdinalID());
// error codes: https://developer.salesforce.com/docs/marketing/marketing-cloud/guide/13000_13099_list_object.html
// Postman showing 13000 ErrorCode with ListNotFound status message in case of unknown lists
if ("OK".equals(drTemp.getStatusCode())) {
// OrdinalID is 0-based
LOGGER.info("Deleted list {} from MC", drTempListID);
} else if ("Error".equalsIgnoreCase(drTemp.getStatusCode())) {
if (drTemp.getErrorCode() == 13000) {
LOGGER.info("List {} not in MC, nothing to delete", drTempListID);
} else {
LOGGER.warn("Could not delete list {}: error code {} status message {}",
drTempListID, drTemp.getErrorCode(), drTemp.getStatusMessage());
}
}
}
Posted by Glen Mazza in Salesforce at 07:00AM Aug 22, 2022 | Tags: marketingcloud | Comments[0]
Saturday February 26, 2022
FuelSDK fork now has latest WSDL
For my FuelSDK fork I found another component that would benefit from updating, namely the Partner API WSDL defining the SOAP requests and responses, used by Apache CXF to generate Java classes representing those objects. The main Salesforce branch is still relying on a copy of the WSDL downloaded in 2017. A comparison of the two WSDLs is showing quite a few new and modified request and response objects since then:
ClientID: CustomerID TaskResult: TblAsyncID SaveOption: TrackChanges AccountUser: IsSendable APIObject: __AdditionalEmailAttribute 1-5. Attribute: no longer inheriting APIObject SubscriberResult: ErrorCodeID TriggeredSendClassEnum TriggeredSendSubClassEnum SenderProfile: FallbackFromAddress deleted: MessagingConfiguration ChatMessagingEventType SalesforceSendActivity ImportDefinition: HasMultipleFiles ImportResultsSummary: NumberRestricted JsonWebKey DirectoryTenant AuditLogUserContext AutomationActivity: SerializedObject AttributeEntityV1 Thumbnail NameIdReference CategorynameIdReference UserBasicsEntity AssetAnyProperty Asset Category ScheduledRequest ScheduledConversation
I did need to make two adjustments to the WSDL copied into my fork, as detailed on the project README and commented in the WSDL. For one of them--a mismatch between what the WSDL claims the SOAP response will be for a particular call and what it actually is, causing a validation exception with CXF--I've sent a help ticket to Marketing Cloud requesting they update their WSDL. This particular issue has been around a long time, apparently, as even the 2017 WSDL needed this manual adjustment.
Posted by Glen Mazza in Salesforce at 06:20AM Feb 26, 2022 | Tags: marketingcloud salesforce | Comments[0]
Monday February 14, 2022
FuelSDK fork now on Gradle and JDK 11
I updated my FuelSDK fork to JDK 11 and switched its build from Maven to Gradle. For those running CXF wsdl2java tasks in Maven and wishing to use Gradle instead, comparing the former pom and current build.gradle for this project should help in showing the process involved.
For the CXF wsdl2java process, I used Francisco Mateo's cxf-codegen-gradle plugin which offers a very nice user's guide. The FuelSDK also uses JavaCC to generate some model classes, which thankfully has its own Gradle plugin, allowing me to complete the build script conversion.
Posted by Glen Mazza in Salesforce at 07:00AM Feb 14, 2022 | Tags: salesforce cxf marketingcloud | Comments[0]
Sunday January 23, 2022
Fork of Marketing Cloud FuelSDK-Java created
I created a fork of Marketing Cloud's FuelSDK to use the latest Log4J (2.17.1 at this time of writing) and Apache CXF (3.5.0). My motivation was to fix the Log4J security issue as posters to the GitHub project were unable at the time to get the Marketing Cloud team's attention on this issue. Since the fork would need testing anyway, I also decided to upgrade CXF from the 3.1.2 dating to July 2015--which probably has its own security problems by now--to the latest 3.5.0 version from December 2021. (As of this writing, the Salesforce Team has upgraded their branch to the JDK 6-friendly Log4J 2.3.2 version, while their CXF version is apparently still at the 2015 version.)
While not all tests currently pass (those that haven't I marked @Ignored in my fork), I believe it is not related to the upgrade, that it would be the same story with main Marketing Cloud branch. I'm reluctant to run the tests fully as many are old and heavily commented out already and as I haven't a development instance of MC I'm concerned about a poorly written test damaging our production setup.
Still, through a couple of weeks at work, the fork has been running fine, we're using it to register and update subscribers and send emails, and it is providing us peace of mind that we're using the latest Log4J and CXF versions.
Posted by Glen Mazza in Salesforce at 05:28PM Jan 23, 2022 | Tags: marketingcloud | Comments[0]
Sunday July 04, 2021
Publishing Salesforce platform events via Apex triggers and REST controllers
For change data capture and push topic events, Salesforce itself generates event messages whenever specified changes to specified objects occur. Salesforce also offers platform and generic events to allow for users to send messages of a specified structure at times of the developer's choosing. For coverage of Platform Events, Salesforce offers a Developer Guide as well as a Trailhead Module for hands-on learning.
Salesforce's generic events have their own guide but generally provide less developer support compared to platform events. For one, it is only with the CometD client that generic events can be listened to, and these messages apparently have a message size limit of 3K compared to the 1 MB provided for platform events. On the other hand, some advantages for generic events include its ability to send any arbitrary JSON as well as specify the users to receive each message. While Platform event messages will stream as JSON, they don't appear to support JSON objects within its payload. Instead, JSON objects need to be streamed as JSON strings and manually unpacked from the client receiving the message.
In the steps below I'll be featuring additional functionality beyond that covered in the Trailhead tutorial, providing examples of including data from multiple object types (here, both Accounts and Contacts in one message) as well as publishing platform events via triggers attached to specified objects and via an Apex REST controller (the latter allowing external systems to request that an event be generated.). My Salesforce Java client from a previous tutorial includes support for calling Apex REST controllers and the separate Change Data Capture (CDC) listener can be modified to read the platform events generated, as CDC events are a specialized case of platform events.
Define the custom platform event. The Trailhead tutorial shows how platform events are defined. Here, I'm creating an AccountAndContacts__e event below, to show how we can create a message consisting of multiple sObject types (here, Account and Contact), as well as generate multiple messages to report on all contacts (there is a 128K limit for contacts per message, but for demonstration purposes below I'll be limiting each message to five contacts). Note the Contact field will exist as a JSON string, containing an array of objects (for this example, we'll have the User's Salesforce ID, name, and email address). From the perspective of the platform event, however, we can define this only as a string without specifying further definition for its contents.
Create an Apex method (and test class) for generating platform events.. This generator will create, for a given account, one platform event message for every five contacts at that account. I keep a "calls" static variable to indicate a platform event getting published, which helps immensely when writing tests. Note as given in the comments this variable's lifetime is just within- and per-transaction and not global as it would be in Java, further helping its usage in testing.
public class AccountAndContactsEventGenerator { // Counter to check that this generator is being called (useful for testing objects calling the generator) // https://salesforce.stackexchange.com/questions/204805/test-that-a-platform-event-was-published/204806 // Note, lifetime of this object per transaction, not system-wide as in Java, see: // https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_classes_static.htm public static Integer calls = 0; public static void publishAccountAndContactsEvents(Account account) { List<AccountAndContacts__e> events = getEventMessages(account); List<Database.SaveResult> results = EventBus.publish(events); for (Database.SaveResult sr : results) { if (sr.isSuccess()) { System.debug('Successfully published event: '+ sr); calls++; } else { System.debug('Error when publishing event: ' + sr); } } } @testVisible private static List<AccountAndContacts__e> getEventMessages(Account account) { List<Contact> contacts = [SELECT FirstName, LastName, Email FROM Contact WHERE AccountId = :account.id]; // Sending one event message for every five contacts at the company List<AccountAndContacts__e> eventList = new List<AccountAndContacts__e>(); AccountAndContacts__e event = initializeNewEvent(account); List<ContactData> contactList = new List<ContactData>(); for (Integer i = 0; i < contacts.size(); i++) { if (i > 0 && math.mod(i, 5) == 0) { event.Contacts__c = JSON.serialize(contactList); contactList.clear(); eventList.add(event); event = initializeNewEvent(account); } ContactData cd = new ContactData(); Contact con = contacts.get(i); cd.userId = con.Id; cd.emailAddress = con.Email; cd.lastName = con.LastName; cd.firstName = con.FirstName; contactList.add(cd); } event.Contacts__c = JSON.serialize(contactList); eventList.add(event); return eventList; } @testVisible private static AccountAndContacts__e initializeNewEvent(Account account) { AccountAndContacts__e event = new AccountAndContacts__e(); event.AccountName__c = account.name; event.AccountSFID__c = account.id; return event; } @testVisible private class ContactData { public String userId { get; set; } public String emailAddress { get; set ; } public String lastName { get; set; } public String firstName { get; set; } } }Test class: The test creates an account with six contacts (so therefore, two messages will be generated) and parses a sample of the AccountAndContacts__e messages generated to confirm they contain the expected contents. Note within Apex test cases it is not necessary to manually delete objects created as they will be rolled back automatically once the test completes.
@IsTest public class AccountAndContactsEventGeneratorTest { @IsTest private static void testEventGeneration() { Account account = new Account(); account.Name = 'Acme Corp'; insert account; List<Contact> contacts = new List<Contact>(); for (Integer i = 0; i < 6; i++) { contacts.add(createContact('First' + i, 'Last' + i, account.id)); } List<AccountAndContacts__e> messages = AccountAndContactsEventGenerator.getEventMessages(account); System.assertEquals(messages.size(), 2); AccountAndContacts__e message = messages.get(0); System.assertEquals(message.AccountName__c, 'Acme Corp'); System.assertEquals(message.AccountSFID__c, account.Id); List<AccountAndContactsEventGenerator.ContactData> firstFiveContacts = (List<AccountAndContactsEventGenerator.ContactData>) JSON.deserialize(messages.get(0).Contacts__c, List<AccountAndContactsEventGenerator.ContactData>.class); System.assertEquals(firstFiveContacts.size(), 5); System.assertEquals(firstFiveContacts.get(1).userId, contacts.get(1).Id); System.assertEquals(firstFiveContacts.get(2).firstName, 'First2'); System.assertEquals(firstFiveContacts.get(3).LastName, 'Last3'); System.assertEquals(firstFiveContacts.get(4).emailAddress, 'last4@yopmail.com'); List<AccountAndContactsEventGenerator.ContactData> sixthContact = (List<AccountAndContactsEventGenerator.ContactData>) JSON.deserialize(messages.get(1).Contacts__c, List<AccountAndContactsEventGenerator.ContactData>.class); System.assertEquals(sixthContact.size(), 1); } private static Contact createContact(String firstName, String lastName, Id accountId) { Contact contact = new Contact(); contact.FirstName = firstName; contact.LastName = lastName; contact.AccountId = accountId; contact.Email = lastName + '@yopmail.com'; insert contact; return contact; } }Create an Apex trigger (and test class) to activate generator based on desired criteria.. Here I'm placing an after-update trigger on the Account object that will trigger the generator if the Account's name has changed. This demonstrates how the generator can be selectively activated based on the nature of changes to an object.
trigger trigger_AccountUpdate on Account (after update) { for (Account a : Trigger.New) { if (!a.Name.equals(Trigger.OldMap.get(a.Id).Name)) { AccountAndContactsEventGenerator.publishAccountAndContactsEvents(a); } } }Test class: The trigger test checks that a platform event message is generated if and only if the Account name changes. As discussed earlier, I rely on the
callsstatic variable in the generator for this.@isTest public class TestAccountUpdate { @isTest static void testUpdateWithNameChangeGeneratesEvent() { Account account = new Account(Name = 'Acme Corp', BillingCity = 'Philadelphia'); insert account; account.name = 'New Acme Corp'; update account; System.assertEquals(1, AccountAndContactsEventGenerator.calls); } @isTest static void testUpdateWithNoNameChangeGeneratesNoEvent() { Account account = new Account(Name = 'Acme Corp', BillingCity = 'Philadelphia'); insert account; account.BillingCity = 'Baltimore'; update account; System.assertEquals(0, AccountAndContactsEventGenerator.calls); } }Create an Apex REST Controller (and test class) to activate generator for a specific Account.. If you're new to Salesforce REST Controllers, recommend its Trailhead tutorial.. It shows convenient ways of making REST calls via cURL and Salesforce's Developer Workbench. Additionally, my Salesforce client has an ApexRestCaller and integrated tests showing a way to call Apex endpoints from Java.
@RestResource(urlMapping='/AccountAndContactsPE/*') global with sharing class AccountAndContactsEventRESTEndpoint { @HttpPost global static void generatePlatformEvent() { RestRequest req = RestContext.request; String accountId = req.requestURI.substring(req.requestURI.lastIndexOf('/')+1); // using List<Account> instead of Account below for safer handling of no matching records // see: https://developer.salesforce.com/forums/?id=906F000000094QZIAY List<Account> accountList = [SELECT Id, Name FROM Account WHERE Id = :accountID]; Account account = (accountList != null && accountList.size() > 0) ? accountList[0] : null; if (account != null) { AccountAndContactsEventGenerator.publishAccountAndContactsEvents(account); } else { System.debug('Could not find Account w/ID = ' + accountId); } } }@isTest public class AccountAndContactsEventEndpointTest { @isTest static void testGeneratePlatformEventMessage() { // Testing method: https://trailhead.salesforce.com/content/learn/modules/apex_integration_services/apex_integration_webservices Account account = new Account(Name = 'RestTestAccount'); insert account; RestRequest request = new RestRequest(); request.httpMethod = 'POST'; request.requestUri = 'https://doesntmatterignored/services/apexrest/AccountAndContactsPE/' + account.id; RestContext.request = request; // call method AccountAndContactsEventRESTEndpoint.generatePlatformEvent(); System.assertEquals(1, AccountAndContactsEventGenerator.calls); } @isTest static void testNoAccountNoPlatformEventMessage() { // creating a contact just to get a SF ID not tied to an account. Contact contact = new Contact(firstname='First', lastname='Last'); // Testing method: https://trailhead.salesforce.com/content/learn/modules/apex_integration_services/apex_integration_webservices Account account = new Account(Name = 'RestTestAccount'); insert account; RestRequest request = new RestRequest(); request.httpMethod = 'POST'; request.requestUri = 'https://doesntmatterignored/services/apexrest/AccountAndContactsPE/' + contact.id; RestContext.request = request; // call method, no PE because no Account with Contact's SF ID AccountAndContactsEventRESTEndpoint.generatePlatformEvent(); System.assertEquals(0, AccountAndContactsEventGenerator.calls); } }Subscribe to the event messages. I added a JSON supporting class and processor for this particular platform event to my Salesforce Event Listener covered in an earlier tutorial. The sample logs the contents of any messages received. Due to the trigger created in this tutorial, all that is needed is to change the name of an account in Salesforce and messages, one for every five contacts at the company, will be sent out and picked up by this listener.
Posted by Glen Mazza in Salesforce at 07:00AM Jul 04, 2021 | Tags: java salesforce | Comments[0]
Sunday May 23, 2021
Creating Chrome- and Firefox-compliant local development certificates
Below shows the steps I followed for creating keys and certificates for local development (at https://localhost:port#) of Tomcat- and Webpack DevServer-powered web applications. The process involves creating a local certificate authority (CA) with a self-signed certificate imported into Firefox and Chrome. Then I created a server key and certificate, the latter signed by the CA, to be used by both application servers. This is for work on a Mac OS with LibreSSL 2.6.5 used for the key commands, the process will vary a bit with other OS's or OpenSSL variants.
Before proceeding, there are a couple of shortcuts for working with self-signed certificates for local development, if perhaps you have only a little bit of development to do and can stand the browser unpleasantries during that time. For Firefox, you can choose to ignore the "self-signed cert" warning, with the development pages continually marked as "not secure" as a consequence. Chrome also provides a couple of options (here and here) for the same. Finally, if your motivation in creating a new key is because you've lost the public key and/or cert for a given private key, see this note on how both can be regenerated from that private key.
Create a Certificate Authority whose certificate will be imported into Firefox and Chrome. Although this certificate will be self-signed, the certificate for the server key that will be used by Tomcat and WDS will be signed by this CA. For these steps, I'm using genpkey to generate the private key and req to sign it, with a lifespan of 825 days as that's apparently the max permitted on MacOS.
(For the commands in this entry, using folders of ../certs and ../certs/ca)
openssl genpkey -algorithm RSA -out ca/MyCA.key -pkeyopt rsa_keygen_bits:2048 -aes-256-cbc openssl req -new -sha256 -key ca/MyCA.key -out ca/MyCA.csr openssl x509 -req -sha256 -days 825 -in ca/MyCA.csr -signkey ca/MyCA.key -out ca/MyCA.crt
Notes:
- The -aes-256-cbc setting provides for key encryption, for which you'll be asked to provide a password.
- "-sha256" is apparently the minimum accepted by Chrome, see here for further discussion.
- If asked for a challenge password, that can apparently be left blank.
- As for the configuration requested during the OpenSSL req command, I used "MyRootCA" as the Common Name for greater readability when viewing certificate chains. Note it needs to be different from the Common Name of any server certificate that it signs.
- Contents of the private key, certificate signing request, and certificate can be viewed using commands such as these:
openssl pkey -in MyCA.key -text -noout openssl req -text -in MyCA.csr -noout openssl x509 -text -in MyCA.crt -noout
Import the CA certificate into Firefox and Chrome.
For Firefox, menu item Firefox -> Preferences -> Privacy & Security -> View Certificates button -> Authorities -> Import MyCA.crt, then select "Trust this CA to identify websites." The CA will be listed on the Authorities tab under the Organization name you gave when creating the CSR.
Chrome uses Apple's Keychain Access to store certificates. It can be activated from menu Chrome -> Preferences -> Privacy & Security -> Security Tab -> Manage Certificates. However, I found it clumsy to work with and simpler to use the command line:
sudo security add-trusted-cert -k /Library/Keychains/System.keychain -d ca/MyCA.crt
Once run, you'll find it under the system keychain, "Certificates" category in Keychain Access.
Create the server key in which you specify the domain name(s) applications using the key will be using. First thing to note is that Chrome requires usage of the subjectAltName extension when creating the key, Common Name alone will not work. There are several ways to configure this extension, the simplest I found that would work with my version of LibreSSL was to use an extension file as explained in the OpenSSL cookbook. (Note "TightBlog" refers to my open source project.)
Place in servercert.ext:
subjectAltName = DNS:localhost
Multiple domains can be specified, just make them comma-delimited.
Then run these commands:
openssl genpkey -algorithm RSA -out tightblog.key -pkeyopt rsa_keygen_bits:2048 openssl req -new -sha256 -key tightblog.key -out tightblog.csr openssl x509 -req -in tightblog.csr -CA ca/MyCA.crt -CAkey ca/MyCA.key -CAcreateserial -out tightblog.crt -days 824 -sha256 -extfile servercert.ext
Configure the keys and/or certs on the development servers. For TightBlog development, the application runs on Tomcat, however I use Webpack DevServer while developing the Vue pages, so I have two servers to configure. SSL information for Tomcat is here and for WDS is here.
For Vue, I create a local-certs.js in the same directory as my vue.config.js which contains:
const fs = require("fs"); module.exports = { key: fs .readFileSync("/Users/gmazza/opensource/certs/tightblog.key") .toString(), cert: fs .readFileSync("/Users/gmazza/opensource/certs/tightblog.crt") .toString() };For Tomcat, I found Jens Grassel's instructions to be useful. He has us create a PKCS #12 key-and-certificate-chain bundle followed by usage of Java keytool to import the bundle into the keystore configured in the Tomcat server.xml file:
openssl pkcs12 -export -in tightblog.crt -inkey tightblog.key -chain -CAfile MyCA.crt -name "MyTomcatCert" -out tightblogForTomcat.p12 keytool -importkeystore -deststorepass changeit -destkeystore /Users/gmazza/.keystore -srckeystore tightblogForTomcat.p12 -srcstoretype PKCS12
For Tomcat, you'll want no more than one alias (here, "MyTomcatCert") in the keystore, or specify the keyAlias in the Tomcat server.xml. The keytool list certs and delete alias commands can help you explore and adjust the Tomcat keystore.
I activated the application in both browsers and checked the URL bar to confirm that the certificates were accepted. For my local development I have the application running on Tomcat at https://localhost:8443/ and the Vue pages running on WDS at https://localhost:8080. Examples showing the Vue URL on Firefox and the Tomcat one on Chrome are as below. Both URLs were accepted by both browsers, but note Firefox does caution that the CA the cert was signed with is not one of the standard CA certs that it ships with.
Posted by Glen Mazza in Programming at 07:00AM May 23, 2021 | Comments[1]
Calendar
About Me
Blog Search
Links
Recent Posts
Recent Comments
- - serxoz on Creating Chrome- and Firefox-compliant local development certificates
- - Nitin Lokhande on Deploying TightBlog on Linode
- - Ralph Callaway on Activating Transport Layer Security (SSL) for CXF web services
- - Vlad BALAN on Using X.509 security with Apache CXF
- - Juan Moreno on Switching from SOAP 1.1 to SOAP 1.2 messages
- - Anant Jaynarayana on Deploying TightBlog on Linode
About Blog
Application Server: Tomcat
Database: MySQL
Hosted on: Linode
SSL Certificate: Let's Encrypt