Glen Mazza's Weblog

https://glenmazza.net/blog/date/20191211 Wednesday December 11, 2019

TightBlog 3.6 Released!

Just-released TightBlog 3.6 (release page/screenshots/installation instructions) now provides an endpoint for embedding GitHub source code in your blog entries, see here for a sample and the Tightblog Wiki for more details. Previously I had to rely on third party tools hosted on other servers to accomplish the same, but happy to report this functionality is now available out-of-the-box with TightBlog.

Posted by Glen Mazza in Programming at 07:00AM Dec 11, 2019 | Comments[0]

https://glenmazza.net/blog/date/20191106 Wednesday November 06, 2019

Improving validation feedback in TightBlog

Tom Homberg provided a nice guide for implementing user-feedback validation within Spring applications, quite helpful for me in improving what I had in TightBlog. He creates a field - message Violation object (e.g., {"Name" "Name is required"}), a list of which is wrapped by a ValidationErrorResponse, the latter of which gets serialized to JSON and sent to the client to display validation errors. For my own implementation, I left the field value blank to display errors not specific to a particular field, and used it for both sending 400-type for user-input problems and generic 500-type messages for system errors.

Implementing this validation for TightBlog's blogger UI, I soon found it helpful to have convenience methods for quick creation of the Violations, ValidationErrorResponses and Spring ResponseEntities for providing feedback to the client:

public static ResponseEntity<ValidationErrorResponse> badRequest(String errorMessage) {
    return badRequest(new Violation(errorMessage));
}

public static ResponseEntity<ValidationErrorResponse> badRequest(Violation error) {
    return badRequest(Collections.singletonList(error));
}

public static ResponseEntity<ValidationErrorResponse> badRequest(List errors) {
    return ResponseEntity.badRequest().body(new ValidationErrorResponse(errors));
}

i18n can be handled via the Locale method argument, one of the parameters automatically provided by Spring:

@Autowired
private MessageSource messages;

@PostMapping(...)
public ResponseEntity doFoo(Locale locale) {
    ...

    if (error) {
        return ValidationErrorResponse.badRequest(messages.getMessage("mediaFile.error.duplicateName", null, locale));
    }
}

On the front-end, I have Angular.js trap the code and then output the error messages (am not presently not using the field names). Below truncated for brevity (full source: JavaScript and JSP):

this.commonErrorResponse = function(response) {   
    self.errorObj = response.data;
}

<div id="errorMessageDiv" class="alert alert-danger" role="alert" ng-show="ctrl.errorObj.errors" ng-cloak>
    <button type="button" class="close" data-ng-click="ctrl.errorObj.errors = null" aria-label="Close">
       <span aria-hidden="true">×</span>
    </button>
    <ul class="list-unstyled">
       <li ng-repeat="item in ctrl.errorObj.errors">{{item.message}}</li>
    </ul>
</div>

Appearance:

400validation

Additionally, I was able to remove a fair amount of per-endpoint boilerplate by creating a single ExceptionHandler for unexpected 500 response code system errors and attaching it to my ControllerAdvice class so it would be used by all REST endpoints. For these types of exceptions usually a generic "System error occurred, please contact Administrator" message is sent to the user. However, I added a UUID that both appears on the client and goes into the logs along with the exception details, making it easy to search the logs for the specific problem. The exception handler (from the TightBlog source):

@ExceptionHandler(value = Exception.class)
// avoiding use of ResponseStatus as it activates Tomcat HTML page (see ResponseStatus JavaDoc)
public ResponseEntity<ValidationErrorResponse> handleException(Exception ex, Locale locale) {
    UUID errorUUID = UUID.randomUUID();
    log.error("Internal Server Error (ID: {}) processing REST call", errorUUID, ex);

    ValidationErrorResponse error = new ValidationErrorResponse();
    error.getErrors().add(new Violation(messages.getMessage(
            "generic.error.check.logs", new Object[] {errorUUID}, locale)));

    return ResponseEntity.status(500).body(error);
}

Screen output:

500errordisplay

Log messaging containing the same UUID:

LogStackTrace

Additional Resources

Posted by Glen Mazza in Programming at 07:00AM Nov 06, 2019 | Comments[0]

https://glenmazza.net/blog/date/20190930 Monday September 30, 2019

TightBlog 3.5.3 Released

Downloads here, deployment instructions here.

Posted by Glen Mazza in Programming at 04:33PM Sep 30, 2019 | Tags:  tightblog | Comments[0]

https://glenmazza.net/blog/date/20190616 Sunday June 16, 2019

TightBlog 3.5 Released!

And this blog is running it! Information here, including migration-from-3.4 notes. Alas, a couple of bugs found when running this version, I hope to get a 3.5.1 out next weekend to fix them. Update: 3.5.1 released!

Posted by Glen Mazza in Programming at 07:57PM Jun 16, 2019 | Comments[0]

https://glenmazza.net/blog/date/20181027 Saturday October 27, 2018

ElasticSearch Notes: Complex Date Filtering, Bulk Updates

Some things learned this past week with ElasticSearch:

Advanced Date Searches: A event search page my company provides for its Pro customers allows for filtering by start date and end date, however some events do not have an end date defined. We decided to have differing business rules on what the start and end dates will filter based on whether or not the event has an end date, specifically:

  • If an event has both start and end dates:
    1. The start date of the range filter, if provided, must be before the end date of the event
    2. The end date of the range filter, if provided, must be after the start date of the event
  • If an event does not have an end date:
    1. The start date of the range filter, if provided, must be before the start date of the event
    2. The end date of the range filter, if provided, must be after the start date of the event

The above business logic had to be implemented in Java but as an intermediate step I first worked out an ElasticSearch query out of it using Kibana. Creating the query first helps immensely in the subsequent conversion to code. For the ElasticSearch query, this is what I came up with (using arbitrary sample dates to test the queries):

GET events-index/_search
{
    "query": {
    "bool": {
        "should" : [
        {"bool" :
            {"must": [
                { "exists": { "field": "eventMeta.dateEnd" }},
                { "range" : { "eventMeta.dateStart": { "lte": "2018-09-01"}}},
                { "range" : { "eventMeta.dateEnd": { "gte": "2018-10-01"}}}
                ]
            }
        },
        {"bool" :
            {"must_not": { "exists": { "field": "eventMeta.dateEnd"}},
             "must": [
                { "range" : { "eventMeta.dateStart": { "gte": "2018-01-01", "lte": "2019-12-31"}}}
                ]
            }
        }
    ]
    }
}
}

As can be seen above, I first used a nested Bool query to separate the two main cases, namely events with and without and end date. The should at the top-level bool acts as an OR, indicating documents fitting either situation are desired. I then added the additional date requirements that need to hold for each specific case.

With the query now available, mapping to Java code using ElasticSearch's QueryBuilders (API) was very pleasantly straightforward, one can see the roughly 1-to-1 mapping of the code to the above query (the capitalized constants in the code refer to the relevant field names in the documents):

private QueryBuilder createEventDatesFilter(DateFilter filter) {

    BoolQueryBuilder mainQuery = QueryBuilders.boolQuery();

    // query modeled as a "should" (OR), divided by events with and without an end date,
    // with different filtering rules for each.
    BoolQueryBuilder hasEndDateBuilder = QueryBuilders.boolQuery();
    hasEndDateBuilder.must().add(QueryBuilders.existsQuery(EVENT_END_DATE));
    hasEndDateBuilder.must().add(fillDates(EVENT_START_DATE, null, filter.getStop()));
    hasEndDateBuilder.must().add(fillDates(EVENT_END_DATE, filter.getStart(), null));
    mainQuery.should().add(hasEndDateBuilder);

    BoolQueryBuilder noEndDateBuilder = QueryBuilders.boolQuery();
    noEndDateBuilder.mustNot().add(QueryBuilders.existsQuery(EVENT_END_DATE));
    noEndDateBuilder.must().add(fillDates(EVENT_START_DATE, filter.getStart(), filter.getStop()));
    mainQuery.should().add(noEndDateBuilder);

    return mainQuery;
}

Bulk Updates: We use a "sortDate" field to indicate the specific date front ends should use for sorting results (whether ascending or descending, and regardless of the actual source of the date used to populate that field). For our news stories we wanted to rely on the last update date for stories that have been updated since their original publish, the published date otherwise. For certain older records loaded it turned out that the sortDate was still at the publishedDate when it should have been set to the updateDate. For research I used the following query to determine the extent of the problem:

GET news-index/_search
{
   "query": {
   "bool": {
      "must": [
         { "exists": { "field": "meta.updateDate" }},
         {
            "script": {
               "script": "doc['meta.dates.sortDate'].value.getMillis() < doc['meta.updateDate'].value.getMillis()"
            }
         }
      ]
   }
    }
}

For the above query I used a two part Bool query, first checking for a non-null updateDate in the first clause and then a script clause to find sortDates preceding updateDates. (I found I needed to use .getMillis() for the inequality check to work.)

Next, I used ES' Update by Query API to do an all-at-once update of the records. The API has two parts, an optional query element to indicate the documents I wish to have updated (strictly speaking, in ES, to be replaced with a document with the requested changes) and a script element to indicate the modifications I want to have done to those documents. For my case:

POST news-index/_update_by_query
{
   "script": {
   "source": "ctx._source.meta.dates.sortDate = ctx._source.meta.updateDate",
   "lang": "painless"
},
   "query": {
      "bool": {
         "must": [
            { "exists": { "field": "meta.updateDate" }},
            {
               "script": {
                  "script": "doc['meta.dates.sortDate'].value.getMillis() < doc['meta.updateDate'].value.getMillis()"
               }
            }
         ]
      }
   }
}

For running your own updates, good to test first by making a do-nothing update in the script (e.g., set sortDate to sortDate) and specifying just one document to be so updated, which can be done by adding a document-specific match requirement to the filter query (e.g., { "match": { "id": "...." }},"). Kibana should report that just one document was "updated", if so switch to the desired update to confirm that single record was updated properly, and then finally remove the match filter to have all desired documents updated.

Posted by Glen Mazza in Programming at 07:00AM Oct 27, 2018 | Comments[0]

https://glenmazza.net/blog/date/20181007 Sunday October 07, 2018

Using functions with a single generic method to convert lists

For converting from a Java collection say List<Foo> to any of several other collections List<Bar1>, List<Bar2>, ... rather than create separate FooListToBar1List, FooListToBar2List, ... methods a single generic FooListToBarList method and a series of Foo->Bar1, Foo->Bar2... converter functions can be more succinctly used. The below example converts a highly simplified List of SaleData objects to separate Lists of Customer and Product information, using a common generic saleDataListToItemList(saleDataList, converterFunction) method along with passed-in converter functions saleDataToCustomer and saleDataToProduct. Of particular note is how the converter functions are specified in the saleDataListToItemList calls. In the case of saleDataToCustomer, which takes two arguments (the SailData object and a Region string), a lambda expression is used, while the Product converter can be specified as a simple method reference due to it having only one parameter (the SailData object).

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.function.Function;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class Main {

    public static void main(String[] args) {

        List saleDataList = new ArrayList<>();
        saleDataList.add(new SaleData("Bob", "radio"));
        saleDataList.add(new SaleData("Sam", "TV"));
        saleDataList.add(new SaleData("George", "laptop"));

        List customerList = saleDataListToItemList(saleDataList, sd -> Main.saleDataToCustomerWithRegion(sd, "Texas"));
        System.out.println("Customers: ");
        customerList.forEach(System.out::println);

        List productList = saleDataListToItemList(saleDataList, Main::saleDataToProduct);
        System.out.println("Products: ");
        productList.forEach(System.out::println);
    }

    private static  List saleDataListToItemList(List sdList, Function converter) {
        // handling potentially null sdList:  https://stackoverflow.com/a/43381747/1207540
        return Optional.ofNullable(sdList).map(List::stream).orElse(Stream.empty()).map(converter).collect(Collectors.toList());
    }

    private static Product saleDataToProduct(SaleData sd) {
        return new Product(sd.getProductName());
    }

    private static Customer saleDataToCustomerWithRegion(SaleData sd, String region) {
        return new Customer(sd.getCustomerName(), region);
    }

    private static class SaleData {
        private String customerName;
        private String productName;

        SaleData(String customerName, String productName) {
            this.customerName = customerName;
            this.productName = productName;
        }

        String getProductName() {
            return productName;
        }

        String getCustomerName() {
            return customerName;
        }

    }

    private static class Product {
        private String name;

        Product(String name) {
            this.name = name;
        }

        @Override
        public String toString() {
            return "Product{" +
                    "name='" + name + '\'' +
                    '}';
        }
    }

    private static class Customer {
        private String name;
        private String region;

        Customer(String name, String region) {
            this.name = name;
            this.region = region;
        }

        @Override
        public String toString() {
            return "Customer{" +
                    "name='" + name + '\'' +
                    ", region='" + region + '\'' +
                    '}';
        }
    }

}

Output from running:

Customers: 
Customer{name='Bob', region='Texas'}
Customer{name='Sam', region='Texas'}
Customer{name='George', region='Texas'}
Products: 
Product{name='radio'}
Product{name='TV'}
Product{name='laptop'}

Posted by Glen Mazza in Programming at 07:00AM Oct 07, 2018 | Comments[0]

https://glenmazza.net/blog/date/20180624 Sunday June 24, 2018

TightBlog 3.0 Released!

My third annual release currently powering this blog. See here for a listing of enhancements over the previous TightBlog 2.0, here for all the enhancements over the original Apache Roller 5.1.0 I had forked in 2015. Screenshots are here.

Posted by Glen Mazza in Programming at 04:36PM Jun 24, 2018 | Comments[1]

https://glenmazza.net/blog/date/20180506 Sunday May 06, 2018

Using SAAJ to call RPC/encoded SOAP web services

The National Weather Service's legacy National Digital Forecast Service (WSDL) uses the older rpc/encoded SOAP binding style not ordinarily supported by JAX-WS implementations (the NWS has since switched to a REST-based API). The WS-I Basic Profile limits binding styles to either Document/literal or RPC/literal, and JAX-WS was designed to honor this limitation. The reason for excluding RPC/encoded was apparently due to compatibility issues involved with encoding, as well as possibly message size and performance issues.

Russell Butek has written an informative article explaining the different SOAP binding styles, their appearance over the wire, and the advantages and disadvantages of each. To show the binding differences between the RPC/encoded and standard Doc/Literal bindings, I've copied an NWS and an Amazon Commerce Service (WSDL) operation below:

<binding name="ndfdXMLBinding" type="tns:ndfdXMLPortType">
   <soap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http"/>
   <operation name="NDFDgen">
      <soap:operation 
         soapAction="http://www.weather.gov/forecasts/xml/DWMLgen/wsdl/ndfdXML.wsdl#NDFDgen" 
         style="rpc"/>
      <input>
         <soap:body use="encoded" 
            namespace="http://www.weather.gov/forecasts/xml/DWMLgen/wsdl/ndfdXML.wsdl" 
            encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/>
      </input>
      <output>
         <soap:body use="encoded" 
            namespace="http://www.weather.gov/forecasts/xml/DWMLgen/wsdl/ndfdXML.wsdl"
            encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/>
      </output>
   </operation>
   ...
</binding>

<binding name="AWSECommerceServiceBinding" type="tns:AWSECommerceServicePortType">
   <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
   <operation name="ItemSearch">
      <soap:operation soapAction="http://soap.amazon.com"/>
      <input>
         <soap:body use="literal"/>
      </input>
      <output>
         <soap:body use="literal"/>
      </output>
   </operation>
   ...
</binding>

In addition to requiring Doc/Lit or RPC/Lit bindings, the WS-I Basic Profile also prohibits the use of the encodingStyle attribute (R1005-R1007) and namespace attributes (R2716-17; R2726), restrictions you can see honored above with the Amazon ItemSearch operation.

Web service implementations that natively support JAX-RPC service calls include Oracle's JAX-RPC implementation as well as Axis 1.x, both long deprecated. Attempting to run Apache CXF's wsdl2java with the NWS' RPC/encoded WSDL returns an "Rpc/encoded wsdls are not supported in JAXWS 2.0" error message. However the SOAP with Attachments API for Java (SAAJ) can be used with the JAX-WS Dispatch interface to create "raw" web service calls to a web service provider that uses RPC/encoded bindings. The following SOAP client provides two such examples of using SAAJ. A simple way to run this example would be to download my intro Web Service tutorial source code, replace its client subproject's WSClient class with the WSClient below and run mvn clean install exec:exec from the client folder.

package client;

import java.io.StringReader;
import java.net.URL;

import javax.xml.namespace.QName;
import javax.xml.soap.MessageFactory;
import javax.xml.soap.Name;
import javax.xml.soap.SOAPBody;
import javax.xml.soap.SOAPBodyElement;
import javax.xml.soap.SOAPElement;
import javax.xml.soap.SOAPFactory;
import javax.xml.soap.SOAPFault;
import javax.xml.soap.SOAPHeader;
import javax.xml.soap.SOAPMessage;
import javax.xml.soap.SOAPPart;
import javax.xml.transform.Source;
import javax.xml.transform.stream.StreamSource;
import javax.xml.ws.Dispatch;
import javax.xml.ws.Service;
import javax.xml.ws.soap.SOAPFaultException;

public class WSClient {

    public static void main (String[] args) {
        WSClient wsc = new WSClient();

        // get forecast by Zip Code
        wsc.getWeatherForecast("19110"); // Philadelphia

        // get another forecast
        wsc.getWeatherForecast("33157"); // Miami
    }

    private void getWeatherForecast(String zipCode) {

        try {
            // Convert the ZIP code to a geocoded value (which is needed
            // as input for the weather data)

            String nsSchema = "http://graphical.weather.gov/xml/DWMLgen/schema/DWML.xsd";

            String soapSchema = "http://schemas.xmlsoap.org/soap/envelope/";

            String xsiSchema
                    = "http://www.w3.org/2001/XMLSchema-instance";

            String encodingStyle
                    = "http://schemas.xmlsoap.org/soap/encoding/";

            String zipRequest = " "
                    + ""
                    +"   "
                    +           ""
                    +               zipCode
                    +           ""
                    +    ""
                    +""
                    +"";

            String wsdl = "https://graphical.weather.gov/xml/SOAP_server/ndfdXMLserver.php?wsdl";
            String targetNS = "https://graphical.weather.gov/xml/DWMLgen/wsdl/ndfdXML.wsdl";

            URL url = new URL(wsdl);
            QName serviceName = new QName(targetNS, "ndfdXML");
            QName portName = new QName(targetNS, "ndfdXMLPort");
            Service service = Service.create(url, serviceName);

            /*
             * JAX-WS Dispatch provides three usage options: -- JAXBContext
             * (unsure if works though for rpc/enc WSDL) -- JAXP Source objects
             * (used here) -- SAAJ SOAPMessages (used in 2nd request below)
             */
            Dispatch dispatch = service.createDispatch(portName,
                    Source.class, Service.Mode.MESSAGE);
            Source zipResponse = dispatch.invoke(
                    new StreamSource(new StringReader(zipRequest)));
            // if using a file for input instead:
            // new StreamSource(new File("myrequest.xml")));

            // use SAAJ to open message -- check if error or valid data
            MessageFactory msgFactory = MessageFactory.newInstance();
            SOAPMessage geocodeMsg = msgFactory.createMessage();
            SOAPPart env = geocodeMsg.getSOAPPart();
            env.setContent(zipResponse);
            // writeTo method outputs SOAPMessage, helpful for debugging
            // geocodeMsg.writeTo(System.out);

            if (geocodeMsg.getSOAPBody().hasFault()) {
                // Copy official error response into our LNF Fault
                SOAPFault fault = geocodeMsg.getSOAPBody().getFault();
                System.out.println("Could not obtain forecast for zipcode "
                        + zipCode + ": "
                        + fault.getFaultString() + "; " + fault.getDetail().getValue());
            }

            // From here: valid geocode is present-- so get weather report next

            /*
             * LatLonListZipCodeResponse is not very helpful; needed information
             * (latLonList) element is html-escaped instead of a real tag, which
             * is suitable for HTML responses but not so helpful when you need
             * to extract the value. So will need to parse string response to
             * get geocode values  
             *      35.1056,-90.007
             *   
             *  
             * 
             */
            String geocodeBuffer = geocodeMsg.getSOAPBody().
                    getElementsByTagName("listLatLonOut")
                    .item(0).getFirstChild().getNodeValue();

            // .getNodeValue() unescapes HTML string
            String geocodeVals = geocodeBuffer.substring(
                    geocodeBuffer.indexOf("") + 12,
                    geocodeBuffer.indexOf(""));
            System.out.println("Geocode Vals for zip code " + zipCode
                    + " are: " + geocodeVals);

            /*
             * NDFDgenLatLonList operation: gets weather data for a given
             * latitude, longitude pair
             *
             * Format of the Message:     38.99,-77.02 
             *  glance
             *    
             */
            SOAPFactory soapFactory = SOAPFactory.newInstance();
            SOAPMessage getWeatherMsg = msgFactory.createMessage();
            SOAPHeader header = getWeatherMsg.getSOAPHeader();
            header.detachNode();  // no header needed
            SOAPBody body = getWeatherMsg.getSOAPBody();
            Name functionCall = soapFactory.createName(
                    "NDFDgenLatLonList", "schNS",
                    nsSchema);
            SOAPBodyElement fcElement = body.addBodyElement(functionCall);
            Name attname = soapFactory.createName("encodingStyle", "S",
                    soapSchema);
            fcElement.addAttribute(attname, soapSchema);
            SOAPElement geocodeElement = fcElement.addChildElement("listLatLon");
            geocodeElement.addTextNode(geocodeVals);
            SOAPElement product = fcElement.addChildElement("product");
            product.addTextNode("glance");

            // make web service call using this SOAPMessage
            Dispatch smDispatch = service.createDispatch(portName,
                    SOAPMessage.class, Service.Mode.MESSAGE);
            SOAPMessage weatherMsg = smDispatch.invoke(getWeatherMsg);
            // weatherMsg.writeTo(System.out); // debugging only

            // Metro needs normalize() command because it breaks
            // up child dwml element into numerous text nodes.
            weatherMsg.getSOAPBody().getElementsByTagName("dwmlOut")
                    .item(0).normalize();

            // First child of dwmlOut is the dwml element that we need.
            // It is the root node of the weather data that we will
            // be using to generate the report.
            String weatherResponse = weatherMsg.getSOAPBody().
                    getElementsByTagName("dwmlOut")
                    .item(0).getFirstChild().getNodeValue();
            System.out.println("WR: " + weatherResponse);
        } catch (SOAPFaultException e) {
            System.out.println("SOAPFaultException: " + e.getFault().getFaultString());
        } catch (Exception e) {
            System.out.println("Exception: " + e.getMessage());
        }
    }
}

In the client code above I used the rather nonintuitive DOM Tree API to get the data elements I needed, for example:

String weatherResponse = weatherMsg.getSOAPBody().getElementsByTagName("dwmlOut")
    .item(0).getFirstChild().getNodeValue();

If you have many such calls to make, another option is to use XPath, see tutorials from Baeldung and TutorialsPoint for more information.

Posted by Glen Mazza in Web Services at 07:00AM May 06, 2018 | Comments[0]


Calendar
« December 2019
Sun Mon Tue Wed Thu Fri Sat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Today
About Me
Java Software Engineer
TightBlog maintainer, Apache CXF committer
Arlington, Virginia USA
gmazza at apache dot org
GitHub LinkedIn
Blog Search
Apache CXF/SOAP tutorial
Blog article index


Today's Blog Hits: 1546

Navigation
About Blog
Blog software: TightBlog 3.6.0
Application Server: Tomcat
Database: MySQL
Hosted on: Linode
SSL Certificate: Let's Encrypt
Installation Instructions