Glen Mazza's Weblog

« Older | Main

https://glenmazza.net/blog/date/20180624 Sunday June 24, 2018

TightBlog 3.0 Released!

My third annual release currently powering this blog. See here for a listing of enhancements over the previous TightBlog 2.0, here for all the enhancements over the original Apache Roller 5.1.0 I had forked in 2015. Screenshots are here.

https://glenmazza.net/blog/date/20180506 Sunday May 06, 2018

Using SAAJ to call RPC/encoded SOAP web services

The National Weather Service's legacy National Digital Forecast Service (WSDL) uses the older rpc/encoded SOAP binding style not ordinarily supported by JAX-WS implementations (the NWS has since switched to a REST-based API). The WS-I Basic Profile limits binding styles to either Document/literal or RPC/literal, and JAX-WS was designed to honor this limitation. The reason for excluding RPC/encoded was apparently due to compatibility issues involved with encoding, as well as possibly message size and performance issues.

Russell Butek has written an informative article explaining the different SOAP binding styles, their appearance over the wire, and the advantages and disadvantages of each. To show the binding differences between the RPC/encoded and standard Doc/Literal bindings, I've copied an NWS and an Amazon Commerce Service (WSDL) operation below:

<binding name="ndfdXMLBinding" type="tns:ndfdXMLPortType">
   <soap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http"/>
   <operation name="NDFDgen">
      <soap:operation 
         soapAction="http://www.weather.gov/forecasts/xml/DWMLgen/wsdl/ndfdXML.wsdl#NDFDgen" 
         style="rpc"/>
      <input>
         <soap:body use="encoded" 
            namespace="http://www.weather.gov/forecasts/xml/DWMLgen/wsdl/ndfdXML.wsdl" 
            encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/>
      </input>
      <output>
         <soap:body use="encoded" 
            namespace="http://www.weather.gov/forecasts/xml/DWMLgen/wsdl/ndfdXML.wsdl"
            encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/>
      </output>
   </operation>
   ...
</binding>

<binding name="AWSECommerceServiceBinding" type="tns:AWSECommerceServicePortType">
   <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
   <operation name="ItemSearch">
      <soap:operation soapAction="http://soap.amazon.com"/>
      <input>
         <soap:body use="literal"/>
      </input>
      <output>
         <soap:body use="literal"/>
      </output>
   </operation>
   ...
</binding>

In addition to requiring Doc/Lit or RPC/Lit bindings, the WS-I Basic Profile also prohibits the use of the encodingStyle attribute (R1005-R1007) and namespace attributes (R2716-17; R2726), restrictions you can see honored above with the Amazon ItemSearch operation.

Web service implementations that natively support JAX-RPC service calls include Oracle's JAX-RPC implementation as well as Axis 1.x, both long deprecated. Attempting to run Apache CXF's wsdl2java with the NWS' RPC/encoded WSDL returns an "Rpc/encoded wsdls are not supported in JAXWS 2.0" error message. However the SOAP with Attachments API for Java (SAAJ) can be used with the JAX-WS Dispatch interface to create "raw" web service calls to a web service provider that uses RPC/encoded bindings. The following SOAP client provides two such examples of using SAAJ. A simple way to run this example would be to download my intro Web Service tutorial source code, replace its client subproject's WSClient class with the WSClient below and run mvn clean install exec:exec from the client folder.

package client;

import java.io.StringReader;
import java.net.URL;

import javax.xml.namespace.QName;
import javax.xml.soap.MessageFactory;
import javax.xml.soap.Name;
import javax.xml.soap.SOAPBody;
import javax.xml.soap.SOAPBodyElement;
import javax.xml.soap.SOAPElement;
import javax.xml.soap.SOAPFactory;
import javax.xml.soap.SOAPFault;
import javax.xml.soap.SOAPHeader;
import javax.xml.soap.SOAPMessage;
import javax.xml.soap.SOAPPart;
import javax.xml.transform.Source;
import javax.xml.transform.stream.StreamSource;
import javax.xml.ws.Dispatch;
import javax.xml.ws.Service;
import javax.xml.ws.soap.SOAPFaultException;

public class WSClient {

    public static void main (String[] args) {
        WSClient wsc = new WSClient();

        // get forecast by Zip Code
        wsc.getWeatherForecast("19110"); // Philadelphia

        // get another forecast
        wsc.getWeatherForecast("33157"); // Miami
    }

    private void getWeatherForecast(String zipCode) {

        try {
            // Convert the ZIP code to a geocoded value (which is needed
            // as input for the weather data)

            String nsSchema = "http://graphical.weather.gov/xml/DWMLgen/schema/DWML.xsd";

            String soapSchema = "http://schemas.xmlsoap.org/soap/envelope/";

            String xsiSchema
                    = "http://www.w3.org/2001/XMLSchema-instance";

            String encodingStyle
                    = "http://schemas.xmlsoap.org/soap/encoding/";

            String zipRequest = " "
                    + ""
                    +"   "
                    +           ""
                    +               zipCode
                    +           ""
                    +    ""
                    +""
                    +"";

            String wsdl = "https://graphical.weather.gov/xml/SOAP_server/ndfdXMLserver.php?wsdl";
            String targetNS = "https://graphical.weather.gov/xml/DWMLgen/wsdl/ndfdXML.wsdl";

            URL url = new URL(wsdl);
            QName serviceName = new QName(targetNS, "ndfdXML");
            QName portName = new QName(targetNS, "ndfdXMLPort");
            Service service = Service.create(url, serviceName);

            /*
             * JAX-WS Dispatch provides three usage options: -- JAXBContext
             * (unsure if works though for rpc/enc WSDL) -- JAXP Source objects
             * (used here) -- SAAJ SOAPMessages (used in 2nd request below)
             */
            Dispatch dispatch = service.createDispatch(portName,
                    Source.class, Service.Mode.MESSAGE);
            Source zipResponse = dispatch.invoke(
                    new StreamSource(new StringReader(zipRequest)));
            // if using a file for input instead:
            // new StreamSource(new File("myrequest.xml")));

            // use SAAJ to open message -- check if error or valid data
            MessageFactory msgFactory = MessageFactory.newInstance();
            SOAPMessage geocodeMsg = msgFactory.createMessage();
            SOAPPart env = geocodeMsg.getSOAPPart();
            env.setContent(zipResponse);
            // writeTo method outputs SOAPMessage, helpful for debugging
            // geocodeMsg.writeTo(System.out);

            if (geocodeMsg.getSOAPBody().hasFault()) {
                // Copy official error response into our LNF Fault
                SOAPFault fault = geocodeMsg.getSOAPBody().getFault();
                System.out.println("Could not obtain forecast for zipcode "
                        + zipCode + ": "
                        + fault.getFaultString() + "; " + fault.getDetail().getValue());
            }

            // From here: valid geocode is present-- so get weather report next

            /*
             * LatLonListZipCodeResponse is not very helpful; needed information
             * (latLonList) element is html-escaped instead of a real tag, which
             * is suitable for HTML responses but not so helpful when you need
             * to extract the value. So will need to parse string response to
             * get geocode values  
             *      35.1056,-90.007
             *   
             *  
             * 
             */
            String geocodeBuffer = geocodeMsg.getSOAPBody().
                    getElementsByTagName("listLatLonOut")
                    .item(0).getFirstChild().getNodeValue();

            // .getNodeValue() unescapes HTML string
            String geocodeVals = geocodeBuffer.substring(
                    geocodeBuffer.indexOf("") + 12,
                    geocodeBuffer.indexOf(""));
            System.out.println("Geocode Vals for zip code " + zipCode
                    + " are: " + geocodeVals);

            /*
             * NDFDgenLatLonList operation: gets weather data for a given
             * latitude, longitude pair
             *
             * Format of the Message:     38.99,-77.02 
             *  glance
             *    
             */
            SOAPFactory soapFactory = SOAPFactory.newInstance();
            SOAPMessage getWeatherMsg = msgFactory.createMessage();
            SOAPHeader header = getWeatherMsg.getSOAPHeader();
            header.detachNode();  // no header needed
            SOAPBody body = getWeatherMsg.getSOAPBody();
            Name functionCall = soapFactory.createName(
                    "NDFDgenLatLonList", "schNS",
                    nsSchema);
            SOAPBodyElement fcElement = body.addBodyElement(functionCall);
            Name attname = soapFactory.createName("encodingStyle", "S",
                    soapSchema);
            fcElement.addAttribute(attname, soapSchema);
            SOAPElement geocodeElement = fcElement.addChildElement("listLatLon");
            geocodeElement.addTextNode(geocodeVals);
            SOAPElement product = fcElement.addChildElement("product");
            product.addTextNode("glance");

            // make web service call using this SOAPMessage
            Dispatch smDispatch = service.createDispatch(portName,
                    SOAPMessage.class, Service.Mode.MESSAGE);
            SOAPMessage weatherMsg = smDispatch.invoke(getWeatherMsg);
            // weatherMsg.writeTo(System.out); // debugging only

            // Metro needs normalize() command because it breaks
            // up child dwml element into numerous text nodes.
            weatherMsg.getSOAPBody().getElementsByTagName("dwmlOut")
                    .item(0).normalize();

            // First child of dwmlOut is the dwml element that we need.
            // It is the root node of the weather data that we will
            // be using to generate the report.
            String weatherResponse = weatherMsg.getSOAPBody().
                    getElementsByTagName("dwmlOut")
                    .item(0).getFirstChild().getNodeValue();
            System.out.println("WR: " + weatherResponse);
        } catch (SOAPFaultException e) {
            System.out.println("SOAPFaultException: " + e.getFault().getFaultString());
        } catch (Exception e) {
            System.out.println("Exception: " + e.getMessage());
        }
    }
}

In the client code above I used the rather nonintuitive DOM Tree API to get the data elements I needed, for example:

String weatherResponse = weatherMsg.getSOAPBody().getElementsByTagName("dwmlOut")
    .item(0).getFirstChild().getNodeValue();

If you have many such calls to make, another option is to use XPath, see tutorials from Baeldung and TutorialsPoint for more information.

https://glenmazza.net/blog/date/20180505 Saturday May 05, 2018

Creating integration tests for SOAP web services

Learn various ways of creating integration tests for SOAP web service providers.

[Read More]

https://glenmazza.net/blog/date/20180310 Saturday March 10, 2018

Compressing SOAP messages during transit

In this article I'll be demonstrating two ways to compress Apache CXF SOAP requests and responses to conserve network resources--Fast Infoset (FI) and GZip compression. We'll modify the introductory DoubleIt web service tutorial to test this functionality along with Wireshark to see how the SOAP requests and responses change as a result. The code modifications necessary are quite minimal, just a few lines over one or two source files.

As a reference point, let's use Wireshark to see the SOAP request and response for a standard DoubleIt call without compression:

POST /doubleit/services/doubleit HTTP/1.1
Content-Type: text/xml; charset=UTF-8
Accept: */*
SOAPAction: ""
User-Agent: Apache-CXF/3.2.2
Cache-Control: no-cache
Pragma: no-cache
Host: localhost:8080
Connection: keep-alive
Content-Length: 224

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <ns2:DoubleIt xmlns:ns2="http://www.example.org/schema/DoubleIt">
            <numberToDouble>10</numberToDouble>
        </ns2:DoubleIt>
    </soap:Body>
</soap:Envelope>
        
HTTP/1.1 200
Content-Type: text/xml;charset=UTF-8
Content-Length: 238
Date: Sat, 10 Mar 2018 16:09:14 GMT

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <ns2:DoubleItResponse xmlns:ns2="http://www.example.org/schema/DoubleIt">
            <doubledNumber>20</doubledNumber>
        </ns2:DoubleItResponse>
    </soap:Body>
</soap:Envelope>

First, let's activate fast infoset, both client- and service-side. The negotiation process involves the SOAP client sending an uncompressed Accept: application/fastinfoset HTTP header in the first SOAP request. If the server is configured to support FI, it will return a compressed SOAP response with a content type of application/fastinfoset. Subsequently all SOAP requests and responses between the client and web service provider will then use FI. However, if the server is not configured to use FI the server will return an uncompressed SOAP response with the usual text/xml Content-Type. The following table shows how to activate FI:

Component How to implement Fast Infoset Compression
CXF Web Service Provider

Add the FastInfoset dependency to your project. Then attach the @FastInfoset feature to either the Service Endpoint Interface (SEI) or SEI implementation:

@WebService(targetNamespace = "http://www.example.org/contract/DoubleIt"... 
@org.apache.cxf.annotations.FastInfoset
public class DoubleItPortTypeImpl implements DoubleItPortType {
    ....  
}

Alternative configuration options: Can configure via the org.apache.cxf.feature.FastInfosetFeature either on the ServerFactoryBean or the cxf:bus. As the FastInfosetFeature just installs the two FI interceptors, those interceptors can also be added programmatically to the web service endpoint.

Note FI will be activated only if the client requests it. The feature's force=true attribute (also settable via the FIStaxOutInterceptor constructor) can be used to force FI from the service to the client without negotiation, however there's no guarantee the client will be able to process it.

CXF SOAP Client

Add the FastInfoset dependency to your project. Then add the FastInfoset interceptors to the SOAP client class:

public class WSClient {

    public static void main (String[] args) {
        DoubleItService service = new DoubleItService();
        DoubleItPortType port = service.getDoubleItPort();

        Client client = ClientProxy.getClient(port);
        client.getInInterceptors().add(new org.apache.cxf.interceptor.FIStaxInInterceptor());
        client.getOutInterceptors().add(new org.apache.cxf.interceptor.FIStaxOutInterceptor());
        ...
    }
}

Alternative: add the FastInfosetFeature to the ClientFactoryBean.

Note FI will be activated only if the service is configured to provide it. FI can be forced from the client to the service without negotiation using the methods described above for the web service provider, however there is no guarantee the service provider will be able to process it.

Now, let's check the results over the wire:

POST /doubleit/services/doubleit HTTP/1.1
Content-Type: text/xml; charset=UTF-8
Accept: application/fastinfoset, */*
SOAPAction: ""
User-Agent: Apache-CXF/3.2.2
Cache-Control: no-cache
Pragma: no-cache
Host: localhost:8080
Connection: keep-alive
Content-Length: 224

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <ns2:DoubleIt xmlns:ns2="http://www.example.org/schema/DoubleIt">
            <numberToDouble>10</numberToDouble>
        </ns2:DoubleIt>
    </soap:Body>
</soap:Envelope>
        
HTTP/1.1 200
Content-Type: application/fastinfoset
Transfer-Encoding: chunked
Date: Sat, 10 Mar 2018 17:39:16 GMT

.....8..soap(http://schemas.xmlsoap.org/soap/envelope/.?...Envelope?...Body8..ns2%http://www.example.org/schema/DoubleIt.?...DoubleItResponse<.doubledNumber.20....POST /doubleit/services/doubleit HTTP/1.1
Content-Type: application/fastinfoset
Accept: application/fastinfoset, */*
SOAPAction: ""
User-Agent: Apache-CXF/3.2.2
Cache-Control: no-cache
Pragma: no-cache
Host: localhost:8080
Connection: keep-alive
Content-Length: 155

.....8..soap(http://schemas.xmlsoap.org/soap/envelope/.?...Envelope?...Body8..ns2%http://www.example.org/schema/DoubleIt.?...DoubleIt<
numberToDouble.0....HTTP/1.1 200
Content-Type: application/fastinfoset
Transfer-Encoding: chunked
Date: Sat, 10 Mar 2018 17:39:16 GMT

.....8..soap(http://schemas.xmlsoap.org/soap/envelope/.?...Envelope?...Body8..ns2%http://www.example.org/schema/DoubleIt.?...DoubleItResponse<.doubledNumber.0....

For GZIP, follow the below table for configuration information. The negotiation process involves the SOAP client sending an Accept-Encoding: gzip HTTP header in the first SOAP request. If the server is configured to support GZIP compression, it will return a compressed SOAP response with the Content-Encoding: gzip HTTP Header. Subsequently all SOAP requests and responses between the client and web service provider will then use GZIP. If the server is not configured to use GZIP then SOAP requests and responses will be sent as normal, uncompressed.

Component How to implement GZIP Compression
CXF Web Service Provider

Attach the @org.apache.cxf.annotations.GZIP annotation to DoubleItPortTypeImpl, or either the GZIPFeature or its two corresponding interceptors as described in the previous table for the FI configuration. Note the annotation, feature and GZIPOutInterceptor all provide an optional threshold value below which SOAP messages will not be compressed, if not provided, the default is 1KB.

CXF SOAP Client

Attach the GZIPFeature or the GZIP interceptors (along with the threshold value, if desired) as described in the previous table for FI.

The results with GZIP are shown below. I set the threshold to zero, both client- and service-side, to ensure compression of all messages. (Note following Wireshark's HTTP stream will uncompress by default, causing one to think no compression is occurring. To see the compressed values as below, right-click any packet part of the SOAP calls and choose "Follow->TCPStream" instead.)

POST /doubleit/services/doubleit HTTP/1.1
Content-Type: text/xml; charset=UTF-8
Accept: */*
Accept-Encoding: gzip;q=1.0, identity; q=0.5, *;q=0
SOAPAction: ""
User-Agent: Apache-CXF/3.2.2
Cache-Control: no-cache
Pragma: no-cache
Host: localhost:8080
Connection: keep-alive
Content-Length: 224

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <ns2:DoubleIt xmlns:ns2="http://www.example.org/schema/DoubleIt">
            <numberToDouble>10</numberToDouble>
        </ns2:DoubleIt>
    </soap:Body>
</soap:Envelope>HTTP/1.1 200
Content-Encoding: gzip
Vary: Accept-Encoding
Content-Type: text/xml;charset=UTF-8
Content-Length: 158
Date: Sat, 10 Mar 2018 18:03:20 GMT

        ..........m.... .D....X.#A.F.^<..T6..,..........d&.fW1.(.aFG.w.w.e..M?.Q...GoX.........)A.U...>.
        ...M...xC..x....M)    \.....:.k33m..:......l......s.l../&.......
        
POST /doubleit/services/doubleit HTTP/1.1
Content-Type: text/xml; charset=UTF-8
Accept: */*
Accept-Encoding: gzip;q=1.0, identity; q=0.5, *;q=0
Content-Encoding: gzip
SOAPAction: ""
User-Agent: Apache-CXF/3.2.2
Cache-Control: no-cache
Pragma: no-cache
Host: localhost:8080
Connection: keep-alive
Content-Length: 150

..........].1..0.E..z..1F!......`.!..:m..q..
.....l.\..8....)....c...4..;....*W.?...k..kf.../k"......5.>A_])E..B...f=|...Ch............u..?..7Y.......HTTP/1.1 200
Content-Encoding: gzip
Vary: Accept-Encoding
Content-Type: text/xml;charset=UTF-8
Content-Length: 157
Date: Sat, 10 Mar 2018 18:03:20 GMT

..........m.... .D....X.. .../...*.{..t.......x..d..*&..)<.Q...]`Y.}7..%.....E../h.C....tZ5...S..;y..wxNW.H..n...........l3.4..6i/..q.[.kC...b........Q.7....

https://glenmazza.net/blog/date/20180304 Sunday March 04, 2018

Replacing JAX-WS handlers with Apache CXF interceptors

Summary: This article shows how to use Apache CXF interceptors as an alternative to JAX-WS handlers for web service providers and SOAP clients.

[Read More]

https://glenmazza.net/blog/date/20180225 Sunday February 25, 2018

Adding JAX-WS handlers to SOAP web services and clients

Summary: JAX-WS handlers provide a way to factor out functionality common for multiple SOAP web service providers and clients. Learn how to add JAX-WS handlers to your Apache CXF-based solutions.

[Read More]

https://glenmazza.net/blog/date/20180218 Sunday February 18, 2018

Sending Custom Metrics from Spring Boot to Datadog

This tutorial shows how Datadog's API can be used to send custom metrics for a Spring Boot web application and see how the results can be viewed graphically from Datadog dashboards. Samantha Drago's blog post provides a background of Datadog custom metrics which require a paid Datadog account. Note as an alternative not covered here, custom metrics can be defined via JMX with Datadog's JMX Integration used to collect them, this integration in particular provides a list of standard metrics that can be used even with the free DD account.

To facilitate metric accumulation and transferring of metrics to Datadog, Spring Boot's ExportMetricReader and ExportMetricWriter implementations will be used. Every 5 milliseconds by default (adjustable via the spring.metrics.export.delay-millis property), all MetricReader implementations marked @ExportMetricReader will have their values read and written to @ExportMetricWriter-registered MetricWriters. The class ("exporter") that handles this within Spring Boot is the MetricCopyExporter, which treats metrics starting with a "counter." as a counter (a metric that reports deltas on a continually growing statistic, like web hits) and anything else as a gauge (an standalone snapshot value at a certain timepoint, such as JVM heap usage.) Note, however, Datadog apparently does not support "counter" type metric collection using its API (everything is treated as a gauge), I'll be showing at the end how a summation function can be used within Datadog to work around that.

Spring Boot already provides several web metrics that can be sent to Datadog without any explicit need to capture those metrics, in particular, the metrics listed here that start with "counter." or "gauge.". These provide commonly requested statistics such as number of calls to a website and average response time in milliseconds. The example below will report those statistics to Datadog along with application-specific "counter.foo" and "gauge.bar" metrics that are maintained by our application.

  1. Create the web application. For our sample, Steps #1 and #2 of the Spring Boot to Kubernetes tutorial can be followed for this. Ensure you can see "Hello World!" at localhost:8080 before proceeding.

  2. Modify the Spring Boot application to send metrics to Datadog. Note for tutorial brevity I'm condensing the number of classes that might otherwise be used to send metrics to DD. Additions/updates to make:

    • In the project build.gradle, the gson JSON library and Apache HTTP Client libraries need to be added to support the API calls to DD:

      build.gradle:
      dependencies {
      	compile('com.google.code.gson:gson:2.8.2')
      	compile('org.apache.httpcomponents:httpclient:4.5.3')
      	...other libraries...
      }
      
    • The DemoMetricReaderWriter.java needs to be included, it serves as both the reader of our application-specific metrics (not those maintained by Spring Boot--those are handled by BufferMetricReader included within the framework) and as the writer of all metrics (app-specific and Spring Boot) to Datadog. Please see the comments within the code for implementation details.

      DemoMetricReaderWriter.java:
      package com.gmazza.demo;
      
      import com.google.gson.Gson;
      import com.google.gson.GsonBuilder;
      import com.google.gson.JsonPrimitive;
      import com.google.gson.JsonSerializer;
      import org.apache.http.HttpEntity;
      import org.apache.http.StatusLine;
      import org.apache.http.client.methods.CloseableHttpResponse;
      import org.apache.http.client.methods.HttpPost;
      import org.apache.http.entity.ByteArrayEntity;
      import org.apache.http.impl.client.CloseableHttpClient;
      import org.apache.http.impl.client.HttpClients;
      import org.apache.http.util.EntityUtils;
      import org.slf4j.Logger;
      import org.slf4j.LoggerFactory;
      import org.springframework.beans.factory.annotation.Value;
      import org.springframework.boot.actuate.metrics.Metric;
      import org.springframework.boot.actuate.metrics.reader.MetricReader;
      import org.springframework.boot.actuate.metrics.writer.Delta;
      import org.springframework.boot.actuate.metrics.writer.MetricWriter;
      import org.springframework.stereotype.Component;
      
      import javax.annotation.PostConstruct;
      import java.io.Closeable;
      import java.io.IOException;
      import java.math.BigDecimal;
      import java.util.ArrayList;
      import java.util.Arrays;
      import java.util.Date;
      import java.util.HashMap;
      import java.util.List;
      import java.util.Map;
      
      @Component
      public class DemoMetricReaderWriter implements MetricReader, MetricWriter, Closeable {
      
          private static final Logger logger = LoggerFactory.getLogger(DemoMetricReaderWriter.class);
      
          private Metric<Integer> accessCounter = null;
      
          private Map<String, Metric<?>> metricMap = new HashMap<>();
      
          private static final String DATADOG_SERIES_API_URL = "https://app.datadoghq.com/api/v1/series";
      
          @Value("${datadog.api.key}")
          private String apiKey = null;
      
          private CloseableHttpClient httpClient;
      
          private Gson gson;
      
          @PostConstruct
          public void init() {
              httpClient = HttpClients.createDefault();
      
              // removes use of scientific notation, see https://stackoverflow.com/a/18892735
              GsonBuilder gsonBuilder = new GsonBuilder();
              gsonBuilder.registerTypeAdapter(Double.class, (JsonSerializer<Double>) (src, typeOfSrc, context) -> {
                  BigDecimal value = BigDecimal.valueOf(src);
                  return new JsonPrimitive(value);
              });
      
              this.gson = gsonBuilder.create();
          }
      
          @Override
          public void close() throws IOException {
              httpClient.close();
          }
      
          // besides the app-specific metrics defined in the below method, Spring Boot also exports metrics
          // via its BufferMetricReader, for those with the "counter." or "gauge.*" prefix here:
          // https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html
          public void updateMetrics(long barGauge) {
              // Using same timestamp for both metrics, makes it easier to match/compare if desired in Datadog
              Date timestamp = new Date();
      
              logger.info("Updating foo-count and bar-gauge of {} for web call", barGauge);
      
              // Updates to values involve creating new Metrics as they are immutable
      
              // Because this Metric starts with a "counter.", MetricCopyExporter used by Spring Boot will treat this
              // as a counter and not a gauge when reading/writing values.
              accessCounter = new Metric<>("counter.foo",
                      accessCounter == null ? 0 : accessCounter.getValue() + 1, timestamp);
              metricMap.put("counter.foo", accessCounter);
      
              // Does not start with "counter.", therefore a gauge to MetricCopyExporter.
              metricMap.put("gauge.bar", new Metric<>("gauge.bar", barGauge, timestamp));
          }
      
          // required by MetricReader
          @Override
          public Metric<?> findOne(String metricName) {
              logger.info("Calling findOne with name of {}", metricName);
              return metricMap.get(metricName);
          }
      
          // required by MetricReader
          @Override
          public Iterable<Metric<?>> findAll() {
              logger.info("Calling findAll(), size of {}", metricMap.size());
              return metricMap.values();
          }
      
          // required by MetricReader
          @Override
          public long count() {
              logger.info("Requesting metricMap size: {}", metricMap.size());
              return metricMap.size();
          }
      
          // required by CounterWriter (in MetricWriter), used only for counters
          @Override
          public void increment(Delta<?> delta) {
              logger.info("Counter being written: {}: {} at {}", delta.getName(), delta.getValue(), delta.getTimestamp());
              if (apiKey != null) {
                  sendMetricToDatadog(delta, "counter");
              }
          }
      
          // required by CounterWriter (in MetricWriter), but implementation optional (MetricCopyExporter doesn't call)
          @Override
          public void reset(String metricName) {
              // not implemented
          }
      
          // required by GaugeWriter (in MetricWriter), used only for gauges
          @Override
          public void set(Metric<?> value) {
              logger.info("Gauge being written: {}: {} at {}", value.getName(), value.getValue(), value.getTimestamp());
              if (apiKey != null) {
                  sendMetricToDatadog(value, "gauge");
              }
          }
      
          // API to send metrics to DD is defined here:
          // https://docs.datadoghq.com/api/?lang=python#post-time-series-points
          private void sendMetricToDatadog(Metric<?> metric, String metricType) {
              // let's add an app prefix to our values to distinguish from other apps in DD
              String dataDogMetricName = "app.glendemo." + metric.getName();
      
              logger.info("Datadog call for metric: {} value: {}", dataDogMetricName, metric.getValue());
      
              Map<String, Object> data = new HashMap<>();
      
              List<List<Object>> points = new ArrayList<>();
              List<Object> singleMetric = new ArrayList<>();
              singleMetric.add(metric.getTimestamp().getTime() / 1000);
              singleMetric.add(metric.getValue().longValue());
              points.add(singleMetric);
              // additional metrics could be added to points list providing params below are same for them
      
              data.put("metric", dataDogMetricName);
              data.put("type", metricType);
              data.put("points", points);
              // InetAddress.getLocalHost().getHostName() may be accurate for your "host" value.
              data.put("host", "localhost:8080");
      
              // optional, just adding to test
              data.put("tags", Arrays.asList("demotag1", "demotag2"));
      
              List<Map<String, Object>> series = new ArrayList<>();
              series.add(data);
      
              Map<String, Object> data2 = new HashMap<>();
              data2.put("series", series);
      
              try {
                  String urlStr = DATADOG_SERIES_API_URL + "?api_key=" + apiKey;
                  String json = gson.toJson(data2);
                  byte[] jsonBytes = json.getBytes("UTF-8");
      
                  HttpPost httpPost = new HttpPost(urlStr);
                  httpPost.addHeader("Content-type", "application/json");
                  httpPost.setEntity(new ByteArrayEntity(jsonBytes));
      
                  try (CloseableHttpResponse response = httpClient.execute(httpPost)) {
                      StatusLine sl = response.getStatusLine();
                      if (sl != null) {
                          // DD sends 202 (accepted) if it's happy
                          if (sl.getStatusCode() == 202) {
                              HttpEntity responseEntity = response.getEntity();
                              EntityUtils.consume(responseEntity);
                          } else {
                              logger.warn("Problem posting to Datadog: {} {}", sl.getStatusCode(), sl.getReasonPhrase());
                          }
                      } else {
                          logger.warn("Problem posting to Datadog: response status line null");
                      }
                  }
      
              } catch (Exception e) {
                  logger.error(e.getMessage(), e);
              }
          }
      }
      
    • The DemoApplication.java file needs updating to wire in the DemoMetricReaderWriter. It's "Hello World" endpoint is also updated to send a duration gauge value (similar to but smaller than the more complete gauge.response.root Spring Boot metric) to the DemoMetricReaderWriter.

      DemoApplication.java:
      package com.gmazza.demo;
      
      import org.springframework.boot.SpringApplication;
      import org.springframework.boot.actuate.autoconfigure.ExportMetricReader;
      import org.springframework.boot.actuate.autoconfigure.ExportMetricWriter;
      import org.springframework.boot.autoconfigure.SpringBootApplication;
      import org.springframework.context.annotation.Bean;
      import org.springframework.web.bind.annotation.RequestMapping;
      import org.springframework.web.bind.annotation.RestController;
      
      @SpringBootApplication
      @RestController
      public class DemoApplication {
      
          public static void main(String[] args) {
              SpringApplication.run(DemoApplication.class, args);
          }
      
          private DemoMetricReaderWriter demoMetricReaderWriter = new DemoMetricReaderWriter();
      
          @Bean
          @ExportMetricReader
          @ExportMetricWriter
          DemoMetricReaderWriter getReader() {
              return demoMetricReaderWriter;
          }
      
          @RequestMapping("/")
          String home() throws Exception {
              long start = System.currentTimeMillis();
      
              // insert up to 2 second delay for a wider range of response times
              Thread.sleep((long) (Math.random() * 2000));
      
              // let that delay become the gauge.bar metric value
              long barValue = System.currentTimeMillis() - start;
      
              demoMetricReaderWriter.updateMetrics(barValue);
              return "Hello World!";
          }
      }
      
    • The application.properties in your resources folder is where you provide your Datadog API key as well as some other settings. A few other spring.metrics.export.* settings are also available.

      application.xml:
      # Just logging will occur if api.key not defined
      datadog.api.key=your_api_key_here
      # Datadog can keep per-second metrics, but using every 15 seconds per Datadog's preference
      spring.metrics.export.delay-millis=15000
      # disabling security for this tutorial (don't do in prod), allows seeing all metrics at http://localhost:8080/metrics
      management.security.enabled=false
      
  3. Make several web calls to http://localhost:8080 from a browser to send metrics to Datadog. May also wish to access metrics at .../metrics a few times, you'll note the app-specific metrics counter.foo and gauge.bar become listed in the web page that is returned, also that accessing /metrics sends additional *.metrics (counter.status.200.metrics and gauge.response.metrics) stats to Datadog. We configured the application in application.properties to send Datadog metrics every 15 seconds, if running in your IDE, you can check the application logging in the Console window to see the metrics being sent.

  4. Log into Datadog and view the metrics sent. Two main options from the left-side Datadog menu: Metrics -> Explorer and Dashboards -> New Dashboard. For the former, one can search on the metric names in the Graph: field (see upper illustration below), with charts of the data appearing immediately to the right. For the latter (lower illustration), I selected "New Timeboard" and added three Timeseries and one Query Value for the two main Spring Boot and two application-specific metrics sent.


    Metrics Explorer

    Datadog TimeBoard

    Again, as the "counter" type is presently not supported via the Datadog API, for dashboards the cumulative sum function can be used to have the counter metrics grow over time in charts:

    Cumulative Sum function

https://glenmazza.net/blog/date/20180212 Monday February 12, 2018

TightBlog 2.0.4 Patch Release

I made a 2.0.4 Patch Release of TightBlog to fix two pressing issues, the blog hit counter was not resetting at the end of each day properly and the "Insert Media File" popup on the blog entry edit page was also not working. Upgrading is as simple as swapping out the 2.0.3 WAR with this one. For first-time installs, see the general installation instructions, Linode-specific instructions are here.

Work on the future TightBlog 3.0 is continuing, it has much simpler blog template extraction, better caching design, and uses Thymeleaf instead of Velocity as the blog page template language. Non-test Java source files have fallen to 126 vs. the 146 in TightBlog 2.0, and one fewer database table is needed (now down to 12).

https://glenmazza.net/blog/date/20180211 Sunday February 11, 2018

Hosting Spring Boot Applications on Kubernetes

Provided here are simple instructions for deploying a "Hello World" Spring Boot application to Kubernetes, assuming usage of Amazon Elastic Container Service (ECS) including its Elastic Container Repository (ECR). Not covered are Kubernetes installation as well as proxy server configuration (i.e., accessibility of your application either externally or within an intranet) which would be specific to your environment.

  1. Create the Spring Boot application via the Spring Initializr. I chose a Gradle app with the Web and Actuator dependencies (the latter to obtain a health check /health URL), as shown in the following illustration.


    References: Getting Started with Spring Boot / Spring Initializr

  2. Import the Spring Boot application generated by Initializr into your favorite Java IDE and modify the DemoApplication.java to expose a "Hello World" endpoint:

    package com.gmazza.demo;
    
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    import org.springframework.boot.*;
    import org.springframework.boot.autoconfigure.*;
    import org.springframework.stereotype.*;
    import org.springframework.web.bind.annotation.*;
    
    @SpringBootApplication
    @RestController
    public class DemoApplication {
    
    	public static void main(String[] args) {
    		SpringApplication.run(DemoApplication.class, args);
    	}
    
    	@RequestMapping("/")
    	String home() {
    		return "Hello World!";
    	}
    }
    

    Let's make sure the application works standalone. From a command-line window in the Demo root folder, run gradle bootRun to activate the application. Ensure you can see "Hello World!" from a browser window at localhost:8080 and the health check at localhost:8080/health ({"status":"UP"}") before proceeding.

  3. Create a Docker Image of the Spring Boot application. Steps:

    1. Create a JAR of the demo application: gradle clean build from the Demo folder will generate a demo-0.0.1-SNAPSHOT.jar in the demo/build/libs folder.

    2. Create a new folder separate from the demo application, any name, say "projdeploy". Copy the demo JAR into this directory and also place there a new file called "Dockerfile" within it having the following code:

      FROM openjdk:8u131-jdk-alpine
      RUN echo "networkaddress.cache.ttl=60" >> /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/java.security
      ADD demo-0.0.1-SNAPSHOT.jar demo.jar
      ENTRYPOINT ["java","-Xmx2000m", "-Dfile.encoding=UTF-8","-jar","demo.jar" ]
      

      The above command creates a docker image building off of the OpenJDK image along with a recommended adjustment to the caching TTL. The ADD command performs a rename of the JAR file, stripping off the version from the name for subsequent use in the ENTRYPOINT command.

    3. Next, we'll generate the docker image. From the projdeploy folder, docker build -t demo:0.0.1-SNAPSHOT. Run the docker images command to view the created image in your local respository:

      $ docker images
      REPOSITORY                                                 TAG                                 IMAGE ID            CREATED             SIZE
      demo                                                       0.0.1-SNAPSHOT                      7139669729bf        10 minutes ago      116MB
      

      Repeated docker build commands with the same repository and tag will just overwrite the previous image. Images can also be deleted using docker rmi -f demo:0.0.1-SNAPSHOT.

  4. Push the target image to ECR. The ECR documentation provides more thorough instructions. Steps:

    1. Install the AWS Command-Line Interface (AWS CLI). Step #1 of AWS guide gives the OS-specific commands to use. In the aws ecr get-login... command you may find it necessary to specify the region where your ECR is hosted (e.g., --region us-west-1). Ensure you can log in from the command line (it will output "Login Succeeded") before continuing.

    2. Create an additional tag for your image to facilitate pushing to ECR, as explained in Step #4 in the ECR w/CLI guide. For this example:

      docker tag demo:0.0.1-SNAPSHOT your_aws_account_id.dkr.ecr.your_ecr_region.amazonaws.com/demo:0.0.1-SNAPSHOT
      

      Note in the above command, the "demo" at the end refers to the name of the ECR repository where the image will ultimately be placed, if not already existing it will need to be created beforehand for the next command to be successful or another existing repository name used. Also, see here for determining your account ID. You may wish to run docker images again to confirm the image was tagged.

    3. Push the newly tagged image to AWS ECR (replacing the "demo" below if you're using another ECR repository):

      docker push your_aws_account_id.dkr.ecr.your_ecr_region.amazonaws.com/demo:0.0.1-SNAPSHOT
      
    4. At this stage, good to confirm that the image was successfully loaded by viewing it in ECR repositories (URL to do so should be https://console.aws.amazon.com/ecs/home?region=your_ecr_region#/repositories.)

  5. Deploy your new application to Kubernetes. Make sure you have kubectl installed locally for this process. Steps:

    1. Create a deployment.yaml for the image. It is in this configuration file that your image's deployment, declare the image to use, and its service and ingress objects. A sample deployment.yaml would be as follows:

      deployment.yaml:

      kind: Deployment
      apiVersion: extensions/v1beta1
      metadata:
        name: demo
      spec:
        replicas: 1
        template:
          metadata:
            labels:
              app: demo
          spec:
            containers:
            - name: demo
              image: aws_acct_id.dkr.ecr.region.amazonaws.com/demo:0.0.1-SNAPSHOT 
              ports:
              - containerPort: 80
              resources:
                requests:
                  memory: "500Mi"
                limits:
                  memory: "1000Mi"
              readinessProbe:
                httpGet:
                  scheme: HTTP
                  path: /health
                  port: 8080
                initialDelaySeconds: 15
                periodSeconds: 5
                timeoutSeconds: 5
                successThreshold: 1
                failureThreshold: 20
              livenessProbe:
                httpGet:
                  scheme: HTTP
                  path: /health
                  port: 8080
                initialDelaySeconds: 15
                periodSeconds: 15
                timeoutSeconds: 10
                successThreshold: 1
                failureThreshold: 3
      ---
      kind: Service
      apiVersion: v1
      metadata:
        name: demo
      spec:
        selector:
          app: demo
        ports:
          - protocol: TCP
            port: 80
            targetPort: 8080
      ---
      kind: Ingress
      apiVersion: extensions/v1beta1
      metadata:
        name: demo
        annotations:
          kubernetes.io/ingress.class: "nginx"
      spec:
        rules:
        - host: demo.myorganization.org
          http:
            paths:
            - path:
              backend:
                serviceName: demo
                servicePort: 80
      

      Take particular note of the bolded deployment image (must match what was deployed to ECR) and the Ingress loadbalancer host, i.e., the URL to be used to access the application.

    2. Deploy the application onto Kubernetes. The basic kubectl create (deploy) command is as follows:

      kubectl --context ??? --namespace ??? create -f deployment.yaml
      

      To determine the correct context and namespace values to use, first enter kubectl config get-contexts to get a table of current contexts, the values will be under in the second column, "Name". If your desired context is not the current one (first column), enter kubectl config use-context context-name to switch to that one. Either way, then enter kubectl get namespaces for a listing of available namespaces under that context, picking one of those or creating a new namespace.

      Once your application is created, good to go to the Kubernetes dashboard to confirm it has successfully deployed. In the "pod" section, click the next-to-last column (the one with the horizontal lines) for the deployed pod to see startup logging including error messages, if any.

    3. Determine the IP address of the deployed application to configure routing. The kubectl --context ??? --namespace ??? get ingresses command (with context and namespace determined as before) will give you a list of configured ingresses and their IP address, configuration of the latter with Route 53 (at a minimum) will probably be needed for accessing your application.

      Once the application URL is accessible, you should be able to retrieve the same "Hello World!" and health check responses you had obtained in the first step from running locally.

    4. To undeploy the application, necessary for redeploying it via kubectl create, the application, service, and ingress can be individually deleted from the Kubernetes Dashboard. As an alternative, the following kubectl commands can be issued to delete the application's deployment, service, and ingress:

      kubectl --context ??? --namespace ??? delete deployment demo
      kubectl --context ??? --namespace ??? delete service demo
      kubectl --context ??? --namespace ??? delete ingress demo
      

      If it is desired to just reload the current application, deletion of the application's pod by default will accomplish that.

https://glenmazza.net/blog/date/20180203 Saturday February 03, 2018

Activating XML schema validation for SOAP calls in Apache CXF

See how the payloads of Apache CXF SOAP requests and responses can be XML schema-validated.

[Read More]