Glen Mazza's Weblog Wednesday December 11, 2019

TightBlog 3.6 Released!

Just-released TightBlog 3.6 (release page/screenshots/installation instructions) now provides an endpoint for embedding GitHub source code in your blog entries, see here for a sample and the Tightblog Wiki for more details. Previously I had to rely on third party tools hosted on other servers to accomplish the same, but happy to report this functionality is now available out-of-the-box with TightBlog.

Posted by Glen Mazza in Programming at 07:00AM Dec 11, 2019 | Comments[0] Wednesday November 06, 2019

Improving validation feedback in TightBlog

Tom Homberg provided a nice guide for implementing user-feedback validation within Spring applications, quite helpful for me in improving what I had in TightBlog. He creates a field - message Violation object (e.g., {"Name" "Name is required"}), a list of which is wrapped by a ValidationErrorResponse, the latter of which gets serialized to JSON and sent to the client to display validation errors. For my own implementation, I left the field value blank to display errors not specific to a particular field, and used it for both sending 400-type for user-input problems and generic 500-type messages for system errors.

Implementing this validation for TightBlog's blogger UI, I soon found it helpful to have convenience methods for quick creation of the Violations, ValidationErrorResponses and Spring ResponseEntities for providing feedback to the client:

public static ResponseEntity<ValidationErrorResponse> badRequest(String errorMessage) {
    return badRequest(new Violation(errorMessage));

public static ResponseEntity<ValidationErrorResponse> badRequest(Violation error) {
    return badRequest(Collections.singletonList(error));

public static ResponseEntity<ValidationErrorResponse> badRequest(List errors) {
    return ResponseEntity.badRequest().body(new ValidationErrorResponse(errors));

i18n can be handled via the Locale method argument, one of the parameters automatically provided by Spring:

private MessageSource messages;

public ResponseEntity doFoo(Locale locale) {

    if (error) {
        return ValidationErrorResponse.badRequest(messages.getMessage("mediaFile.error.duplicateName", null, locale));

On the front-end, I have Angular.js trap the code and then output the error messages (am not presently not using the field names). Below truncated for brevity (full source: JavaScript and JSP):

this.commonErrorResponse = function(response) {   
    self.errorObj =;

<div id="errorMessageDiv" class="alert alert-danger" role="alert" ng-show="ctrl.errorObj.errors" ng-cloak>
    <button type="button" class="close" data-ng-click="ctrl.errorObj.errors = null" aria-label="Close">
       <span aria-hidden="true">×</span>
    <ul class="list-unstyled">
       <li ng-repeat="item in ctrl.errorObj.errors">{{item.message}}</li>



Additionally, I was able to remove a fair amount of per-endpoint boilerplate by creating a single ExceptionHandler for unexpected 500 response code system errors and attaching it to my ControllerAdvice class so it would be used by all REST endpoints. For these types of exceptions usually a generic "System error occurred, please contact Administrator" message is sent to the user. However, I added a UUID that both appears on the client and goes into the logs along with the exception details, making it easy to search the logs for the specific problem. The exception handler (from the TightBlog source):

@ExceptionHandler(value = Exception.class)
// avoiding use of ResponseStatus as it activates Tomcat HTML page (see ResponseStatus JavaDoc)
public ResponseEntity<ValidationErrorResponse> handleException(Exception ex, Locale locale) {
    UUID errorUUID = UUID.randomUUID();
    log.error("Internal Server Error (ID: {}) processing REST call", errorUUID, ex);

    ValidationErrorResponse error = new ValidationErrorResponse();
    error.getErrors().add(new Violation(messages.getMessage(
            "generic.error.check.logs", new Object[] {errorUUID}, locale)));

    return ResponseEntity.status(500).body(error);

Screen output:


Log messaging containing the same UUID:


Additional Resources

Posted by Glen Mazza in Programming at 07:00AM Nov 06, 2019 | Comments[0] Monday September 30, 2019

TightBlog 3.5.3 Released

Downloads here, deployment instructions here.

Posted by Glen Mazza in Programming at 04:33PM Sep 30, 2019 | Tags:  tightblog | Comments[0] Sunday June 16, 2019

TightBlog 3.5 Released!

And this blog is running it! Information here, including migration-from-3.4 notes. Alas, a couple of bugs found when running this version, I hope to get a 3.5.1 out next weekend to fix them. Update: 3.5.1 released!

Posted by Glen Mazza in Programming at 07:57PM Jun 16, 2019 | Comments[0] Saturday October 27, 2018

ElasticSearch Notes: Complex Date Filtering, Bulk Updates

Some things learned this past week with ElasticSearch:

Advanced Date Searches: A event search page my company provides for its Pro customers allows for filtering by start date and end date, however some events do not have an end date defined. We decided to have differing business rules on what the start and end dates will filter based on whether or not the event has an end date, specifically:

  • If an event has both start and end dates:
    1. The start date of the range filter, if provided, must be before the end date of the event
    2. The end date of the range filter, if provided, must be after the start date of the event
  • If an event does not have an end date:
    1. The start date of the range filter, if provided, must be before the start date of the event
    2. The end date of the range filter, if provided, must be after the start date of the event

The above business logic had to be implemented in Java but as an intermediate step I first worked out an ElasticSearch query out of it using Kibana. Creating the query first helps immensely in the subsequent conversion to code. For the ElasticSearch query, this is what I came up with (using arbitrary sample dates to test the queries):

GET events-index/_search
    "query": {
    "bool": {
        "should" : [
        {"bool" :
            {"must": [
                { "exists": { "field": "eventMeta.dateEnd" }},
                { "range" : { "eventMeta.dateStart": { "lte": "2018-09-01"}}},
                { "range" : { "eventMeta.dateEnd": { "gte": "2018-10-01"}}}
        {"bool" :
            {"must_not": { "exists": { "field": "eventMeta.dateEnd"}},
             "must": [
                { "range" : { "eventMeta.dateStart": { "gte": "2018-01-01", "lte": "2019-12-31"}}}

As can be seen above, I first used a nested Bool query to separate the two main cases, namely events with and without and end date. The should at the top-level bool acts as an OR, indicating documents fitting either situation are desired. I then added the additional date requirements that need to hold for each specific case.

With the query now available, mapping to Java code using ElasticSearch's QueryBuilders (API) was very pleasantly straightforward, one can see the roughly 1-to-1 mapping of the code to the above query (the capitalized constants in the code refer to the relevant field names in the documents):

private QueryBuilder createEventDatesFilter(DateFilter filter) {

    BoolQueryBuilder mainQuery = QueryBuilders.boolQuery();

    // query modeled as a "should" (OR), divided by events with and without an end date,
    // with different filtering rules for each.
    BoolQueryBuilder hasEndDateBuilder = QueryBuilders.boolQuery();
    hasEndDateBuilder.must().add(fillDates(EVENT_START_DATE, null, filter.getStop()));
    hasEndDateBuilder.must().add(fillDates(EVENT_END_DATE, filter.getStart(), null));

    BoolQueryBuilder noEndDateBuilder = QueryBuilders.boolQuery();
    noEndDateBuilder.must().add(fillDates(EVENT_START_DATE, filter.getStart(), filter.getStop()));

    return mainQuery;

Bulk Updates: We use a "sortDate" field to indicate the specific date front ends should use for sorting results (whether ascending or descending, and regardless of the actual source of the date used to populate that field). For our news stories we wanted to rely on the last update date for stories that have been updated since their original publish, the published date otherwise. For certain older records loaded it turned out that the sortDate was still at the publishedDate when it should have been set to the updateDate. For research I used the following query to determine the extent of the problem:

GET news-index/_search
   "query": {
   "bool": {
      "must": [
         { "exists": { "field": "meta.updateDate" }},
            "script": {
               "script": "doc['meta.dates.sortDate'].value.getMillis() < doc['meta.updateDate'].value.getMillis()"

For the above query I used a two part Bool query, first checking for a non-null updateDate in the first clause and then a script clause to find sortDates preceding updateDates. (I found I needed to use .getMillis() for the inequality check to work.)

Next, I used ES' Update by Query API to do an all-at-once update of the records. The API has two parts, an optional query element to indicate the documents I wish to have updated (strictly speaking, in ES, to be replaced with a document with the requested changes) and a script element to indicate the modifications I want to have done to those documents. For my case:

POST news-index/_update_by_query
   "script": {
   "source": "ctx._source.meta.dates.sortDate = ctx._source.meta.updateDate",
   "lang": "painless"
   "query": {
      "bool": {
         "must": [
            { "exists": { "field": "meta.updateDate" }},
               "script": {
                  "script": "doc['meta.dates.sortDate'].value.getMillis() < doc['meta.updateDate'].value.getMillis()"

For running your own updates, good to test first by making a do-nothing update in the script (e.g., set sortDate to sortDate) and specifying just one document to be so updated, which can be done by adding a document-specific match requirement to the filter query (e.g., { "match": { "id": "...." }},"). Kibana should report that just one document was "updated", if so switch to the desired update to confirm that single record was updated properly, and then finally remove the match filter to have all desired documents updated.

Posted by Glen Mazza in Programming at 07:00AM Oct 27, 2018 | Comments[0] Sunday October 07, 2018

Using functions with a single generic method to convert lists

For converting from a Java collection say List<Foo> to any of several other collections List<Bar1>, List<Bar2>, ... rather than create separate FooListToBar1List, FooListToBar2List, ... methods a single generic FooListToBarList method and a series of Foo->Bar1, Foo->Bar2... converter functions can be more succinctly used. The below example converts a highly simplified List of SaleData objects to separate Lists of Customer and Product information, using a common generic saleDataListToItemList(saleDataList, converterFunction) method along with passed-in converter functions saleDataToCustomer and saleDataToProduct. Of particular note is how the converter functions are specified in the saleDataListToItemList calls. In the case of saleDataToCustomer, which takes two arguments (the SailData object and a Region string), a lambda expression is used, while the Product converter can be specified as a simple method reference due to it having only one parameter (the SailData object).

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.function.Function;

public class Main {

    public static void main(String[] args) {

        List saleDataList = new ArrayList<>();
        saleDataList.add(new SaleData("Bob", "radio"));
        saleDataList.add(new SaleData("Sam", "TV"));
        saleDataList.add(new SaleData("George", "laptop"));

        List customerList = saleDataListToItemList(saleDataList, sd -> Main.saleDataToCustomerWithRegion(sd, "Texas"));
        System.out.println("Customers: ");

        List productList = saleDataListToItemList(saleDataList, Main::saleDataToProduct);
        System.out.println("Products: ");

    private static  List saleDataListToItemList(List sdList, Function converter) {
        // handling potentially null sdList:
        return Optional.ofNullable(sdList).map(List::stream).orElse(Stream.empty()).map(converter).collect(Collectors.toList());

    private static Product saleDataToProduct(SaleData sd) {
        return new Product(sd.getProductName());

    private static Customer saleDataToCustomerWithRegion(SaleData sd, String region) {
        return new Customer(sd.getCustomerName(), region);

    private static class SaleData {
        private String customerName;
        private String productName;

        SaleData(String customerName, String productName) {
            this.customerName = customerName;
            this.productName = productName;

        String getProductName() {
            return productName;

        String getCustomerName() {
            return customerName;


    private static class Product {
        private String name;

        Product(String name) {
   = name;

        public String toString() {
            return "Product{" +
                    "name='" + name + '\'' +

    private static class Customer {
        private String name;
        private String region;

        Customer(String name, String region) {
   = name;
            this.region = region;

        public String toString() {
            return "Customer{" +
                    "name='" + name + '\'' +
                    ", region='" + region + '\'' +


Output from running:

Customer{name='Bob', region='Texas'}
Customer{name='Sam', region='Texas'}
Customer{name='George', region='Texas'}

Posted by Glen Mazza in Programming at 07:00AM Oct 07, 2018 | Comments[0] Sunday June 24, 2018

TightBlog 3.0 Released!

My third annual release currently powering this blog. See here for a listing of enhancements over the previous TightBlog 2.0, here for all the enhancements over the original Apache Roller 5.1.0 I had forked in 2015. Screenshots are here.

Posted by Glen Mazza in Programming at 04:36PM Jun 24, 2018 | Comments[1] Sunday February 18, 2018

Sending Custom Metrics from Spring Boot to Datadog

This tutorial shows how Datadog's API can be used to send custom metrics for a Spring Boot web application and see how the results can be viewed graphically from Datadog dashboards. Samantha Drago's blog post provides a background of Datadog custom metrics which require a paid Datadog account. Note as an alternative not covered here, custom metrics can be defined via JMX with Datadog's JMX Integration used to collect them, this integration in particular provides a list of standard metrics that can be used even with the free DD account.

To facilitate metric accumulation and transferring of metrics to Datadog, Spring Boot's ExportMetricReader and ExportMetricWriter implementations will be used. Every 5 milliseconds by default (adjustable via the spring.metrics.export.delay-millis property), all MetricReader implementations marked @ExportMetricReader will have their values read and written to @ExportMetricWriter-registered MetricWriters. The class ("exporter") that handles this within Spring Boot is the MetricCopyExporter, which treats metrics starting with a "counter." as a counter (a metric that reports deltas on a continually growing statistic, like web hits) and anything else as a gauge (an standalone snapshot value at a certain timepoint, such as JVM heap usage.) Note, however, Datadog apparently does not support "counter" type metric collection using its API (everything is treated as a gauge), I'll be showing at the end how a summation function can be used within Datadog to work around that.

Spring Boot already provides several web metrics that can be sent to Datadog without any explicit need to capture those metrics, in particular, the metrics listed here that start with "counter." or "gauge.". These provide commonly requested statistics such as number of calls to a website and average response time in milliseconds. The example below will report those statistics to Datadog along with application-specific "" and "" metrics that are maintained by our application.

  1. Create the web application. For our sample, Steps #1 and #2 of the Spring Boot to Kubernetes tutorial can be followed for this. Ensure you can see "Hello World!" at localhost:8080 before proceeding.

  2. Modify the Spring Boot application to send metrics to Datadog. Note for tutorial brevity I'm condensing the number of classes that might otherwise be used to send metrics to DD. Additions/updates to make:

    • In the project build.gradle, the gson JSON library and Apache HTTP Client libraries need to be added to support the API calls to DD:

      dependencies {
      	...other libraries...
    • The needs to be included, it serves as both the reader of our application-specific metrics (not those maintained by Spring Boot--those are handled by BufferMetricReader included within the framework) and as the writer of all metrics (app-specific and Spring Boot) to Datadog. Please see the comments within the code for implementation details.
      package com.gmazza.demo;
      import org.apache.http.HttpEntity;
      import org.apache.http.StatusLine;
      import org.apache.http.client.methods.CloseableHttpResponse;
      import org.apache.http.client.methods.HttpPost;
      import org.apache.http.entity.ByteArrayEntity;
      import org.apache.http.impl.client.CloseableHttpClient;
      import org.apache.http.impl.client.HttpClients;
      import org.apache.http.util.EntityUtils;
      import org.slf4j.Logger;
      import org.slf4j.LoggerFactory;
      import org.springframework.beans.factory.annotation.Value;
      import org.springframework.boot.actuate.metrics.Metric;
      import org.springframework.boot.actuate.metrics.reader.MetricReader;
      import org.springframework.boot.actuate.metrics.writer.Delta;
      import org.springframework.boot.actuate.metrics.writer.MetricWriter;
      import org.springframework.stereotype.Component;
      import javax.annotation.PostConstruct;
      import java.math.BigDecimal;
      import java.util.ArrayList;
      import java.util.Arrays;
      import java.util.Date;
      import java.util.HashMap;
      import java.util.List;
      import java.util.Map;
      public class DemoMetricReaderWriter implements MetricReader, MetricWriter, Closeable {
          private static final Logger logger = LoggerFactory.getLogger(DemoMetricReaderWriter.class);
          private Metric<Integer> accessCounter = null;
          private Map<String, Metric<?>> metricMap = new HashMap<>();
          private static final String DATADOG_SERIES_API_URL = "";
          private String apiKey = null;
          private CloseableHttpClient httpClient;
          private Gson gson;
          public void init() {
              httpClient = HttpClients.createDefault();
              // removes use of scientific notation, see
              GsonBuilder gsonBuilder = new GsonBuilder();
              gsonBuilder.registerTypeAdapter(Double.class, (JsonSerializer<Double>) (src, typeOfSrc, context) -> {
                  BigDecimal value = BigDecimal.valueOf(src);
                  return new JsonPrimitive(value);
              this.gson = gsonBuilder.create();
          public void close() throws IOException {
          // besides the app-specific metrics defined in the below method, Spring Boot also exports metrics
          // via its BufferMetricReader, for those with the "counter." or "gauge.*" prefix here:
          public void updateMetrics(long barGauge) {
              // Using same timestamp for both metrics, makes it easier to match/compare if desired in Datadog
              Date timestamp = new Date();
    "Updating foo-count and bar-gauge of {} for web call", barGauge);
              // Updates to values involve creating new Metrics as they are immutable
              // Because this Metric starts with a "counter.", MetricCopyExporter used by Spring Boot will treat this
              // as a counter and not a gauge when reading/writing values.
              accessCounter = new Metric<>("",
                      accessCounter == null ? 0 : accessCounter.getValue() + 1, timestamp);
              metricMap.put("", accessCounter);
              // Does not start with "counter.", therefore a gauge to MetricCopyExporter.
              metricMap.put("", new Metric<>("", barGauge, timestamp));
          // required by MetricReader
          public Metric<?> findOne(String metricName) {
    "Calling findOne with name of {}", metricName);
              return metricMap.get(metricName);
          // required by MetricReader
          public Iterable<Metric<?>> findAll() {
    "Calling findAll(), size of {}", metricMap.size());
              return metricMap.values();
          // required by MetricReader
          public long count() {
    "Requesting metricMap size: {}", metricMap.size());
              return metricMap.size();
          // required by CounterWriter (in MetricWriter), used only for counters
          public void increment(Delta<?> delta) {
    "Counter being written: {}: {} at {}", delta.getName(), delta.getValue(), delta.getTimestamp());
              if (apiKey != null) {
                  sendMetricToDatadog(delta, "counter");
          // required by CounterWriter (in MetricWriter), but implementation optional (MetricCopyExporter doesn't call)
          public void reset(String metricName) {
              // not implemented
          // required by GaugeWriter (in MetricWriter), used only for gauges
          public void set(Metric<?> value) {
    "Gauge being written: {}: {} at {}", value.getName(), value.getValue(), value.getTimestamp());
              if (apiKey != null) {
                  sendMetricToDatadog(value, "gauge");
          // API to send metrics to DD is defined here:
          private void sendMetricToDatadog(Metric<?> metric, String metricType) {
              // let's add an app prefix to our values to distinguish from other apps in DD
              String dataDogMetricName = "app.glendemo." + metric.getName();
    "Datadog call for metric: {} value: {}", dataDogMetricName, metric.getValue());
              Map<String, Object> data = new HashMap<>();
              List<List<Object>> points = new ArrayList<>();
              List<Object> singleMetric = new ArrayList<>();
              singleMetric.add(metric.getTimestamp().getTime() / 1000);
              // additional metrics could be added to points list providing params below are same for them
              data.put("metric", dataDogMetricName);
              data.put("type", metricType);
              data.put("points", points);
              // InetAddress.getLocalHost().getHostName() may be accurate for your "host" value.
              data.put("host", "localhost:8080");
              // optional, just adding to test
              data.put("tags", Arrays.asList("demotag1", "demotag2"));
              List<Map<String, Object>> series = new ArrayList<>();
              Map<String, Object> data2 = new HashMap<>();
              data2.put("series", series);
              try {
                  String urlStr = DATADOG_SERIES_API_URL + "?api_key=" + apiKey;
                  String json = gson.toJson(data2);
                  byte[] jsonBytes = json.getBytes("UTF-8");
                  HttpPost httpPost = new HttpPost(urlStr);
                  httpPost.addHeader("Content-type", "application/json");
                  httpPost.setEntity(new ByteArrayEntity(jsonBytes));
                  try (CloseableHttpResponse response = httpClient.execute(httpPost)) {
                      StatusLine sl = response.getStatusLine();
                      if (sl != null) {
                          // DD sends 202 (accepted) if it's happy
                          if (sl.getStatusCode() == 202) {
                              HttpEntity responseEntity = response.getEntity();
                          } else {
                              logger.warn("Problem posting to Datadog: {} {}", sl.getStatusCode(), sl.getReasonPhrase());
                      } else {
                          logger.warn("Problem posting to Datadog: response status line null");
              } catch (Exception e) {
                  logger.error(e.getMessage(), e);
    • The file needs updating to wire in the DemoMetricReaderWriter. It's "Hello World" endpoint is also updated to send a duration gauge value (similar to but smaller than the more complete gauge.response.root Spring Boot metric) to the DemoMetricReaderWriter.
      package com.gmazza.demo;
      import org.springframework.boot.SpringApplication;
      import org.springframework.boot.actuate.autoconfigure.ExportMetricReader;
      import org.springframework.boot.actuate.autoconfigure.ExportMetricWriter;
      import org.springframework.boot.autoconfigure.SpringBootApplication;
      import org.springframework.context.annotation.Bean;
      import org.springframework.web.bind.annotation.RequestMapping;
      import org.springframework.web.bind.annotation.RestController;
      public class DemoApplication {
          public static void main(String[] args) {
    , args);
          private DemoMetricReaderWriter demoMetricReaderWriter = new DemoMetricReaderWriter();
          DemoMetricReaderWriter getReader() {
              return demoMetricReaderWriter;
          String home() throws Exception {
              long start = System.currentTimeMillis();
              // insert up to 2 second delay for a wider range of response times
              Thread.sleep((long) (Math.random() * 2000));
              // let that delay become the metric value
              long barValue = System.currentTimeMillis() - start;
              return "Hello World!";
    • The in your resources folder is where you provide your Datadog API key as well as some other settings. A few other spring.metrics.export.* settings are also available.

      # Just logging will occur if api.key not defined
      # Datadog can keep per-second metrics, but using every 15 seconds per Datadog's preference
      # disabling security for this tutorial (don't do in prod), allows seeing all metrics at http://localhost:8080/metrics
  3. Make several web calls to http://localhost:8080 from a browser to send metrics to Datadog. May also wish to access metrics at .../metrics a few times, you'll note the app-specific metrics and become listed in the web page that is returned, also that accessing /metrics sends additional *.metrics (counter.status.200.metrics and gauge.response.metrics) stats to Datadog. We configured the application in to send Datadog metrics every 15 seconds, if running in your IDE, you can check the application logging in the Console window to see the metrics being sent.

  4. Log into Datadog and view the metrics sent. Two main options from the left-side Datadog menu: Metrics -> Explorer and Dashboards -> New Dashboard. For the former, one can search on the metric names in the Graph: field (see upper illustration below), with charts of the data appearing immediately to the right. For the latter (lower illustration), I selected "New Timeboard" and added three Timeseries and one Query Value for the two main Spring Boot and two application-specific metrics sent.

    Metrics Explorer

    Datadog TimeBoard

    Again, as the "counter" type is presently not supported via the Datadog API, for dashboards the cumulative sum function can be used to have the counter metrics grow over time in charts:

    Cumulative Sum function

Posted by Glen Mazza in Programming at 07:00AM Feb 18, 2018 | Comments[0]

« December 2019
Sun Mon Tue Wed Thu Fri Sat
About Me
Java Software Engineer
TightBlog maintainer, Apache CXF committer
Arlington, Virginia USA
gmazza at apache dot org
GitHub LinkedIn
Blog Search
Apache CXF/SOAP tutorial
Blog article index

Today's Blog Hits: 1562

About Blog
Blog software: TightBlog 3.6.0
Application Server: Tomcat
Database: MySQL
Hosted on: Linode
SSL Certificate: Let's Encrypt
Installation Instructions