Glen Mazza's Weblog

This entry provides a simple example of using Spring Shell (within Spring Boot) and Micrometer to send custom metrics to Datadog. For a fuller example with Docker and the Datadog Agent, I recommend Datadog Learning Center's free Datadog 101: Developer course. This course will also provide you a free two-week training Datadog account which you can use to receive custom metrics for this example, useful if you don't care to test against your company account. Note custom metrics ordinarily carry billing costs, requiring a paid Datadog account.

Spring Boot already provides numerous web metrics in several areas that can be sent to Datadog without explicit need to capture them. The jvm.* properties, for example, are readable in Datadog's Metrics Explorer, filtering by statistic:value for this example in the "from" field. For custom metrics, we'll have the Spring Shell app commands modify a Timer and a Gauge.

1. Create the Spring Shell application. From Spring Initializr, choose a Java JAR app with Spring Shell and Datadog as dependencies. For some reason I needed to choose the Spring Boot 2.7.x series for an app to download. Prior to running the demo in your IDE (I use IntelliJ), the management.metrics.export.datadog.apiKey=... value needs to be added to the main/resources/application.properties file. Your API key can be determined by logging into Datadog, and from the bottom of the left-side menu, click on your name, then Organization Settings, then Access, the API Keys.

2. Create the shell commands to add to the timer and gauge values:

   package com.example.demo;
   
   import io.micrometer.core.instrument.MeterRegistry;
   import io.micrometer.core.instrument.Tags;
   import io.micrometer.core.instrument.Timer;
   import org.springframework.shell.standard.ShellComponent;
   import org.springframework.shell.standard.ShellMethod;
   
   import java.util.ArrayList;
   import java.util.List;
   import java.util.concurrent.TimeUnit;
   
   @ShellComponent
   public class MyCommands {
   
       private final Timer timer;
   
       private final List<Integer> integerList;
   
       public MyCommands(MeterRegistry registry) {
           timer = registry.timer("demoapp.timer", Tags.empty());
           integerList = registry.gauge("demoapp.listsize", Tags.empty(), new ArrayList<>(), List::size);
       }
   
       @ShellMethod("Note a timer event of given duration in seconds")
       public String timer(int seconds) {
           timer.record(seconds, TimeUnit.SECONDS);
           return "Timer event noted";
       }
       @ShellMethod("Add an element to list")
       public String listAdd() {
           integerList.add(10);
           return "List has " + integerList.size() + " elements";
       }
   
       @ShellMethod("Remove an element from list (if possible)")
       public String listRemove() {
           if (integerList.size() > 0) {
               integerList.remove(integerList.size() - 1);
           }
           return "List has " + integerList.size() + " elements";
       }
   }
   

   For the above we're keeping a gauge on the size of the list, and for the timer, we provide the number of seconds that an arbitrary event took.

3. Run the application and enter several timer #secs, list-add, and list-remove commands. Run -> Run Application from the IntelliJ menu should work fine to enter the commands in the IDE's Console view. To keep the connection with Datadog, keep the command-line app running, even if you're not entering commands. After 2 or 3 minutes, check Datadog's Metrics Explorer to confirm that the demoapp.timer and demoapp.listsize metrics are getting received:

   

   A dashboard can be created to show both properties at once (with separate average time and counts given for the Timer):

   

Resources

• Metrics - Spring Boot docs
• Custom metrics can also be defined via JMX with Datadog's JMX Integration (tutorial)
• Micrometer tutorial - by Baeldung, using Netflix Atlas instead of Datadog