Skip to content
Open
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
b98b630
feat(config): update OpenTelemetry OTLP
foukou19 Oct 5, 2026
76f3ec9
feat(config): Adapt webhooks security configuration
foukou19 Oct 7, 2026
bf81dd8
fix: resolution conflit sur webhooks.md
foukou19 Oct 8, 2026
e46ae70
feat(config): update webhook.md
foukou19 Oct 8, 2026
4ee4d47
Merge branch 'main' into feat/idp-core-observability-otel-starter
foukou19 Oct 9, 2026
bb69f19
feat(config): update application.yaml
foukou19 Oct 9, 2026
6a1d961
Update docs/src/features/webhooks.md
foukou19 Oct 9, 2026
fee9111
feat(config): update application.yaml
foukou19 Oct 13, 2026
92d43ba
feat(config): update application.yaml
foukou19 Oct 13, 2026
5924d23
feat(observability): add custom webhook security metrics
foukou19 Oct 13, 2026
4bad155
feat(observability): add custom webhook security metrics
foukou19 Oct 13, 2026
a598f66
feat(config): add custom webhook ingestion metrics
foukou19 Oct 14, 2026
4f91fed
Merge branch 'main' into feat/idp-core-observability
foukou19 Oct 14, 2026
ad98b66
feat(config): centralize webhook metrics and tracing
foukou19 Oct 15, 2026
372e20f
Merge branch 'main' into feat/idp-core-observability
foukou19 Oct 15, 2026
d55eb94
feat(config): centralize webhook metrics and tracing
foukou19 Oct 15, 2026
7d67e93
feat(config): update webhook metrics
foukou19 Oct 16, 2026
ec92e1d
feat(config): update camel tracing configuration
foukou19 Oct 19, 2026
7059a37
feat(config): update CamelTracingConfiguration
foukou19 Oct 19, 2026
04fe456
feat(config): update EntityController.java
foukou19 Oct 29, 2026
556bae2
Merge branch 'main' into feat/idp-core-observability
foukou19 Oct 29, 2026
7a32662
feat(config): update EntityController.java
foukou19 Oct 29, 2026
5530fae
Merge branch 'main' into feat/idp-core-observability
foukou19 Nov 3, 2026
eaea15b
feat(config): update EntityController.java
foukou19 Nov 4, 2026
1226447
feat(config): Keep Camel Micrometer route metrics active
foukou19 Nov 4, 2026
d32fb09
feat(config): attach micrometer route policy factory to camel context
foukou19 Nov 5, 2026
57e617d
feat(config): separate Camel OTel tracing from Micrometer OTLP metric…
foukou19 Nov 5, 2026
00b757a
feat(config): remove Camel Micrometer metrics scope from this PR
foukou19 Nov 5, 2026
48c55c4
Merge branch 'main' into feat/idp-core-observability
foukou19 Nov 5, 2026
4e512fd
Merge branch 'main' into feat/idp-core-observability
brandPittCode Nov 12, 2026
8f47512
Merge branch 'main' into feat/idp-core-observability
brandPittCode Nov 13, 2026
9371a07
feat(config): Centralize ingestion metric constants and typed attribu…
foukou19 Nov 17, 2026
8853bbd
feat(config): Centralize ingestion metric constants and typed attribu…
foukou19 Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 82 additions & 41 deletions docs/src/deployment/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,48 +28,83 @@ The app exposes health endpoints and uses OTLP export for traces and metrics whe

## Metrics

### Metrics exported by Spring Boot and Micrometer

The application exposes the standard Micrometer metrics and OTLP export can be enabled through environment variables.

```yaml
management:
tracing:
sampling:
probability: ${OTEL_TRACES_SAMPLER_ARG:0.1}
export:
enabled: ${OTEL_TRACING_ENABLED:false}

otlp:
metrics:
export:
url: ${OTEL_EXPORTER_OTLP_METRICS_ENDPOINT:${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4318}/v1/metrics}
enabled: ${OTEL_METRICS_EXPORT_ENABLED:false}
transport: ${OTEL_EXPORTER_OTLP_METRICS_PROTOCOL:${OTEL_EXPORTER_OTLP_PROTOCOL:http/protobuf}}

opentelemetry:
tracing:
export:
otlp:
endpoint: ${OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4318}/v1/traces}
transport: ${OTEL_EXPORTER_OTLP_TRACES_TRANSPORT:${OTEL_EXPORTER_OTLP_PROTOCOL:http}}
resource-attributes:
deployment.environment: ${OTEL_ENV:local}
service.namespace: ${OTEL_SERVICE_NAMESPACE:idp-core}
service.name: ${OTEL_SERVICE_NAME:internal-developer-platform}
### Native OpenTelemetry metrics

The application uses `io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter` to configure the SDK, metric readers, and OTLP exporters. You do not need application-defined `OpenTelemetry` or `SdkMeterProvider` beans. The ingestion processors use the auto-configured instance.

```yaml title="application.yml"
otel:
metrics:
exporter: ${OTEL_METRICS_EXPORTER:otlp}
exporter:
otlp:
endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4318}
protocol: ${OTEL_EXPORTER_OTLP_PROTOCOL:http/protobuf}
```

### Custom business metrics
The previous Spring Boot starter exported Micrometer metrics. Its SDK configuration could attach an existing meter provider, but did not create one for these native instruments. Setting `management.opentelemetry.metrics.export.otlp.*` did not activate collection. A `DefaultMeter` therefore discarded custom measurements even while traces worked.

Business-level metrics are intentionally separate from Camel route telemetry. The application adds custom counters only for functional decisions that Camel does not expose as a reusable business signal.
`camel-opentelemetry2-starter` provides Camel tracing, not automatic base route metrics. This correction restores the existing custom metrics; it does not add Camel base metrics or export Micrometer registries.
Comment thread
foukou19 marked this conversation as resolved.
Resolved

Examples:
### Diagnose missing metrics

1. Restart the application and inspect `[OTEL SDK CHECK]`. `DefaultMeter` means no-op collection; `SdkMeter` confirms a native SDK meter, not successful delivery to Datadog.
2. Temporarily set `OTEL_METRICS_EXPORTER=console,otlp` and `OTEL_METRIC_EXPORT_INTERVAL=5000`. The starter includes the logging exporter. Console and OTLP exporters can coexist without a manually created reader.
3. Send a webhook, then wait for the export interval. Look for `idp.webhook.security.requests.total` or `idp.webhook.mapping.applied.total`. A filtered mapping still records attempts and filtered results, but does not create an entity.
4. If console metrics appear but Datadog receives nothing, check exporter errors, collector metrics routing, authentication, and Datadog filters. Configure required request headers with `OTEL_EXPORTER_OTLP_METRICS_HEADERS`; resource attributes are not HTTP headers.
5. Restore `OTEL_METRICS_EXPORTER=otlp` and the normal interval after diagnosis. Console metrics can contain operational identifiers and generate large logs.

> [!WARNING]
> On platforms that allocate CPU only during requests, background export can pause between requests. Check CPU allocation and instance lifetime if collection works but periodic export is delayed.

For HTTP/protobuf, set `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` to the full collector URL, including `/v1/metrics`. Inside a container, `localhost` points to the application container. Use the collector service name, or `host.docker.internal` if the collector runs on your macOS host through Docker Desktop. The repository's Docker Compose file does not start a collector.

### Verify the export path

- `idp.webhook.security.requests.total{connector, security_type}`
- `idp.webhook.security.result{connector, security_type, result}`
- `idp.webhook.security.validation.errors.total{connector, security_type, reason}`
Run the focused tests without a collector or Datadog credentials:

These metrics reflect the real security decision made by the application and should not duplicate the standard Camel `route.*` metrics.
```bash
mvn -Dspring.profiles.active=test -Dtest=OpenTelemetryMetricsAutoConfigurationTest test
Comment thread
foukou19 marked this conversation as resolved.
Resolved
```

The tests distinguish these checks:

- **SDK collection**: an in-memory exporter receives a custom counter and histogram with the expected measurements. A separate test checks the warning when you disable the SDK.
- **OTLP HTTP transport**: the real exporter sends a `protobuf` request to a local server on an ephemeral port. The test checks the method, path, content type, configured header, and encoded metric, connector, and service identifiers. An HTTP 200 response completes the export; an HTTP 400 response reports failure. This check does not decode numeric points or exercise periodic scheduling.
- **Datadog delivery**: these tests do not contact your deployed collector or Datadog. After deployment, send a webhook and confirm reception at the collector, then in Datadog. Check that the collector enables an OTLP receiver and a metrics pipeline connected to the Datadog exporter, or that your Datadog Agent enables OTLP metrics ingestion. A successful HTTP response alone does not prove downstream delivery.

> [!WARNING]
> Do not use `SdkMeterProvider.forceFlush().isSuccess()` as proof of HTTP acceptance. In this SDK version, the reader can complete the flush successfully even when the exporter receives HTTP 400. The transport tests observe the actual exporter result; in deployment, check exporter errors and collector reception.

### Custom webhook ingestion metrics

The application emits custom OpenTelemetry metrics in the webhook ingestion pipeline to capture business decisions and failures that are not exposed by the standard Camel route telemetry.

The following metrics are produced by the ingestion, security and decoding processors:

| Metric name | Attributes | Meaning |
|--------------------------------------------------|-----------------------------------------|------------------------------------------------------------------------------|
| `idp.webhook.security.requests.total` | `connector`, `security_type` | Number of security checks executed for a connector. |
| `idp.webhook.security.result` | `connector`, `security_type`, `result` | Result of the security decision (`allowed`, `rejected`, `error`). |
Comment thread
Copilot marked this conversation as resolved.
Resolved
| `idp.webhook.security.validation.errors.total` | `connector`, `security_type`, `reason` | Security validation failures and malformed configuration errors. |
| `idp.webhook.payload.decoded.total` | `connector`, `encoding` | Successful payload decodes for each encoding step. |
| `idp.webhook.payload.decode.errors.total` | `connector`, `reason` | Payload decode failures, including unsupported or corrupted encodings. |
| `idp.webhook.payload.size.bytes` | `connector` | Histogram of decoded payload size in bytes. |
| `idp.webhook.mapping.applied.total` | `connector`, `mapping_id`, `action` | Mapping execution attempts. |
| `idp.webhook.mapping.filtered.total` | `connector`, `mapping_id`, `reason` | Mapping results rejected by the filter stage. |
| `idp.webhook.mapping.result` | `connector`, `mapping_id`, `result` | Mapping outcome (`MATCHED`, `FILTERED_OUT`, `ERROR`). |
| `idp.webhook.mapping.errors.total` | `connector`, `mapping_id`, `error_type` | Mapping exceptions and runtime failures. |
| `idp.webhook.mapping.latency.ms` | `connector`, `mapping_id` | Mapping duration histogram in milliseconds. |
| `idp.webhook.entity.created.total` | `template`, `connector` | Number of entities created during ingestion. |
| `idp.webhook.entity.updated.total` | `template`, `connector` | Number of entities updated during ingestion. |
| `idp.webhook.entity.deleted.total` | `template`, `connector` | Number of entities deleted during ingestion. |
| `idp.webhook.entity.patch.total` | `template`, `connector`, `action` | Number of patch operations applied. |
| `idp.webhook.entity.validation.errors.total` | `template`, `connector`, `reason` | Entity validation and business-rule failures. |
| `idp.webhook.entity.notfound.total` | `template`, `connector`, `action` | Missing entity operations during ingestion. |
| `idp.webhook.entity.conflict.total` | `template`, `connector`, `reason` | Conflict scenarios such as duplicate entities. |
| `idp.webhook.functional.validation.failed.total` | `connector`, `template`, `reason` | Functional validation failures captured after a mapping or entity operation. |

These metric names are centralized in `IngestionObservabilityConstants` and are intentionally aligned with the connector, template, action, and reason dimensions used across the webhook processing pipeline.

## Logging

Expand All @@ -85,19 +120,23 @@ OpenTelemetry tracing is enabled through Spring Boot configuration. The applicat

| Variable | Description | Default |
| --- | --- | --- |
| `OTEL_TRACING_ENABLED` | Enables tracing export | `false` |
| `OTEL_METRICS_EXPORT_ENABLED` | Enables OTLP metrics export | `false` |
| `OTEL_TRACES_EXPORTER` | Trace exporter; `none` disables export | `otlp` (`none` locally) |
| `OTEL_METRICS_EXPORTER` | Metric exporters; `console,otlp` for diagnosis | `otlp` |
| `OTEL_METRIC_EXPORT_INTERVAL` | Metric export interval in milliseconds | `60000` |
| `OTEL_LOGS_EXPORTER` | Log exporter | `none` |
| `OTEL_TRACES_SAMPLER_ARG` | Trace sampling ratio | `0.1` |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Base OTLP endpoint | `http://localhost:4318` |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Full traces endpoint | `<base>/v1/traces` |
| `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | Full metrics endpoint | `<base>/v1/metrics` |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | Generic OTLP protocol fallback | `http/protobuf` for metrics, `http` for traces |
| `OTEL_EXPORTER_OTLP_TRACES_TRANSPORT` | Trace-specific transport override | `OTEL_EXPORTER_OTLP_PROTOCOL` or `http` |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | OTLP protocol, `http/protobuf` or `grpc` | `http/protobuf` |
| `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` | Trace-specific protocol override | `OTEL_EXPORTER_OTLP_PROTOCOL` |
| `OTEL_EXPORTER_OTLP_METRICS_PROTOCOL` | Metrics-specific transport override | `OTEL_EXPORTER_OTLP_PROTOCOL` |
| `OTEL_RESOURCE_ATTRIBUTES` | Resource metadata injected by the environment | - |

The signal-specific variables win over the generic fallback, so local development and hosted deployments can both work without hard-coded vendor configuration.

Replace the old `OTEL_TRACING_ENABLED` and `OTEL_METRICS_EXPORT_ENABLED` flags with exporter selection (`otlp` or `none`). Replace `*_TRANSPORT=http` with `*_PROTOCOL=http/protobuf`. The official starter reads `otel.*`, not the former `management.opentelemetry.*` export settings. The test profile disables network export for all signals.

### Important trace attributes

The application sets trace attributes for the webhook pipeline, including the connector identifier and the security decision.
Expand All @@ -119,7 +158,9 @@ The app exposes health endpoints used by Docker and Kubernetes.
| `/actuator/health/liveness` | Liveness probe |
| `/actuator/health/readiness` | Readiness probe |

## Next steps
---

## Next Steps

- **[Configuration](configuration.md)** - Environment and Spring Boot settings
- **[Docker](docker.md)** - Docker deployment and OTLP examples
Expand Down
31 changes: 28 additions & 3 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,14 @@
<!-- Swagger version management to avoid conflicts -->
<dependencyManagement>
<dependencies>
<!-- Align the native metrics SDK and the official OTel Spring Boot starter. -->
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-instrumentation-bom</artifactId>
<version>2.32.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-bom</artifactId>
Expand Down Expand Up @@ -180,6 +188,12 @@
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-testing</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.mock-server</groupId>
<artifactId>mockserver-client-java</artifactId>
Expand Down Expand Up @@ -268,7 +282,7 @@

<!-- JSLT: JVM-native JSON transformation language used as the DSL engine for dynamic mappings -->
<!-- ADR 0004: JSLT chosen over JQ for performance, robust syntax checking and JVM-native
execution -->
execution -->
<dependency>
<groupId>com.schibsted.spt.data</groupId>
<artifactId>jslt</artifactId>
Expand Down Expand Up @@ -332,21 +346,32 @@
<artifactId>camel-rest-starter</artifactId>
</dependency>

<!-- Autoconfigures native OTel metrics, readers and exporters, not Micrometer metrics. -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-opentelemetry</artifactId>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-spring-boot-starter</artifactId>
</dependency>

<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-opentelemetry2-starter</artifactId>
</dependency>

<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api</artifactId>
</dependency>

<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-micrometer-starter</artifactId>
</dependency>

<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-otlp</artifactId>
</dependency>

</dependencies>

<build>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@
import com.decathlon.idp_core.infrastructure.adapters.api.mapper.entity.SearchFilterMapper;
import com.decathlon.idp_core.infrastructure.adapters.common.model.ErrorResponse;

import io.opentelemetry.api.trace.Span;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.enums.ParameterIn;
Expand Down Expand Up @@ -178,6 +179,11 @@ public class EntityController {
public Page<EntityDtoOut> getEntities(@RequestParam(defaultValue = "0") int page,
@RequestParam(required = false) Integer size, @PathVariable String templateIdentifier,
@RequestParam(required = false) String q) {
Span currentSpan = Span.current();
currentSpan.setAttribute("idp.entity.template", templateIdentifier);
currentSpan.setAttribute("idp.entity.page", page);
currentSpan.setAttribute("idp.entity.filter.query", q == null ? "" : q);

PaginationCriteria paginationCriteria = new PaginationCriteria(page, resolvePageSize(size),
null);
EntityFilter filter = entityFilterDslParser.parse(q);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
package com.decathlon.idp_core.infrastructure.adapters.ingestion.configuration;

import org.apache.camel.CamelContext;
import org.apache.camel.component.micrometer.eventnotifier.MicrometerExchangeEventNotifier;
import org.apache.camel.component.micrometer.eventnotifier.MicrometerRouteEventNotifier;
import org.apache.camel.component.micrometer.messagehistory.MicrometerMessageHistoryFactory;
import org.apache.camel.component.micrometer.routepolicy.MicrometerRoutePolicyFactory;
import org.apache.camel.spring.boot.CamelContextConfiguration;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import io.micrometer.core.instrument.MeterRegistry;
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.propagation.ContextPropagators;
import lombok.extern.slf4j.Slf4j;

@Configuration
@Slf4j
public class CamelTracingConfiguration {
Comment thread
foukou19 marked this conversation as resolved.
Resolved

static final String CAMEL_INSTRUMENTATION_SCOPE = "camel";

@Bean

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Null-safe OpenTelemetry fallback in Camel tracing

In CamelTracingConfiguration.java:20-29, pass OpenTelemetry::noop as the default supplier so sliced tests or contexts without the OTel starter do not fail with a NullPointerException:

Suggested change
@Bean
@Bean
public Tracer camelTracer(ObjectProvider<OpenTelemetry> openTelemetryProvider) {
return openTelemetryProvider.getIfAvailable(OpenTelemetry::noop)
.getTracer(CAMEL_INSTRUMENTATION_SCOPE);
}
@Bean
public ContextPropagators camelContextPropagators(
ObjectProvider<OpenTelemetry> openTelemetryProvider) {
return openTelemetryProvider.getIfAvailable(OpenTelemetry::noop).getPropagators();
}

public Tracer camelTracer(ObjectProvider<OpenTelemetry> openTelemetryProvider) {
return openTelemetryProvider.getIfAvailable().getTracer(CAMEL_INSTRUMENTATION_SCOPE);
}

@Bean
public ContextPropagators camelContextPropagators(
ObjectProvider<OpenTelemetry> openTelemetryProvider) {
return openTelemetryProvider.getIfAvailable().getPropagators();
}

@Bean
public CamelContextConfiguration camelTelemetryBridge(Tracer camelTracer,
ContextPropagators camelContextPropagators, MeterRegistry meterRegistry) {
return new CamelContextConfiguration() {
@Override
public void beforeApplicationStart(CamelContext camelContext) {
// 1. Preserve the native OTel tracing bridge.
camelContext.getRegistry().bind("camelTracer", camelTracer);
camelContext.getRegistry().bind("camelContextPropagators", camelContextPropagators);

// 2. Emit camel.* metrics into Micrometer so they can be exported through the
// OTLP registry.
MicrometerRoutePolicyFactory routePolicyFactory = new MicrometerRoutePolicyFactory();
routePolicyFactory.setMeterRegistry(meterRegistry);
camelContext.addRoutePolicyFactory(routePolicyFactory);

MicrometerMessageHistoryFactory messageHistoryFactory = new MicrometerMessageHistoryFactory();
messageHistoryFactory.setMeterRegistry(meterRegistry);
camelContext.setMessageHistoryFactory(messageHistoryFactory);

MicrometerExchangeEventNotifier exchangeNotifier = new MicrometerExchangeEventNotifier();
exchangeNotifier.setMeterRegistry(meterRegistry);
camelContext.getManagementStrategy().addEventNotifier(exchangeNotifier);

MicrometerRouteEventNotifier routeNotifier = new MicrometerRouteEventNotifier();
routeNotifier.setMeterRegistry(meterRegistry);
camelContext.getManagementStrategy().addEventNotifier(routeNotifier);

log.info("[TELEMETRY BRIDGE] Registered Camel OTel Tracer and Micrometer route metrics.");
}

@Override
public void afterApplicationStart(CamelContext camelContext) {
// No-op.
}
};
}
}
Loading
Loading