Knowledge Processing in Cisco Observability Platform

Course of Huge Quantities of MELT Knowledge

Cisco Observability Platform s designed to ingest and course of huge quantities of MELT (Metrics, Occasions, Logs and Traces) knowledge. It’s constructed on prime of open requirements like OpenTelemetry to make sure interoperability.

What units it aside is its provision of extensions, empowering our companions and clients to tailor each side of its performance to their distinctive wants. Our focus right now is unveiling the intricacies of customizations particularly tailor-made for knowledge processing. It’s anticipated that you’ve got an understanding of the platform fundamentals, like Versatile Metadata Mannequin (FMM) and resolution growth. Let’s dive in!

The information processing pipeline has numerous levels that result in knowledge storage. As MELT knowledge strikes by the pipeline, it’s processed, reworked, and enriched, and finally lands within the knowledge retailer the place it may be queried with Unified Question Language (UQL):

Every stage marked with a gear icon permits customization of particular logic. Moreover, the platform permits the creation of solely customized post-processing logic when knowledge can not be altered.

To streamline customization whereas sustaining flexibility, we’re embracing a brand new method: workflows, faucets, and plugins, using the CNCF Serverless Workflow specification with JSONata because the default expression language. Since Serverless Workflows are designed utilizing open requirements, we’re extensively using CloudEvents and OpenAPI specs. By leveraging these open requirements, we guarantee compatibility and ease of growth.

Knowledge processing levels that permit knowledge mutation are referred to as faucets, and their customizations plugins. Every faucet declares an enter and output JSON schema for its plugins. Plugins are anticipated to supply an output that adheres to the faucet’s output schema. A faucet is accountable for merging outputs from all its plugins and producing a brand new occasion, which is a modified model of an authentic occasion. Faucets can solely be authored by the platform, whereas plugins will be created by any resolution in addition to common customers of the platform.

Workflows are meant for post-processing and thus can solely subscribe to triggers (see under). Workflow use circumstances vary from easy occasion counting to classy machine studying mannequin inferences. Anybody can writer workflows.

This abstraction permits builders to motive by way of a single occasion, with out exposing the complexity of the underlying stream processing, and use acquainted properly documented requirements, each of which decrease the barrier of entry.

Every knowledge processing stage communicates with different levels through occasions, which permits us to decouple shoppers and producers and seamlessly rearrange the levels ought to the necessity come up.
Every occasion has an related class, which determines whether or not a selected stage can subscribe to or publish that occasion. There are two public classes for data-related occasions:

• knowledge:remark – a class of occasions with publish-only permissions which will be considered side-effects of processing the unique occasion, for instance, an entity derived from useful resource attributes in OpenTelemetry metric packet. Observations are indicated with upward ‘publish’ arrows within the above diagram. Faucets, workflows and plugins can all produce observations. Observations can solely be subscribed to by particular faucets.
• knowledge:set off – subscribe-only occasions which might be emitted after all of the mutations have accomplished. Triggers are indicated with a lightning ‘set off’ icon within the above diagram. Solely workflows (post-processing logic) can subscribe to triggers, and solely particular faucets can publish them.

There are 5 remark occasion varieties within the platform:

• entity.noticed – FMM entity was found whereas processing some knowledge. It may be a brand new entity or an replace to an current entity. Every replace from the identical supply absolutely replaces the earlier one.
• affiliation.noticed – FMM affiliation was found whereas processing some knowledge. Relying on the cardinality of the affiliation the replace logic differs
• extension.noticed – FMM extension attributes have been found whereas processing some knowledge. A goal entity should exist already.
• measurement.acquired – a measurement occasion which contributes to a selected FMM metric. These measurements will likely be aggregated right into a metric in Metric aggregation faucet. Aggregation logic is dependent upon the metric’s content material kind.
• occasion.acquired – raises a brand new FMM occasion. This occasion can even be processed by the Occasion processing faucet, similar to externally ingested occasions.

There are 3 set off occasion varieties within the platform, one for every knowledge sort: metric.enriched, occasion.enriched, hint.encriched. All three occasions are emitted from the ultimate ‘Tag enrichment’ faucet.

Every occasion is registered in a platform’s information retailer, in order that they’re simply discoverable. To record all obtainable occasions, merely use fsoc to question them, i.e., to get all triggers:

fsoc information get --type=contracts:cloudevent --filter="knowledge.class eq 'knowledge:set off'" --layer-type=TENANT

Be aware that every one occasion varieties are versioned to permit for evolution and are certified with platform resolution identifier for isolation. For instance, a totally certified id of measurement.acquired occasion is platform:measurement.acquired.v1

Let’s illustrate the above ideas with an easy instance. Take into account a workflow designed to rely well being rule violations for Kubernetes workloads and APM companies. The logic of the workflow will be damaged down into a number of steps:

1. Subscribe to the set off occasion
2. Validate occasion kind and entity relevance
3. Publish a measurement occasion counting violations whereas retaining severity

Improvement Instruments

Builders can make the most of numerous instruments to help in workflow growth, similar to web-based editors or IDEs.

It’s essential to make sure expressions and logic are legitimate by unit assessments and validation towards outlined schemas.

To assist in that, you may write unit assessments by using said, see an instance for this workflow.
On-line JSONata editor will also be a useful software in writing your expressions.

A weblog on workflow testing is coming quickly!

Step by Step Information

Create the workflow DSL

Present a singular identifier and a reputation to your workflow:

id: violations-counter
model: '1.0.0'
specVersion: '0.8'
identify: Violations Counter

Discover the set off occasion

Let’s question our set off utilizing fsoc:

fsoc information get --type=contracts:cloudevent --object-id=platform:occasion.enriched.v1 --layer-type=TENANT

Output:

kind: occasion.enriched.v1
description: Signifies that an occasion was enriched with topology tags
dataschema: contracts:jsonSchema/platform:occasion.v1
class: knowledge:set off
extensions:
  - contracts:cloudeventExtension/platform:entitytypes
  - contracts:cloudeventExtension/platform:supply

Subscribe to the occasion

To subscribe to this occasion, it’s good to add an occasion definition and occasion state referencing this definition (observe a nature of the reference to the occasion – it have to be certified with its information kind):

occasions:
  - identify: EventReceived
    kind: contracts:cloudevent/platform:occasion.enriched.v1
    sort: consumed
    dataOnly: false
    supply: platform
states:
  - identify: event-received
    kind: occasion
    onEvents:
      - eventRefs:
          - EventReceived

Examine the occasion

Because the knowledge in workflows is acquired in JSON format, occasion knowledge is described in JSON schema.

Let’s take a look at the JSON schema of this occasion (referenced in dataschema), so you already know what to anticipate in our workflow:

fsoc information get --type=contracts:jsonSchema --object-id=platform:occasion.v1 --layer-type=TENANT

End result:
$schema: http://json-schema.org/draft-07/schema#
title: Occasion
$id: occasion.v1
kind: object
required:
  - entities
  - kind
  - timestamp
properties:
  entities:
    kind: array
    minItems: 1
    gadgets:
      $ref: '#/definitions/EntityReference'
  kind:
    $ref: '#/definitions/TypeReference'
  timestamp:
    kind: integer
    description: The timestamp in milliseconds
  spanId:
    kind: string
    description: Span id
  traceId:
    kind: string
    description: Hint id
  uncooked:
    kind: string
    description: The uncooked physique of the occasion report
  attributes:
    $ref: '#/definitions/Attributes'
  tags:
    $ref: '#/definitions/Tags'
additionalProperties: false
definitions:
  Tags:
    kind: object
    propertyNames:
      minLength: 1
      maxLength: 256
    additionalProperties:
      kind: string
  Attributes:
    kind: object
    propertyNames:
      minLength: 1
      maxLength: 256
    additionalProperties:
      kind:
        - string
        - quantity
        - boolean
        - object
        - array
  EntityReference:
    kind: object
    required:
      - id
      - kind
    properties:
      id:
        kind: string
      kind:
        $ref: '#/definitions/TypeReference'
      additionalProperties: false
  TypeReference:
    kind: string
    description: A totally certified FMM kind reference
    instance: k8s:pod

It’s easy – a single occasion, with a number of entity references. Since dataOnly=false, the payload of the occasion will likely be enclosed within the knowledge discipline, and extension attributes can even be obtainable to the workflow.
Since we all know the precise FMM occasion kind we’re occupied with, you can too question its definition to grasp the attributes that the workflow will likely be receiving and their semantics:

fsoc information get --type=fmm:occasion --filter="knowledge.identify eq "healthrule.violation" and knowledge.namespace.identify eq "alerting"" --layer-type=TENANT

Validate occasion relevance

You’ll want to make sure that the occasion you obtain is of the proper FMM occasion kind, and that referenced entities are related. To do that, you may write an expression in JSONata after which use it in an motion situation:

capabilities:
  - identify: checkType
    kind: expression
    operation: |-
      knowledge.kind="alerting:healthrule.violation" and (
          'k8s:deployment' in knowledge.entities.kind or
          'k8s:statefulset' in knowledge.entities.kind or
          'k8s:daemonset' in knowledge.entities.kind or
          'k8s:cronjob' in knowledge.entities.kind or
          'k8s:managed_job' in knowledge.entities.kind or
          'apm:service' in knowledge.entities.kind
      )
 states:
  - identify: event-received
    kind: occasion
    onEvents:
      - eventRefs:
          - EventReceived
        actions:
          - identify: createMeasurement
            situation: ${ fn:checkType }

Create and publish an occasion

Let’s discover the measurement remark occasion that it’s good to publish:

fsoc information get --type=contracts:cloudevent --object-id=platform:measurement.acquired.v1 --layer-type=TENANT

Output:

kind: measurement.acquired.v1
description: Signifies that measurements have been acquired. Measurements are then aggregated right into a metric.
dataschema: contracts:jsonSchema/platform:measurement.v1
class: knowledge:remark
extensions:
  - contracts:cloudeventExtension/platform:supply

Now let’s take a look at the measurement schema so you know the way to supply a measurement occasion:

fsoc information get --type=contracts:jsonSchema --object-id=platform:measurement.v1 --layer-type=TENANT

Output:

$schema: http://json-schema.org/draft-07/schema#
title: Measurements for a selected metric
$id: measurement.v1
kind: object
required:
  - entity
  - kind
  - measurements
properties:
  entity:
    $ref: '#/definitions/EntityReference'
  kind:
    $ref: '#/definitions/TypeReference'
  attributes:
    $ref: '#/definitions/Attributes'
  measurements:
    kind: array
    minItems: 1
    description: Measurement values with timestamp for use for metric computation
    gadgets:
      kind: object
      required:
        - timestamp
      oneOf:
        - required:
            - intValue
        - required:
            - doubleValue
      properties:
        timestamp:
          kind: integer
          description: The timestamp in milliseconds
        intValue:
          kind: integer
          description: Lengthy worth for use for metric computation.
        doubleValue:
          kind: quantity
          description: Double Measurement worth for use for metric computation.
      additionalProperties: false
additionalProperties: false
definitions:
  Attributes:
    kind: object
    propertyNames:
      minLength: 1
      maxLength: 256
    additionalProperties:
      kind:
        - string
        - quantity
        - boolean
  EntityReference:
    kind: object
    required:
      - id
      - kind
    properties:
      id:
        kind: string
      kind:
        $ref: '#/definitions/TypeReference'
      additionalProperties: false
  TypeReference:
    kind: string
    description: A totally certified FMM kind identify
    instance: k8s:pod

Create a measurement

Let’s create one other expression that takes the enter occasion and generates a measurement as per the above schema, and use it in an motion within the occasion state:

capabilities:
  ...
  - identify: createMeasurement
    kind: expression
    operation: |-
      {
          'entity': knowledge.entities[0],
          'kind': 'sampleworkflow:healthrule.violation.rely',
          'attributes': {
              'violation_severity': knowledge.attributes.violation_severity
          },
          'measurements': [
              {
                  'timestamp': data.timestamp,
                  'intValue': 
$exists(data.attributes.'event_details.condition_details.violation_count')? data.attributes.'event_details.condition_details.violation_count': 1
              }
          ]
      }
states:
  - identify: event-received
    kind: occasion
    onEvents:
      - eventRefs:
          - EventReceived
        actions:
          - identify: createMeasurement
            situation: '${ fn:checkType }'
            functionRef: createMeasurement
            actionDataFilter:
              toStateData: '${ measurement }'

Right here we’re preserving the violation_severity attribute from the unique occasion and associating the measurement with the identical entity.

The state execution will lead to a measurement discipline created by createMeasurement motion, however provided that the occasion was attention-grabbing primarily based on the situation.

Be aware that since we’re utilizing a brand new FMM metric kind – sampleworkflow:healthrule.violation.rely – we have to register it through the extension on the goal entity varieties. See full resolution linked under for particulars.

Publish an occasion

The subsequent step is to examine if the measurement was certainly created, and produce an occasion if it was. To do this, we are going to use a swap state:

states:
  - identify: event-received
    kind: occasion
    onEvents:
      - eventRefs:
          - EventReceived
        actions:
          - identify: createMeasurement
            situation: ${ fn:checkType }
            functionRef:
              refName: createMeasurement
            actionDataFilter:
              toStateData: ${ measurement }
    transition: check-measurement
  - identify: check-measurement
    kind: swap
    dataConditions:
      - situation: ${ measurement != null }
        finish:
          terminate: true
          produceEvents:
            - eventRef: CreateMeasurement
              knowledge: ${ measurement }
    defaultCondition:
      finish: true

That’s it! You’ll be able to bundle your workflow in an answer, push your resolution, subscribe to it, and examine the metrics by navigating to the metric explorer at https://<your tenant>.observe.appdynamics.com/discover/cco/metric-explorer 

An instance graph sliced by violation_severity

In conclusion, the extensibility of the Cisco Observability Platform empowers builders to tailor knowledge processing to their particular necessities effectively. Whether or not it’s customizing processing logic or implementing complicated workflows, the platform supplies the mandatory instruments and adaptability.

Able to study extra? Go to examples repo to discover additional and begin customizing your knowledge processing workflows right now.

Share: