Verify the "First Water through the Pipes" end-to-end integration between Datacore (T1 Adapter) and the DiaT Production Substrate.
The integrated run successfully demonstrated the full synchronous path from a Datacore-compiled job to a live worker execution.
integrated-t1-1779072910
| import logging | |
| import time | |
| from typing import Any, List, Mapping, Optional | |
| from aws_durable_execution_sdk_python import DurableContext, durable_execution, durable_step | |
| from diat.protocol.transform import ( | |
| EngineName, | |
| EnginePolicy, | |
| EnginePolicyMode, |
This pass validates the TransformJob JSON produced by the current Datacore compiler against the DiaT protocol and execution runtime.
Extracted from Datacore branch origin/codex/datacore-t1-diat-compiler (at 2b6157a).
{
| # SSH Agent | |
| if [[ ! $(ps -ef | grep "[s]sh-agent") ]]; then | |
| echo "Starting SSH Agent" | |
| eval $(ssh-agent -s) | |
| else | |
| ssh_agent_pid=$(pidof ssh-agent) | |
| socket=$(find /tmp/ssh* | grep agent) | |
| short_sock=$(find /tmp/ssh* | grep agent | cut -d '.' -f 2) | |
| if [[ "${short_sock::-2}" == "${ssh_agent_pid::-2}" ]]; then | |
| echo "Found ssh-agent running with pid ${ssh_agent_pid} and matching" |
| #!/usr/bin/env python | |
| import boto3 | |
| import json | |
| from boto3.dynamodb.conditions import Key, Attr | |
| from datetime import datetime | |
| DIMENSION = 'logicalCpuCount' | |
| TABLE_NAME = 'ms-usage-metering' |
| #!/usr/bin/env python3 | |
| # | |
| # cc-viz.py | |
| # | |
| # A hastily-thrown together collection of functions that gets confluent cloud metrics into datadog, graphite, or prometheus (or any combination of those) | |
| # | |
| GRAPHITE_HOST = '10.1.2.3' | |
| GRAPHITE_PORT = 2003 | |
| GRAPHITE_METRIC_PREFIX = 'confluent.cloud.kafka' |
| FIXED_HOURLY_COSTS = { | |
| 'DISKS': { | |
| 'aws': { | |
| 'us-east-1': { | |
| 'io1': 0.125 / 30 / 24, | |
| 'gp2': 0.1 / 30 / 24, | |
| 'st1': 0.045 / 30 / 24, | |
| 'sc1': 0.025 / 30 / 24 | |
| }, | |
| 'us-east-2': { |
| ### how to upload to gist with bash / curl | |
| DESC="file that I'm uploading" | |
| FNAME="file_to_upload" | |
| FPATH="/tmp/file_to_upload" | |
| # 1. Somehow sanitize the file content | |
| # Remove \r (from Windows end-of-lines), | |
| # Replace tabs by \t | |
| # Replace " by \" |
| ##################################### | |
| Instance: 10.74.147.10:7203 | |
| ##################################### | |
| Matching: 1/350. Bean name: kafka.network:type=RequestMetrics,name=RequestsPerSec,request=OffsetCommit,version=6 - Attribute name: Count - Attribute type: long | |
| Matching: 2/350. Bean name: kafka.network:type=RequestMetrics,name=ResponseSendTimeMs,request=FetchConsumer - Attribute name: 50thPercentile - Attribute type: double | |
| Matching: 3/350. Bean name: kafka.network:type=RequestMetrics,name=ResponseSendTimeMs,request=FetchConsumer - Attribute name: Mean - Attribute type: double | |
| Matching: 4/350. Bean name: kafka.network:type=RequestMetrics,name=ResponseSendTimeMs,request=FetchConsumer - Attribute name: 99thPercentile - Attribute type: double | |
| Matching: 5/350. Bean name: kafka.network:type=RequestMetrics,name=ResponseSendTimeMs,request=FetchConsumer - Attribute name: 999thPercentile - Attribute type: double |
| ##################################### | |
| Instance: 10.74.147.10:7203 | |
| ##################################### | |
| Not Matching: Bean name: kafka.log:type=Log,name=Size,topic=lkc-4vndj__confluent-ksql-ksql-prodtransient_5013047390911865834_1548275444821-KSTREAM-AGGREGATE-STATE-STORE-0000000006-repartition,partition=1 - Attribute name: Value - Attribute type: java.lang.Object | |
| Not Matching: Bean name: kafka.cluster:type=Partition,name=InSyncReplicasCount,topic=__consumer_offsets,partition=37 - Attribute name: Value - Attribute type: java.lang.Object | |
| Not Matching: Bean name: kafka.network:type=RequestMetrics,name=ResponseQueueTimeMs,request=ListGroups - Attribute name: Count - Attribute type: long | |
| Not Matching: Bean name: kafka.network:type=RequestMetrics,name=ResponseQueueTimeMs,request=ListGroups - Attribute name: Min - Attribute type: double | |
| Not Matching: Bean name: kafka.network:type=RequestMetrics,name=ResponseQueueTimeMs,request=ListGroups - Attribute name: 50thPercentile - A |