Glue / Client / publish_data_quality_result_v2

publish_data_quality_result_v2

Glue.Client.publish_data_quality_result_v2(**kwargs)

Note

This operation is intended for internal use. Do not call it directly.

AWS Glue Data Quality uses this operation to publish a data quality result.

See also: AWS API Documentation

Request Syntax

response = client.publish_data_quality_result_v2(
    Dimensions={
        'string': 'string'
    },
    Tags={
        'string': 'string'
    },
    ExecutionStartedOn=datetime(2015, 1, 1),
    ExecutionCompletedOn=datetime(2015, 1, 1),
    Score=123.0,
    RuleResults=[
        {
            'Name': 'string',
            'Description': 'string',
            'EvaluationMessage': 'string',
            'Result': 'PASS'|'FAIL'|'ERROR',
            'EvaluatedMetrics': {
                'string': 123.0
            },
            'EvaluatedRule': 'string',
            'RuleMetrics': {
                'string': 123.0
            },
            'Labels': {
                'string': 'string'
            }
        },
    ],
    AnalyzerResults=[
        {
            'Name': 'string',
            'Description': 'string',
            'EvaluationMessage': 'string',
            'EvaluatedMetrics': {
                'string': 123.0
            },
            'EvaluatedDistributions': {
                'string': {
                    'BinEdges': [
                        'string',
                    ],
                    'Count': [
                        123,
                    ],
                    'DataType': 'string'
                }
            }
        },
    ],
    Observations=[
        {
            'Description': 'string',
            'MetricBasedObservation': {
                'MetricName': 'string',
                'StatisticId': 'string',
                'MetricValues': {
                    'ActualValue': 123.0,
                    'ExpectedValue': 123.0,
                    'LowerLimit': 123.0,
                    'UpperLimit': 123.0
                },
                'NewRules': [
                    'string',
                ]
            }
        },
    ],
    AggregatedMetrics={
        'TotalRowsProcessed': 123.0,
        'TotalRowsPassed': 123.0,
        'TotalRowsFailed': 123.0,
        'TotalRulesProcessed': 123.0,
        'TotalRulesPassed': 123.0,
        'TotalRulesFailed': 123.0
    },
    Statistics=[
        {
            'StatisticId': 'string',
            'ProfileId': 'string',
            'RunIdentifier': {
                'RunId': 'string',
                'JobRunId': 'string'
            },
            'StatisticName': 'string',
            'DoubleValue': 123.0,
            'DistributionValue': {
                'BinEdges': [
                    'string',
                ],
                'Count': [
                    123,
                ],
                'DataType': 'string'
            },
            'EvaluationLevel': 'Dataset'|'Column'|'Multicolumn',
            'ColumnsReferenced': [
                'string',
            ],
            'ReferencedDatasets': [
                'string',
            ],
            'StatisticProperties': {
                'string': 'string'
            },
            'RecordedOn': datetime(2015, 1, 1),
            'InclusionAnnotation': {
                'Value': 'INCLUDE'|'EXCLUDE',
                'LastModifiedOn': datetime(2015, 1, 1)
            }
        },
    ],
    RunConfiguration={
        'ObservationsEnabled': 'ALL'|'NONE',
        'ObservationMode': 'SCHEDULED'|'FIXED'
    },
    ClientToken='string',
    EvaluationContext='string',
    Descriptors={
        'string': 'string'
    },
    RulesetName='string',
    DataQualitySecurityConfiguration='string',
    Telemetry={
        'VcpuHours': 123.0,
        'FlagsEnabled': {
            'string': 'string'
        },
        'LibraryVersion': 'string',
        'SparkVersion': 'string',
        'HostRuntimeVersion': 'string',
        'Engine': 'SPARK'|'PYTHON'
    }
)
Parameters:

  • Dimensions (dict) –

    [REQUIRED]

    The dimensions for the result.

    • (string) –

      • (string) –

  • Tags (dict) –

    The tags for the result.

    • (string) –

      • (string) –

  • ExecutionStartedOn (datetime) – The time execution started for the result.

  • ExecutionCompletedOn (datetime) – The time execution completed for the result.

  • Score (float) –

    [REQUIRED]

    The score for the result.

  • RuleResults (list) –

    [REQUIRED]

    The rule level results.

    • (dict) –

      Describes the result of the evaluation of a data quality rule.

      • Name (string) –

        The name of the data quality rule.

      • Description (string) –

        A description of the data quality rule.

      • EvaluationMessage (string) –

        An evaluation message.

      • Result (string) –

        A pass or fail status for the rule.

      • EvaluatedMetrics (dict) –

        A map of metrics associated with the evaluation of the rule.

        • (string) –

          • (float) –

      • EvaluatedRule (string) –

        The evaluated rule.

      • RuleMetrics (dict) –

        A map containing metrics associated with the evaluation of the rule based on row-level results.

        • (string) –

          • (float) –

      • Labels (dict) –

        A map containing labels assigned to the data quality rule.

        • (string) –

          • (string) –

  • AnalyzerResults (list) –

    The analyzer level results.

    • (dict) –

      Describes the result of the evaluation of a data quality analyzer.

      • Name (string) –

        The name of the data quality analyzer.

      • Description (string) –

        A description of the data quality analyzer.

      • EvaluationMessage (string) –

        An evaluation message.

      • EvaluatedMetrics (dict) –

        A map of metrics associated with the evaluation of the analyzer.

        • (string) –

          • (float) –

      • EvaluatedDistributions (dict) –

        A map of distribution metrics associated with the evaluation of the analyzer.

        • (string) –

          • (dict) –

            The distribution data for a statistic.

            • BinEdges (list) –

              The bin edge values for the distribution.

              • (string) –

            • Count (list) –

              The frequency count for each bin in the distribution.

              • (integer) –

            • DataType (string) –

              The data type of the column for the distribution.

  • Observations (list) –

    The observations for the result.

    • (dict) –

      Describes the observation generated after evaluating the rules and analyzers.

      • Description (string) –

        A description of the data quality observation.

      • MetricBasedObservation (dict) –

        An object of type MetricBasedObservation representing the observation that is based on evaluated data quality metrics.

        • MetricName (string) –

          The name of the data quality metric used for generating the observation.

        • StatisticId (string) –

          The Statistic ID.

        • MetricValues (dict) –

          An object of type DataQualityMetricValues representing the analysis of the data quality metric value.

          • ActualValue (float) –

            The actual value of the data quality metric.

          • ExpectedValue (float) –

            The expected value of the data quality metric according to the analysis of historical data.

          • LowerLimit (float) –

            The lower limit of the data quality metric value according to the analysis of historical data.

          • UpperLimit (float) –

            The upper limit of the data quality metric value according to the analysis of historical data.

        • NewRules (list) –

          A list of new data quality rules generated as part of the observation based on the data quality metric value.

          • (string) –

  • AggregatedMetrics (dict) –

    The aggregated metrics for the result.

    • TotalRowsProcessed (float) –

      The total number of rows that were processed during the data quality evaluation.

    • TotalRowsPassed (float) –

      The total number of rows that passed all applicable data quality rules.

    • TotalRowsFailed (float) –

      The total number of rows that failed one or more data quality rules.

    • TotalRulesProcessed (float) –

      The total number of data quality rules that were evaluated.

    • TotalRulesPassed (float) –

      The total number of data quality rules that passed their evaluation criteria.

    • TotalRulesFailed (float) –

      The total number of data quality rules that failed their evaluation criteria.

  • Statistics (list) –

    The statistics for the result.

    • (dict) –

      Summary information about a statistic.

      • StatisticId (string) –

        The Statistic ID.

      • ProfileId (string) –

        The Profile ID.

      • RunIdentifier (dict) –

        The Run Identifier

        • RunId (string) –

          The Run ID.

        • JobRunId (string) –

          The Job Run ID.

      • StatisticName (string) –

        The name of the statistic.

      • DoubleValue (float) –

        The value of the statistic.

      • DistributionValue (dict) –

        The distribution value for the statistic.

        • BinEdges (list) –

          The bin edge values for the distribution.

          • (string) –

        • Count (list) –

          The frequency count for each bin in the distribution.

          • (integer) –

        • DataType (string) –

          The data type of the column for the distribution.

      • EvaluationLevel (string) –

        The evaluation level of the statistic. Possible values: Dataset, Column, Multicolumn.

      • ColumnsReferenced (list) –

        The list of columns referenced by the statistic.

        • (string) –

      • ReferencedDatasets (list) –

        The list of datasets referenced by the statistic.

        • (string) –

      • StatisticProperties (dict) –

        A StatisticPropertiesMap, which contains a NameString and DescriptionString

        • (string) –

          • (string) –

      • RecordedOn (datetime) –

        The timestamp when the statistic was recorded.

      • InclusionAnnotation (dict) –

        The inclusion annotation for the statistic.

        • Value (string) –

          The inclusion annotation value.

        • LastModifiedOn (datetime) –

          The timestamp when the inclusion annotation was last modified.

  • RunConfiguration (dict) –

    The run configuration for the result.

    • ObservationsEnabled (string) –

      Whether observations are enabled for the run. Valid values are:

      • ALL

      • NONE

    • ObservationMode (string) –

      The observation mode for the run. Valid values are:

      • SCHEDULED

      • FIXED

  • ClientToken (string) –

    The client token for the request.

    This field is autopopulated if not provided.

  • EvaluationContext (string) –

    [REQUIRED]

    The evaluation context for the result.

  • Descriptors (dict) –

    The descriptors for the result.

    • (string) –

      • (string) –

  • RulesetName (string) – The ruleset name for the result.

  • DataQualitySecurityConfiguration (string) – The security configuration for the result.

  • Telemetry (dict) –

    • VcpuHours (float) –

    • FlagsEnabled (dict) –

      • (string) –

        • (string) –

    • LibraryVersion (string) –

    • SparkVersion (string) –

    • HostRuntimeVersion (string) –

    • Engine (string) –

Return type:

dict

Returns:

Response Syntax

{
    'ResultId': 'string'
}

Response Structure

  • (dict) –

    • ResultId (string) –

      The ID of the result.

Exceptions