Glue / Client / get_data_quality_evaluation_config

get_data_quality_evaluation_config

Glue.Client.get_data_quality_evaluation_config(**kwargs)

Retrieves a data quality evaluation configuration.

See also: AWS API Documentation

Request Syntax

response = client.get_data_quality_evaluation_config(
    Name='string'
)
Parameters:

Name (string) –

[REQUIRED]

The name of the data quality evaluation configuration.

Return type:

dict

Returns:

Response Syntax

{
    'Name': 'string',
    'EvaluationRunMode': 'ON_DEMAND'|'EVALUATION_CONFIG',
    'CreatedOn': datetime(2015, 1, 1),
    'LastModifiedOn': datetime(2015, 1, 1),
    'Level': 'CATALOG'|'DATABASE'|'TABLE',
    'DataSource': {
        'GlueTables': {
            'CatalogId': 'string',
            'DatabaseName': 'string',
            'TableNamePattern': 'string',
            'PreProcessingQuery': 'string',
            'ConnectionName': 'string',
            'AdditionalOptions': {
                'string': 'string'
            }
        }
    },
    'RulesetName': 'string',
    'ManagedRulesetName': 'SIMPLE'|'ADVANCED',
    'Triggers': [
        'ONCE'|'SCHEDULE'|'METADATA_CHANGE',
    ],
    'ScheduleExpression': 'string',
    'StartAt': datetime(2015, 1, 1),
    'EndAt': datetime(2015, 1, 1),
    'PauseRuns': True|False,
    'Role': 'string',
    'NumberOfWorkers': 123,
    'Timeout': 123,
    'AdditionalRunOptions': {
        'CloudWatchMetricsEnabled': True|False,
        'ResultsS3Prefix': 'string',
        'CompositeRuleEvaluationMethod': 'COLUMN'|'ROW',
        'RowLevelResults': {
            'MaxRowsToWrite': 123,
            'ResultType': 'ALL'|'PASSED_ONLY'|'FAILED_ONLY',
            'CatalogTableConfig': {
                'DatabaseName': 'string',
                'TableName': 'string',
                'S3Location': 'string',
                'CatalogId': 'string'
            }
        },
        'ProfilingResults': {
            'WriteProfilingResultsEnabled': True|False,
            'CatalogTableConfig': {
                'DatabaseName': 'string',
                'TableName': 'string',
                'S3Location': 'string',
                'CatalogId': 'string'
            },
            'DistributionResults': {
                'WriteDistributionResultsEnabled': True|False,
                'CatalogTableConfig': {
                    'DatabaseName': 'string',
                    'TableName': 'string',
                    'S3Location': 'string',
                    'CatalogId': 'string'
                }
            }
        },
        'ObservationScope': 'ALL'|'NONE',
        'ObservationMode': 'SCHEDULED'|'FIXED',
        'DataQualityRuleResults': {
            'WriteDataQualityRuleResultsEnabled': True|False,
            'CatalogTableConfig': {
                'DatabaseName': 'string',
                'TableName': 'string',
                'S3Location': 'string',
                'CatalogId': 'string'
            }
        },
        'ObservationResults': {
            'WriteObservationResultsEnabled': True|False,
            'CatalogTableConfig': {
                'DatabaseName': 'string',
                'TableName': 'string',
                'S3Location': 'string',
                'CatalogId': 'string'
            }
        },
        'CustomLogGroupPrefix': 'string'
    },
    'AdditionalDataSources': {
        'string': {
            'GlueTable': {
                'DatabaseName': 'string',
                'TableName': 'string',
                'CatalogId': 'string',
                'ConnectionName': 'string',
                'AdditionalOptions': {
                    'string': 'string'
                }
            },
            'DataQualityGlueTable': {
                'DatabaseName': 'string',
                'TableName': 'string',
                'CatalogId': 'string',
                'ConnectionName': 'string',
                'AdditionalOptions': {
                    'string': 'string'
                },
                'PreProcessingQuery': 'string'
            }
        }
    }
}

Response Structure

  • (dict) –

    • Name (string) –

      The name of the data quality evaluation configuration.

    • EvaluationRunMode (string) –

      The evaluation run mode of the configuration’s results, which is EVALUATION_CONFIG.

    • CreatedOn (datetime) –

      The time when the configuration was created.

    • LastModifiedOn (datetime) –

      The time when the configuration was last modified.

    • Level (string) –

      The Glue Data Catalog hierarchical level that the configuration applies to.

    • DataSource (dict) –

      The catalog, database, or table that the configuration applies to.

      • GlueTables (dict) –

        The Glue Data Catalog tables that the configuration applies to.

        • CatalogId (string) –

          A unique identifier for the Glue Data Catalog.

        • DatabaseName (string) –

          A database name in the Glue Data Catalog.

        • TableNamePattern (string) –

          For the CATALOG and DATABASE levels, a glob pattern, such as orders_*, that selects tables. If you don’t specify a pattern, every table at the level is in scope. For the TABLE level, the name of the table.

        • PreProcessingQuery (string) –

          SQL Query of SparkSQL format that can be used to pre-process the data for the table in Glue Data Catalog, before running the Data Quality Operation.

          For the CATALOG and DATABASE levels, use the {{DATABASE_NAME}} and {{TABLE_NAME}} placeholders, which are filled in for each table.

        • ConnectionName (string) –

          The name of the connection to the Glue Data Catalog.

        • AdditionalOptions (dict) –

          Options for reading the table. The following keys are supported:

          • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

          • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

          • (string) –

            • (string) –

    • RulesetName (string) –

      The name of the data quality ruleset that tables are evaluated with.

    • ManagedRulesetName (string) –

      The built-in ruleset that tables are evaluated with.

    • Triggers (list) –

      When the configuration evaluates tables:

      ONCE

      Each table is evaluated one time, including when tables are added to a database or catalog.

      SCHEDULE

      Each table is evaluated on the cron or rate expression in ScheduleExpression.

      METADATA_CHANGE

      A table is evaluated whenever its Glue Data Catalog metadata changes.

      • (string) –

    • ScheduleExpression (string) –

      The cron or rate expression, in UTC, for the SCHEDULE trigger.

    • StartAt (datetime) –

      The time from which the schedule runs, inclusive.

    • EndAt (datetime) –

      The time at which the schedule stops running, exclusive.

    • PauseRuns (boolean) –

      Specifies whether evaluations are paused.

    • Role (string) –

      The Amazon Resource Name (ARN) of the IAM role that evaluation runs assume.

    • NumberOfWorkers (integer) –

      The number of G.1X workers to be used in the run. The default is 5.

    • Timeout (integer) –

      The timeout for a run in minutes. This is the maximum time that a run can consume resources before it is terminated and enters TIMEOUT status. The default is 2,880 minutes (48 hours).

    • AdditionalRunOptions (dict) –

      Additional run options you can specify for an evaluation run.

      • CloudWatchMetricsEnabled (boolean) –

        Specifies whether to enable CloudWatch metrics.

      • ResultsS3Prefix (string) –

        Prefix for Amazon S3 to store results.

      • CompositeRuleEvaluationMethod (string) –

        Set the evaluation method for composite rules in the ruleset to ROW/COLUMN

      • RowLevelResults (dict) –

        The configuration for writing row-level evaluation results to a Glue Data Catalog table.

        • MaxRowsToWrite (integer) –

          The maximum number of rows to write in the results.

        • ResultType (string) –

          The result type to include in the row-level results output.

        • CatalogTableConfig (dict) –

          The Glue Data Catalog table configuration for storing the results.

          • DatabaseName (string) –

            The name of the database in the Glue Data Catalog.

          • TableName (string) –

            The name of the table in the Glue Data Catalog.

          • S3Location (string) –

            The Amazon S3 location for storing the results.

          • CatalogId (string) –

            A unique identifier for the Glue Data Catalog.

      • ProfilingResults (dict) –

        The configuration for writing profiling results to a Glue Data Catalog table.

        • WriteProfilingResultsEnabled (boolean) –

          Set to true to write profiling results.

        • CatalogTableConfig (dict) –

          The Glue Data Catalog table configuration for storing the profiling results.

          • DatabaseName (string) –

            The name of the database in the Glue Data Catalog.

          • TableName (string) –

            The name of the table in the Glue Data Catalog.

          • S3Location (string) –

            The Amazon S3 location for storing the results.

          • CatalogId (string) –

            A unique identifier for the Glue Data Catalog.

        • DistributionResults (dict) –

          The configuration for writing distribution results.

          • WriteDistributionResultsEnabled (boolean) –

            Set to true to write distribution results.

          • CatalogTableConfig (dict) –

            The Glue Data Catalog table configuration for storing the distribution results.

            • DatabaseName (string) –

              The name of the database in the Glue Data Catalog.

            • TableName (string) –

              The name of the table in the Glue Data Catalog.

            • S3Location (string) –

              The Amazon S3 location for storing the results.

            • CatalogId (string) –

              A unique identifier for the Glue Data Catalog.

      • ObservationScope (string) –

        The scope of the observation for the evaluation run. Specifies whether anomaly detection is enabled or disabled.

      • ObservationMode (string) –

        The observation mode for the evaluation run. Specifies how anomaly detection bounds are calculated.

      • DataQualityRuleResults (dict) –

        The configuration for writing rule results to a Glue Data Catalog table.

        • WriteDataQualityRuleResultsEnabled (boolean) –

          Set to true to write data quality rule results.

        • CatalogTableConfig (dict) –

          The Glue Data Catalog table configuration for storing the rule results.

          • DatabaseName (string) –

            The name of the database in the Glue Data Catalog.

          • TableName (string) –

            The name of the table in the Glue Data Catalog.

          • S3Location (string) –

            The Amazon S3 location for storing the results.

          • CatalogId (string) –

            A unique identifier for the Glue Data Catalog.

      • ObservationResults (dict) –

        The configuration for writing observation results to a Glue Data Catalog table.

        • WriteObservationResultsEnabled (boolean) –

          Set to true to write observation results.

        • CatalogTableConfig (dict) –

          The Glue Data Catalog table configuration for storing the observation results.

          • DatabaseName (string) –

            The name of the database in the Glue Data Catalog.

          • TableName (string) –

            The name of the table in the Glue Data Catalog.

          • S3Location (string) –

            The Amazon S3 location for storing the results.

          • CatalogId (string) –

            A unique identifier for the Glue Data Catalog.

      • CustomLogGroupPrefix (string) –

        A custom prefix for the CloudWatch log group names. When specified, evaluation run logs are written to <CustomLogGroupPrefix>/error and <CustomLogGroupPrefix>/output instead of the default /aws-glue/data-quality/error and /aws-glue/data-quality/output log groups.

    • AdditionalDataSources (dict) –

      A map of reference strings to additional data sources you can specify for an evaluation run.

      • (string) –

        • (dict) –

          A data source (an Glue table) for which you want data quality results.

          • GlueTable (dict) –

            An Glue table.

            • DatabaseName (string) –

              A database name in the Glue Data Catalog.

            • TableName (string) –

              A table name in the Glue Data Catalog.

            • CatalogId (string) –

              A unique identifier for the Glue Data Catalog.

            • ConnectionName (string) –

              The name of the connection to the Glue Data Catalog.

            • AdditionalOptions (dict) –

              Additional options for the table. Currently there are two keys supported:

              • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

              • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

              • (string) –

                • (string) –

          • DataQualityGlueTable (dict) –

            An Glue table for Data Quality Operations.

            • DatabaseName (string) –

              A database name in the Glue Data Catalog.

            • TableName (string) –

              A table name in the Glue Data Catalog.

            • CatalogId (string) –

              A unique identifier for the Glue Data Catalog.

            • ConnectionName (string) –

              The name of the connection to the Glue Data Catalog.

            • AdditionalOptions (dict) –

              Additional options for the table. Currently there are two keys supported:

              • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

              • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

              • (string) –

                • (string) –

            • PreProcessingQuery (string) –

              SQL Query of SparkSQL format that can be used to pre-process the data for the table in Glue Data Catalog, before running the Data Quality Operation.

Exceptions