AWS Glue ETL Testing: Unit Tests, Data Quality Rules and Athena Checks

Many Indian IT projects for US and European clients run on AWS, and "ETL testing in AWS" usually means testing AWS Glue jobs that move data through S3 into Redshift or other targets. This guide shows what to test and how.

How ETL Works on AWS

Sources (RDS, files, APIs)
      |   (Glue jobs / DMS)
   S3 raw zone  --crawler-->  Glue Data Catalog
      |   (Glue PySpark jobs)
   S3 curated zone (Parquet)  -->  Redshift / Athena / QuickSight

AWS Glue provides serverless Spark jobs, crawlers that infer schemas into the Data Catalog, and job bookmarks for incremental processing. See the ETL tools list.

What to Test in an AWS Glue Pipeline

Unit Testing Glue Transformation Logic

Keep business logic in plain PySpark functions so you can test them without AWS:

# transforms.py
from pyspark.sql import functions as F
def clean_orders(df):
    return (df.withColumn("city", F.initcap(F.trim("city")))
              .filter(F.col("amount") > 0)
              .dropDuplicates(["order_id"]))

# test_transforms.py   -> pytest
from pyspark.sql import SparkSession
from transforms import clean_orders

def test_clean_orders():
    spark = SparkSession.builder.master("local[1]").getOrCreate()
    df = spark.createDataFrame(
        [(1, " pune ", 100.0), (1, " pune ", 100.0), (2, "delhi", -5.0)],
        ["order_id", "city", "amount"])
    out = clean_orders(df).collect()
    assert len(out) == 1 and out[0]["city"] == "Pune"

More on pytest frameworks: Python ETL testing framework.

Running and Testing Glue Jobs Locally

AWS publishes Glue libraries and Docker images (for example amazon/aws-glue-libs) so developers and testers can run Glue scripts and pytest suites locally before deploying, which saves cost and time. Use small sample files in a local folder or a test S3 bucket.

AWS Glue Data Quality Rules

Glue Data Quality lets you declare rules in DQDL (Data Quality Definition Language) and run them on Data Catalog tables or inside jobs:

Rules = [
    RowCount > 0,
    IsComplete "order_id",
    IsUnique "order_id",
    ColumnValues "status" in ["NEW", "PAID", "SHIPPED", "CANCELLED"],
    ColumnValues "amount" > 0
]

Results can stop a pipeline or raise alerts. Similar ideas: data quality checks.

SQL Validation with Amazon Athena

Athena runs SQL directly on S3 data registered in the Data Catalog, so testers can reconcile layers without loading anything:

-- Raw vs curated count for one partition
SELECT COUNT(*) FROM raw_db.orders WHERE dt = '2026-10-01';
SELECT COUNT(*) FROM curated_db.orders WHERE dt = '2026-10-01';

-- Duplicates in curated output
SELECT order_id, COUNT(*) FROM curated_db.orders
WHERE dt = '2026-10-01' GROUP BY order_id HAVING COUNT(*) > 1;

-- Records lost between raw and curated
SELECT order_id FROM raw_db.orders WHERE dt = '2026-10-01' AND amount > 0
EXCEPT
SELECT order_id FROM curated_db.orders WHERE dt = '2026-10-01';

Testing Incremental Loads (Job Bookmarks)

• Run the job, add new files, run again: only new files should be processed.
• Re-run without new data: output should not change.
• Reset the bookmark in a test environment and confirm a full reprocess gives the same result.

AWS Glue ETL Testing Interview Questions

1. How do you test an AWS Glue job?

Unit test the PySpark logic locally, run the job on sample data, validate output with Athena or Redshift SQL, check Data Catalog schema and job logs, and test incremental runs.

2. What are Glue job bookmarks?

State that Glue keeps between runs so a job processes only new data.

3. What is a Glue crawler?

A component that scans data in S3 or databases and creates or updates table definitions in the Data Catalog.

4. How do you validate data in S3?

Query it with Athena using the Data Catalog, comparing counts, totals and keys between zones.

5. What is Glue Data Quality?

A feature for defining data quality rules in DQDL and evaluating them on tables or within jobs.

Data testing skills: SQL for testers · data migration testing · Power BI testing · big data testing · data quality checks · ETL tools list · ADF & Databricks testing · database testing interview questions · Python for testers · data reconciliation · SQL joins · test data management · ETL defects examples · Snowflake ETL testing · data lake testing

More guides for Indian students: ETL full form · ETL testing with example · ETL testing jobs in India · ETL testing course in India · ETL training in Hyderabad, Bangalore & Pune · best ETL testing course on Udemy · ETL testing roadmap · ETL tester vs ETL developer · interview questions for freshers · free ETL testing mock test · SQL for testers · QA to data engineer · ETL testing kya hai (Hindi) · ETL testing projects for practice · ETL testing notes · work from home ETL jobs

Frequently Asked Questions

How do you do ETL testing in AWS?

Unit test Glue PySpark logic, run jobs on sample data, validate S3 and Redshift outputs with SQL in Athena or Redshift, check schemas in the Glue Data Catalog and test incremental runs with job bookmarks.

Can AWS Glue jobs be tested locally?

Yes. AWS provides Glue libraries and Docker images that let you run Glue scripts and pytest suites on your machine.

What is AWS Glue Data Quality?

A Glue feature that evaluates declarative data quality rules written in DQDL, such as completeness, uniqueness and allowed values.

Which SQL tool is used to test data in S3?

Amazon Athena, which runs SQL on S3 data registered in the Glue Data Catalog.

Do ETL testers need AWS skills?

For cloud projects, basic AWS knowledge (S3, Glue, Athena, Redshift) is increasingly expected on top of SQL and ETL testing fundamentals.

AN

Written by

Asim Noaman Lodhi

Certified Google Partner · QA Consultant · 12+ Years IT

QA consultant specializing in ETL testing and data quality. Trained 913+ students to transition into data testing roles through hands-on, real-world instruction.

4.5 Rating 913+ Students Google Partner 79 Lectures