AWS Glue 5.1 Now Integrated with Amazon SageMaker Unified Studio, Enhancing Data Processing Capabilities
Amazon SageMaker Unified Studio now offers enhanced data processing capabilities through integration with AWS Glue 5.1. This update allows data engineers and data scientists to leverage the latest Apache Spark runtime and open table format libraries directly within the Unified Studio environment, streamlining data workflows and accelerating insights.
Key Benefits of AWS Glue 5.1 in Amazon SageMaker Unified Studio
- Updated Spark Runtime: Run jobs on Apache Spark 3.5.6, providing performance improvements and access to the newest features.
- Python 3.11 and Scala 2.12.18 Support: Utilize the latest versions of Python and Scala for data processing tasks.
- Enhanced Open Table Format Libraries: Benefit from updated libraries including Apache Iceberg 1.10.0, Apache Hudi 1.0.2 and Delta Lake 3.3.2.
- Versatile Job Support: The integration applies to Visual ETL jobs, notebook jobs, and code-based jobs, offering flexibility across various workflows.
Availability and Regional Support
AWS Glue 5.1 within Amazon SageMaker Unified Studio is currently available in the following regions:
- US East (N. Virginia)
- US East (Ohio)
- US West (Oregon)
- Europe (Ireland)
- Europe (Stockholm)
- Europe (Frankfurt)
- Europe (Spain)
- Asia Pacific (Hong Kong)
- Asia Pacific (Singapore)
- Asia Pacific (Sydney)
- Asia Pacific (Tokyo)
- Asia Pacific (Malaysia)
- Asia Pacific (Thailand)
- Asia Pacific (Mumbai)
- South America (Sao Paulo)
Getting Started
To utilize AWS Glue 5.1 in Amazon SageMaker Unified Studio, simply select “Glue 5.1” from the version dropdown menu within the job settings when creating data processing jobs. Further details can be found in the Amazon SageMaker Unified Studio documentation and the AWS Glue documentation.
Recent Expansions of AWS Glue 5.1
AWS Glue 5.1 has recently expanded its availability to eighteen additional regions, including Africa (Cape Town), Asia Pacific (Hyderabad, Jakarta, Melbourne, Osaka, Seoul, Taipei), Canada (Calgary, Central), Europe (London, Milan, Paris, Zurich), Israel (Tel Aviv), Mexico (Central), Middle East (Bahrain, UAE), and US West (N. California). AWS Glue 5.1 is now available in thirty-three AWS Regions.
Additional Updates in AWS Glue 5.1
- Apache Iceberg Support: Includes support for Apache Iceberg Materialized View and Apache Iceberg format version 3.0.
- Fine-Grained Access Control: Extends AWS Lake Formation fine-grained access control to write operations for Spark DataFrames and Spark SQL.
- Full-Table Access Control: Adds full-table access control in Apache Spark for Apache Hudi and Delta Lake tables.
These updates enhance performance, security, and governance capabilities for data lakes.