The AWS Glue Data Catalog is a fully managed, Apache Hive Metastore compatible, metadata repository. It serves as a unified view of all your data across multiple data lakes, databases, and Amazon S3 storage. It provides a centralized source for storing, accessing, and managing metadata, thus enabling effective data exploration and discovery.
Setting Up AWS Glue Data Catalog
Follow these steps to create a catalog:
- Login to the AWS Management Console and navigate to the AWS Glue service.
- Choose ‘Add tables’ to add tables to your catalog. There are multiple ways to add tables:
- Add tables manually: You can choose this to define your data schema manually.
- Use a crawler: A crawler connects to your data source and infers schemas using built-in classifiers.
- If you’re adding tables manually, fill in the necessary information regarding your table, like database, table name, and schema.
- If you’re using a crawler, follow the wizard to specify the data store, IAM roles, and schedule for crawling.
Adding Tables Manually
Beneath is a sample step-by-step procedure to add tables manually:
# Navigate to the AWS Glue console
# Choose 'Databases' in the left navigation pane.
# Choose a database or create a new one. Then, select 'Tables in the database'.
# Choose 'Add tables', then select 'Add table manually'.
Then fill in the information:
- Table Name
- Classification (optional): The data format
- Stored as Subdirectories: Select this if your data is stored in subdirectories
- Location: The location of your data in Amazon S3.
- Input Format Class, Output Format Class, Serialization Library.
You can then define the schema by adding columns, indicating their types, and adding an optional comment.
Using a Crawler
A crawler automatically extracts metadata and creates table definitions in the Data Catalog. Here are the steps to create a crawler:
# Navigate to the AWS Glue console
# Choose 'Crawlers' in the left navigation pane.
# Select 'Add Crawler'.
# Specify the Data Store.
# Specify an existing IAM role or create a new one for the crawler to access the data store.
# Define the crawler's runtime properties, like schedule, output, etc.
Once you have a crawler in place, it accesses configured data stores to extract metadata and then creates table definitions in the Data Catalog.
Creating AWS Glue Data Catalog Using AWS SDK
Though the AWS Management Console provides a simple, non-coding option, you can also create and manage your AWS Glue Data Catalog using the AWS Software Development Kit (SDK). Here’s a Python-based sample code snippet with Boto3, AWS SDK for Python:
import boto3
glue = boto3.client('glue')
response = glue.create_database(
DatabaseInput={
'Name': 'string',
'Description': 'string',
'LocationUri': 'string',
}
)
Conclusion
Creating an AWS Glue Data Catalog can drastically improve your data governance and management. It provides features such as data discovery, data lineage, and data cataloging, which are of enormous benefit for any data engineer. As you work through your AWS Certified Data Engineer – Associate (DEA-C01) exam preparation, understanding how to create and work with AWS Glue Data Catalog will be vital.
Practice Test
True or False: You must create a data catalog in order to use AWS Glue.
Answer: False.
Explanation: A data catalog is not necessary to use AWS Glue, but it can make dataset management and discovery easier.
Which AWS services can assist in creating a data catalog?
- a) AWS Glue
- b) AWS Athena
- c) S3
- d) All of the above
Answer: d) All of the above.
Explanation: All these AWS services can be utilized for creating or operating a data catalog.
True or False: You can only create a data catalog that includes data stored in AWS.
Answer: False.
Explanation: AWS data catalog can include metadata from diverse databases both in AWS and on-premise storage.
Which statement(s) are true about data catalog:
- a) It organizes data in a consistent format.
- b) It only documents structured data.
- c) It helps in building and maintaining data lakes.
- d) It only contains AWS datasets.
Answer: a) It organizes data in a consistent format and c) It helps in building and maintaining data lakes.
Explanation: A data catalog organizes data in a consistent format and it can document both structured and unstructured data. It is not only limited to AWS datasets.
True or False: You can use IAM policies to control access to the data catalog.
Answer: True.
Explanation: With Amazon’s Identity and Access Management (IAM) service, you can manage access to the AWS Glue Data Catalog.
Which AWS service allows you to run SQL queries on your data catalog?
- a) AWS Glue
- b) Amazon Athena
- c) Amazon Redshift
- d) AWS Lake Formation
Answer: b) Amazon Athena.
Explanation: Amazon Athena is a service that lets users analyze data in Amazon S3 using standard SQL, working directly with a data catalog.
True or False: AWS Glue Crawler can be employed to populate the AWS Glue Data Catalog.
Answer: True.
Explanation: AWS Glue crawler is used to connect to a source, extract metadata, and create table definitions in the AWS Glue Data Catalog.
Which of the following is not a function of a data catalog?
- a) Data classification
- b) Data organization
- c) Data encryption
- d) Data discovery
Answer: c) Data encryption.
Explanation: Data encryption is related to data security and not a function of a data catalog, which is more concerned with data organization and discovery.
Is a data catalog most useful for organizations with small datasets?
- a) True
- b) False
Answer: b) False.
Explanation: A data catalog is generally most beneficial for organizations with large, diverse datasets that require management and discovery.
True or False: In AWS Glue, a database can contain tables from different sources.
Answer: True.
Explanation: In AWS Glue, a database is a set of associated table definitions, organized into a logical group. These tables can be from different data sources.
Interview Questions
What is the primary function of a data catalog in AWS?
A data catalog in AWS functions as a central repository where metadata from data sources is stored. It enables users to discover, understand and manage data.
What AWS service should be used to create a data catalog?
AWS Glue is the service used for creating a data catalog. It creates a unified metadata repository across various data sources.
How does the AWS Glue data catalog handle Schema discovery?
AWS Glue data catalog automatically infers and suggests schemas based on the source data, whenever it crawls a data store.
Can the AWS Glue data catalog be shared across different AWS accounts?
Yes, the AWS Glue data catalog can be shared across multiple AWS accounts, enabling those accounts to provide a consistent view of the data.
Is it possible to search data in AWS Glue data catalog?
Yes, AWS Glue data catalog enables you to run queries and perform search operations on your data.
Which AWS service integrates with AWS Glue and can utilize the data catalog for running queries?
Amazon Athena is a service that seamlessly integrates with AWS Glue and it can use the data catalog as a central meta-data repository to run SQL queries.
What is the role of a ‘Crawler’ in AWS Glue and data catalog creation?
A Crawler is a program that connects to a data store, progresses through a prioritized list of classifiers to determine the schema for your data, and then creates metadata tables in the AWS Glue Data Catalog.
Why is a Data Catalog significant in Data Engineering?
Data Catalog helps in efficient data discovery, enables data governance, and improves data quality. It serves as a foundation for data-driven decision making.
How do you ensure security in an AWS Glue data catalog?
Security in an AWS Glue data catalog is ensured by AWS Identity and Access Management (IAM), which you can use to control access.
What happens when new data sources or newly partitioned data is added in a data store crawled by AWS Glue?
AWS Glue detects new data and newly partitioned data during a crawl. It populates the catalog with the new findings, keeping the metadata up to date.
What AWS service can you use to transform data using the AWS Glue Data Catalog?
You can use AWS Glue ETL (Extract, Transform, Load) jobs to transform the data using the AWS Glue Data Catalog.
How are database entities represented in AWS Glue Data Catalog?
Entities in a database are represented as metadata tables in AWS Glue Data Catalog.
How can you manage access to individual tables in the AWS Glue Data Catalog?
You can manage access to individual tables in the AWS Glue Data Catalog using AWS Identity and Access Management (IAM) policies.
Can the AWS Glue Data Catalog be used as a metadata store?
Yes, the AWS Glue Data Catalog can be used as a metadata repository for services like Amazon Athena and Amazon Redshift Spectrum.
How do you remove a database from AWS Glue Data Catalog?
You can remove a database from the AWS Glue Data Catalog using the AWS Management Console, the AWS CLI, or the AWS Glue API.
extutorials.com