Amazon Practice Questions, Discussions & Exam Topics by our Authors
A data engineer is processing and analyzing multiple terabytes of raw data that is in Amazon S3. The data engineer needs to clean and prepare the data. Then the data engineer needs to load the data into Amazon Redshift for analytics.The data engineer needs a solution that will give data analysts the ability to perform complex queries. The solution must eliminate the need to perfo...
To meet the requirements of cleaning and preparing large amounts of raw data and loading it into Amazon Redshift for analytics with minimal operational overhead, let’s evaluate the options:
Option A: Amazon EMR, AWS Step Functions, and Amazon QuickSight
- Amazon EMR: Amazon EMR is a powerful tool for processing large datasets using Apache Hadoop, Spark, or other frameworks. While it’s highly scalable, it requires setting up and managing the cluster, which adds operational complexity.
- AWS Step Functions: While Step Functions is great for orchestrating workflows, it doesn’t directly simplify the process of data preparation or transformation. It would still need an integration with tools like AWS Lambda or Amazon EMR, which leads to increased complexity.
- Amazon QuickSight: QuickSight is a BI tool, but it is not suitable for performing complex queries directly on raw or transformed data. It is better suited for visualization and dashboarding, not for querying large datasets in a Redshift data warehouse.
This option is rejected because it requires managing an EMR cluster and doesn’t provide a streamlined, low-maintenance ETL process.
Option B: AWS Glue DataBrew, AWS Glue, and Amazon Redshift
- AWS Glue DataBrew: AWS Glue DataBrew is a powerful no-code data preparation tool that enables data engineers to clean and transform data without writing code. It integrates well with S3 and is easy to use for data preparation.
- AWS Glue: AWS Glue can be used to load the prepared data into Amazon Redshift. It is a fully managed ETL service, meaning minimal infrastructure management and scalability are built-in. It is designed for easy integration with Redshift.
- Amazon Redshift: Redshift is a fully managed data warehouse, which is ideal for running complex queries on large datasets, making it a good fit for analytics.
This option is ideal because it eliminates the need to manage infrastructure, simplifies the ETL process, and makes it easy to run complex queries on prepared data. It also leverages fully managed services, reducing operational overhead.
Option C: AWS Lambda, Amazon Kinesis Data Firehose, and Amazon Athena
- AWS Lambda: Lambda can be used for small, serverless...
Author: Amira99 · Last updated Jun 25, 2026
A company uses an AWS Lambda function to transfer files from a legacy SFTP environment to Amazon S3 buckets. The Lambda function is VPC enabled to ensure that all communications between the Lambda function and other AVS services that are in the same VPC environment will occur over a secure network.The Lambda function is able to connect to the SFTP environment successfully. However, when the Lambda function attempts to upload files to the S3 ...
Let's evaluate the options based on the requirement to resolve the timeout issue for the Lambda function when uploading files to Amazon S3. The Lambda function is VPC-enabled and is encountering timeout errors while trying to upload to S3, indicating a networking issue between the Lambda function and S3.
Option A: Create a NAT gateway in the public subnet of the VPC
- NAT Gateway: A NAT gateway allows instances in a private subnet to connect to the internet. While a NAT gateway can help route traffic to the internet, it introduces a cost, as it is a managed service that charges for usage (both for data processing and traffic).
- Relevance: The Lambda function is trying to access Amazon S3, which is a public AWS service. Using a NAT gateway for this scenario adds unnecessary complexity and cost because it is designed for outbound internet access for private subnet resources, not specifically for AWS services like S3.
This option is rejected because it’s not the most cost-effective solution and introduces unnecessary complexity and charges for the NAT gateway.
Option B: Create a VPC gateway endpoint for Amazon S3
- VPC Gateway Endpoint: A VPC gateway endpoint is a secure and cost-effective way to connect a VPC directly to Amazon S3 without the need for a public internet route. It allows resources in the VPC to access S3 over the AWS backbone network rather than the public internet.
- Relevance: The Lambda function needs to upload files to S3, and a VPC gateway endpoint ensures that traffic to and from S3 remains within the AWS network, which is more secure and avoids timeout issues that could arise from routing through the internet. This is the most cost-effective option because it doesn't involve any data transfer charges outside of the AWS infrastruct...
Author: FrozenWolf2022 · Last updated Jun 25, 2026
A company reads data from customer databases that run on Amazon RDS. The databases contain many inconsistent fields. For example, a customer record field that iPnamed place_id in one database is named location_id in another database. The company needs to link customer records across different databas...
Let's evaluate the options to determine which solution provides the least operational overhead while meeting the requirements for linking customer records across different Amazon RDS databases, especially when fields do not match.
Option A: Create a provisioned Amazon EMR cluster to process and analyze data in the databases. Connect to the Apache Zeppelin notebook. Use the FindMatches transform to find duplicate records in the data.
- Amazon EMR: EMR is a powerful solution for processing large datasets, and it can use tools like Apache Spark and Hadoop. However, managing an EMR cluster requires a significant amount of operational overhead, including cluster provisioning, management, and scaling.
- Apache Zeppelin: While Zeppelin can be used for interactive data processing, this also requires setup and management.
- FindMatches: The FindMatches transform is a good tool for identifying duplicates, but combining it with EMR adds complexity that may not be necessary for this use case.
This option is rejected because it involves too much infrastructure management (EMR cluster) for relatively straightforward data transformation tasks, making it less ideal in terms of operational overhead.
Option B: Create an AWS Glue crawler to crawl the databases. Use the FindMatches transform to find duplicate records in the data. Evaluate and tune the transform by evaluating the performance and results.
- AWS Glue: AWS Glue is a fully managed ETL service that simplifies data preparation, transformation, and loading. A Glue crawler automatically discovers and catalogs data in the databases.
- FindMatches Transform: The FindMatches transform in AWS Glue provides a managed, serverless solution for deduplicating and matching records, making it highly suitable for this task. It abstracts much of the complexity involved in data matching and linking.
- Operational Overhead: AWS Glue is fully managed, meaning there’s no need to handle cluster provisioning, scaling, or infrastructure. It's a cost-effective and less operationally heavy solution for the task at hand.
This option is selected because it leverages AWS Glue’s fully managed service for discovering and transforming data with minimal operational overhead. The FindMatches transform is directly suited for iden...
Author: VioletCheetah55 · Last updated Jun 25, 2026
A finance company receives data from third-party data providers and stores the data as objects in an Amazon S3 bucket.The company ran an AWS Glue crawler on the objects to create a data catalog. The AWS Glue crawler created multiple tables. However, the company expected that the crawler would create only one table.The company needs ...
In order for the AWS Glue crawler to create only one table, the key factor is ensuring that the data schema and structure across all objects is consistent. Let’s analyze the options based on this requirement:
A) Ensure that the object format, compression type, and schema are the same for each object.
- Reasoning: The AWS Glue crawler uses metadata (schema, object format, and compression type) to determine how to catalog the data. If all objects have the same format, compression type, and schema, the crawler can recognize that all the objects belong to the same dataset and create a single table.
- Why selected: This option ensures full consistency across all data objects, which is critical for the crawler to group all objects into one table. If these aspects are consistent, the crawler will be able to handle the objects as a single logical unit.
B) Ensure that the object format and schema are the same for each object. Do not enforce consistency for the compression type of each object.
- Reasoning: Ensuring that the object format and schema are the same is important for creating a single table, but inconsistency in the compression type may still cause the AWS Glue crawler to treat objects as distinct entities, potentially resulting in multiple tables.
- Why rejected: While it is true that object format and schema consistency are crucial, varying compression types could still cause the AWS Glue crawler to treat objects differently and create multiple tables. This solution could work in some scenarios but is less reliable than option A.
C) Ensure that the schema is the same for each object. Do not enforce consistency for the file format and compression type of each object.
- Reasoning: Schema consistency is indeed important, but if the file format or compression type varies, the crawler m...
Author: Carlos Garcia · Last updated Jun 25, 2026
An application consumes messages from an Amazon Simple Queue Service (Amazon SQS) queue. The application experiences occasional downtime. As a result of the downtime, messages within the queue expire and are deleted after 1 day. The message deletions c...
To minimize data loss in this scenario, we need to ensure that messages are preserved in the queue long enough to be processed, even if the application experiences downtime. Let's evaluate each option to determine the best solutions for minimizing data loss.
Option A: Increase the message retention period
- Message Retention Period: The retention period determines how long messages are kept in the SQS queue before they are automatically deleted. By default, the retention period is 4 days, but it can be extended up to 14 days. If messages are deleted after 1 day due to downtime, increasing the retention period would allow more time for the application to process messages before they are removed.
This option is selected because increasing the retention period ensures that messages are not deleted prematurely, providing more time for the application to consume them and reducing the risk of data loss during downtime.
Option B: Increase the visibility timeout
- Visibility Timeout: The visibility timeout controls how long a message remains invisible to other consumers after it has been received by a consumer. Increasing the visibility timeout prevents other consumers from attempting to process the message while it is being worked on. However, it does not directly address the issue of messages being deleted due to expiration. If the application is down for an extended period, increasing the visibility timeout does not prevent the message from expiring in the queue.
This option is rejected because it does not impact the message retention period or prevent messages from expiring if the application is not processing them in time.
Option C: Attach a dead-letter queue (DLQ) to the SQS queue
- Dead-Letter Queue (DLQ): A DLQ is used to capture messages that cannot be successfully processed after multiple attempts. If a message cannot be processed (e.g., due to application downtime), it is moved to the DLQ. While this helps handle undeliverable messages, it does not prevent messages from being deleted due to expiration if they a...
Author: Ella · Last updated Jun 25, 2026
A company is creating near real-time dashboards to visualize time series data. The company ingests data into Amazon Managed Streaming for Apache Kafka (Amazon MSK). A customized data pipeline consumes the data. The pipeline then writes data to Amazon Keyspaces (for Apache Cassandra), Amazon OpenSearch Service...
To determine the best solution, we must focus on achieving the lowest latency for visualizing the data in real time. Let's analyze each option:
A) Create OpenSearch Dashboards by using the data from OpenSearch Service.
- Reasoning: This solution involves using OpenSearch Service, which is designed for search and real-time analytics on large datasets. Since the data is already written to OpenSearch Service, OpenSearch Dashboards can immediately visualize the data with minimal latency.
- Key factors: OpenSearch Service is well-suited for time-series data and near real-time analytics. The latency between ingestion and visualization will be minimal because it leverages OpenSearch, which is optimized for search and analytics workloads.
- Why this is good: Real-time visualization is directly supported with OpenSearch Dashboards. The data is already in OpenSearch, so there is no need for additional steps like querying or cataloging.
- Why other options are rejected:
- Option B and C involve querying Amazon S3, which introduces potential latency compared to querying a service like OpenSearch that is optimized for low-latency search queries.
- Option D involves using S3 Select, which can introduce overhead due to the need for cataloging and querying from S3.
B) Use Amazon Athena with an Apache Hive metastore to query the Avro objects in Amazon S3. Use Amazon Managed Grafana to connect to Athena and to create the dashboards.
- Reasoning: Athena provides a serverless query service for querying data directly from Amazon S3. However, Athena is typically not the best option for real-time dashboards due to the overhead involved in querying large datasets stored in S3. Additionally, while Amazon Managed Grafana is a powerful tool for visualizations, the latency in que...
Author: Suresh · Last updated Jun 25, 2026
A company stores petabytes of data in thousands of Amazon S3 buckets in the S3 Standard storage class. The data supports analytics workloads that have unpredictable and variable data access patterns.The company does not access some data for months. However, the company must be able to retrieve all data within milli...
To determine the best solution, we need to focus on optimizing storage costs while ensuring the data can be retrieved within milliseconds. Let's evaluate each option:
A) Use S3 Storage Lens standard metrics to determine when to move objects to more cost-optimized storage classes. Create S3 Lifecycle policies for the S3 buckets to move objects to cost-optimized storage classes. Continue to refine the S3 Lifecycle policies in the future to optimize storage costs.
- Reasoning: S3 Storage Lens provides detailed insights into storage usage and access patterns, but this option requires manual adjustments of Lifecycle policies over time, which can introduce operational overhead. Refining these policies can be cumbersome, especially at the scale of petabytes of data.
- Why this is rejected: While this approach allows for optimization over time, it involves continuous management and adjustments, making it not ideal for minimizing operational overhead. The data retrieval within milliseconds requirement is not specifically addressed by this option.
B) Use S3 Storage Lens activity metrics to identify S3 buckets that the company accesses infrequently. Configure S3 Lifecycle rules to move objects from S3 Standard to the S3 Standard-Infrequent Access (S3 Standard-IA) and S3 Glacier storage classes based on the age of the data.
- Reasoning: S3 Standard-IA is intended for infrequent access but can still offer relatively fast access times. S3 Glacier provides more cost optimization but with retrieval times that can range from minutes to hours, which may not meet the "milliseconds" retrieval requirement for certain use cases.
- Why this is rejected: S3 Glacier introduces retrieval times of minutes or hours, which wou...
Author: Zara · Last updated Jun 25, 2026
A media company wants to use Amazon OpenSearch Service to analyze rea-time data about popular musical artists and songs. The company expects to ingest millions of new data events every day. The new data events will arrive through an Amazon Kinesis data stream. The company must transform the data and then ingest ...
To determine the best solution, we need to consider minimizing operational overhead, while ensuring the data is transformed and ingested into OpenSearch Service efficiently. Let's evaluate each option:
A) Use Amazon Kinesis Data Firehose and an AWS Lambda function to transform the data and deliver the transformed data to OpenSearch Service.
- Reasoning: Kinesis Data Firehose can automatically stream data to destinations like OpenSearch Service without much operational management. By integrating an AWS Lambda function, you can apply transformations to the data before it is ingested into OpenSearch. This method allows for a serverless architecture, reducing operational overhead and managing scaling automatically.
- Why this is selected: This option has low operational overhead because Kinesis Data Firehose handles the delivery, scaling, and retry logic automatically. Lambda is highly scalable fo...
Author: Sofia · Last updated Jun 25, 2026
A company stores customer data tables that include customer addresses in an AWS Lake Formation data lake. To comply with new regulations, the company must ensure that users cannot access data for customers who are in Canada.The company needs a solution that will prevent user acc...
To meet the requirement of preventing access to data for customers in Canada with the least operational effort, we need to consider the best way to restrict access while ensuring compliance. Let's evaluate each option:
A) Set a row-level filter to prevent user access to a row where the country is Canada.
- Reasoning: A row-level filter allows for restricting access based on specific row values—in this case, where the country is "Canada." This is a common approach to enforce fine-grained access control on data.
- Why this is selected: Row-level filtering directly addresses the requirement to prevent access to rows based on the country. Using AWS Lake Formation, the company can define and enforce these filters with minimal operational effort. It allows granular control over which rows a user can access based on the country, and it integrates seamlessly with Lake Formation's existing data access controls.
- Why other options are rejected:
- Opti...
Author: StarlightBear · Last updated Jun 25, 2026
A company has implemented a lake house architecture in Amazon Redshift. The company needs to give users the ability to authenticate into Redshift query editor by using a third-party identity provider (IdP).A data engineer must set up the ...
To enable users to authenticate into Amazon Redshift Query Editor using a third-party identity provider (IdP), the first step involves configuring Amazon Redshift to recognize and trust the external IdP for user authentication. Let's evaluate each option based on this requirement.
A) Register the third-party IdP as an identity provider in the configuration settings of the Redshift cluster.
- Reasoning: This option suggests registering the third-party IdP directly in the configuration settings of the Redshift cluster, which is the correct approach. Amazon Redshift supports integration with external identity providers (IdPs) such as SAML-based IdPs for user authentication. By registering the third-party IdP, the data engineer can allow users to authenticate using the IdP credentials in Redshift Query Editor.
- Why selected: This is the correct first step because Amazon Redshift allows direct integration with IdPs through its authentication settings. Once the IdP is registered, users can authenticate with their IdP credentials.
B) Register the third-party IdP as an identity provider from within Amazon Redshift.
- Reasoning: This option is almost similar to option A, but it specifies the action of registering from within Amazon Redshift, which may be interpreted as registering within the Redshift console or using APIs. However, in practice, registering the IdP is typically done via the cluster configuration, which is what option A suggests.
- Why rejected: While this is a valid step, it lacks the clear reference to configuration settings, making it less clear than option A, which more explicitly describes th...
Author: SolarFalcon11 · Last updated Jun 25, 2026
A company currently uses a provisioned Amazon EMR cluster that includes general purpose Amazon EC2 instances. The EMR cluster uses EMR managed scaling between one to five task nodes for the company=E2=80=99s long-running Apache Spark extract, transform, and load (ETL) job. The company runs the ETL job every day.When the company runs the ETL job, the EMR cluster quickly scales up to five nodes. The EMR cluster often reaches maximum CPU usage, but the memory...
The company is looking to optimize the EMR cluster configuration in order to reduce the costs associated with running the daily ETL job. We need to focus on cost-effective ways to handle the workload based on the observed usage patterns (high CPU usage, low memory usage). Let’s evaluate the options based on these factors.
A) Increase the maximum number of task nodes for EMR managed scaling to 10
- Why it's not ideal: Increasing the maximum number of task nodes would result in higher costs since the cluster could scale to 10 nodes. This would be an unnecessary cost because the CPU is already maxed out while memory usage remains low. The real problem is the cluster's CPU usage, not the lack of task nodes. More nodes will not solve the issue of CPU saturation and would just add extra cost.
- Conclusion: Not a cost-effective solution.
B) Change the task node type from general purpose EC2 instances to memory optimized EC2 instances
- Why it's not ideal: The memory usage remains low (under 30%), so switching to memory-optimized instances would be overkill. Memory-optimized instances typically cost more, and since the workload doesn't require additional memory, this change would be wasteful in terms of both resources and cost.
- Conclusion: Not a cost-effective solution.
C) Switch the task node type from general purpose EC2 instances to compute-optimized EC2 instances
- Why it’s the best choice: The primary issue is that the CPU usage is consistently h...
Author: Zara · Last updated Jun 25, 2026
A company uploads .csv files to an Amazon S3 bucket. The company=E2=80=99s data platform team has set up an AWS Glue crawler to perform data discovery and to create the tables and schemas.An AWS Glue job writes processed data from the tables to an Amazon Redshift database. The AWS Glue job handles column mapping and creates the Amazon Redshift tables in the Redshift database appropriately.If the company reruns the AWS Glue job for any reason, du...
The company wants to prevent the insertion of duplicate records into Amazon Redshift when rerunning the AWS Glue job. Each option involves a different approach to solving this issue. Let’s evaluate each solution and consider the best approach based on the goal of avoiding duplicates while efficiently updating the Redshift tables.
A) Modify the AWS Glue job to copy the rows into a staging Redshift table. Add SQL commands to update the existing rows with new values from the staging Redshift table.
- Why it works: This approach involves copying data to a staging table in Redshift first and then performing an update (or an upsert) on the existing records in the target table. This avoids duplicates by ensuring only new or modified records are inserted or updated in the destination table. Using a staging table is a common practice in ETL workflows as it helps in isolating and controlling the data before it is merged into the main table.
- Why it’s a good choice: This is an effective and reliable way to prevent duplicates and ensure data integrity. It allows for full control over how records are updated in Redshift (via SQL commands such as `UPDATE` or `MERGE`), which addresses the problem of data duplication.
- Conclusion: This is a solid approach to ensure records are updated without duplication, and it provides flexibility for complex transformations or deduplication logic.
B) Modify the AWS Glue job to load the previously inserted data into a MySQL database. Perform an upsert operation in the MySQL database. Copy the results to the Amazon Redshift tables.
- Why it’s not ideal: Introducing MySQL as an intermediary step adds unnecessary complexity to the workflow. The data already resides in Redshift, and transferring it through MySQL introduces an extra layer of management, which is not needed and increases both the operational overhead and the risk of errors.
- Why it’s rejected: The solution complicates the process unnecessarily, as Redshift itself is capable of handling upserts directly. MySQL doesn’t offer any significant advantage in this case, and the added step would likely increase the overall cost and mainte...
Author: Maya2022 · Last updated Jun 25, 2026
A company is using Amazon Redshift to build a data warehouse solution. The company is loading hundreds of files into a fact table that is in a Redshift cluster.The company wants the data warehouse solution to achieve the greatest possible throughput. The solution must use cl...
The goal is to achieve the greatest possible throughput and use cluster resources optimally when loading data into a Redshift fact table. Let's evaluate the options based on how they impact performance and resource utilization.
A) Use multiple COPY commands to load the data into the Redshift cluster.
- Why it’s not ideal: Although using multiple `COPY` commands can parallelize the data load, the overhead of multiple commands can degrade performance. Each `COPY` command is a separate transaction, and splitting the workload into multiple commands may not utilize the Redshift resources as efficiently as one well-optimized `COPY` command.
- Why it’s rejected: While this approach may offer parallelism, it is not the most optimal approach for throughput, as the overhead from multiple commands will reduce the overall performance.
B) Use S3DistCp to load multiple files into Hadoop Distributed File System (HDFS). Use an HDFS connector to ingest the data into the Redshift cluster.
- Why it’s not ideal: This option introduces additional complexity and infrastructure by moving data through HDFS before loading it into Redshift. Redshift has native tools (like `COPY`) optimized for loading data directly from Amazon S3. Using HDFS as an intermediary step only adds extra processing overhead and doesn’t provide any significant throughput benefit.
- Why it’s rejected: Adding HDFS introduces unnecessary steps in the data loading process and complicates the solution without providing an optimal increase in throughput.
C) Use a number of INSERT statements equal to the number of Redshift cluste...
Author: Isabella · Last updated Jun 25, 2026
A company ingests data from multiple data sources and stores the data in an Amazon S3 bucket. An AWS Glue extract, transform, and load (ETL) job transforms the data and writes the transformed data to an Amazon S3 based data lake. The company uses Amazon Athena to query the data that is in the data lake.The company n...
The company needs to identify matching records in a data lake where records do not have a common unique identifier. Let's evaluate each option based on how well it meets the requirement of finding matching records without a unique identifier.
A) Use Amazon Macie pattern matching as part of the ETL job.
- Why it’s not ideal: Amazon Macie is primarily designed for data privacy and security, specifically to identify and protect sensitive data, such as personally identifiable information (PII). While Macie can be useful for pattern matching in the context of data privacy, it is not suited for matching general records without a unique identifier. Macie does not have the capabilities to match records across datasets that lack a common identifier in the way that is needed here.
- Why it’s rejected: This approach does not solve the problem of matching records without a unique identifier. It’s more focused on security use cases than record matching.
B) Train and use the AWS Glue PySpark Filter class in the ETL job.
- Why it’s not ideal: The `Filter` class in PySpark is used for filtering data based on conditions, but it does not inherently provide a way to match records that don’t have a common unique identifier. Matching records without unique identifiers typically requires more sophisticated logic, such as fuzzy matching or similarity-based matching, which the `Filter` class cannot provide directly.
- Why it’s rejected: While `Filter` can help with filtering records based on conditions, it is not designed for the complex task of matching records that lack a unique identifier.
C) Partition tables and use the ETL job to partition the data on a unique identifier.
- Why it’s n...
Author: Amira99 · Last updated Jun 25, 2026
A data engineer is using an AWS Glue crawler to catalog data that is in an Amazon S3 bucket. The S3 bucket contains both .csv and json files. The data engineer configured the crawler to exclude the .json files from the catalog.When the data engineer runs queries in Amazon Athena, the queries also process the excluded .json files. The data engineer wants to resolve this issue. The data engineer ne...
The data engineer wants to ensure that the excluded `.json` files are not processed by Athena queries, while maintaining access to the `.csv` files. Let’s evaluate each option and see which provides the shortest query times and solves the issue efficiently.
A) Adjust the AWS Glue crawler settings to ensure that the AWS Glue crawler also excludes .json files.
- Why it's not ideal: The data engineer has already configured the crawler to exclude `.json` files, but the issue is that the `.json` files are still being processed by Athena queries. The AWS Glue crawler settings impact the cataloging process, but Athena queries may still access files directly from S3. This approach would only affect the cataloging, but it doesn't resolve the issue with Athena queries still processing the `.json` files.
- Why it's rejected: While adjusting the crawler may improve cataloging, it does not directly address the query processing issue in Athena.
B) Use the Athena console to ensure the Athena queries also exclude the .json files.
- Why it's not ideal: Athena queries work on data in S3 based on the catalog created by AWS Glue, and the exclusion of files must happen at the data source level, not just at the query level. Athena itself does not have a built-in feature to "exclude" certain file types during query execution once the data is cataloged, especially if the file types are part of the same table. This approach would not provide an optimal solution for preventing `.json` files from being processed during queries.
- Why it's rejected: Athena queries would still scan the entire table and might attempt to process the `.json` files unless they are excluded from the catalog in a more definitive manner.
C) Relocate the .json files to a different path within the S3 bucket.
- Why it works: By relocating the `.json` files to a different path in the S3 bucket, the AWS Glue crawler can be configured to exclude that path, preventing Athena queries from ...
Author: Ahmed · Last updated Jun 25, 2026
A data engineer set up an AWS Lambda function to read an object that is stored in an Amazon S3 bucket. The object is encrypted by an AWS KMS key.The data engineer configured the Lambda function=E2=80=99s execution role to access the S3 bucket. However, the Lambda fu...
Let's break down the options and their reasoning.
A) The data engineer misconfigured the permissions of the S3 bucket. The Lambda function could not access the object.
- This is unlikely because the problem is specifically related to decryption, not access to the S3 bucket itself. The Lambda function's execution role has already been configured to access the S3 bucket. Since it’s specified that the issue occurs when trying to read the encrypted object, the problem is not about basic permissions to access the object in S3.
- Rejected: Incorrect because the issue is not with accessing the object in the S3 bucket itself, but with decryption.
B) The Lambda function is using an outdated SDK version, which caused the read failure.
- This is a possible cause, but it’s unlikely. Lambda functions typically use updated versions of SDKs, and unless explicitly specified, the Lambda function would use the latest SDK version. It’s more probable that the issue stems from missing permissions to access the KMS key.
- Rejected: Outdated SDKs are rarely the cause of decryption issues, especially when AWS SDKs handle encryption and decryption seamlessly.
...
Author: Aria · Last updated Jun 25, 2026
A data engineer has implemented data quality rules in 1,000 AWS Glue Data Catalog tables. Because of a recent change in business requirements, the data engineer must edit the data quality rules.Ho...
Let's break down the options and their reasoning.
A) Create a pipeline in AWS Glue ETL to edit the rules for each of the 1,000 Data Catalog tables. Use an AWS Lambda function to call the corresponding AWS Glue job for each Data Catalog table.
- This option involves creating a pipeline and triggering AWS Glue ETL jobs for each Data Catalog table, which is complex and requires a significant amount of operational overhead. For 1,000 tables, the number of jobs and the complexity of managing them would be high, making this solution not optimal in terms of efficiency.
- Rejected: High operational overhead due to the need to manage multiple AWS Glue jobs for each table.
B) Create an AWS Lambda function that makes an API call to AWS Glue Data Quality to make the edits.
- This option directly addresses the need to edit data quality rules for all the tables by utilizing a Lambda function to call AWS Glue Data Quality APIs. This is efficient because AWS Lambda can process API calls in bulk or in a loop without needing to manage individual jobs. It also scales well for 1,000 tables, providing a low-overhead, serverless approach to make the required changes.
- Selected: This is t...
Author: StarlightBear · Last updated Jun 25, 2026
Two developers are working on separate application releases. The developers have created feature branches named Branch A and Branch B by using a GitHub repository=E2=80=99s master branch as the source.The developer for Branch A deployed code to the production system. The code for Branch B will merge into a master branch in the following week=E2=80=...
Let's break down the options and analyze the best command for the developer working on Branch B.
A) `git diff branchB master; git commit -m`
- This command would display the differences between Branch B and the master branch, but it does not handle synchronization or updating of Branch B with the latest changes from master. Additionally, it’s followed by `git commit -m`, which would suggest that you would be committing something to Branch B, but this doesn't seem necessary or relevant in this context.
- Rejected: This command is not helpful for synchronizing Branch B with master before creating a pull request.
B) `git pull master`
- `git pull master` fetches changes from the remote master branch and merges them into the current branch. However, this is not the most optimal approach because it can result in a merge commit, which could lead to a messy history. It’s better to use a rebase to make the history cleaner and linear, especially when preparing a pull request to master.
- Rejected: Merging master into Branch B is not ideal because ...
Author: Jack · Last updated Jun 25, 2026
A company stores employee data in Amazon Resdshift. A table names Employee uses columns named Region ID, Department ID, and Role ID as a compound sort key.Which queries will MOST incre...
To optimize queries that use a compound sort key in Amazon Redshift, the query should leverage the sort key columns efficiently. A compound sort key stores data in a sorted order based on the specified columns, which means queries that filter by the leftmost columns in the sort key will benefit the most in terms of performance.
Understanding Compound Sort Keys
A compound sort key uses multiple columns to determine the order of the data, and the columns are processed from left to right. In this case, the sort key is defined on the following columns, in order:
1. Region ID
2. Department ID
3. Role ID
A) `Select from Employee where Region ID = 'North America';`
- Reasoning: This query uses the Region ID as the filter, which is the leftmost column in the compound sort key. Since the data is stored sorted by Region ID, this query will take full advantage of the sort key and filter quickly.
- Why selected: This is an optimal query for the compound sort key because it directly filters based on the first column of the sort key, which will help Amazon Redshift quickly locate the relevant rows.
B) `Select from Employee where Region ID = 'North America' and Department ID = 20;`
- Reasoning: This query filters by both Region ID and Department ID. Since Region ID is the first column in the sort key, and Department ID is the second, this query will also benefit from the compound sort key. Redshift can efficiently scan the data based on both columns' sorted order.
- Why selected: This query benefits from both the first and second columns of the sort key, improving performance as Redshift can filter using the sorted regions and then narrow down further by department.
C) `Select from Employee where Department ID = 20 and Region ID = 'North America';`
- Reasoning: This query filters by Department ID first and then by Region ID. Since Department ID...
Author: William · Last updated Jun 25, 2026
A company receives test results from testing facilities that are located around the world. The company stores the test results in millions of 1 KB JSON files in an Amazon S3 bucket. A data engineer needs to process the files, convert them into Apache Parquet format, and load them into Amazon Redshift tables. The data engineer uses AWS Glue to process the files, AWS Step Functions to orchestrate the processes, and Amazon EventBridge to schedule jobs.The company r...
Let's analyze each option to determine the best solution for reducing data processing time.
A) Use AWS Lambda to group the raw input files into larger files. Write the larger files back to Amazon S3. Use AWS Glue to process the files. Load the files into the Amazon Redshift tables.
- Reasoning: AWS Lambda could be used to group smaller JSON files into larger files (e.g., larger JSON files or compressed formats like gzip). Larger files are more efficient to process because AWS Glue would read fewer files, and the cost of starting Glue jobs would be lower. However, Lambda's execution time is limited, so it might not be ideal for large-scale file processing.
- Rejected: Although this reduces the number of files to process, Lambda's execution limits and the potential overhead of re-writing the files back to S3 could make this solution inefficient for large datasets.
B) Use the AWS Glue dynamic frame file-grouping option to ingest the raw input files. Process the files. Load the files into the Amazon Redshift tables.
- Reasoning: The AWS Glue dynamic frame file-grouping option allows Glue to group files automatically during the ingestion process, which reduces the number of files being processed and speeds up the overall job. This solution takes advantage of Glue’s built-in file management capabilities to improve data processing time by reducing file overhead.
- Selected: This is the most efficient approach because it automates file grouping without additional services, leveraging Glue’s native capabilities to optimize the process. This minimizes the overhead involved with processing millions of small files and ensures that files are grouped dynamically for optimal performance.
C) Use the...
Author: Amelia · Last updated Jun 25, 2026
A data engineer uses Amazon Managed Workflows for Apache Airflow (Amazon MWAA) to run data pipelines in an AWS account.A workflow recently failed to run. The data engineer needs to use Apache Airflow logs to diagnose the failure...
To diagnose the cause of the workflow failure in Amazon MWAA, the data engineer should focus on logs that provide insights into the execution and processing of tasks in the workflow. Let's go over the log types one by one:
A) YourEnvironmentName-WebServer Logs:
- Scenario: This log type is primarily useful for diagnosing issues related to the web server, such as issues with the Airflow UI, interactions with the user interface, or API calls.
- Reason for rejection: It will not help in understanding why a specific task or workflow failed, as these logs mainly provide details about user-facing operations like web requests and interface interactions.
B) YourEnvironmentName-Scheduler Logs:
- Scenario: These logs are focused on the scheduler component of Apache Airflow. They provide information on scheduling the workflows, such as when DAGs are triggered, and whether they start or finish properly.
- Reason for rejection: While they are helpful for understanding if a DAG was triggered properly, these logs don’t provide specific information about the individual task executions or failures within a DAG, which is key for tro...
Author: Layla · Last updated Jun 25, 2026
A finance company uses Amazon Redshift as a data warehouse. The company stores the data in a shared Amazon S3 bucket. The company uses Amazon Redshift Spectrum to access the data that is stored in the S3 bucket. The data comes from certified third-party data providers. Each third-party data provider has unique connection details.To comply with regulations, the company must ensure that...
To ensure that none of the data stored in the Amazon S3 bucket is accessible from outside the company's AWS environment, the company needs to ensure private network access to both Amazon Redshift and the S3 data, and possibly restrict external access to the data from third-party providers.
Key requirements:
- Ensure data is only accessible within the company's AWS environment.
- Protect access from third-party data providers.
- Enable access to data using Amazon Redshift Spectrum for querying the data stored in S3.
Option Analysis:
A) Replace the existing Redshift cluster with a new Redshift cluster that is in a private subnet. Use an interface VPC endpoint to connect to the Redshift cluster. Use a NAT gateway to give Redshift access to the S3 bucket.
- Reasoning: This option ensures that Amazon Redshift operates within a private subnet and can access the S3 bucket securely using a VPC endpoint. This approach avoids the need for the Redshift cluster to access the internet through a public IP, which ensures that no data is exposed outside the AWS environment. A NAT gateway is necessary for allowing outbound access from the private subnet to S3, ensuring connectivity without external exposure.
- Why selected: By using a private subnet for Redshift, along with a VPC endpoint and NAT gateway, this solution ensures all access to data is confined to the AWS environment, preventing external access while allowing the Redshift cluster to interact with the S3 data through Redshift Spectrum.
B) Create an AWS CloudHSM hardware security module (HSM) for each data provider. Encrypt each data provider's data by using the corresponding HSM for each data provider.
- Reasoning: AWS CloudHSM is used for secure key management and hardware-based encryption. While it ensures strong encryption, it does not directly address the need to restrict access to data stored in S3 or prevent external access to the AWS environment.
- Why rejected: CloudHSM helps with encryption but does not enforce network or access control restrictions. It also adds unnecessary complexity without solving the core requirement of preventing external access to the data in S3 or Redshift. Encryption alone does not provide access control.
C) Turn on enhanced VPC routing for the Amazon Redshift cluster. Set up an AWS Direct Connect connection and configure a connection between each data provider and the compan...
Author: Isabella1 · Last updated Jun 25, 2026
Files from multiple data sources arrive in an Amazon S3 bucket on a regular basis. A data engineer wants to ingest new files into Amazon Redshift in near real time when the ne...
To meet the requirement of ingesting new files into Amazon Redshift in near real time when the new files arrive in the S3 bucket, the solution must automatically trigger the ingestion process as soon as the files are available. Let’s analyze the options:
A) Use the query editor v2 to schedule a COPY command to load new files into Amazon Redshift.
- Rejected reason: This solution involves scheduling a command to load files into Amazon Redshift using the COPY command. While this approach can load data into Redshift, it does not offer near real-time ingestion. It relies on a scheduled process, which means there will be a delay between when new files arrive and when they are ingested. This doesn't meet the requirement of near real-time ingestion of files as soon as they arrive.
B) Use the zero-ETL integration between Amazon Aurora and Amazon Redshift to load new files into Amazon Redshift.
- Rejected reason: The zero-ETL integration between Amazon Aurora and Amazon Redshift is designed for continuous data transfer between Aurora and Redshift. However, this solution does not apply to data that is stored in S3. This approach is specific to Amazon Aurora and would not work with files stored in S3,...
Author: Daniel · Last updated Jun 25, 2026
A technology company currently uses Amazon Kinesis Data Streams to collect log data in real time. The company wants to use Amazon Redshift for downstream real-time queries and to enrich the log data.Wh...
To select the best solution for ingesting data from Amazon Kinesis Data Streams into Amazon Redshift with the least operational overhead, we need to focus on minimizing complexity and manual processes. Let's evaluate the options:
A) Set up an Amazon Kinesis Data Firehose delivery stream to send data to a Redshift provisioned cluster table.
- Reasoning: Amazon Kinesis Data Firehose is a fully managed service that can deliver streaming data to Amazon Redshift with minimal configuration and management. It automatically batches, buffers, and compresses the data as it is delivered to Redshift, making this a low-maintenance solution. Additionally, Firehose integrates seamlessly with Redshift, allowing real-time ingestion without requiring complex custom applications.
- Why selected: This solution has the least operational overhead as it is fully managed, requires minimal setup, and handles the data transformation and loading directly into Redshift with no need for additional infrastructure or manual batch processes.
B) Set up an Amazon Kinesis Data Firehose delivery stream to send data to Amazon S3. Configure a Redshift provisioned cluster to load data every minute.
- Reasoning: While this solution leverages Kinesis Data Firehose and S3 as an intermediary, it requires manual configuration to periodically load data from S3 into Redshift. The scheduled data load process introduces additional operational complexity because the company would need to handle frequent S3-to-Redshift loads, which could become cumbersome as the volume of data increases.
- Why rejected: While this option is technically valid, it introduces extra operational overhead because the data needs to be loaded from S3 into Redshift periodically, making it more complex than directly delivering data to Redshift.
...
Author: Liam · Last updated Jun 25, 2026
A company maintains a data warehouse in an on-premises Oracle database. The company wants to build a data lake on AWS. The company wants to load data warehouse tables into Amazon S3 and synchronize the tables with incremental data that arrives from the data warehouse every day.Each table has a column that contains monotonically increasing values. The size of each table is less than 50 GB. The data warehouse tables are refreshed every nigh...
Solution Evaluation:
The company has the requirement to synchronize data from the on-premises Oracle database to Amazon S3, specifically focusing on incremental data loading with minimal operational overhead. The solution must ensure daily updates while also keeping operational tasks simple.
Option A: Use an AWS Database Migration Service (AWS DMS) full load plus CDC job to load tables that contain monotonically increasing data columns from the on-premises data warehouse to Amazon S3. Use custom logic in AWS Glue to append the daily incremental data to a full-load copy that is in Amazon S3.
- Reasoning: AWS DMS can handle the full load and change data capture (CDC) processes, ensuring that the initial migration of data is done efficiently. After the initial load, AWS Glue would append incremental data based on the monotonically increasing column each day. This approach is a hybrid solution, combining DMS for the initial load and incremental data capture, and AWS Glue for custom logic to manage data updates.
- Why selected: This approach provides incremental data synchronization and handles the full-load data efficiently with minimal manual intervention. DMS can take care of CDC with minimal setup, and AWS Glue allows for automated processing of the incremental data. Although it requires some custom logic, it offers operational flexibility and scalability.
Option B: Use an AWS Glue Java Database Connectivity (JDBC) connection. Configure a job bookmark for a column that contains monotonically increasing values. Write custom logic to append the daily incremental data to a full-load copy that is in Amazon S3.
- Reasoning: AWS Glue can connect to the database via JDBC, and the job bookmark feature can track changes for incremental data. However, this solution requires custom logic to append the incremental data to the existing full-load copy in S3. The use of job bookmarks may also require careful configuration, and managing the custom logic for incremental updates might add complexity.
- ...
Author: Olivia · Last updated Jun 25, 2026
A company is building a data lake for a new analytics team. The company is using Amazon S3 for storage and Amazon Athena for query analysis. All data that is in Amazon S3 is in Apache Parquet format.The company is running a new Oracle database as a source system in the company=E2=80=99s data center. The company has 70 tables in the Oracle database. All the tables have primary keys. Data can occ...
Let's go through each option to evaluate which one meets the requirements with the least effort, considering key factors like effort, scalability, ease of integration, and future maintainability:
Option A: Create an Apache Sqoop job in Amazon EMR to read the data from the Oracle database. Configure the Sqoop job to write the data to Amazon S3 in Parquet format.
- Pros:
- Apache Sqoop is specifically designed to import data from relational databases like Oracle into Hadoop-based storage systems, so it works well for bulk data transfer.
- Amazon EMR can handle the workload efficiently.
- Sqoop supports writing to various formats, including Parquet.
- Cons:
- The solution requires setting up and maintaining an Amazon EMR cluster, which adds complexity and operational overhead.
- It may require custom scripting for incremental data updates, which could be cumbersome to manage.
- It doesn’t provide an out-of-the-box solution for handling changes in the source database (like updates or deletes).
This option would be more complex to manage and would require additional work to handle incremental changes (as the company needs to handle occasional changes to data).
Option B: Create an AWS Glue connection to the Oracle database. Create an AWS Glue bookmark job to ingest the data incrementally and to write the data to Amazon S3 in Parquet format.
- Pros:
- AWS Glue provides an integrated, serverless environment for ETL (Extract, Transform, Load) jobs, which reduces the management overhead.
- Glue has built-in support for Parquet format, and Glue bookmarks help to track changes, enabling incremental loads.
- Glue can integrate seamlessly with Amazon S3 and supports various data formats, including Parquet.
- It requires less setup and maintenance than setting up a custom solution like Apache Sqoop on EMR.
- Cons:
- While Glue simplifies the integration, it may require some initial setup to define the schema and configure the job, but once set up, it handles regular ETL tasks efficiently.
- Performance can be a concern for very large datasets, depending on the job configurations.
AWS Glue is a managed service that can handle the incremental data ingestion seamlessly, making this a strong contender for the least effort solution.
Option C: Create an AWS Database Migration Service (AWS DMS) task for ongoing replication. Set the Oracle database as the source. Set Amazon S3 as the target. Configure the task to write the data in Parquet f...
Author: MysticJaguar44 · Last updated Jun 25, 2026
A transportation company wants to track vehicle movements by capturing geolocation records. The records are 10 bytes in size. The company receives up to 10.000 records every second. Data transmission delays of a few minutes are acceptable because of unreliable network conditions.The transportation company wants to use Amazon Kinesis Data Streams to ingest the geolocation data. The company needs a reliable mechanism to send data t...
Let's analyze each of the options in relation to the requirements of the transportation company:
Option A: Kinesis Agent
- Pros:
- Kinesis Agent is a pre-built, open-source application that is easy to deploy. It's particularly suited for streamlining data ingestion from local files, logs, or other systems.
- It can automatically push data to Kinesis Data Streams or Kinesis Data Firehose.
- It handles retries and basic error handling in case of transmission delays or network issues.
- Cons:
- Kinesis Agent is generally more suited for file-based data sources, such as logs or data stored in files, rather than real-time streaming of small records (like geolocation data).
- It doesn’t provide the same level of throughput control and efficiency optimization as some of the other options.
- It's less configurable compared to the KPL when trying to maximize throughput efficiency.
When to use: Best for log or file-based data ingestion, but not ideal for maximizing throughput efficiency for high-frequency data like geolocation records.
Option B: Kinesis Producer Library (KPL)
- Pros:
- KPL is specifically designed for sending high-volume, high-throughput data to Kinesis Data Streams with minimal operational overhead.
- It automatically batches records to maximize throughput, thus optimizing the use of available shard capacity, which is critical for this use case (sending 10,000 records per second).
- It handles retries, error management, and backpressure effectively.
- It is ideal for real-time, low-latency streaming use cases.
- Cons:
- KPL requires custom application development, so there is more initial setup compared to some other options.
- You need to configure the KPL with appropriate buffer sizes and retry strategies, which adds some complexity.
When to use: The ideal option when you need to efficiently stream a high volume of data to Kinesis, especially for low-latency, real-time use cases. This option fits the requirement to maximize throughput efficiency and send geolocation data.
Option C: Amazon Kinesis Data Firehose
- Pros:
- Kinesis Data Firehose is a fully ...
Author: Jack · Last updated Jun 25, 2026
An investment company needs to manage and extract insights from a volume of semi-structured data that grows continuously.A data engineer needs to deduplicate the semi-structured data, remove records that are duplicates, and remove common mis...
Let's evaluate each of the options based on the investment company's needs: deduplication, handling semi-structured data, and removing common misspellings, all while minimizing operational overhead.
Option A: Use the FindMatches feature of AWS Glue to remove duplicate records.
- Pros:
- FindMatches is a feature within AWS Glue that can perform deduplication of records based on fuzzy matching, which is ideal for identifying and removing duplicates and misspellings.
- AWS Glue is a fully managed ETL service, meaning it abstracts away much of the operational overhead.
- It can handle semi-structured data (like JSON, Parquet) well, making it suitable for this scenario.
- Glue automatically scales to handle growing data volumes, which aligns with the company's requirement for managing continuously growing data.
- Cons:
- While it’s effective for deduplication, it requires configuring AWS Glue jobs, but this is minimal compared to manually handling deduplication logic.
When to use: This is the most suitable option as it directly addresses the deduplication requirement and handles both exact matches and fuzzy matching for misspelled records with minimal manual intervention.
Option B: Use non-Windows functions in Amazon Athena to remove duplicate records.
- Pros:
- Amazon Athena can query semi-structured data stored in S3 using standard SQL.
- It is serverless, so there’s no need to manage infrastructure.
- Athena can remove exact duplicates using SQL commands like `DISTINCT`.
- Cons:
- Athena does not have native fuzzy matching capabilities for misspellings. While you can use SQL to deduplicate exact matches, misspelled duplicates would require additional custom logic and possibly complicated SQL queries (e.g., using regular expressions or approximate string matching).
- For large-scale, continuously growing data, Athena may not be the most efficient or least operationally burdensome solution compared to fully managed services like AWS Glue, especially if ongoing processing is needed.
When to use: This is viable if the primary need is for deduplication of exact matches and if misspellings are not a significant concern. However, additional e...
Author: Noah · Last updated Jun 25, 2026
A company is building an inventory management system and an inventory reordering system to automatically reorder products. Both systems use Amazon Kinesis Data Streams. The inventory management system uses the Amazon Kinesis Producer Library (KPL) to publish data to a stream. The inventory reordering system uses the Amazon Kinesis Client Library (KCL) to consume data from the stream. The company configures the stream to scale up and down as needed.Before the company deploys ...
Let's evaluate each factor in terms of the cause of duplicated data in the inventory reordering system:
Option A: The producer experienced network-related timeouts.
- Pros:
- If the producer (using KPL) experiences network timeouts, it may retry sending records to the stream.
- Kinesis Producer Library (KPL) automatically retries failed record sends and ensures that the data is successfully placed into the stream. However, retries could result in duplicate records if the producer does not detect that the record was already successfully placed into the stream.
- Cons:
- KPL is designed to handle retries gracefully and is meant to avoid sending duplicate data unless the retry mechanism fails. The network timeouts on their own would not directly lead to duplicates, as the KPL handles such retries.
When to use: This could be a contributing factor but is not the most likely cause of duplicates, as the KPL should generally ensure idempotency for retrying records.
Option B: The streams value for the IteratorAgeMilliseconds metric was too high.
- Pros:
- The IteratorAgeMilliseconds metric indicates how far behind the consumer is from the latest data in the stream. A high IteratorAge might suggest that the consumer is not processing data as quickly as the producer is publishing, causing the consumer to reprocess the same data.
- Cons:
- A high IteratorAge itself doesn’t cause duplication directly. It simply indicates lag, which might result in the consumer missing some records or being behind in processing. Duplication would occur if the consumer retried reading the same data due to improper handling of the record's state or sequence.
When to use: While a high IteratorAge could contribute to an issue in record processing, it's not the root cause of data duplication. The duplication is more likely related to how the consumer (using KCL) handles the stream.
Option C: There was a change in the number of shards, record processors, or both.
- Pros:
- Scaling the stream by changing the number of shards can cause duplication if the KCL doesn't properly handle the reassignment of shard processing. When the number of shards changes, KCL's record processors may be reassigned, and in some cases, the same record can be consumed more than once.
- This is a common cause of duplication in systems using Kinesis and KCL because KCL uses checkpoints to track its progress, a...
Author: Ravi Patel · Last updated Jun 25, 2026
An ecommerce company operates a complex order fulfilment process that spans several operational systems hosted in AWS. Each of the operational systems has a Java DatabaseConnectivity (JDBC)-compliant relational database where the latest processing state is captured.The company needs to give an operations team the ability to track o...
Let's break down each of the options to determine the solution that meets the requirement with the least development overhead:
Option A: Use AWS Glue to build ingestion pipelines from the operational systems into Amazon Redshift. Build dashboards in Amazon QuickSight that track the orders.
- Pros:
- Amazon Redshift is a powerful data warehouse that is optimized for analytical queries, making it well-suited for tracking and analyzing large datasets, such as orders.
- AWS Glue can easily extract, transform, and load (ETL) data from relational databases to Redshift, with built-in support for JDBC.
- Amazon QuickSight can seamlessly connect to Redshift and provide robust dashboarding capabilities.
- Cons:
- Setting up Redshift and Glue for incremental data ingestion can require some effort in terms of ETL pipeline design and configuration.
- For tracking orders in near real-time on an hourly basis, Redshift's batch-oriented architecture may not provide the best performance, especially if the operational systems have frequent updates.
- The initial setup and configuration of Redshift could be more complex and require careful monitoring.
While this option is a solid choice for handling large-scale data with analytical needs, it could involve more setup compared to other options, particularly for continuous or near real-time data tracking.
Option B: Use AWS Glue to build ingestion pipelines from the operational systems into Amazon DynamoDB. Build dashboards in Amazon QuickSight that track the orders.
- Pros:
- Amazon DynamoDB is a managed NoSQL database that supports high availability and can handle high throughput, making it suitable for applications that need to access and update order tracking in real-time.
- AWS Glue can build ETL pipelines to move data from relational databases into DynamoDB.
- QuickSight integrates easily with DynamoDB, and dashboards can be created to visualize order data.
- Cons:
- DynamoDB is not ideal for running complex queries or analytics at scale (compared to Redshift). Its query capabilities are more limited, and dashboards may not perform well with large datasets or complex join operations.
- DynamoDB can handle high-throughput, but querying and aggregating large volumes of order data might not be as efficient for detailed analysis.
While DynamoDB provides low-latency access and could be suitable for fast updates, it is not the best choice for complex analytical queries, especially when it comes to visualization of large-scale data across multiple operational systems.
Option C: Use AWS Database Migration Service (AWS DMS) to capture changed records in the operational systems. Publish the changes to an Amazon DynamoDB table in a different AWS region from the source database. Build Grafana dashboards that tra...
Author: Emma · Last updated Jun 25, 2026
A data engineer needs to use Amazon Neptune to develop graph applications.Which programming languages should the engineer...
To develop graph applications on Amazon Neptune, the engineer needs to use graph query languages designed for graph databases. Let’s evaluate each option:
A) Gremlin
Reason for Selection: Gremlin is a graph traversal language that is supported by Amazon Neptune. Neptune provides full support for the Apache TinkerPop 3 specification, which includes Gremlin for graph traversal. Gremlin is designed specifically for graph databases and is ideal for traversing nodes and edges in a graph structure. It is a popular choice for querying graph databases that follow the property-graph model, making it very suitable for use with Amazon Neptune.
B) SQL
Reason for Rejection: SQL is a relational query language used primarily for querying relational databases (RDBMS). Amazon Neptune, being a graph database, is not designed to support SQL for graph-specific queries. SQL is not well-suited for performing graph-specific operations, such as node and edge traversals, which are central to graph databases. Therefore, SQL is not the right tool for developing graph applications on Amazon Neptune.
C) ANSI SQL
Reason for Rejection: ANSI SQL is a standard for relational databases, much like SQL. As previously mentioned, Amazon Neptune is a graph data...
Author: Sofia · Last updated Jun 25, 2026
A mobile gaming company wants to capture data from its gaming app. The company wants to make the data available to three internal consumers of the data. The data records are approximately 20 KB in size.The company wants to achieve optimal throughput from each device that runs the gaming app. Additionally, the company wants to develop an application to proc...
To meet the mobile gaming company's requirements, let's break down each option in detail:
A) Configure the mobile app to call the PutRecords API operation to send data to Amazon Kinesis Data Streams. Use the enhanced fan-out feature with a stream for each internal consumer.
Reason for Selection:
- Kinesis Data Streams is designed for high-throughput data ingestion, and calling the `PutRecords` API operation to send data is suitable for capturing records of approximately 20 KB in size, which is efficient for Kinesis Data Streams.
- Enhanced fan-out provides dedicated throughput for each internal consumer. This is critical because it ensures that each of the three internal consumers has independent and dedicated throughput without sharing resources. Enhanced fan-out is an ideal solution for multiple consumers requiring parallel processing of data streams with high throughput.
- The setup also allows the company to process data in real time and efficiently distribute data to internal consumers.
B) Configure the mobile app to call the PutRecordBatch API operation to send data to Amazon Kinesis Data Firehose. Submit an AWS Support case to turn on dedicated throughput for the company’s AWS account. Allow each internal consumer to access the stream.
Reason for Rejection:
- Kinesis Data Firehose is primarily designed for real-time data streaming to other destinations like S3, Redshift, or Elasticsearch. However, it doesn't support dedicated throughput for each internal consumer natively. This could make it harder to guarantee the specific throughput for each consumer as required.
- Also, the PutRecordBatch operation is more useful for batching data, but Firehose does not provide the level of granularity for managing throughput per consumer, unlike Kinesis Data Streams with enhanced fan-out.
C) Configure the mobile app to use the Amazon Kinesis Producer Library (KPL) to send data to Amazon Kinesis Data Firehose. Use the e...
Author: Oscar · Last updated Jun 25, 2026
A retail company uses an Amazon Redshift data warehouse and an Amazon S3 bucket. The company ingests retail order data into the S3 bucket every day.The company stores all order data at a single path within the S3 bucket. The data has more than 100 columns. The company ingests the order data from a third-party application that generates more than 30 files in CSV format every day. Each CSV file is between 50 and 70 MB in size.The company uses Amazon Redshift Spectrum to run queries that select sets of columns. Users aggregate metrics based on daily orders. Recen...
To resolve the performance issues when querying data using Amazon Redshift Spectrum, we need to focus on optimizing how the data is stored, partitioned, and queried. Let’s evaluate each option in detail:
A) Configure the third-party application to create the files in a columnar format.
Reason for Selection:
- Columnar formats like Parquet or ORC are highly efficient for querying specific columns, especially in Amazon Redshift Spectrum. Since the company queries only a subset of columns from the 100+ columns in the dataset, columnar formats store data more efficiently by reducing the amount of data that needs to be read during queries.
- Columnar storage reduces I/O and improves query performance because only the necessary columns are read. This format is optimized for analytical workloads, which fits the company’s use case of aggregating metrics based on daily orders.
- The files in CSV format are row-based, leading to unnecessary data being read and increasing query time. Switching to a columnar format will address the performance degradation effectively.
B) Develop an AWS Glue ETL job to convert the multiple daily CSV files to one file for each day.
Reason for Rejection:
- Although consolidating multiple files into a single daily file can help with query performance by reducing the number of files that need to be scanned, it does not address the underlying issue of inefficient file formats for querying. It still involves reading from a non-columnar CSV format, which leads to high I/O.
- This step adds complexity to the process (additional Glue job development), and while it might provide some performance improvement, it won’t be as effective as switching to a columnar format, especially when paired with partitioning.
C) Partition the order data in the S3 bucket based on order date.
Reason for Selection:
- Partitioning the data in Amazon S3 by order date (or another logical partition key) helps Amazon Redshift Spectrum query only the relevant partitions, reducing the amount of data scanned and improving performance.
- This is es...
Author: IceDragon2023 · Last updated Jun 25, 2026
A company stores customer records in Amazon S3. The company must not delete or modify the customer record data for 7 years after each record is created. The root user also must not have the ability to delete or modify the data.A data...
To meet the company's requirement to prevent data from being deleted or modified for 7 years and ensure that even the root user cannot delete or modify the data, let’s evaluate each option in detail:
A) Enable governance mode on the S3 bucket. Use a default retention period of 7 years.
Reason for Rejection:
- Governance mode allows users with specific permissions (but not the root user) to bypass object lock settings, meaning that even though the data would be locked for 7 years, an authorized user could potentially override this setting and modify or delete the data.
- Since the company requires that the root user cannot modify or delete the data, governance mode does not provide the necessary level of protection because the root user has more permissive access in governance mode.
B) Enable compliance mode on the S3 bucket. Use a default retention period of 7 years.
Reason for Selection:
- Compliance mode in Amazon S3 Object Lock ensures that once data is locked, it cannot be modified or deleted by any user, including the root user, for the duration of the retention period (in this case, 7 years).
- This mode provides the highest level of security, guaranteeing that the data will remain immutable for the required period, meeting the company's requirement that no one (including the root user) can alter or delete the data.
- Compliance mode is specifically designed for regulatory compliance use cases, such as this one where data must not be modified or deleted for ...
Author: Harper · Last updated Jun 25, 2026
A data engineer needs to create a new empty table in Amazon Athena that has the same schema as an existing table named old_table.Which SQL sta...
To create a new empty table in Amazon Athena with the same schema as an existing table (`old_table`), the goal is to replicate the structure (schema) of the old table without copying any data. Let's evaluate each option:
A) CREATE TABLE new_table AS SELECT FROM old_table;
Reason for Rejection:
- This query would create a new table `new_table` by selecting all the data from `old_table`. While this creates a table with the same schema, it also copies all the data from `old_table` into `new_table`.
- The requirement specifies that the new table should be empty, so this option is not suitable because it will populate the new table with data from the old one.
B) INSERT INTO new_table SELECT FROM old_table;
Reason for Rejection:
- This option assumes that `new_table` already exists and is an empty table. It would insert all the data from `old_table` into `new_table`.
- The requirement is to create a new empty table with the same schema, so the insertion of data is not needed, and this option does not address the creation of a table with the same schema...
Author: BlazingPhoenix22 · Last updated Jun 25, 2026
A data engineer needs to create an Amazon Athena table based on a subset of data from an existing Athena table named cities_world. The cities_world table contains cities that are located around the world. The data engineer must create a new table named cities_us to contain only the cit...
Let's analyze the options provided one by one:
A) INSERT INTO cities_usa (city, state) SELECT city, state FROM cities_world WHERE country = 'usa';
- Analysis: This option uses an `INSERT INTO` statement to insert the selected data (cities located in the US) into the `cities_usa` table. It only selects the `city` and `state` columns from the `cities_world` table, where the `country` is equal to "usa".
- Why it works: The correct syntax for inserting data into an existing table is `INSERT INTO`, and selecting rows with a specific condition using `WHERE` (in this case, `country = 'usa'`) is the right approach for filtering the data.
- Why it's preferred: This query is appropriate as it inserts the filtered data into an existing `cities_usa` table without modifying the structure of either table.
B) MOVE city, state FROM cities_world TO cities_usa WHERE country = 'usa';
- Analysis: The `MOVE` statement does not exist in SQL syntax, specifically not in Amazon Athena. Athena supports the SQL language that does not include a `MOVE` command.
- Why it's rejected: This is not valid SQL syntax and would throw an error when executed in Athena. Therefore, it cannot be used.
C) INSERT INTO cities_usa SELECT city, state FROM cities_world WHERE country = 'usa';
- Analysis: Similar to option A, this query inserts data into the `cities_usa` table based on the selected columns (`city` and `state`) from `cities_world` where the `country` is 'usa'.
- Why it works: This query is correctly formatted for inserting data into a table in Athena and is a valid approach. It selects and inserts the necessary data as required.
- Why it's similar to option A: ...
Author: ShadowWolf101 · Last updated Jun 25, 2026
A company implements a data mesh that has a central governance account. The company needs to catalog all data in the governance account. The governance account uses AWS Lake Formation to centrally share data and grant access permissions.The company has created a new data product that includes a group of Amazon Redshift Serverless tables. A data engineer needs to share the data product with a marketing team. The marketing team must have access to only a subset of columns. The data engineer needs to share the same data product wit...
Let's go through the options one by one:
A) Create views of the tables that need to be shared. Include only the required columns.
- Analysis: Creating views is a common approach when you need to share specific columns or subsets of data from a larger table. A view allows the data engineer to expose only the columns that should be visible to a specific user or team (in this case, the marketing team and the compliance team).
- Why it works: By creating views with only the necessary columns, the data engineer can control the access to data, ensuring that both teams see only their respective required columns. This satisfies the requirement of granting different subsets of data to different teams.
- Why it's preferred: Views offer a flexible, secure way to share only a subset of data without modifying the underlying tables or creating redundant copies of data. This approach is commonly used in scenarios where data sharing needs to be granular and controlled.
B) Create an Amazon Redshift data share that includes the tables that need to be shared.
- Analysis: Creating an Amazon Redshift data share allows data to be shared across different Redshift clusters. However, this does not inherently provide the ability to limit access to specific columns from the shared tables.
- Why it's rejected: While data sharing via Redshift is useful for cross-account or cross-cluster data access, it does not allow fine-grained control over specific columns. This option alone wouldn't satisfy the requirement to provide different subsets of columns to different teams. So, it's not a complete solution.
C) Create an Amazon Redshift managed VPC endpoint in the marketing team's account. Grant the marketing team access to the views.
- Analysis: A managed VPC endpoint allows secure connectivity between Redshift clusters across accounts. This step might be necessary if the marketing team is in a different AWS account than the central governance account. However, it doesn't directly address the need for column-level access control.
- Why it's rejected: While it could be part of the solution for network connectivity and cro...
Author: StarryEagle42 · Last updated Jun 25, 2026
A company has a data lake in Amazon S3. The company uses AWS Glue to catalog data and AWS Glue Studio to implement data extract, transform, and load (ETL) pipelines.The company needs to ensure that data quality issues are checked every time the pipelines run. A data engineer must enhance the existing pipelines to eva...
Let's analyze the options one by one:
A) Add a new transform that is defined by a SQL query to each Glue ETL job. Use the SQL query to implement a ruleset that includes the data quality rules that need to be evaluated.
- Analysis: Adding a SQL query transform is a way to check data quality, but SQL queries alone would require a custom implementation for defining data quality rules, and they would not have built-in integration with data quality frameworks.
- Why it's rejected: While it can be done, SQL alone does not provide a dedicated mechanism for data quality checks and is not the most efficient way to evaluate predefined thresholds. It would involve more manual effort for rule creation, lack of flexibility, and would not be as easily maintained as other options specifically designed for data quality.
B) Add a new Evaluate Data Quality transform to each Glue ETL job. Use Data Quality Definition Language (DQDL) to implement a ruleset that includes the data quality rules that need to be evaluated.
- Analysis: The "Evaluate Data Quality" transform is a built-in feature in AWS Glue. It uses the Data Quality Definition Language (DQDL) to define and evaluate rules. This is a purpose-built solution for handling data quality, where you can specify rules for thresholds, completeness, and other aspects of data quality.
- Why it works: This option directly meets the requirement to ensure that data quality issues are checked with minimal effort. The DQDL is specifically designed to be simple, and AWS Glue’s native integration with this transform streamlines implementation, making it the least effortful solution. It is efficient and designed for this exact use case.
- Why it's preferred: It requires minimal implementation effort because it’s fully integrated with AWS Glue, and you don't need to worry about third-party libraries or manual rule handling. It's a clean, declarative approach to handling data quality within Glue pipelines.
C) Add a new custom transform to each Glue ETL job. Use the PyDeequ library to implement a ruleset that includes the data quality rules that need t...
Author: Isabella · Last updated Jun 25, 2026
A company has an application that uses a microservice architecture. The company hosts the application on an Amazon Elastic Kubernetes Services (Amazon EKS) cluster.The company wants to set up a robust monitoring system for the application. The company needs to analyze the logs from the EKS cluster and the application. The company needs to correlate the cluster's logs with the application's traces ...
Let's analyze the options to determine the best approach for setting up a robust monitoring system for the application and correlating logs with traces:
A) Use FluentBit to collect logs. Use OpenTelemetry to collect traces.
- Analysis: FluentBit is a lightweight log shipper commonly used to collect logs from Kubernetes clusters, such as Amazon EKS. OpenTelemetry is a framework used to collect, generate, and export traces from distributed systems, including Kubernetes.
- Why it works: FluentBit can efficiently collect logs from the EKS cluster, and OpenTelemetry can collect application traces. Both tools are widely supported in modern microservice architectures and can integrate seamlessly with cloud monitoring tools.
- Why it's preferred: This option directly addresses both log collection and trace collection with minimal custom development. Both FluentBit and OpenTelemetry are compatible with AWS services and are designed to work well in Kubernetes environments.
- Why it's better than other options: FluentBit is a cost-effective, simple way to collect logs, and OpenTelemetry provides native tracing functionality. Additionally, both are well-supported within AWS monitoring systems, and the integration with Amazon CloudWatch (which stores logs and traces) is straightforward.
B) Use Amazon CloudWatch to collect logs. Use Amazon Kinesis to collect traces.
- Analysis: Amazon CloudWatch can be used to collect and store logs from various AWS resources, including EKS clusters. However, Amazon Kinesis is generally used for stream processing and would require additional setup and custom development to collect traces.
- Why it's rejected: Kinesis is not primarily designed for trace collection. While it is great for real-time data streaming, it is not the best tool for correlating application traces. Amazon CloudWatch itself has integrated tracing capabilities through CloudWatch ServiceLens, which provides application monitoring and trace collection.
C) Use Amazon CloudWatch to collect logs. Use Amazon Managed Streaming for Apache Kafka (Amazon MSK) to collect traces.
- Analysis: While Amazon CloudWatch is an excellent choice for collecting logs, using Amazon MSK (Kafka) to collect traces introduces unnecessary complexity. Kafka is generally used for message streaming, not trace collection.
- Why it's rejected: Kafka does not provide native support for application tracing and requ...
Author: John · Last updated Jun 25, 2026
A company has a gaming application that stores data in Amazon DynamoDB tables. A data engineer needs to ingest the game data into an Amazon OpenSearch Service cluster. Data upda...
Let's analyze each option and determine the best solution to ingest game data from Amazon DynamoDB into Amazon OpenSearch Service in near real time:
A) Use AWS Step Functions to periodically export data from the Amazon DynamoDB tables to an Amazon S3 bucket. Use an AWS Lambda function to load the data into Amazon OpenSearch Service.
- Analysis: This solution involves periodic exports of data using Step Functions, which will then be loaded into OpenSearch using Lambda. However, the key issue is that it uses a periodic batch process rather than handling real-time data updates. This approach does not ensure near real-time updates, as there would be delays between each export and data processing cycle.
- Why it's rejected: This option does not meet the near-real-time requirement, as the data is processed periodically. For near real-time updates, a more continuous, event-driven solution is needed.
B) Configure an AWS Glue job to have a source of Amazon DynamoDB and a destination of Amazon OpenSearch Service to transfer data in near real time.
- Analysis: AWS Glue is generally used for batch processing and ETL workflows. While it supports various data sources and destinations, including DynamoDB and OpenSearch, it is not optimized for real-time or near real-time data updates. Glue jobs are typically scheduled and may not be the best fit for continuous, near real-time data transfer.
- Why it's rejected: AWS Glue is not ideal for near real-time data transfer, as it is generally used for batch-based ETL workflows. Real-time updates would require a more immediate data flow mechanism.
C) Use Amazon DynamoDB Streams to capture table changes. Use an AWS Lambda function to process and update the data in Amazon OpenSearch Service.
- Analysis: This solution leverages DynamoDB Streams, which captures changes (inserts, updates, deletes) in real time as they occur in DynamoDB tables. A Lambda fu...
Author: Sophia Clark · Last updated Jun 25, 2026
A company uses Amazon Redshift as its data warehouse service. A data engineer needs to design a physical data model.The data engineer encounters a de-normalized table that is growing in size. The table does not have a suitable column to use as the distribution key.Wh...
When designing a physical data model for Amazon Redshift, the distribution style determines how data is distributed across the compute nodes. The choice of distribution style has a significant impact on query performance and maintenance overhead. Here's a breakdown of each distribution style and how it fits the scenario of a growing, de-normalized table without a suitable column for a distribution key:
A) ALL Distribution:
- How it works: This style distributes a full copy of the table to all compute nodes.
- Advantages: It can be very efficient for small dimension tables or lookup tables because all nodes have the entire data, thus avoiding shuffling during joins.
- Disadvantages: The maintenance overhead can be high when the table grows, especially if it contains a large number of rows, as the full copy of the table is distributed to every node. This increases storage and can degrade performance as the table size grows.
- When to use: It's best suited for small lookup or dimension tables, not large fact tables or tables that are growing in size.
B) EVEN Distribution:
- How it works: Data is distributed evenly across the nodes without considering any column. This is typically used when there isn't a good choice of a distribution key.
- Advantages: It ensures even distribution of data, which can prevent "hotspots" on specific nodes. There’s minimal maintenance overhead.
- Disadvantages: Since there's no relation to join patterns or data, it could lead to unnecessary shuffling during queries, especially for large fact tables.
- When to use: This is a good option when the table does not have a natural distribution key and is not frequently joined with other tables. It can also be suitable when there’s no performance requirement for joins or when the table is not growing rapidly.
C) AUTO Distribution:
- How it works: Amazon Redshift automatically selects the most appropriate distribution style based on the size of the table. For small tables, it chooses ALL distribution, for large tables it s...
Author: Michael · Last updated Jun 25, 2026
A retail company is expanding its operations globally. The company needs to use Amazon QuickSight to accurately calculate currency exchange rates for financial reports. The company has an existing dashboard that includes a visual that is based on an analysis of a dataset that contains global currency values and exchange rates.A data engineer needs to ensure that exchange rates are calculated with a precision of four decimal places. The cal...
To meet the requirement of ensuring that currency exchange rates are calculated with four decimal places, precomputed, and materialized in Amazon QuickSight's SPICE (Super-fast, Parallel, In-memory Calculation Engine), we need to consider the context in which the calculation should be defined and how QuickSight processes calculations.
Evaluation of options:
1. A) Define and create the calculated field in the dataset:
- Description: Calculated fields in the dataset are computed during the dataset's refresh process, before any analysis or visualizations. These fields are materialized in SPICE and stored as part of the dataset.
- Pros: This ensures that the calculations are precomputed, and results are stored in SPICE, allowing for fast and efficient access during any analysis or visualization. It also ensures the calculation uses the desired precision (e.g., four decimal places).
- Cons: If the calculation logic needs to change frequently or is only needed for a specific analysis, it might require re-refreshing the dataset, which could lead to overhead.
- Selected Option: This is the most optimal choice because it ensures that the exchange rate calculations are precomputed, stored, and available with high precision, leveraging SPICE’s performance.
2. B) Define and create the calculated field in the analysis:
- Description: This allows the calculated field to be created directly within an analysis, but the calculation occurs in real-time when the analysis is viewed, rather than being precomputed in SPICE.
- Pros: Allows for dynamic and flexible calculations in the context of the analysis, without needing to alter the dataset.
- Cons: Since the calculations aren't precomputed and stored in SPICE, performance may be slower, and the calculations won't be as efficient for large datasets. Additionally, t...
Author: Deepak · Last updated Jun 25, 2026
A company has three subsidiaries. Each subsidiary uses a different data warehousing solution. The first subsidiary hosts its data warehouse in Amazon Redshift. The second subsidiary uses Teradata Vantage on AWS. The third subsidiary uses Google BigQuery.The company wants to aggregate all the data into a central Amazon S3 data lake. The company wants to use Apache Iceberg as the table format.A data engineer needs to build a new pipeline to connect to...
Analysis of the Solution Options
To address the problem, we need a solution that can connect to all three data sources (Amazon Redshift, Teradata, and Google BigQuery), run transformations, join the data, and write it to an Apache Iceberg table in Amazon S3. Additionally, the solution should have the least operational effort, meaning minimal manual intervention and complexity.
Let’s break down the options and evaluate each one:
A) Use native Amazon Redshift, Teradata, and BigQuery connectors to build the pipeline in AWS Glue. Use native AWS Glue transforms to join the data. Run a Merge operation on the data lake Iceberg table.
- Advantages:
- AWS Glue supports native connectors for Amazon Redshift and Google BigQuery, making it easier to integrate.
- AWS Glue provides built-in transformation capabilities and supports various table formats, including Apache Iceberg.
- AWS Glue automates much of the ETL process, reducing operational effort.
- Disadvantages:
- While AWS Glue does support a native connector for Redshift and BigQuery, Teradata would require a custom connector, which could increase complexity.
- Glue transforms are somewhat limited in handling complex data processing, and building transformations might require more effort, especially for joining data from different sources.
- When to use:
- This option is appropriate when the focus is on a managed ETL service with minimal infrastructure setup. However, the need for a custom connector for Teradata and possible complexity in managing transformations makes this a bit cumbersome.
B) Use the Amazon Athena federated query connectors for Amazon Redshift, Teradata, and BigQuery to build the pipeline in Athena. Write a SQL query to read from all the data sources, join the data, and run a Merge operation on the data lake Iceberg table.
- Advantages:
- Athena supports federated queries, which allows querying data across multiple sources without moving data.
- SQL queries can be used to read from all the sources, simplifying the logic needed for transformations.
- Athena can be configured to work with Iceberg tables, so this solution can directly write data in the desired format.
- Disadvantages:
- Federated queries are useful for querying, but they can become inefficient or slower when handling large datasets, especially when involving multiple data sources.
- Athena may not handle complex transformation logic as easily as a more feature-rich solution like AWS Glue or Apache Spark.
- When to use:
- Best used when you want an easy-to-use, serverless solution for querying across different sources with minimal operational overhead. However, performance can be a concern when working with large datasets or needing complex data processing.
C) Use the native Amazon Redshift connector, the Java Database Connectivity (JDBC) connector for Teradata, and the open source Apache Spark BigQuery connector to build the pipeline in Amazon EM...
Author: Akash · Last updated Jun 25, 2026
A data engineer needs to onboard a new data producer into AWS. The data producer needs to migrate data products to AWS.The data producer maintains many data pipelines that support a business application. Each pipeline must have service accounts and their corresponding credentials. The data engineer must establish a secure connection from the data producer's on-premises data cen...
Let's evaluate the options based on the requirements:
1. The data engineer needs to establish a secure connection from the on-premises data center to AWS.
2. The data engineer must ensure that no public internet is used for data transfer.
3. Each pipeline requires service accounts with corresponding credentials.
Option A: Instruct the new data producer to create Amazon Machine Images (AMIs) on Amazon Elastic Container Service (Amazon ECS) to store the code base of the application. Create security groups in a public subnet that allow connections only to the on-premises data center.
- Pros: ECS and AMIs are great for managing application code and containers. Security groups can control access to instances.
- Cons: The solution involves using public subnets and doesn't address the core requirement of avoiding the public internet for data transfer. Additionally, storing service account credentials directly within the containers is not secure.
- Rejection Reason: The data transfer would still involve public internet connectivity, which violates the requirement for avoiding public internet access.
Option B: Create an AWS Direct Connect connection to the on-premises data center. Store the service account credentials in AWS Secrets Manager.
- Pros: AWS Direct Connect provides a dedicated, secure connection from an on-premises data center to AWS, eliminating the use of the public internet. AWS Secrets Manager can securely store and manage service account credentials.
- Cons: This option doesn't mention specific actions regarding data transfer or providing the data producer with an easy way to manage data pipelines in AWS.
- Selected Option: This solution addresses all requirements securely: it avoids the public internet, ensures secure storage of credentials, and establishes the necessary secure connection. The use of AWS Direct Connect ensures high bandwidth, secure data transfer...
Author: Maya · Last updated Jun 25, 2026
A data engineer configured an AWS Glue Data Catalog for data that is stored in Amazon S3 buckets. The data engineer needs to configure the Data Catalog to receive incremental updates.The data engineer sets up event notifications for the S3 bucket and creates an Amazon Simple Queue Service (Amazon SQS) queue to receive the ...
To meet the requirement of configuring the AWS Glue Data Catalog to receive incremental updates with least operational overhead, we need to choose a combination of solutions that will automate the update process based on S3 events without adding complexity. Let's analyze each option based on key factors like automation, maintenance overhead, and how well they fit with S3 event-based processing.
Option A: Create an S3 event-based AWS Glue crawler to consume events from the SQS queue.
- Reasoning: This option is ideal because it directly integrates AWS Glue with the event-driven architecture of S3 and SQS. The crawler can be triggered automatically by the SQS queue when an event is received (i.e., when new data is added to S3). The crawler will only update the Glue Data Catalog when necessary, so it’s efficient and scales with minimal manual intervention. This option is well-suited for automated incremental updates.
- Key Factors: Fully automated process, minimal manual intervention, integrates well with Glue, low operational overhead.
Option B: Define a time-based schedule to run the AWS Glue crawler, and perform incremental updates to the Data Catalog.
- Reasoning: While this is a valid approach, it introduces unnecessary overhead because it would run the crawler at fixed intervals, regardless of whether there were changes in the S3 data. It lacks the event-driven trigger that would ensure updates happen only when necessary. This is not as efficient as the event-based approach, which only runs when data changes.
- Key Factors: More operational overhead (since you need to define and maintain the schedule), not as efficient as an event-driven approach.
Option C: Use an AWS Lambda function to directly update the Data Catalog based on S3 events that the SQS queue receives.
- Reasoning: This option can also automate the process, but it adds complexity. The Lambda function would need to parse the SQS events, trigger an update in the Glue Data Catalog, and ha...
Author: Sofia2021 · Last updated Jun 25, 2026
A company uses AWS Glue Data Catalog to index data that is uploaded to an Amazon S3 bucket every day. The company uses a daily batch processes in an extract, transform, and load (ETL) pipeline to upload data from external sources into the S3 bucket.The company runs a daily report on the S3 data. Some days, the company runs the report before all the daily data has been uploaded to the S3 bucket. A data engineer must be able to send a me...
To meet the requirement of identifying incomplete data with the least operational overhead, we need a solution that is automated, integrated with existing services, and does not require significant infrastructure setup or maintenance. Let's evaluate each option based on these factors.
A) Create data quality checks for the source datasets that the daily reports use. Create a new AWS managed Apache Airflow cluster. Run the data quality checks by using Airflow tasks that run data quality queries on the columns' data type and the presence of null values. Configure Airflow Directed Acyclic Graphs (DAGs) to send an email notification that informs the data engineer about the incomplete datasets to the SNS topic.
- Explanation: Apache Airflow is a powerful orchestration tool, but managing an Airflow cluster requires significant operational overhead. It involves setting up, maintaining, and scaling the cluster, which increases complexity compared to other serverless solutions.
- Why rejected: While it’s an effective tool for orchestrating complex workflows, Airflow introduces operational overhead and complexity that is unnecessary for this use case. The goal is to minimize operational overhead, so Airflow would not be the best option here.
B) Create data quality checks on the source datasets that the daily reports use. Create a new Amazon EMR cluster. Use Apache Spark SQL to create Apache Spark jobs in the EMR cluster that run data quality queries on the columns' data type and the presence of null values. Orchestrate the ETL pipeline by using an AWS Step Functions workflow. Configure the workflow to send an email notification that informs the data engineer about the incomplete datasets to the SNS topic.
- Explanation: Amazon EMR is a scalable big data platform, and while it is powerful, it requires management of clusters, which adds significant overhead in terms of both cost and maintenance. Using Spark jobs for data quality checks also introduces unnecessary complexity.
- Why rejected: Like Airflow, EMR is overkill for this task. It requires maintaining clusters and managing complex resources. It's a great tool for large-scale data processing but introduces more operational overhead than needed here.
C) Create data quality checks on the source datasets that the daily reports use. Create data quality actions by using AWS Glue workflows to confirm the com...
Author: Ryan · Last updated Jun 25, 2026
A company stores customer data that contains personally identifiable information (PII) in an Amazon Redshift cluster. The company's marketing, claims, and analytics teams need to be able to access the customer data.The marketing team should have access to obfuscated claim information but should have full access to customer contact information. The claims team should have access to customer information for each claim that t...
Let's break down the options and evaluate them based on the scenario of securing access to customer data with least administrative overhead while meeting the specific access requirements for the marketing, claims, and analytics teams.
Option A: Create a separate Redshift cluster for each team. Load only the required data for each team. Restrict access to clusters based on the teams.
- Why rejected: This option would involve managing multiple Redshift clusters, which increases both operational complexity and cost. Each cluster would require separate data loads, maintenance, and security configurations. This solution is not efficient or scalable, especially for scenarios where teams might need to interact with overlapping datasets.
- Not ideal because: It requires significant administrative overhead and is inefficient, as the company would have to manage multiple clusters, which adds complexity without providing much benefit in data access control.
Option B: Create views that include required fields for each of the data requirements. Grant the teams access only to the view that each team requires.
- Why selected: This solution is effective and scalable. By creating views that contain only the necessary data fields for each team (e.g., obfuscated data for the analytics team, full contact data for the marketing team, and claim-specific data for the claims team), you can enforce the access requirements directly in Redshift. Views can be easily managed and modified without significant overhead, and they help isolate the teams from unnecessary data access.
- Why other options are rejected:
- It is straightforward to implement, as it only requires the creation of views and granting appropriate permissions.
- It provides flexibility, as any changes to access requirements can be managed by modifying views instead of creating new clusters or roles.
- The solution is scalable because more views can be added as required without needing complex restructuring.
...
Author: ShadowWolf101 · Last updated Jun 25, 2026
A financial company recently added more features to its mobile app. The new features required the company to create a new topic in an existing Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster.A few days after the company added the new topic, Amazon CloudWatc...
To address the CloudWatch alarm on the RootDiskUsed metric for an Amazon MSK cluster, we need to identify the underlying issue and select a solution that addresses the disk space usage specifically, which is related to storage capacity. Let's evaluate each option based on this requirement:
A) Expand the storage of the MSK broker. Configure the MSK cluster storage to expand automatically.
- Explanation: This option directly addresses the issue by expanding the storage available to the MSK brokers. Configuring the MSK cluster to expand automatically is also a good practice because it ensures that storage will be increased automatically as disk usage grows, preventing future alarms related to disk space. MSK brokers store the actual Kafka data, so running out of disk space would trigger the alarm.
- Why selected: This is the most effective and appropriate solution, as it directly solves the disk space issue by increasing the storage allocated to the MSK brokers. Configuring automatic expansion ensures that the storage is managed dynamically with minimal administrative effort. It also addresses the alarm on the RootDiskUsed metric, which is related to disk space usage on the MSK brokers.
B) Expand the storage of the Apache ZooKeeper nodes.
- Explanation: Apache ZooKeeper nodes are essential for managing the Kafka cluster's metadata and coordination. However, RootDiskUsed specifically refers to the MSK brokers’ storage, not ZooKeeper’s. ZooKeeper typically doesn’t store Kafka data in the same way as brokers do, so expanding its storage would not address the disk space alarm triggered by the brokers' storage.
- Why rejected: ZooKeeper nodes may require storage expansion in some cases, b...
Author: Jack · Last updated Jun 25, 2026
A data engineer needs to build an enterprise data catalog based on the company's Amazon S3 buckets and Amazon RDS databases. The data catalog must include storage format metadata for the data...
To meet the requirement of building an enterprise data catalog that includes storage format metadata for data in Amazon S3 buckets and Amazon RDS databases, let's evaluate each option based on effort, scalability, and automation.
A) Use an AWS Glue crawler to scan the S3 buckets and RDS databases and build a data catalog. Use data stewards to inspect the data and update the data catalog with the data format.
- Explanation: This solution involves using an AWS Glue crawler to scan data in S3 and RDS and create a data catalog. However, it relies on manual inspection and updates by data stewards to determine the data format. This introduces unnecessary operational overhead, as it requires human intervention to inspect and update the catalog after it’s created.
- Why rejected: This solution adds more manual effort by requiring data stewards to update the data format, which is inefficient and not the least effort approach. It also doesn't fully automate the process of identifying and storing the data format in the catalog.
B) Use an AWS Glue crawler to build a data catalog. Use AWS Glue crawler classifiers to recognize the format of data and store the format in the catalog.
- Explanation: AWS Glue crawlers are designed to automatically discover and catalog data stored in S3 and RDS. The crawlers use built-in classifiers to automatically detect the storage format (e.g., Parquet, CSV, JSON, etc.) and include it in the catalog. This solution offers automation and efficiency, as the crawler identifies data formats without requiring manual intervention.
- Why selected: This solution automates the process of discovering both the data and its format with minimal effort. AWS Glue is well-integrated with both S3 and RDS, and the classifiers are designed to recognize...
Author: Joseph · Last updated Jun 25, 2026
A company analyzes data in a data lake every quarter to perform inventory assessments. A data engineer uses AWS Glue DataBrew to detect any personally identifiable formation (PII) about customers within the data. The company's privacy policy considers some custom categories of information to be PII. However, the categories are not included in standard DataBrew data quality rules.The data engineer needs to modify the ...
Let's break down each option based on the key factors:
A) Manually review the data for custom PII categories:
- Key Factors: High operational overhead, prone to human error, and not scalable.
- Why rejected: This option requires manual intervention and would be time-consuming, inefficient, and error-prone. Additionally, it's not suitable for large datasets across multiple datasets in a data lake. This does not meet the requirement of reducing operational overhead.
B) Implement custom data quality rules in DataBrew. Apply the custom rules across datasets:
- Key Factors: Leverages AWS Glue DataBrew’s data quality rules, which can be automated and reusable, reducing operational overhead.
- Why selected: AWS Glue DataBrew allows the implementation of custom data quality rules, and applying them across datasets can be automated, minimizing manual effort. This solution integrates directly into the existing workflow with minimal added complexity. It provides a low-overhead solution to scan for the custom PII categories across datasets in the data lake and fits well with the tool already being used.
C) Develop custom Python scripts to detect the custom PII categories. Call the scripts from DataBrew:
- Key Factors: Requires coding, introduces custom development, and adds overhead for maintenance and scali...
