Amazon Practice Questions, Discussions & Exam Topics by our Authors
A company stores employee data in Amazon Resdshift. A table names Employee uses columns named Region ID, Department ID, and Role ID as a compound sort key.Which queries will MOST incre...
To optimize queries that use a compound sort key in Amazon Redshift, the query should leverage the sort key columns efficiently. A compound sort key stores data in a sorted order based on the specified columns, which means queries that filter by the leftmost columns in the sort key will benefit the most in terms of performance.
Understanding Compound Sort Keys
A compound sort key uses multiple columns to determine the order of the data, and the columns are processed from left to right. In this case, the sort key is defined on the following columns, in order:
1. Region ID
2. Department ID
3. Role ID
A) `Select from Employee where Region ID = 'North America';`
- Reasoning: This query uses the Region ID as the filter, which is the leftmost column in the compound sort key. Since the data is stored sorted by Region ID, this query will take full advantage of the sort key and filter quickly.
- Why selected: This is an optimal query for the compound sort key because it directly filters based on the first column of the sort key, which will help Amazon Redshift quickly locate the relevant rows.
B) `Select from Employee where Region ID = 'North America' and Department ID = 20;`
- Reasoning: This query filters by both Region ID and Department ID. Since Region ID is the first column in the sort key, and Department ID is the second, this query will also benefit from the compound sort key. Redshift can efficiently scan the data based on both columns' sorted order.
- Why selected: This query benefits from both the first and second columns of the sort key, improving performance as Redshift can filter using the sorted regions and then narrow down further by department.
C) `Select from Employee where Department ID = 20 and Region ID = 'North America';`
- Reasoning: This query filters by Department ID first and then by Region ID. Since Department ID...
Author: RadiantPhoenixX · Last updated Jul 30, 2026
A company receives test results from testing facilities that are located around the world. The company stores the test results in millions of 1 KB JSON files in an Amazon S3 bucket. A data engineer needs to process the files, convert them into Apache Parquet format, and load them into Amazon Redshift tables. The data engineer uses AWS Glue to process the files, AWS Step Functions to orchestrate the processes, and Amazon EventBridge to schedule jobs.The company r...
Let's analyze each option to determine the best solution for reducing data processing time.
A) Use AWS Lambda to group the raw input files into larger files. Write the larger files back to Amazon S3. Use AWS Glue to process the files. Load the files into the Amazon Redshift tables.
- Reasoning: AWS Lambda could be used to group smaller JSON files into larger files (e.g., larger JSON files or compressed formats like gzip). Larger files are more efficient to process because AWS Glue would read fewer files, and the cost of starting Glue jobs would be lower. However, Lambda's execution time is limited, so it might not be ideal for large-scale file processing.
- Rejected: Although this reduces the number of files to process, Lambda's execution limits and the potential overhead of re-writing the files back to S3 could make this solution inefficient for large datasets.
B) Use the AWS Glue dynamic frame file-grouping option to ingest the raw input files. Process the files. Load the files into the Amazon Redshift tables.
- Reasoning: The AWS Glue dynamic frame file-grouping option allows Glue to group files automatically during the ingestion process, which reduces the number of files being processed and speeds up the overall job. This solution takes advantage of Glue’s built-in file management capabilities to improve data processing time by reducing file overhead.
- Selected: This is the most efficient approach because it automates file grouping without additional services, leveraging Glue’s native capabilities to optimize the process. This minimizes the overhead involved with processing millions of small files and ensures that files are grouped dynamically for optimal performance.
C) Use the...
Author: IronLion88 · Last updated Jul 30, 2026
A data engineer uses Amazon Managed Workflows for Apache Airflow (Amazon MWAA) to run data pipelines in an AWS account.A workflow recently failed to run. The data engineer needs to use Apache Airflow logs to diagnose the failure...
To diagnose the cause of the workflow failure in Amazon MWAA, the data engineer should focus on logs that provide insights into the execution and processing of tasks in the workflow. Let's go over the log types one by one:
A) YourEnvironmentName-WebServer Logs:
- Scenario: This log type is primarily useful for diagnosing issues related to the web server, such as issues with the Airflow UI, interactions with the user interface, or API calls.
- Reason for rejection: It will not help in understanding why a specific task or workflow failed, as these logs mainly provide details about user-facing operations like web requests and interface interactions.
B) YourEnvironmentName-Scheduler Logs:
- Scenario: These logs are focused on the scheduler component of Apache Airflow. They provide information on scheduling the workflows, such as when DAGs are triggered, and whether they start or finish properly.
- Reason for rejection: While they are helpful for understanding if a DAG was triggered properly, these logs don’t provide specific information about the individual task executions or failures within a DAG, which is key for tro...
Author: Aarav2020 · Last updated Jul 30, 2026
A finance company uses Amazon Redshift as a data warehouse. The company stores the data in a shared Amazon S3 bucket. The company uses Amazon Redshift Spectrum to access the data that is stored in the S3 bucket. The data comes from certified third-party data providers. Each third-party data provider has unique connection details.To comply with regulations, the company must ensure that...
To ensure that none of the data stored in the Amazon S3 bucket is accessible from outside the company's AWS environment, the company needs to ensure private network access to both Amazon Redshift and the S3 data, and possibly restrict external access to the data from third-party providers.
Key requirements:
- Ensure data is only accessible within the company's AWS environment.
- Protect access from third-party data providers.
- Enable access to data using Amazon Redshift Spectrum for querying the data stored in S3.
Option Analysis:
A) Replace the existing Redshift cluster with a new Redshift cluster that is in a private subnet. Use an interface VPC endpoint to connect to the Redshift cluster. Use a NAT gateway to give Redshift access to the S3 bucket.
- Reasoning: This option ensures that Amazon Redshift operates within a private subnet and can access the S3 bucket securely using a VPC endpoint. This approach avoids the need for the Redshift cluster to access the internet through a public IP, which ensures that no data is exposed outside the AWS environment. A NAT gateway is necessary for allowing outbound access from the private subnet to S3, ensuring connectivity without external exposure.
- Why selected: By using a private subnet for Redshift, along with a VPC endpoint and NAT gateway, this solution ensures all access to data is confined to the AWS environment, preventing external access while allowing the Redshift cluster to interact with the S3 data through Redshift Spectrum.
B) Create an AWS CloudHSM hardware security module (HSM) for each data provider. Encrypt each data provider's data by using the corresponding HSM for each data provider.
- Reasoning: AWS CloudHSM is used for secure key management and hardware-based encryption. While it ensures strong encryption, it does not directly address the need to restrict access to data stored in S3 or prevent external access to the AWS environment.
- Why rejected: CloudHSM helps with encryption but does not enforce network or access control restrictions. It also adds unnecessary complexity without solving the core requirement of preventing external access to the data in S3 or Redshift. Encryption alone does not provide access control.
C) Turn on enhanced VPC routing for the Amazon Redshift cluster. Set up an AWS Direct Connect connection and configure a connection between each data provider and the compan...
Author: Ella · Last updated Jul 30, 2026
Files from multiple data sources arrive in an Amazon S3 bucket on a regular basis. A data engineer wants to ingest new files into Amazon Redshift in near real time when the ne...
To meet the requirement of ingesting new files into Amazon Redshift in near real time when the new files arrive in the S3 bucket, the solution must automatically trigger the ingestion process as soon as the files are available. Let’s analyze the options:
A) Use the query editor v2 to schedule a COPY command to load new files into Amazon Redshift.
- Rejected reason: This solution involves scheduling a command to load files into Amazon Redshift using the COPY command. While this approach can load data into Redshift, it does not offer near real-time ingestion. It relies on a scheduled process, which means there will be a delay between when new files arrive and when they are ingested. This doesn't meet the requirement of near real-time ingestion of files as soon as they arrive.
B) Use the zero-ETL integration between Amazon Aurora and Amazon Redshift to load new files into Amazon Redshift.
- Rejected reason: The zero-ETL integration between Amazon Aurora and Amazon Redshift is designed for continuous data transfer between Aurora and Redshift. However, this solution does not apply to data that is stored in S3. This approach is specific to Amazon Aurora and would not work with files stored in S3,...
Author: Ella · Last updated Jul 30, 2026
A technology company currently uses Amazon Kinesis Data Streams to collect log data in real time. The company wants to use Amazon Redshift for downstream real-time queries and to enrich the log data.Wh...
To select the best solution for ingesting data from Amazon Kinesis Data Streams into Amazon Redshift with the least operational overhead, we need to focus on minimizing complexity and manual processes. Let's evaluate the options:
A) Set up an Amazon Kinesis Data Firehose delivery stream to send data to a Redshift provisioned cluster table.
- Reasoning: Amazon Kinesis Data Firehose is a fully managed service that can deliver streaming data to Amazon Redshift with minimal configuration and management. It automatically batches, buffers, and compresses the data as it is delivered to Redshift, making this a low-maintenance solution. Additionally, Firehose integrates seamlessly with Redshift, allowing real-time ingestion without requiring complex custom applications.
- Why selected: This solution has the least operational overhead as it is fully managed, requires minimal setup, and handles the data transformation and loading directly into Redshift with no need for additional infrastructure or manual batch processes.
B) Set up an Amazon Kinesis Data Firehose delivery stream to send data to Amazon S3. Configure a Redshift provisioned cluster to load data every minute.
- Reasoning: While this solution leverages Kinesis Data Firehose and S3 as an intermediary, it requires manual configuration to periodically load data from S3 into Redshift. The scheduled data load process introduces additional operational complexity because the company would need to handle frequent S3-to-Redshift loads, which could become cumbersome as the volume of data increases.
- Why rejected: While this option is technically valid, it introduces extra operational overhead because the data needs to be loaded from S3 into Redshift periodically, making it more complex than directly delivering data to Redshift.
...
Author: ElectricLionX · Last updated Jul 30, 2026
A company maintains a data warehouse in an on-premises Oracle database. The company wants to build a data lake on AWS. The company wants to load data warehouse tables into Amazon S3 and synchronize the tables with incremental data that arrives from the data warehouse every day.Each table has a column that contains monotonically increasing values. The size of each table is less than 50 GB. The data warehouse tables are refreshed every nigh...
Solution Evaluation:
The company has the requirement to synchronize data from the on-premises Oracle database to Amazon S3, specifically focusing on incremental data loading with minimal operational overhead. The solution must ensure daily updates while also keeping operational tasks simple.
Option A: Use an AWS Database Migration Service (AWS DMS) full load plus CDC job to load tables that contain monotonically increasing data columns from the on-premises data warehouse to Amazon S3. Use custom logic in AWS Glue to append the daily incremental data to a full-load copy that is in Amazon S3.
- Reasoning: AWS DMS can handle the full load and change data capture (CDC) processes, ensuring that the initial migration of data is done efficiently. After the initial load, AWS Glue would append incremental data based on the monotonically increasing column each day. This approach is a hybrid solution, combining DMS for the initial load and incremental data capture, and AWS Glue for custom logic to manage data updates.
- Why selected: This approach provides incremental data synchronization and handles the full-load data efficiently with minimal manual intervention. DMS can take care of CDC with minimal setup, and AWS Glue allows for automated processing of the incremental data. Although it requires some custom logic, it offers operational flexibility and scalability.
Option B: Use an AWS Glue Java Database Connectivity (JDBC) connection. Configure a job bookmark for a column that contains monotonically increasing values. Write custom logic to append the daily incremental data to a full-load copy that is in Amazon S3.
- Reasoning: AWS Glue can connect to the database via JDBC, and the job bookmark feature can track changes for incremental data. However, this solution requires custom logic to append the incremental data to the existing full-load copy in S3. The use of job bookmarks may also require careful configuration, and managing the custom logic for incremental updates might add complexity.
- ...
Author: Noah Williams · Last updated Jul 30, 2026
A company is building a data lake for a new analytics team. The company is using Amazon S3 for storage and Amazon Athena for query analysis. All data that is in Amazon S3 is in Apache Parquet format.The company is running a new Oracle database as a source system in the company=E2=80=99s data center. The company has 70 tables in the Oracle database. All the tables have primary keys. Data can occ...
Let's go through each option to evaluate which one meets the requirements with the least effort, considering key factors like effort, scalability, ease of integration, and future maintainability:
Option A: Create an Apache Sqoop job in Amazon EMR to read the data from the Oracle database. Configure the Sqoop job to write the data to Amazon S3 in Parquet format.
- Pros:
- Apache Sqoop is specifically designed to import data from relational databases like Oracle into Hadoop-based storage systems, so it works well for bulk data transfer.
- Amazon EMR can handle the workload efficiently.
- Sqoop supports writing to various formats, including Parquet.
- Cons:
- The solution requires setting up and maintaining an Amazon EMR cluster, which adds complexity and operational overhead.
- It may require custom scripting for incremental data updates, which could be cumbersome to manage.
- It doesn’t provide an out-of-the-box solution for handling changes in the source database (like updates or deletes).
This option would be more complex to manage and would require additional work to handle incremental changes (as the company needs to handle occasional changes to data).
Option B: Create an AWS Glue connection to the Oracle database. Create an AWS Glue bookmark job to ingest the data incrementally and to write the data to Amazon S3 in Parquet format.
- Pros:
- AWS Glue provides an integrated, serverless environment for ETL (Extract, Transform, Load) jobs, which reduces the management overhead.
- Glue has built-in support for Parquet format, and Glue bookmarks help to track changes, enabling incremental loads.
- Glue can integrate seamlessly with Amazon S3 and supports various data formats, including Parquet.
- It requires less setup and maintenance than setting up a custom solution like Apache Sqoop on EMR.
- Cons:
- While Glue simplifies the integration, it may require some initial setup to define the schema and configure the job, but once set up, it handles regular ETL tasks efficiently.
- Performance can be a concern for very large datasets, depending on the job configurations.
AWS Glue is a managed service that can handle the incremental data ingestion seamlessly, making this a strong contender for the least effort solution.
Option C: Create an AWS Database Migration Service (AWS DMS) task for ongoing replication. Set the Oracle database as the source. Set Amazon S3 as the target. Configure the task to write the data in Parquet f...
Author: Aria · Last updated Jul 30, 2026
A transportation company wants to track vehicle movements by capturing geolocation records. The records are 10 bytes in size. The company receives up to 10.000 records every second. Data transmission delays of a few minutes are acceptable because of unreliable network conditions.The transportation company wants to use Amazon Kinesis Data Streams to ingest the geolocation data. The company needs a reliable mechanism to send data t...
Let's analyze each of the options in relation to the requirements of the transportation company:
Option A: Kinesis Agent
- Pros:
- Kinesis Agent is a pre-built, open-source application that is easy to deploy. It's particularly suited for streamlining data ingestion from local files, logs, or other systems.
- It can automatically push data to Kinesis Data Streams or Kinesis Data Firehose.
- It handles retries and basic error handling in case of transmission delays or network issues.
- Cons:
- Kinesis Agent is generally more suited for file-based data sources, such as logs or data stored in files, rather than real-time streaming of small records (like geolocation data).
- It doesn’t provide the same level of throughput control and efficiency optimization as some of the other options.
- It's less configurable compared to the KPL when trying to maximize throughput efficiency.
When to use: Best for log or file-based data ingestion, but not ideal for maximizing throughput efficiency for high-frequency data like geolocation records.
Option B: Kinesis Producer Library (KPL)
- Pros:
- KPL is specifically designed for sending high-volume, high-throughput data to Kinesis Data Streams with minimal operational overhead.
- It automatically batches records to maximize throughput, thus optimizing the use of available shard capacity, which is critical for this use case (sending 10,000 records per second).
- It handles retries, error management, and backpressure effectively.
- It is ideal for real-time, low-latency streaming use cases.
- Cons:
- KPL requires custom application development, so there is more initial setup compared to some other options.
- You need to configure the KPL with appropriate buffer sizes and retry strategies, which adds some complexity.
When to use: The ideal option when you need to efficiently stream a high volume of data to Kinesis, especially for low-latency, real-time use cases. This option fits the requirement to maximize throughput efficiency and send geolocation data.
Option C: Amazon Kinesis Data Firehose
- Pros:
- Kinesis Data Firehose is a fully ...
Author: Aditya · Last updated Jul 30, 2026
An investment company needs to manage and extract insights from a volume of semi-structured data that grows continuously.A data engineer needs to deduplicate the semi-structured data, remove records that are duplicates, and remove common mis...
Let's evaluate each of the options based on the investment company's needs: deduplication, handling semi-structured data, and removing common misspellings, all while minimizing operational overhead.
Option A: Use the FindMatches feature of AWS Glue to remove duplicate records.
- Pros:
- FindMatches is a feature within AWS Glue that can perform deduplication of records based on fuzzy matching, which is ideal for identifying and removing duplicates and misspellings.
- AWS Glue is a fully managed ETL service, meaning it abstracts away much of the operational overhead.
- It can handle semi-structured data (like JSON, Parquet) well, making it suitable for this scenario.
- Glue automatically scales to handle growing data volumes, which aligns with the company's requirement for managing continuously growing data.
- Cons:
- While it’s effective for deduplication, it requires configuring AWS Glue jobs, but this is minimal compared to manually handling deduplication logic.
When to use: This is the most suitable option as it directly addresses the deduplication requirement and handles both exact matches and fuzzy matching for misspelled records with minimal manual intervention.
Option B: Use non-Windows functions in Amazon Athena to remove duplicate records.
- Pros:
- Amazon Athena can query semi-structured data stored in S3 using standard SQL.
- It is serverless, so there’s no need to manage infrastructure.
- Athena can remove exact duplicates using SQL commands like `DISTINCT`.
- Cons:
- Athena does not have native fuzzy matching capabilities for misspellings. While you can use SQL to deduplicate exact matches, misspelled duplicates would require additional custom logic and possibly complicated SQL queries (e.g., using regular expressions or approximate string matching).
- For large-scale, continuously growing data, Athena may not be the most efficient or least operationally burdensome solution compared to fully managed services like AWS Glue, especially if ongoing processing is needed.
When to use: This is viable if the primary need is for deduplication of exact matches and if misspellings are not a significant concern. However, additional e...
Author: David · Last updated Jul 30, 2026
A company is building an inventory management system and an inventory reordering system to automatically reorder products. Both systems use Amazon Kinesis Data Streams. The inventory management system uses the Amazon Kinesis Producer Library (KPL) to publish data to a stream. The inventory reordering system uses the Amazon Kinesis Client Library (KCL) to consume data from the stream. The company configures the stream to scale up and down as needed.Before the company deploys ...
Let's evaluate each factor in terms of the cause of duplicated data in the inventory reordering system:
Option A: The producer experienced network-related timeouts.
- Pros:
- If the producer (using KPL) experiences network timeouts, it may retry sending records to the stream.
- Kinesis Producer Library (KPL) automatically retries failed record sends and ensures that the data is successfully placed into the stream. However, retries could result in duplicate records if the producer does not detect that the record was already successfully placed into the stream.
- Cons:
- KPL is designed to handle retries gracefully and is meant to avoid sending duplicate data unless the retry mechanism fails. The network timeouts on their own would not directly lead to duplicates, as the KPL handles such retries.
When to use: This could be a contributing factor but is not the most likely cause of duplicates, as the KPL should generally ensure idempotency for retrying records.
Option B: The streams value for the IteratorAgeMilliseconds metric was too high.
- Pros:
- The IteratorAgeMilliseconds metric indicates how far behind the consumer is from the latest data in the stream. A high IteratorAge might suggest that the consumer is not processing data as quickly as the producer is publishing, causing the consumer to reprocess the same data.
- Cons:
- A high IteratorAge itself doesn’t cause duplication directly. It simply indicates lag, which might result in the consumer missing some records or being behind in processing. Duplication would occur if the consumer retried reading the same data due to improper handling of the record's state or sequence.
When to use: While a high IteratorAge could contribute to an issue in record processing, it's not the root cause of data duplication. The duplication is more likely related to how the consumer (using KCL) handles the stream.
Option C: There was a change in the number of shards, record processors, or both.
- Pros:
- Scaling the stream by changing the number of shards can cause duplication if the KCL doesn't properly handle the reassignment of shard processing. When the number of shards changes, KCL's record processors may be reassigned, and in some cases, the same record can be consumed more than once.
- This is a common cause of duplication in systems using Kinesis and KCL because KCL uses checkpoints to track its progress, a...
Author: Ella · Last updated Jul 30, 2026
An ecommerce company operates a complex order fulfilment process that spans several operational systems hosted in AWS. Each of the operational systems has a Java DatabaseConnectivity (JDBC)-compliant relational database where the latest processing state is captured.The company needs to give an operations team the ability to track o...
Let's break down each of the options to determine the solution that meets the requirement with the least development overhead:
Option A: Use AWS Glue to build ingestion pipelines from the operational systems into Amazon Redshift. Build dashboards in Amazon QuickSight that track the orders.
- Pros:
- Amazon Redshift is a powerful data warehouse that is optimized for analytical queries, making it well-suited for tracking and analyzing large datasets, such as orders.
- AWS Glue can easily extract, transform, and load (ETL) data from relational databases to Redshift, with built-in support for JDBC.
- Amazon QuickSight can seamlessly connect to Redshift and provide robust dashboarding capabilities.
- Cons:
- Setting up Redshift and Glue for incremental data ingestion can require some effort in terms of ETL pipeline design and configuration.
- For tracking orders in near real-time on an hourly basis, Redshift's batch-oriented architecture may not provide the best performance, especially if the operational systems have frequent updates.
- The initial setup and configuration of Redshift could be more complex and require careful monitoring.
While this option is a solid choice for handling large-scale data with analytical needs, it could involve more setup compared to other options, particularly for continuous or near real-time data tracking.
Option B: Use AWS Glue to build ingestion pipelines from the operational systems into Amazon DynamoDB. Build dashboards in Amazon QuickSight that track the orders.
- Pros:
- Amazon DynamoDB is a managed NoSQL database that supports high availability and can handle high throughput, making it suitable for applications that need to access and update order tracking in real-time.
- AWS Glue can build ETL pipelines to move data from relational databases into DynamoDB.
- QuickSight integrates easily with DynamoDB, and dashboards can be created to visualize order data.
- Cons:
- DynamoDB is not ideal for running complex queries or analytics at scale (compared to Redshift). Its query capabilities are more limited, and dashboards may not perform well with large datasets or complex join operations.
- DynamoDB can handle high-throughput, but querying and aggregating large volumes of order data might not be as efficient for detailed analysis.
While DynamoDB provides low-latency access and could be suitable for fast updates, it is not the best choice for complex analytical queries, especially when it comes to visualization of large-scale data across multiple operational systems.
Option C: Use AWS Database Migration Service (AWS DMS) to capture changed records in the operational systems. Publish the changes to an Amazon DynamoDB table in a different AWS region from the source database. Build Grafana dashboards that tra...
Author: Layla · Last updated Jul 30, 2026
A data engineer needs to use Amazon Neptune to develop graph applications.Which programming languages should the engineer...
To develop graph applications on Amazon Neptune, the engineer needs to use graph query languages designed for graph databases. Let’s evaluate each option:
A) Gremlin
Reason for Selection: Gremlin is a graph traversal language that is supported by Amazon Neptune. Neptune provides full support for the Apache TinkerPop 3 specification, which includes Gremlin for graph traversal. Gremlin is designed specifically for graph databases and is ideal for traversing nodes and edges in a graph structure. It is a popular choice for querying graph databases that follow the property-graph model, making it very suitable for use with Amazon Neptune.
B) SQL
Reason for Rejection: SQL is a relational query language used primarily for querying relational databases (RDBMS). Amazon Neptune, being a graph database, is not designed to support SQL for graph-specific queries. SQL is not well-suited for performing graph-specific operations, such as node and edge traversals, which are central to graph databases. Therefore, SQL is not the right tool for developing graph applications on Amazon Neptune.
C) ANSI SQL
Reason for Rejection: ANSI SQL is a standard for relational databases, much like SQL. As previously mentioned, Amazon Neptune is a graph data...
Author: ElectricLionX · Last updated Jul 30, 2026
A mobile gaming company wants to capture data from its gaming app. The company wants to make the data available to three internal consumers of the data. The data records are approximately 20 KB in size.The company wants to achieve optimal throughput from each device that runs the gaming app. Additionally, the company wants to develop an application to proc...
To meet the mobile gaming company's requirements, let's break down each option in detail:
A) Configure the mobile app to call the PutRecords API operation to send data to Amazon Kinesis Data Streams. Use the enhanced fan-out feature with a stream for each internal consumer.
Reason for Selection:
- Kinesis Data Streams is designed for high-throughput data ingestion, and calling the `PutRecords` API operation to send data is suitable for capturing records of approximately 20 KB in size, which is efficient for Kinesis Data Streams.
- Enhanced fan-out provides dedicated throughput for each internal consumer. This is critical because it ensures that each of the three internal consumers has independent and dedicated throughput without sharing resources. Enhanced fan-out is an ideal solution for multiple consumers requiring parallel processing of data streams with high throughput.
- The setup also allows the company to process data in real time and efficiently distribute data to internal consumers.
B) Configure the mobile app to call the PutRecordBatch API operation to send data to Amazon Kinesis Data Firehose. Submit an AWS Support case to turn on dedicated throughput for the company’s AWS account. Allow each internal consumer to access the stream.
Reason for Rejection:
- Kinesis Data Firehose is primarily designed for real-time data streaming to other destinations like S3, Redshift, or Elasticsearch. However, it doesn't support dedicated throughput for each internal consumer natively. This could make it harder to guarantee the specific throughput for each consumer as required.
- Also, the PutRecordBatch operation is more useful for batching data, but Firehose does not provide the level of granularity for managing throughput per consumer, unlike Kinesis Data Streams with enhanced fan-out.
C) Configure the mobile app to use the Amazon Kinesis Producer Library (KPL) to send data to Amazon Kinesis Data Firehose. Use the e...
Author: Sofia · Last updated Jul 30, 2026
A retail company uses an Amazon Redshift data warehouse and an Amazon S3 bucket. The company ingests retail order data into the S3 bucket every day.The company stores all order data at a single path within the S3 bucket. The data has more than 100 columns. The company ingests the order data from a third-party application that generates more than 30 files in CSV format every day. Each CSV file is between 50 and 70 MB in size.The company uses Amazon Redshift Spectrum to run queries that select sets of columns. Users aggregate metrics based on daily orders. Recen...
To resolve the performance issues when querying data using Amazon Redshift Spectrum, we need to focus on optimizing how the data is stored, partitioned, and queried. Let’s evaluate each option in detail:
A) Configure the third-party application to create the files in a columnar format.
Reason for Selection:
- Columnar formats like Parquet or ORC are highly efficient for querying specific columns, especially in Amazon Redshift Spectrum. Since the company queries only a subset of columns from the 100+ columns in the dataset, columnar formats store data more efficiently by reducing the amount of data that needs to be read during queries.
- Columnar storage reduces I/O and improves query performance because only the necessary columns are read. This format is optimized for analytical workloads, which fits the company’s use case of aggregating metrics based on daily orders.
- The files in CSV format are row-based, leading to unnecessary data being read and increasing query time. Switching to a columnar format will address the performance degradation effectively.
B) Develop an AWS Glue ETL job to convert the multiple daily CSV files to one file for each day.
Reason for Rejection:
- Although consolidating multiple files into a single daily file can help with query performance by reducing the number of files that need to be scanned, it does not address the underlying issue of inefficient file formats for querying. It still involves reading from a non-columnar CSV format, which leads to high I/O.
- This step adds complexity to the process (additional Glue job development), and while it might provide some performance improvement, it won’t be as effective as switching to a columnar format, especially when paired with partitioning.
C) Partition the order data in the S3 bucket based on order date.
Reason for Selection:
- Partitioning the data in Amazon S3 by order date (or another logical partition key) helps Amazon Redshift Spectrum query only the relevant partitions, reducing the amount of data scanned and improving performance.
- This is es...
Author: Layla · Last updated Jul 30, 2026
A company stores customer records in Amazon S3. The company must not delete or modify the customer record data for 7 years after each record is created. The root user also must not have the ability to delete or modify the data.A data...
To meet the company's requirement to prevent data from being deleted or modified for 7 years and ensure that even the root user cannot delete or modify the data, let’s evaluate each option in detail:
A) Enable governance mode on the S3 bucket. Use a default retention period of 7 years.
Reason for Rejection:
- Governance mode allows users with specific permissions (but not the root user) to bypass object lock settings, meaning that even though the data would be locked for 7 years, an authorized user could potentially override this setting and modify or delete the data.
- Since the company requires that the root user cannot modify or delete the data, governance mode does not provide the necessary level of protection because the root user has more permissive access in governance mode.
B) Enable compliance mode on the S3 bucket. Use a default retention period of 7 years.
Reason for Selection:
- Compliance mode in Amazon S3 Object Lock ensures that once data is locked, it cannot be modified or deleted by any user, including the root user, for the duration of the retention period (in this case, 7 years).
- This mode provides the highest level of security, guaranteeing that the data will remain immutable for the required period, meeting the company's requirement that no one (including the root user) can alter or delete the data.
- Compliance mode is specifically designed for regulatory compliance use cases, such as this one where data must not be modified or deleted for ...
Author: Lucas · Last updated Jul 30, 2026
A data engineer needs to create a new empty table in Amazon Athena that has the same schema as an existing table named old_table.Which SQL sta...
To create a new empty table in Amazon Athena with the same schema as an existing table (`old_table`), the goal is to replicate the structure (schema) of the old table without copying any data. Let's evaluate each option:
A) CREATE TABLE new_table AS SELECT FROM old_table;
Reason for Rejection:
- This query would create a new table `new_table` by selecting all the data from `old_table`. While this creates a table with the same schema, it also copies all the data from `old_table` into `new_table`.
- The requirement specifies that the new table should be empty, so this option is not suitable because it will populate the new table with data from the old one.
B) INSERT INTO new_table SELECT FROM old_table;
Reason for Rejection:
- This option assumes that `new_table` already exists and is an empty table. It would insert all the data from `old_table` into `new_table`.
- The requirement is to create a new empty table with the same schema, so the insertion of data is not needed, and this option does not address the creation of a table with the same schema...
Author: Ava · Last updated Jul 30, 2026
A data engineer needs to create an Amazon Athena table based on a subset of data from an existing Athena table named cities_world. The cities_world table contains cities that are located around the world. The data engineer must create a new table named cities_us to contain only the cit...
Let's analyze the options provided one by one:
A) INSERT INTO cities_usa (city, state) SELECT city, state FROM cities_world WHERE country = 'usa';
- Analysis: This option uses an `INSERT INTO` statement to insert the selected data (cities located in the US) into the `cities_usa` table. It only selects the `city` and `state` columns from the `cities_world` table, where the `country` is equal to "usa".
- Why it works: The correct syntax for inserting data into an existing table is `INSERT INTO`, and selecting rows with a specific condition using `WHERE` (in this case, `country = 'usa'`) is the right approach for filtering the data.
- Why it's preferred: This query is appropriate as it inserts the filtered data into an existing `cities_usa` table without modifying the structure of either table.
B) MOVE city, state FROM cities_world TO cities_usa WHERE country = 'usa';
- Analysis: The `MOVE` statement does not exist in SQL syntax, specifically not in Amazon Athena. Athena supports the SQL language that does not include a `MOVE` command.
- Why it's rejected: This is not valid SQL syntax and would throw an error when executed in Athena. Therefore, it cannot be used.
C) INSERT INTO cities_usa SELECT city, state FROM cities_world WHERE country = 'usa';
- Analysis: Similar to option A, this query inserts data into the `cities_usa` table based on the selected columns (`city` and `state`) from `cities_world` where the `country` is 'usa'.
- Why it works: This query is correctly formatted for inserting data into a table in Athena and is a valid approach. It selects and inserts the necessary data as required.
- Why it's similar to option A: ...
Author: Olivia · Last updated Jul 30, 2026
A company implements a data mesh that has a central governance account. The company needs to catalog all data in the governance account. The governance account uses AWS Lake Formation to centrally share data and grant access permissions.The company has created a new data product that includes a group of Amazon Redshift Serverless tables. A data engineer needs to share the data product with a marketing team. The marketing team must have access to only a subset of columns. The data engineer needs to share the same data product wit...
Let's go through the options one by one:
A) Create views of the tables that need to be shared. Include only the required columns.
- Analysis: Creating views is a common approach when you need to share specific columns or subsets of data from a larger table. A view allows the data engineer to expose only the columns that should be visible to a specific user or team (in this case, the marketing team and the compliance team).
- Why it works: By creating views with only the necessary columns, the data engineer can control the access to data, ensuring that both teams see only their respective required columns. This satisfies the requirement of granting different subsets of data to different teams.
- Why it's preferred: Views offer a flexible, secure way to share only a subset of data without modifying the underlying tables or creating redundant copies of data. This approach is commonly used in scenarios where data sharing needs to be granular and controlled.
B) Create an Amazon Redshift data share that includes the tables that need to be shared.
- Analysis: Creating an Amazon Redshift data share allows data to be shared across different Redshift clusters. However, this does not inherently provide the ability to limit access to specific columns from the shared tables.
- Why it's rejected: While data sharing via Redshift is useful for cross-account or cross-cluster data access, it does not allow fine-grained control over specific columns. This option alone wouldn't satisfy the requirement to provide different subsets of columns to different teams. So, it's not a complete solution.
C) Create an Amazon Redshift managed VPC endpoint in the marketing team's account. Grant the marketing team access to the views.
- Analysis: A managed VPC endpoint allows secure connectivity between Redshift clusters across accounts. This step might be necessary if the marketing team is in a different AWS account than the central governance account. However, it doesn't directly address the need for column-level access control.
- Why it's rejected: While it could be part of the solution for network connectivity and cro...
Author: IceDragon2023 · Last updated Jul 30, 2026
A company has a data lake in Amazon S3. The company uses AWS Glue to catalog data and AWS Glue Studio to implement data extract, transform, and load (ETL) pipelines.The company needs to ensure that data quality issues are checked every time the pipelines run. A data engineer must enhance the existing pipelines to eva...
Let's analyze the options one by one:
A) Add a new transform that is defined by a SQL query to each Glue ETL job. Use the SQL query to implement a ruleset that includes the data quality rules that need to be evaluated.
- Analysis: Adding a SQL query transform is a way to check data quality, but SQL queries alone would require a custom implementation for defining data quality rules, and they would not have built-in integration with data quality frameworks.
- Why it's rejected: While it can be done, SQL alone does not provide a dedicated mechanism for data quality checks and is not the most efficient way to evaluate predefined thresholds. It would involve more manual effort for rule creation, lack of flexibility, and would not be as easily maintained as other options specifically designed for data quality.
B) Add a new Evaluate Data Quality transform to each Glue ETL job. Use Data Quality Definition Language (DQDL) to implement a ruleset that includes the data quality rules that need to be evaluated.
- Analysis: The "Evaluate Data Quality" transform is a built-in feature in AWS Glue. It uses the Data Quality Definition Language (DQDL) to define and evaluate rules. This is a purpose-built solution for handling data quality, where you can specify rules for thresholds, completeness, and other aspects of data quality.
- Why it works: This option directly meets the requirement to ensure that data quality issues are checked with minimal effort. The DQDL is specifically designed to be simple, and AWS Glue’s native integration with this transform streamlines implementation, making it the least effortful solution. It is efficient and designed for this exact use case.
- Why it's preferred: It requires minimal implementation effort because it’s fully integrated with AWS Glue, and you don't need to worry about third-party libraries or manual rule handling. It's a clean, declarative approach to handling data quality within Glue pipelines.
C) Add a new custom transform to each Glue ETL job. Use the PyDeequ library to implement a ruleset that includes the data quality rules that need t...
Author: Aria · Last updated Jul 30, 2026
A company has an application that uses a microservice architecture. The company hosts the application on an Amazon Elastic Kubernetes Services (Amazon EKS) cluster.The company wants to set up a robust monitoring system for the application. The company needs to analyze the logs from the EKS cluster and the application. The company needs to correlate the cluster's logs with the application's traces ...
Let's analyze the options to determine the best approach for setting up a robust monitoring system for the application and correlating logs with traces:
A) Use FluentBit to collect logs. Use OpenTelemetry to collect traces.
- Analysis: FluentBit is a lightweight log shipper commonly used to collect logs from Kubernetes clusters, such as Amazon EKS. OpenTelemetry is a framework used to collect, generate, and export traces from distributed systems, including Kubernetes.
- Why it works: FluentBit can efficiently collect logs from the EKS cluster, and OpenTelemetry can collect application traces. Both tools are widely supported in modern microservice architectures and can integrate seamlessly with cloud monitoring tools.
- Why it's preferred: This option directly addresses both log collection and trace collection with minimal custom development. Both FluentBit and OpenTelemetry are compatible with AWS services and are designed to work well in Kubernetes environments.
- Why it's better than other options: FluentBit is a cost-effective, simple way to collect logs, and OpenTelemetry provides native tracing functionality. Additionally, both are well-supported within AWS monitoring systems, and the integration with Amazon CloudWatch (which stores logs and traces) is straightforward.
B) Use Amazon CloudWatch to collect logs. Use Amazon Kinesis to collect traces.
- Analysis: Amazon CloudWatch can be used to collect and store logs from various AWS resources, including EKS clusters. However, Amazon Kinesis is generally used for stream processing and would require additional setup and custom development to collect traces.
- Why it's rejected: Kinesis is not primarily designed for trace collection. While it is great for real-time data streaming, it is not the best tool for correlating application traces. Amazon CloudWatch itself has integrated tracing capabilities through CloudWatch ServiceLens, which provides application monitoring and trace collection.
C) Use Amazon CloudWatch to collect logs. Use Amazon Managed Streaming for Apache Kafka (Amazon MSK) to collect traces.
- Analysis: While Amazon CloudWatch is an excellent choice for collecting logs, using Amazon MSK (Kafka) to collect traces introduces unnecessary complexity. Kafka is generally used for message streaming, not trace collection.
- Why it's rejected: Kafka does not provide native support for application tracing and requ...
Author: IceDragon2023 · Last updated Jul 30, 2026
A company has a gaming application that stores data in Amazon DynamoDB tables. A data engineer needs to ingest the game data into an Amazon OpenSearch Service cluster. Data upda...
Let's analyze each option and determine the best solution to ingest game data from Amazon DynamoDB into Amazon OpenSearch Service in near real time:
A) Use AWS Step Functions to periodically export data from the Amazon DynamoDB tables to an Amazon S3 bucket. Use an AWS Lambda function to load the data into Amazon OpenSearch Service.
- Analysis: This solution involves periodic exports of data using Step Functions, which will then be loaded into OpenSearch using Lambda. However, the key issue is that it uses a periodic batch process rather than handling real-time data updates. This approach does not ensure near real-time updates, as there would be delays between each export and data processing cycle.
- Why it's rejected: This option does not meet the near-real-time requirement, as the data is processed periodically. For near real-time updates, a more continuous, event-driven solution is needed.
B) Configure an AWS Glue job to have a source of Amazon DynamoDB and a destination of Amazon OpenSearch Service to transfer data in near real time.
- Analysis: AWS Glue is generally used for batch processing and ETL workflows. While it supports various data sources and destinations, including DynamoDB and OpenSearch, it is not optimized for real-time or near real-time data updates. Glue jobs are typically scheduled and may not be the best fit for continuous, near real-time data transfer.
- Why it's rejected: AWS Glue is not ideal for near real-time data transfer, as it is generally used for batch-based ETL workflows. Real-time updates would require a more immediate data flow mechanism.
C) Use Amazon DynamoDB Streams to capture table changes. Use an AWS Lambda function to process and update the data in Amazon OpenSearch Service.
- Analysis: This solution leverages DynamoDB Streams, which captures changes (inserts, updates, deletes) in real time as they occur in DynamoDB tables. A Lambda fu...
Author: VioletCheetah55 · Last updated Jul 30, 2026
A company uses Amazon Redshift as its data warehouse service. A data engineer needs to design a physical data model.The data engineer encounters a de-normalized table that is growing in size. The table does not have a suitable column to use as the distribution key.Wh...
When designing a physical data model for Amazon Redshift, the distribution style determines how data is distributed across the compute nodes. The choice of distribution style has a significant impact on query performance and maintenance overhead. Here's a breakdown of each distribution style and how it fits the scenario of a growing, de-normalized table without a suitable column for a distribution key:
A) ALL Distribution:
- How it works: This style distributes a full copy of the table to all compute nodes.
- Advantages: It can be very efficient for small dimension tables or lookup tables because all nodes have the entire data, thus avoiding shuffling during joins.
- Disadvantages: The maintenance overhead can be high when the table grows, especially if it contains a large number of rows, as the full copy of the table is distributed to every node. This increases storage and can degrade performance as the table size grows.
- When to use: It's best suited for small lookup or dimension tables, not large fact tables or tables that are growing in size.
B) EVEN Distribution:
- How it works: Data is distributed evenly across the nodes without considering any column. This is typically used when there isn't a good choice of a distribution key.
- Advantages: It ensures even distribution of data, which can prevent "hotspots" on specific nodes. There’s minimal maintenance overhead.
- Disadvantages: Since there's no relation to join patterns or data, it could lead to unnecessary shuffling during queries, especially for large fact tables.
- When to use: This is a good option when the table does not have a natural distribution key and is not frequently joined with other tables. It can also be suitable when there’s no performance requirement for joins or when the table is not growing rapidly.
C) AUTO Distribution:
- How it works: Amazon Redshift automatically selects the most appropriate distribution style based on the size of the table. For small tables, it chooses ALL distribution, for large tables it s...
Author: IronLion88 · Last updated Jul 30, 2026
A retail company is expanding its operations globally. The company needs to use Amazon QuickSight to accurately calculate currency exchange rates for financial reports. The company has an existing dashboard that includes a visual that is based on an analysis of a dataset that contains global currency values and exchange rates.A data engineer needs to ensure that exchange rates are calculated with a precision of four decimal places. The cal...
To meet the requirement of ensuring that currency exchange rates are calculated with four decimal places, precomputed, and materialized in Amazon QuickSight's SPICE (Super-fast, Parallel, In-memory Calculation Engine), we need to consider the context in which the calculation should be defined and how QuickSight processes calculations.
Evaluation of options:
1. A) Define and create the calculated field in the dataset:
- Description: Calculated fields in the dataset are computed during the dataset's refresh process, before any analysis or visualizations. These fields are materialized in SPICE and stored as part of the dataset.
- Pros: This ensures that the calculations are precomputed, and results are stored in SPICE, allowing for fast and efficient access during any analysis or visualization. It also ensures the calculation uses the desired precision (e.g., four decimal places).
- Cons: If the calculation logic needs to change frequently or is only needed for a specific analysis, it might require re-refreshing the dataset, which could lead to overhead.
- Selected Option: This is the most optimal choice because it ensures that the exchange rate calculations are precomputed, stored, and available with high precision, leveraging SPICE’s performance.
2. B) Define and create the calculated field in the analysis:
- Description: This allows the calculated field to be created directly within an analysis, but the calculation occurs in real-time when the analysis is viewed, rather than being precomputed in SPICE.
- Pros: Allows for dynamic and flexible calculations in the context of the analysis, without needing to alter the dataset.
- Cons: Since the calculations aren't precomputed and stored in SPICE, performance may be slower, and the calculations won't be as efficient for large datasets. Additionally, t...
Author: Victoria · Last updated Jul 30, 2026
A company has three subsidiaries. Each subsidiary uses a different data warehousing solution. The first subsidiary hosts its data warehouse in Amazon Redshift. The second subsidiary uses Teradata Vantage on AWS. The third subsidiary uses Google BigQuery.The company wants to aggregate all the data into a central Amazon S3 data lake. The company wants to use Apache Iceberg as the table format.A data engineer needs to build a new pipeline to connect to...
Analysis of the Solution Options
To address the problem, we need a solution that can connect to all three data sources (Amazon Redshift, Teradata, and Google BigQuery), run transformations, join the data, and write it to an Apache Iceberg table in Amazon S3. Additionally, the solution should have the least operational effort, meaning minimal manual intervention and complexity.
Let’s break down the options and evaluate each one:
A) Use native Amazon Redshift, Teradata, and BigQuery connectors to build the pipeline in AWS Glue. Use native AWS Glue transforms to join the data. Run a Merge operation on the data lake Iceberg table.
- Advantages:
- AWS Glue supports native connectors for Amazon Redshift and Google BigQuery, making it easier to integrate.
- AWS Glue provides built-in transformation capabilities and supports various table formats, including Apache Iceberg.
- AWS Glue automates much of the ETL process, reducing operational effort.
- Disadvantages:
- While AWS Glue does support a native connector for Redshift and BigQuery, Teradata would require a custom connector, which could increase complexity.
- Glue transforms are somewhat limited in handling complex data processing, and building transformations might require more effort, especially for joining data from different sources.
- When to use:
- This option is appropriate when the focus is on a managed ETL service with minimal infrastructure setup. However, the need for a custom connector for Teradata and possible complexity in managing transformations makes this a bit cumbersome.
B) Use the Amazon Athena federated query connectors for Amazon Redshift, Teradata, and BigQuery to build the pipeline in Athena. Write a SQL query to read from all the data sources, join the data, and run a Merge operation on the data lake Iceberg table.
- Advantages:
- Athena supports federated queries, which allows querying data across multiple sources without moving data.
- SQL queries can be used to read from all the sources, simplifying the logic needed for transformations.
- Athena can be configured to work with Iceberg tables, so this solution can directly write data in the desired format.
- Disadvantages:
- Federated queries are useful for querying, but they can become inefficient or slower when handling large datasets, especially when involving multiple data sources.
- Athena may not handle complex transformation logic as easily as a more feature-rich solution like AWS Glue or Apache Spark.
- When to use:
- Best used when you want an easy-to-use, serverless solution for querying across different sources with minimal operational overhead. However, performance can be a concern when working with large datasets or needing complex data processing.
C) Use the native Amazon Redshift connector, the Java Database Connectivity (JDBC) connector for Teradata, and the open source Apache Spark BigQuery connector to build the pipeline in Amazon EM...
Author: Carlos Garcia · Last updated Jul 30, 2026
A data engineer needs to onboard a new data producer into AWS. The data producer needs to migrate data products to AWS.The data producer maintains many data pipelines that support a business application. Each pipeline must have service accounts and their corresponding credentials. The data engineer must establish a secure connection from the data producer's on-premises data cen...
Let's evaluate the options based on the requirements:
1. The data engineer needs to establish a secure connection from the on-premises data center to AWS.
2. The data engineer must ensure that no public internet is used for data transfer.
3. Each pipeline requires service accounts with corresponding credentials.
Option A: Instruct the new data producer to create Amazon Machine Images (AMIs) on Amazon Elastic Container Service (Amazon ECS) to store the code base of the application. Create security groups in a public subnet that allow connections only to the on-premises data center.
- Pros: ECS and AMIs are great for managing application code and containers. Security groups can control access to instances.
- Cons: The solution involves using public subnets and doesn't address the core requirement of avoiding the public internet for data transfer. Additionally, storing service account credentials directly within the containers is not secure.
- Rejection Reason: The data transfer would still involve public internet connectivity, which violates the requirement for avoiding public internet access.
Option B: Create an AWS Direct Connect connection to the on-premises data center. Store the service account credentials in AWS Secrets Manager.
- Pros: AWS Direct Connect provides a dedicated, secure connection from an on-premises data center to AWS, eliminating the use of the public internet. AWS Secrets Manager can securely store and manage service account credentials.
- Cons: This option doesn't mention specific actions regarding data transfer or providing the data producer with an easy way to manage data pipelines in AWS.
- Selected Option: This solution addresses all requirements securely: it avoids the public internet, ensures secure storage of credentials, and establishes the necessary secure connection. The use of AWS Direct Connect ensures high bandwidth, secure data transfer...
Author: VenomousSerpent42 · Last updated Jul 30, 2026
A data engineer configured an AWS Glue Data Catalog for data that is stored in Amazon S3 buckets. The data engineer needs to configure the Data Catalog to receive incremental updates.The data engineer sets up event notifications for the S3 bucket and creates an Amazon Simple Queue Service (Amazon SQS) queue to receive the ...
To meet the requirement of configuring the AWS Glue Data Catalog to receive incremental updates with least operational overhead, we need to choose a combination of solutions that will automate the update process based on S3 events without adding complexity. Let's analyze each option based on key factors like automation, maintenance overhead, and how well they fit with S3 event-based processing.
Option A: Create an S3 event-based AWS Glue crawler to consume events from the SQS queue.
- Reasoning: This option is ideal because it directly integrates AWS Glue with the event-driven architecture of S3 and SQS. The crawler can be triggered automatically by the SQS queue when an event is received (i.e., when new data is added to S3). The crawler will only update the Glue Data Catalog when necessary, so it’s efficient and scales with minimal manual intervention. This option is well-suited for automated incremental updates.
- Key Factors: Fully automated process, minimal manual intervention, integrates well with Glue, low operational overhead.
Option B: Define a time-based schedule to run the AWS Glue crawler, and perform incremental updates to the Data Catalog.
- Reasoning: While this is a valid approach, it introduces unnecessary overhead because it would run the crawler at fixed intervals, regardless of whether there were changes in the S3 data. It lacks the event-driven trigger that would ensure updates happen only when necessary. This is not as efficient as the event-based approach, which only runs when data changes.
- Key Factors: More operational overhead (since you need to define and maintain the schedule), not as efficient as an event-driven approach.
Option C: Use an AWS Lambda function to directly update the Data Catalog based on S3 events that the SQS queue receives.
- Reasoning: This option can also automate the process, but it adds complexity. The Lambda function would need to parse the SQS events, trigger an update in the Glue Data Catalog, and ha...
Author: Leo · Last updated Jul 30, 2026
A company uses AWS Glue Data Catalog to index data that is uploaded to an Amazon S3 bucket every day. The company uses a daily batch processes in an extract, transform, and load (ETL) pipeline to upload data from external sources into the S3 bucket.The company runs a daily report on the S3 data. Some days, the company runs the report before all the daily data has been uploaded to the S3 bucket. A data engineer must be able to send a me...
To meet the requirement of identifying incomplete data with the least operational overhead, we need a solution that is automated, integrated with existing services, and does not require significant infrastructure setup or maintenance. Let's evaluate each option based on these factors.
A) Create data quality checks for the source datasets that the daily reports use. Create a new AWS managed Apache Airflow cluster. Run the data quality checks by using Airflow tasks that run data quality queries on the columns' data type and the presence of null values. Configure Airflow Directed Acyclic Graphs (DAGs) to send an email notification that informs the data engineer about the incomplete datasets to the SNS topic.
- Explanation: Apache Airflow is a powerful orchestration tool, but managing an Airflow cluster requires significant operational overhead. It involves setting up, maintaining, and scaling the cluster, which increases complexity compared to other serverless solutions.
- Why rejected: While it’s an effective tool for orchestrating complex workflows, Airflow introduces operational overhead and complexity that is unnecessary for this use case. The goal is to minimize operational overhead, so Airflow would not be the best option here.
B) Create data quality checks on the source datasets that the daily reports use. Create a new Amazon EMR cluster. Use Apache Spark SQL to create Apache Spark jobs in the EMR cluster that run data quality queries on the columns' data type and the presence of null values. Orchestrate the ETL pipeline by using an AWS Step Functions workflow. Configure the workflow to send an email notification that informs the data engineer about the incomplete datasets to the SNS topic.
- Explanation: Amazon EMR is a scalable big data platform, and while it is powerful, it requires management of clusters, which adds significant overhead in terms of both cost and maintenance. Using Spark jobs for data quality checks also introduces unnecessary complexity.
- Why rejected: Like Airflow, EMR is overkill for this task. It requires maintaining clusters and managing complex resources. It's a great tool for large-scale data processing but introduces more operational overhead than needed here.
C) Create data quality checks on the source datasets that the daily reports use. Create data quality actions by using AWS Glue workflows to confirm the com...
Author: MoonlitPantherX · Last updated Jul 30, 2026
A company stores customer data that contains personally identifiable information (PII) in an Amazon Redshift cluster. The company's marketing, claims, and analytics teams need to be able to access the customer data.The marketing team should have access to obfuscated claim information but should have full access to customer contact information. The claims team should have access to customer information for each claim that t...
Let's break down the options and evaluate them based on the scenario of securing access to customer data with least administrative overhead while meeting the specific access requirements for the marketing, claims, and analytics teams.
Option A: Create a separate Redshift cluster for each team. Load only the required data for each team. Restrict access to clusters based on the teams.
- Why rejected: This option would involve managing multiple Redshift clusters, which increases both operational complexity and cost. Each cluster would require separate data loads, maintenance, and security configurations. This solution is not efficient or scalable, especially for scenarios where teams might need to interact with overlapping datasets.
- Not ideal because: It requires significant administrative overhead and is inefficient, as the company would have to manage multiple clusters, which adds complexity without providing much benefit in data access control.
Option B: Create views that include required fields for each of the data requirements. Grant the teams access only to the view that each team requires.
- Why selected: This solution is effective and scalable. By creating views that contain only the necessary data fields for each team (e.g., obfuscated data for the analytics team, full contact data for the marketing team, and claim-specific data for the claims team), you can enforce the access requirements directly in Redshift. Views can be easily managed and modified without significant overhead, and they help isolate the teams from unnecessary data access.
- Why other options are rejected:
- It is straightforward to implement, as it only requires the creation of views and granting appropriate permissions.
- It provides flexibility, as any changes to access requirements can be managed by modifying views instead of creating new clusters or roles.
- The solution is scalable because more views can be added as required without needing complex restructuring.
...
Author: Ryan · Last updated Jul 30, 2026
A financial company recently added more features to its mobile app. The new features required the company to create a new topic in an existing Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster.A few days after the company added the new topic, Amazon CloudWatc...
To address the CloudWatch alarm on the RootDiskUsed metric for an Amazon MSK cluster, we need to identify the underlying issue and select a solution that addresses the disk space usage specifically, which is related to storage capacity. Let's evaluate each option based on this requirement:
A) Expand the storage of the MSK broker. Configure the MSK cluster storage to expand automatically.
- Explanation: This option directly addresses the issue by expanding the storage available to the MSK brokers. Configuring the MSK cluster to expand automatically is also a good practice because it ensures that storage will be increased automatically as disk usage grows, preventing future alarms related to disk space. MSK brokers store the actual Kafka data, so running out of disk space would trigger the alarm.
- Why selected: This is the most effective and appropriate solution, as it directly solves the disk space issue by increasing the storage allocated to the MSK brokers. Configuring automatic expansion ensures that the storage is managed dynamically with minimal administrative effort. It also addresses the alarm on the RootDiskUsed metric, which is related to disk space usage on the MSK brokers.
B) Expand the storage of the Apache ZooKeeper nodes.
- Explanation: Apache ZooKeeper nodes are essential for managing the Kafka cluster's metadata and coordination. However, RootDiskUsed specifically refers to the MSK brokers’ storage, not ZooKeeper’s. ZooKeeper typically doesn’t store Kafka data in the same way as brokers do, so expanding its storage would not address the disk space alarm triggered by the brokers' storage.
- Why rejected: ZooKeeper nodes may require storage expansion in some cases, b...
Author: Lucas Carter · Last updated Jul 30, 2026
A data engineer needs to build an enterprise data catalog based on the company's Amazon S3 buckets and Amazon RDS databases. The data catalog must include storage format metadata for the data...
To meet the requirement of building an enterprise data catalog that includes storage format metadata for data in Amazon S3 buckets and Amazon RDS databases, let's evaluate each option based on effort, scalability, and automation.
A) Use an AWS Glue crawler to scan the S3 buckets and RDS databases and build a data catalog. Use data stewards to inspect the data and update the data catalog with the data format.
- Explanation: This solution involves using an AWS Glue crawler to scan data in S3 and RDS and create a data catalog. However, it relies on manual inspection and updates by data stewards to determine the data format. This introduces unnecessary operational overhead, as it requires human intervention to inspect and update the catalog after it’s created.
- Why rejected: This solution adds more manual effort by requiring data stewards to update the data format, which is inefficient and not the least effort approach. It also doesn't fully automate the process of identifying and storing the data format in the catalog.
B) Use an AWS Glue crawler to build a data catalog. Use AWS Glue crawler classifiers to recognize the format of data and store the format in the catalog.
- Explanation: AWS Glue crawlers are designed to automatically discover and catalog data stored in S3 and RDS. The crawlers use built-in classifiers to automatically detect the storage format (e.g., Parquet, CSV, JSON, etc.) and include it in the catalog. This solution offers automation and efficiency, as the crawler identifies data formats without requiring manual intervention.
- Why selected: This solution automates the process of discovering both the data and its format with minimal effort. AWS Glue is well-integrated with both S3 and RDS, and the classifiers are designed to recognize...
Author: David · Last updated Jul 30, 2026
A company analyzes data in a data lake every quarter to perform inventory assessments. A data engineer uses AWS Glue DataBrew to detect any personally identifiable formation (PII) about customers within the data. The company's privacy policy considers some custom categories of information to be PII. However, the categories are not included in standard DataBrew data quality rules.The data engineer needs to modify the ...
Let's break down each option based on the key factors:
A) Manually review the data for custom PII categories:
- Key Factors: High operational overhead, prone to human error, and not scalable.
- Why rejected: This option requires manual intervention and would be time-consuming, inefficient, and error-prone. Additionally, it's not suitable for large datasets across multiple datasets in a data lake. This does not meet the requirement of reducing operational overhead.
B) Implement custom data quality rules in DataBrew. Apply the custom rules across datasets:
- Key Factors: Leverages AWS Glue DataBrew’s data quality rules, which can be automated and reusable, reducing operational overhead.
- Why selected: AWS Glue DataBrew allows the implementation of custom data quality rules, and applying them across datasets can be automated, minimizing manual effort. This solution integrates directly into the existing workflow with minimal added complexity. It provides a low-overhead solution to scan for the custom PII categories across datasets in the data lake and fits well with the tool already being used.
C) Develop custom Python scripts to detect the custom PII categories. Call the scripts from DataBrew:
- Key Factors: Requires coding, introduces custom development, and adds overhead for maintenance and scali...
Author: VenomousSerpent42 · Last updated Jul 30, 2026
A company receives a data file from a partner each day in an Amazon S3 bucket. The company uses a daily AWS Glue extract, transform, and load (ETL) pipeline to clean and transform each data file. The output of the ETL pipeline is written to a CSV file named Daily.csv in a second S3 bucket.Occasionally, the daily data file is empty or is missing values for required fields. When the file is missing data, the company can use the previous day=E2=80=99s CSV fil...
Let’s break down each option based on key factors like effort, scalability, and effectiveness in addressing the scenario:
A) Invoke an AWS Lambda function to check the file for missing data and to fill in missing values in required fields:
- Key Factors: This introduces additional complexity with Lambda function management, error handling, and integrating it into the pipeline. Lambda would be used for checking missing data, but there’s additional overhead in managing it and handling error cases.
- Why rejected: While Lambda could be effective for checking and fixing missing data, it introduces extra components that increase complexity and maintenance. It doesn't directly address the core issue (i.e., whether the file is complete or not) in a streamlined manner as other options can.
B) Configure the AWS Glue ETL pipeline to use AWS Glue Data Quality rules. Develop rules in Data Quality Definition Language (DQDL) to check for missing values in required fields and empty files:
- Key Factors: AWS Glue Data Quality rules can automate the validation of data completeness, checking for missing or empty fields, and rejecting invalid files. This could ensure that the ETL pipeline works only with valid data before it overwrites the previous day's CSV.
- Why selected: This is a direct and automated solution. By configuring Glue’s built-in data quality rules, the company can ensure that the data is valid before proceeding with overwriting the file. Data Quality rules also provide an easy-to-use, scalable solution with low maintenance overhead, as it’s a native feature in Glue.
C) Use AWS Glue Studio to change the code in the ETL pipeline to fill in any missing values in the required fields...
Author: Ethan · Last updated Jul 30, 2026
A marketing company uses Amazon S3 to store marketing data. The company uses versioning in some buckets. The company runs several jobs to read and load data into the buckets.To help cost-optimize its storage, the company wants to gather information about incomplete multipart uploads and outda...
Let's evaluate the options based on the requirement to gather information about incomplete multipart uploads and outdated versions in S3 buckets while minimizing operational effort:
Option A: Use AWS CLI to gather the information.
- Why rejected: Using the AWS CLI to manually gather information about incomplete multipart uploads or outdated versions involves writing custom scripts and running them periodically. While possible, this option requires manual effort and does not scale well as the company needs to automate the process. It also doesn't provide a centralized view or easy integration for tracking over time.
- Not ideal because: This option involves more operational effort, including script management, scheduling, and monitoring. It is error-prone and less efficient than automated services.
Option B: Use Amazon S3 Inventory configurations reports to gather the information.
- Why selected: S3 Inventory can be configured to provide detailed reports about the objects in a bucket, including versions and incomplete multipart uploads. This solution is easy to set up and provides comprehensive, regular reports with minimal operational overhead. The reports can be scheduled to run periodically and are automatically delivered to a designated S3 bucket.
- Why other options are rejected:
- S3 Inventory is designed specifically for gathering information about the objects in an S3 bucket, including outdated versions and incomplete multipart uploads.
- It provides a low-maintenance, automated approach for the company to receive regular reports with the ...
Author: Zara · Last updated Jul 30, 2026
A company needs a solution to manage costs for an existing Amazon DynamoDB table. The company also needs to control the size of the table. The solution must not disrupt any ongoing read or write operations. The company wants to use a solution that automatically delete...
Let’s evaluate the options based on factors like ease of implementation, automation, and ongoing maintenance:
A) Use the DynamoDB TTL feature to automatically expire data based on timestamps:
- Key Factors: DynamoDB’s Time to Live (TTL) feature is designed to automatically delete expired items based on a timestamp attribute. When TTL is enabled, DynamoDB automatically removes expired data without requiring custom logic or manual intervention. TTL runs in the background and does not disrupt ongoing read or write operations.
- Why selected: This solution perfectly aligns with the company’s requirement of automatically deleting data after a certain period (1 month) with the least operational maintenance. Once TTL is configured, it automatically manages the deletion of expired items without any manual intervention, reducing complexity and maintenance effort. It’s efficient and integrates seamlessly into the existing DynamoDB table.
B) Configure a scheduled Amazon EventBridge rule to invoke an AWS Lambda function to check for data that is older than 1 month. Configure the Lambda function to delete old data:
- Key Factors: While this solution would also automate the deletion of data, it requires creating and managing an EventBridge rule, a Lambda function, and ensuring that the function efficiently scans and deletes data. This solution requires more maintenance, monitoring, and scaling as the table size grows.
- Why rejected: This solution introduces extra components (EventBridge rule and Lambda function) that require ongoing monitoring and maintenance. It also needs periodic invocation to check and delete data, which increases operational overhead compared to the simpler TTL solution.
C) Configure a stream on the DynamoDB table to invoke an AWS Lambda function. Configure the Lambda function to delete data...
Author: Vivaan · Last updated Jul 30, 2026
A company uses Amazon S3 to store data and Amazon QuickSight to create visualizations,The company has an S3 bucket in an AWS account named Hub-Account. The S3 bucket is encrypted by an AWS Key Management Service (AWS KMS) key. The company's QuickSight instance is in a separate account named BI-Account.The company updates the S3 bucket policy to grant access to the QuickSight service role. ...
Let's analyze each option based on the requirements:
A) Use the existing AWS KMS key to encrypt connections from QuickSight to the S3 bucket.
- Rejected: AWS KMS keys are used for encryption/decryption of data stored in S3, but they don't directly handle the encryption of connections. This option is irrelevant in this context because it doesn't pertain to granting cross-account access or permission management for accessing encrypted data in the S3 bucket.
B) Add the S3 bucket as a resource that the QuickSight service role can access.
- Selected: This is a necessary step. Since the S3 bucket is in a different AWS account (Hub-Account) and QuickSight operates in the BI-Account, adding the S3 bucket as a resource the QuickSight service role can access ensures that QuickSight has permission to interact with the S3 bucket. This is done via the S3 bucket policy.
C) Use AWS Resource Access Manager (AWS RAM) to share the S3 bucket with the BI-Account account.
- Rejected: AWS RAM is typically used for sharing resources such as VPCs or subnets, not S3 buckets. S3 buckets can be shared using policies or IAM roles, but AWS RAM is not applicable for sharing S3 buckets. This option does not fulfill the need for cross-account access to S3.
D) A...
Author: Mia · Last updated Jul 30, 2026
A car sales company maintains data about cars that are listed for sale in an area. The company receives data about new car listings from vendors who upload the data daily as compressed files into Amazon S3. The compressed files are up to 5 KB in size. The company wants to see the most up-to-date listings as soon as the data is uploaded to Amazon S3.A data engineer must automate and orchestrate the data processing workflow of the listings to feed a dashboard. The...
Let's break down each solution to understand which one will meet the requirements most cost-effectively, focusing on scalability, automation, orchestration, and cost.
Option Analysis:
A) Use an Amazon EMR cluster to process incoming data. Use AWS Step Functions to orchestrate workflows. Use Apache Hive for one-time queries and analytical reporting. Use Amazon OpenSearch Service to bulk ingest the data into compute-optimized instances. Use OpenSearch Dashboards in OpenSearch Service for the dashboard.
- Rejected: While EMR and OpenSearch Service can handle the job, EMR clusters (especially provisioned ones) can be expensive to run continuously. Moreover, OpenSearch Service might not be the best fit for managing large datasets with frequent querying and analysis. The need for compute-optimized instances adds complexity and unnecessary cost. The use of Hive for analytical reporting is more suited for large-scale processing, which might be overkill for a small dataset (5 KB files). This option is not cost-efficient for the described needs.
B) Use a provisioned Amazon EMR cluster to process incoming data. Use AWS Step Functions to orchestrate workflows. Use Amazon Athena for one-time queries and analytical reporting. Use Amazon QuickSight for the dashboard.
- Rejected: While this option includes AWS Athena for querying and Amazon QuickSight for the dashboard, the use of a provisioned EMR cluster is not ideal for cost efficiency. Provisioned clusters in EMR are expensive to run continuously, especially given the small size of the files (5 KB). Athena is well-suited for querying, but the EMR cluster incurs unnecessary costs due to its provisioning and compute power needs for such small data files.
C) Use AWS Glue to process incoming data. Use AWS Step Functions to orchestrate workflows....
Author: Andrew · Last updated Jul 30, 2026
A company has AWS resources in multiple AWS Regions. The company has an Amazon EFS file system in each Region where the company operates. The company=E2=80=99s data science team operates within only a single Region. The data that the data science team works with must remain within the team's Region.A data engineer needs to create a single dataset by processing files that are in each of the company's Regional EFS file systems. The...
Let's analyze the options based on the requirement to create a single dataset from files in multiple AWS Regions while ensuring the data remains within the data science team's Region. The goal is to process the data using AWS Step Functions orchestrating Lambda functions with the least effort.
Option Analysis:
A) Peer the VPCs that host the EFS file systems in each Region with the VPC that is in the data science team's Region. Enable EFS file locking. Configure the Lambda functions in the data science team's Region to mount each of the Region-specific file systems. Use the Lambda functions to process the data.
- Rejected: This solution requires complex VPC peering between Regions, which can be difficult to manage and maintain, especially when it comes to networking between different Regions. Mounting EFS file systems across Regions is not ideal due to potential latency issues and the fact that EFS is designed to work within a single region. Also, enabling EFS file locking for cross-region access can add complexity and potential performance overhead. This solution is more complex and less efficient.
B) Configure each of the Regional EFS file systems to replicate data to the data science team's Region. In the data science team's Region, configure the Lambda functions to mount the replica file systems. Use the Lambda functions to process the data.
- Rejected: EFS does not natively support cross-region replication like Amazon S3. Setting up replication manually or using third-party solutions would add unnecessary complexity. Additionally, maintaining consistency across multiple regions and ensuring data synchronization could become cumbersome and inefficient. This approach introduces more operational overhead and would be difficult to manage at scale.
C) Deploy the Lambda functions to ...
Author: ShadowWolf101 · Last updated Jul 30, 2026
A company hosts its applications on Amazon EC2 instances. The company must use SSL/TLS connections that encrypt data in transit to communicate securely with AWS infrastructure that is managed by a customer.A data engineer needs to implement a solution to simplify the generation, distribution, and rotation of digital certificates. The s...
Let's break down each option and evaluate which will meet the requirements with the least operational overhead for simplifying the generation, distribution, and rotation of digital certificates while ensuring automatic renewal and deployment of SSL/TLS certificates.
Option Analysis:
A) Store self-managed certificates on the EC2 instances.
- Rejected: While it is possible to store self-managed certificates directly on EC2 instances, this approach introduces significant operational overhead. The company would need to manually manage certificate generation, distribution, and rotation, which could become cumbersome and error-prone. There is no automated process for renewing or deploying certificates, making this option less desirable. Additionally, manually handling certificates on EC2 instances can lead to security risks if not properly managed.
B) Use AWS Certificate Manager (ACM).
- Selected: AWS Certificate Manager (ACM) is the ideal solution here. ACM simplifies the generation, distribution, and rotation of SSL/TLS certificates. It supports automated certificate renewal and can be used to deploy certificates to Amazon EC2 instances and other AWS services such as Elastic Load Balancers and CloudFront distributions. ACM automates the certificate lifecycle management, reducing operational overhead and increasing security. It integrates seamlessly with other AWS services, making it the most efficient and low-maintenance option for the described requirements.
C) Implement custom automation scripts in AWS Secrets Manager. ...
Author: Noah · Last updated Jul 30, 2026
A company saves customer data to an Amazon S3 bucket. The company uses server-side encryption with AWS KMS keys (SSE-KMS) to encrypt the bucket. The dataset includes personally identifiable information (PII) such as social security numbers and account details.Data that is tagged as PII must be masked before the company uses customer data for analysis. Some users must have secure access to the PII data during the pre-processing phase. The ...
Let's evaluate each option based on the requirements:
- Masking PII data before analysis
- Securing PII data during processing
- Low-maintenance solution for managing the entire engineering pipeline
Option Analysis:
A) Use AWS Glue DataBrew to perform extract, transform, and load (ETL) tasks that mask the PII data before analysis.
- Selected: AWS Glue DataBrew is a visual, no-code data preparation tool that allows you to clean and transform data efficiently, including masking PII data. It can be integrated with Amazon S3, so you can automatically perform data masking during the ETL process before the data is used for analysis. This solution is low maintenance because it automates the transformation process and handles the data masking securely.
B) Use Amazon GuardDuty to monitor access patterns for the PII data that is used in the engineering pipeline.
- Rejected: Amazon GuardDuty is a security monitoring service that detects unusual or unauthorized access patterns or malicious activity within your AWS environment. While GuardDuty is important for security, it does not directly help with masking PII data or securing it during the processing phase. Its role is more about detecting anomalies and security threats rather than ensuring compliance with data masking requirements.
C) Configure an Amazon Macie discovery job for the S3 bucket.
- Rejected: Amazon Macie is a service designed to discover and classify sensitive data (such as PII) in S3 buckets. It helps ide...
Author: Ethan · Last updated Jul 30, 2026
A data engineer is launching an Amazon EMR cluster. The data that the data engineer needs to load into the new cluster is currently in an Amazon S3 bucket. The data engineer needs to ensure that data is encrypted both at rest and in transit.The data that is in the S3 bucket is encrypted by an AWS Key Management Service (AW...
Let's analyze each of the options to determine the best solution based on the requirements of ensuring encryption both at rest and in transit for data loaded into an Amazon EMR cluster from an Amazon S3 bucket.
Key Requirements:
- At-rest encryption: The data in the S3 bucket is encrypted using AWS Key Management Service (AWS KMS) keys.
- In-transit encryption: The data must be encrypted during transfer from Amazon S3 to the EMR cluster. This can be managed with a PEM file for Secure Socket Layer (SSL) encryption.
Evaluation of Options:
Option A:
- Create an Amazon EMR security configuration. Specify the appropriate AWS KMS key for at-rest encryption for the S3 bucket.
- Create a second security configuration specifying the Amazon S3 path of the PEM file for in-transit encryption.
- Create the EMR cluster and attach both security configurations to the cluster.
Analysis:
- You can use two different security configurations for at-rest and in-transit encryption.
- While possible, creating and attaching multiple security configurations to the same cluster is not the standard or optimal approach for encryption. AWS prefers consolidating encryption settings within a single security configuration to minimize complexity and reduce the chances of errors.
Rejection Reason: Multiple configurations could lead to unnecessary complexity, so this option is not ideal.
Option B:
- Create an Amazon EMR security configuration. Specify the appropriate AWS KMS key for local disk encryption for the S3 bucket.
- Specify the Amazon S3 path of the PEM file for in-transit encryption.
- Use the security configuration during EMR cluster creation.
Analysis:
- This option suggests local disk encryption for S3 data, which is incorrect because S3 data is already encrypted at rest using AWS KMS. Local disk encryption refers to encrypting the local storage of the EMR nodes, not the data in S3.
Rejection Reason: Misinterpretation of encryption scopes (local disk vs. S3 encryption). It doesn't fulfill the at-rest encrypti...
Author: Aarav · Last updated Jul 30, 2026
A retail company is using an Amazon Redshift cluster to support real-time inventory management. The company has deployed an ML model on a real-time endpoint in Amazon SageMaker.The company wants to make real-time inventory recommendations. The company also wa...
To evaluate the options, we need to focus on the core requirements: real-time inventory recommendations and future inventory predictions. The company already has an ML model deployed on a real-time endpoint in Amazon SageMaker, and it's using Amazon Redshift for inventory management.
Key Requirements:
1. Real-time recommendations: This implies that the system needs to make predictions or provide recommendations instantly based on incoming data (e.g., inventory changes).
2. Future inventory predictions: This requires forecasting future inventory needs, likely using historical data and machine learning models.
Now, let's evaluate each option.
Option A: Use Amazon Redshift ML to generate inventory recommendations.
- Analysis:
- Amazon Redshift ML allows you to use machine learning models directly within Redshift using SQL.
- It enables users to build, train, and deploy models for predictions directly in Redshift.
- Using Redshift ML to generate real-time inventory recommendations is a direct and effective solution for the use case, especially since it's already integrated with the Redshift cluster, where the inventory data is stored.
Selected Reasoning: This is a strong option because Redshift ML can handle both real-time recommendations and predictions without needing external calls or complex integration.
Option B: Use SQL to invoke a remote SageMaker endpoint for prediction.
- Analysis:
- This approach involves using SQL within Redshift to call a remote SageMaker endpoint for real-time predictions.
- Redshift can invoke external services via SQL, which means predictions could be fetched from the SageMaker real-time endpoint.
- This is a valid solution because it allows the Redshift cluster to retrieve predictions from SageMaker, leveraging the real-time ML model deployed there. However, calling a remote endpoint can introduce latency and potential reliability concerns, especially in high-volume use cases, as it's dependent on network calls and external systems.
Selected Reasoning: This option can work but comes with additional complexity and potential latency due to invoking an external service.
Option C: Use Amazon Redshift ML to schedule regular data exports for offline model training.
- Analysis:
- Amazon Red...
Author: Kai99 · Last updated Jul 30, 2026
A company stores CSV files in an Amazon S3 bucket. A data engineer needs to process the data in the CSV files and store the processed data in a new S3 bucket.The process needs to rename a column, remove specific columns, ignore the second row of each file, create a new column based on the values of the first row of the ...
Let's analyze the problem and each potential solution based on the requirements:
Key Requirements:
1. Rename a column.
2. Remove specific columns.
3. Ignore the second row of each file.
4. Create a new column based on values in the first row.
5. Filter by a numeric value of a column.
The goal is to process CSV files in an S3 bucket and output the processed data to a new S3 bucket. We want to meet the requirements with the least development effort.
Evaluation of Options:
Option A: Use AWS Glue Python jobs to read and transform the CSV files.
- Analysis:
- AWS Glue Python jobs allow you to write custom code in Python to transform the data.
- While this provides full flexibility to perform any transformation (e.g., renaming columns, removing columns, ignoring rows, creating new columns, and filtering), it requires significant development effort. The data engineer would need to manually code and test all transformations, especially for handling the second row and numeric filtering.
Rejection Reason: While this option is flexible, it requires more development effort than the other options, making it less suitable if the goal is to minimize development time.
Option B: Use an AWS Glue custom crawler to read and transform the CSV files.
- Analysis:
- AWS Glue crawlers are designed to infer the schema of data and create metadata for it, but they are not intended for data transformation. A crawler can detect the structure of the CSV files and catalog them, but it does not support complex transformations such as renaming columns, filtering rows, or creating new columns.
Rejection Reason: Crawlers are meant for cataloging and inferring metadata, not for performing data transformations. Thus, this solution will not meet the processing needs described in the requirements.
Option C: Use an AWS Glue workflow to build a set of jobs to crawl and transform the CSV files.
- Analysis:
- AWS Glue workflows allow you to orchestrate multiple Glue jobs...
Author: Liam123 · Last updated Jul 30, 2026
A company uses Amazon Redshift as its data warehouse. Data encoding is applied to the existing tables of the data warehouse. A data engineer discovers that the compression encoding applied to some of the tables is not the best fit for the data.The data engineer needs t...
Key Requirement:
The data engineer needs to improve data encoding for tables in an Amazon Redshift data warehouse where the compression encoding is sub-optimal. The goal is to identify better encoding methods and apply them to optimize storage and performance.
Evaluation of Options:
Option A: Run the ANALYZE command against the identified tables. Manually update the compression encoding of columns based on the output of the command.
- Analysis:
- The ANALYZE command in Redshift is primarily used to update statistics about the data to help the query planner make better decisions. It does not help with compression or encoding selection.
- This option would require the manual identification and update of compression encoding, which is time-consuming and not automated.
Rejection Reason: While the ANALYZE command helps with query performance, it does not optimize or recommend changes for compression encoding. This option is not the most efficient for improving compression encoding.
Option B: Run the ANALYZE COMPRESSION command against the identified tables. Manually update the compression encoding of columns based on the output of the command.
- Analysis:
- ANALYZE COMPRESSION is a specific Redshift command that analyzes the current compression encoding of columns and suggests more optimal encoding based on the data distribution and type.
- The output of this command provides recommendations for improving compression encoding, which can then be applied to optimize storage and query performance.
- Manually updating the encoding based on the output can be done, but this is still a manual process.
Selected Reasoning: This is the most appropriate solution because ANALYZE COMPRESSION provides targeted insights into the best compression encodin...
Author: Leah · Last updated Jul 30, 2026
The company stores a large volume of customer records in Amazon S3. To comply with regulations, the company must be able to access new customer records immediately for the first 30 days after the records are created. The company accesses records that are older than 30 days infrequently....
Key Requirements:
- Immediate access to new customer records for the first 30 days.
- Infrequent access to records older than 30 days.
- The goal is to cost-optimize S3 storage.
Evaluation of Options:
Option A: Apply a lifecycle policy to transition records to S3 Standard Infrequent-Access (S3 Standard-IA) storage after 30 days.
- Analysis:
- S3 Standard-IA is designed for infrequent access to data, offering lower cost compared to S3 Standard for data that is accessed less often.
- By applying a lifecycle policy, the company can automatically move customer records to S3 Standard-IA after 30 days, which would reduce storage costs for older records while still ensuring immediate access for the first 30 days.
- This approach balances cost-efficiency and access needs perfectly.
Selected Reasoning: This is the most cost-effective solution because it automatically optimizes storage costs based on access patterns while meeting the requirement for quick access within the first 30 days.
Option B: Use S3 Intelligent-Tiering storage.
- Analysis:
- S3 Intelligent-Tiering automatically moves data between two access tiers (frequent and infrequent) based on access patterns, without the need for a lifecycle policy.
- It is useful for datasets with unpredictable access patterns, but it incurs additional charges for monitoring and automation, which may not be necessary in this scenario where the data access is predictable (accessed immediately for 30 days, infrequent after that).
Rejection Reason: While S3 Intelligent-Tiering is useful for unpredictable access, it could be less cost-efficient for this use case, as the data access pattern is relatively predictable, and lifecycle policies with S3 Standard-IA can offer a more cost-effective solution.
Option C: Transition records to S3 Glacier Deep Archive storage after 30 days.
- Analysis:
- S3 Gl...
Author: Emma Brown · Last updated Jul 30, 2026
A data engineer is using Amazon QuickSight to build a dashboard to report a company=E2=80=99s revenue in multiple AWS Regions. The data engineer wants the dashboard to display the total revenue for a Region, regardl...
To meet the requirement of displaying the total revenue for a region, regardless of the drill-down levels shown in the visual, the solution needs to focus on performing calculations that allow control over how the data is aggregated across different levels. Let's break down the options:
A) Create a table calculation:
- Table calculations are typically used to compute values based on the current view in the dashboard, such as applying calculations on the visible rows or columns. However, table calculations depend on the current drill-down context, and they are not ideal when you want to show a constant total that ignores drill-downs.
- Why rejected: This option cannot be guaranteed to provide the total revenue across regions without being affected by the drill-down level. It does not offer control over how the data should be aggregated independent of the view level.
B) Create a simple calculated field:
- Simple calculated fields are used for basic mathematical operations or logic based on the data. However, they will be affected by the current drill-down level in the dashboard, which is not what is needed here.
- Why rejected: This option only calculates based on the current data view (drill-down level), so it doesn't work for aggregating totals across regions independent of the drill-down context.
C) Create a level-aware calculation - aggregate (LAC-A) function:
- LAC-A functions allow you to perform calculations on a hig...
Author: Amelia · Last updated Jul 30, 2026
A retail company stores customer data in an Amazon S3 bucket. Some of the customer data contains personally identifiable information (PII) about customers. The company must not share PII data with business partners.A data engineer must determine whether a dataset contains PII before making obje...
To meet the requirement of determining whether a dataset contains PII before sharing it, the solution should automate the process of identifying and flagging sensitive data (PII) with minimal manual intervention. Let's analyze each option:
A) Configure the S3 bucket and S3 objects to allow access to Amazon Macie. Use automated sensitive data discovery in Macie:
- Amazon Macie is a fully managed service that helps identify, classify, and protect sensitive data, such as PII, in S3. Macie uses machine learning to detect PII across S3 objects and can automatically scan and flag content as sensitive.
- Why selected: Macie is designed specifically for this use case, providing an automated solution to identify PII with minimal manual intervention. Once configured, Macie will continuously monitor and classify data in the S3 bucket, making it the most efficient and automated option for detecting PII data in S3.
B) Configure AWS CloudTrail to monitor S3 PUT operations. Inspect the CloudTrail trails to identify operations that save PII:
- AWS CloudTrail records API calls made to AWS services, including S3 operations like PUT. However, CloudTrail only records metadata about the request (who, what, when, and where) but does not inspect the content of the objects being uploaded to S3.
- Why rejected: CloudTrail cannot help identify PII in the actual content of the files; it only logs the event data. Therefore, it does not address the core requirement of identifying sensitive data within the S3 objects themselves.
C) Create an AWS Lambda f...
Author: IceDragon2023 · Last updated Jul 30, 2026
A data engineer needs to create an empty copy of an existing table in Amazon Athena to perform data processing tasks. The existing table in Athe...
Let’s analyze each query option to determine the one that creates an empty copy of the existing table:
A) CREATE TABLE new_table LIKE old_table;
- Rejected: This query will create a new table with the same schema as `old_table` (i.e., same columns and data types), but it will not copy any data from the `old_table`. It only replicates the table structure, which is useful if you just want the table definition, but it doesn't specifically meet the requirement of creating an empty copy with no data, so this option is close but not ideal.
B) CREATE TABLE new_table AS SELECT FROM old_table WITH NO DATA;
- Selected: This query creates an empty table (`new_table`) with the same structure as `old_table` and no data. The `WITH NO DATA` clause ensures that the new table will have the same schema as `old_table`, but no data is copied over. This is exactly what is required in the scenario: an empty copy of the existing table.
C) CREATE TABLE new_table AS SELECT FROM old_table;
- Rejected: This query will create a new table `new_table` and copy all data from `old_ta...
Author: Vivaan · Last updated Jul 30, 2026
A company has a data lake in Amazon S3. The company collects AWS CloudTrail logs for multiple applications. The company stores the logs in the data lake, catalogs the logs in AWS Glue, and partitions the logs based on the year. The company uses Amazon Athena to analyze the logs.Recently, customers reported that a query on one of the Athena tables d...
To troubleshoot the issue where a query in Athena is not returning data, we need to focus on possible issues related to how Athena is interacting with the data in Amazon S3, particularly with respect to partitioning and table metadata. Let's review the options:
A) Confirm that Athena is pointing to the correct Amazon S3 location.
- Why selected: The first step is to ensure that Athena is correctly configured to point to the correct S3 location where the logs are stored. If Athena is querying the wrong location (e.g., the logs have been moved or there is a misconfiguration), the query would not return any data.
- This is a straightforward check to confirm that the table's location matches the S3 path where the logs are actually stored. If there is a mismatch, no data will be available for querying.
B) Increase the query timeout duration.
- Why rejected: The issue described is that no data is being returned, not that the query is taking too long. Increasing the timeout would be useful if the query is timing out, but in this case, the problem is likely related to data availability or metadata issues, not performance. Thus, this step is not relevant for resolving the issue where no data is returned.
C) Use the MSCK REPAIR TABLE command.
- Why selected: The issue could be related to Athena not recognizing the correct partitions, especially if the logs are partitioned by year. The `MSCK REPAIR ...
Author: Noah · Last updated Jul 30, 2026
A data engineer wants to orchestrate a set of extract, transform, and load (ETL) jobs that run on AWS. The ETL jobs contain tasks that must run Apache Spark jobs on Amazon EMR, make API calls to Salesforce, and load data into Amazon Redshift.The ETL jobs need to handle failures and retr...
Let's analyze each service option based on the requirements:
A) Amazon Managed Workflows for Apache Airflow (Amazon MWAA)
- Rejected: Amazon MWAA is a managed service that runs Apache Airflow workflows. While Airflow is great for orchestrating ETL jobs and can handle dependencies, retries, and failure management, it requires more setup and management of the environment. Additionally, while MWAA can run Spark jobs on EMR, it may not directly integrate as smoothly with all of the AWS services (like Salesforce or Redshift) compared to other services. It is a good option for more complex workflows, but the other options may be more directly suited for the given task, especially with Python integration.
B) AWS Step Functions
- Selected: AWS Step Functions is ideal for orchestrating tasks that require integration with various AWS services. Step Functions allows you to define workflows with state machines, supporting task failures and retries, and integrates directly with Amazon EMR, Amazon Redshift, and other AWS services. It also provides native support for invoking Lambda functions (which can be used to make API calls to external services like Salesforce). Python is also fully supported, making it a great choice for the ETL orchestration requirements. Step Functions is designed specifically to handle workflows with retries, error handling, and integration across AWS services, making it the best option for this scenario.
C) AWS Glue
- Rejected: AWS Glue is a fully managed ETL service, and while it is designed for ETL jobs, it is p...
