Important Databricks Certified Generative AI Engineer Associate Exam Questions

Answer: C

Explanation:

The task described in the question involves enhancing customer experience by automatically handling common customer inquiries using an LLM-powered AI solution. This requires the system to process input data (customer inquiries) and generate personalized, relevant responses efficiently. Let's evaluate the options step-by-step in the context of Databricks Generative AI Engineer principles, which emphasize leveraging LLMs for tasks like question answering, summarization, and retrieval-augmented generation (RAG).

Option A: Input: Customer reviews; Output: Group the reviews by users and aggregate per-user average rating, then respond

This option focuses on analyzing customer reviews to compute average ratings per user. While this might be useful for sentiment analysis or user profiling, it does not directly address the goal of handling common customer inquiries or improving response times for personalized interactions. Customer reviews are typically feedback data, not real-time inquiries requiring immediate responses.

Databricks Reference: Databricks documentation on LLMs (e.g., 'Building LLM Applications with Databricks') emphasizes that LLMs excel at tasks like question answering and conversational responses, not just aggregation or statistical analysis of reviews.

Option B: Input: Customer service chat logs; Output: Group the chat logs by users, followed by summarizing each user's interactions, then respond

This option uses chat logs as input, which aligns with customer service scenarios. However, the output---grouping by users and summarizing interactions---focuses on user-specific summaries rather than directly addressing inquiries. While summarization is an LLM capability, this approach lacks the specificity of finding answers to common questions, which is central to the problem.

Databricks Reference: Per Databricks' 'Generative AI Cookbook,' LLMs can summarize text, but for customer service, the emphasis is on retrieval and response generation (e.g., RAG workflows) rather than user interaction summaries alone.

Option C: Input: Customer service chat logs; Output: Find the answers to similar questions and respond with a summary

This option uses chat logs (real customer inquiries) as input and tasks the LLM with identifying answers to similar questions, then providing a summarized response. This directly aligns with the goal of handling common inquiries efficiently while maintaining personalization (by referencing past interactions or similar cases). It leverages LLM capabilities like semantic search, retrieval, and response generation, which are core to Databricks' LLM workflows.

Databricks Reference: From Databricks documentation ('Building LLM-Powered Applications,' 2023), an exact extract states: 'For customer support use cases, LLMs can be used to retrieve relevant answers from historical data like chat logs and generate concise, contextually appropriate responses.' This matches Option C's approach of finding answers and summarizing them.

Option D: Input: Customer reviews; Output: Classify review sentiment

This option focuses on sentiment classification of reviews, which is a valid LLM task but unrelated to handling customer inquiries or improving response times in a conversational context. It's more suited for feedback analysis than real-time customer service.

Databricks Reference: Databricks' 'Generative AI Engineer Guide' notes that sentiment analysis is a common LLM task, but it's not highlighted for real-time conversational applications like customer support.

Conclusion: Option C is the best fit because it uses relevant input (chat logs) and defines an LLM task (finding answers and summarizing) that meets the requirements of improving response times and maintaining personalized interaction. This aligns with Databricks' recommended practices for LLM-powered customer service solutions, such as retrieval-augmented generation (RAG) workflows.

Answer: A

Explanation:

The task is to prevent toxic or unsafe responses in an LLM system using the Foundation Model API with minimal effort. Let's assess the options.

Option A: Host Llama Guard on Foundation Model API and use it to detect unsafe responses

Llama Guard is a safety-focused model designed to detect toxic or unsafe content. Hosting it via the Foundation Model API (a Databricks service) integrates seamlessly with the existing system, requiring minimal setup (just deployment and a check step), and leverages provisioned throughput for performance.

Databricks Reference: 'Foundation Model API supports hosting safety models like Llama Guard to filter outputs efficiently' ('Foundation Model API Documentation,' 2023).

Option B: Add some LLM calls to their chain to detect unsafe content before returning text

Using additional LLM calls (e.g., prompting an LLM to classify toxicity) increases latency, complexity, and effort (crafting prompts, chaining logic), and lacks the specificity of a dedicated safety model.

Databricks Reference: 'Ad-hoc LLM checks are less efficient than purpose-built safety solutions' ('Building LLM Applications with Databricks').

Option C: Add a regex expression on inputs and outputs to detect unsafe responses

Regex can catch simple patterns (e.g., profanity) but fails for nuanced toxicity (e.g., sarcasm, context-dependent harm), requiring significant manual effort to maintain and update rules.

Databricks Reference: 'Regex-based filtering is limited for complex safety needs' ('Generative AI Cookbook').

Option D: Ask users to report unsafe responses

User reporting is reactive, not preventive, and places burden on users rather than the system. It doesn't limit unsafe outputs proactively and requires additional effort for feedback handling.

Databricks Reference: 'Proactive guardrails are preferred over user-driven monitoring' ('Databricks Generative AI Engineer Guide').

Conclusion: Option A (Llama Guard on Foundation Model API) is the least-effort, most effective approach, leveraging Databricks' infrastructure for seamless safety integration.

Answer: A

Explanation:

The task is to choose between LSH and HNSW for a vector database index, prioritizing semantic accuracy. The evaluation must assess how well each method retrieves semantically relevant results. Let's evaluate the options.

Option A: Compare the cosine similarities of the embeddings of returned results against those of a representative sample of test inputs

Cosine similarity measures semantic closeness between vectors, directly assessing retrieval accuracy in a vector database. Comparing returned results' embeddings to test inputs' embeddings evaluates how well LSH or HNSW preserves semantic relationships, aligning with the priority.

Databricks Reference: 'Cosine similarity is a standard metric for evaluating vector search accuracy' ('Databricks Vector Search Documentation,' 2023).

Option B: Compare the Bilingual Evaluation Understudy (BLEU) scores of returned results for a representative sample of test inputs

BLEU evaluates text generation (e.g., translations), not vector retrieval accuracy. It's irrelevant for indexing performance.

Databricks Reference: 'BLEU applies to generative tasks, not retrieval' ('Generative AI Cookbook').

Option C: Compare the Recall-Oriented-Understudy for Gisting Evaluation (ROUGE) scores of returned results for a representative sample of test inputs

ROUGE is for summarization evaluation, not vector search. It doesn't measure semantic accuracy in retrieval.

Databricks Reference: 'ROUGE is unsuited for vector database evaluation' ('Building LLM Applications with Databricks').

Option D: Compare the Levenshtein distances of returned results against a representative sample of test inputs

Levenshtein distance measures string edit distance, not semantic similarity in embeddings. It's inappropriate for vector-based retrieval.

Databricks Reference: No specific support for Levenshtein in vector search contexts.

Conclusion: Option A (cosine similarity) is the correct approach, directly evaluating semantic accuracy in vector retrieval, as recommended by Databricks for Vector Search assessments.

Answer: A

Explanation:

The task is to extend a cinema chatbot to provide movie showtime information using a RAG application, leveraging user location and a continuously updated Delta table, with minimal effort and high performance. Let's evaluate the options.

Option A: Create a Feature Serving Endpoint from a FeatureSpec that references an online store synced from the Delta table. Query the Feature Serving Endpoint as part of the agent logic / tool implementation

Databricks Feature Serving provides low-latency access to real-time data from Delta tables via an online store. Syncing the Delta table to a Feature Serving Endpoint allows the chatbot to query showtimes efficiently, integrating seamlessly into the RAG agent's tool logic. This leverages Databricks' native infrastructure, minimizing effort and ensuring performance.

Databricks Reference: 'Feature Serving Endpoints provide real-time access to Delta table data with low latency, ideal for production systems' ('Databricks Feature Engineering Guide,' 2023).

Option B: Query the Delta table directly via a SQL query constructed from the user's input using a text-to-SQL LLM in the agent logic / tool

Using a text-to-SQL LLM to generate queries adds complexity (e.g., ensuring accurate SQL generation) and latency (LLM inference + SQL execution). While feasible, it's less performant and requires more effort than a pre-built serving solution.

Databricks Reference: 'Direct SQL queries are flexible but may introduce overhead in real-time applications' ('Building LLM Applications with Databricks').

Option C: Write the Delta table contents to a text column, then embed those texts using an embedding model and store these in the vector index. Look up the information based on the embedding as part of the agent logic / tool implementation

Converting structured Delta table data (e.g., showtimes) into text, embedding it, and using vector search is inefficient for structured lookups. It's effort-intensive (preprocessing, embedding) and less precise than direct queries, undermining performance.

Databricks Reference: 'Vector search excels for unstructured data, not structured tabular lookups' ('Databricks Vector Search Documentation').

Option D: Set up a task in Databricks Workflows to write the information in the Delta table periodically to an external database such as MySQL and query the information from there as part of the agent logic / tool implementation

Exporting to an external database (e.g., MySQL) adds setup effort (workflow, external DB management) and latency (periodic updates vs. real-time). It's less performant and more complex than using Databricks' native tools.

Databricks Reference: 'Avoid external systems when Delta tables provide real-time data natively' ('Databricks Workflows Guide').

Conclusion: Option A minimizes effort by using Databricks Feature Serving for real-time, low-latency access to the Delta table, ensuring high performance in a production-ready RAG chatbot.

Answer: A

Explanation:

The system answers questions on unfolding news topics using articles and social media, with a concern about toxic outputs from toxic inputs. A guardrail must limit toxicity in the LLM's responses. Let's evaluate the options.

Option A: Use only approved social media and news accounts to prevent unexpected toxic data from getting to the LLM

Curating input sources (e.g., verified accounts) reduces exposure to toxic content at the data ingestion stage, directly limiting toxic outputs. This is a proactive guardrail aligned with data quality control.

Databricks Reference: 'Control input data quality to mitigate unwanted LLM behavior, such as toxicity' ('Building LLM Applications with Databricks,' 2023).

Option B: Implement rate limiting

Rate limiting controls request frequency, not content quality. It prevents overload but doesn't address toxicity in social media inputs or outputs.

Databricks Reference: Rate limiting is for performance, not safety: 'Use rate limits to manage compute load' ('Generative AI Cookbook').

Option C: Reduce the amount of context items the system will include in consideration for its response

Reducing context might limit exposure to some toxic items but risks losing relevant information, and it doesn't specifically target toxicity. It's an indirect, imprecise fix.

Databricks Reference: Context reduction is for efficiency, not safety: 'Adjust context size based on performance needs' ('Databricks Generative AI Engineer Guide').

Option D: Log all LLM system responses and perform a batch toxicity analysis monthly

Logging and analyzing responses is reactive, identifying toxicity after it occurs rather than preventing it. Monthly analysis doesn't limit real-time toxic outputs.

Databricks Reference: Monitoring is for auditing, not prevention: 'Log outputs for post-hoc analysis, but use input filters for safety' ('Building LLM-Powered Applications').

Conclusion: Option A is the most effective guardrail, proactively filtering toxic inputs from unverified sources, which aligns with Databricks' emphasis on data quality as a primary safety mechanism for LLM systems.