Google has made headlines for its new directive that impacts contractors responsible for evaluating the responses of its AI, Gemini. Recent reports indicate that the technology company has disabled the option for these contractors to bypass prompts falling outside their areas of expertise. This decision raises concerns about the potential deterioration of response quality, particularly on complex topics, amidst ongoing issues related to artificial intelligence (AI) hallucinations.
Google Reportedly Not Allowing Contractors to Skip Gemini Prompts
According to a report from TechCrunch, Google has issued an updated internal guideline to contractors engaged with the Gemini project. Sources familiar with the memo assert that contractors are now required to attempt to evaluate queries, even if they lack sufficient knowledge to accurately appraise the responses.
The evaluation responsibilities for Gemini’s answers have been outsourced to GlobalLogic, a subsidiary of Hitachi. Contractors evaluating the AI’s outputs typically possess expertise in specific fields, including coding, mathematics, and medicine. Their role involves analyzing technical prompts to assess the AI’s responses based on several criteria, including accuracy and truthfulness.
Prior to this guideline revision, contractors were permitted to skip prompts that did not fall within their areas of expertise. This practice ensured that evaluations were conducted by individuals qualified to understand and critique the technical nature of Gemini’s responses, thereby helping to mitigate the risk of AI hallucinations—a common challenge faced by AI systems.
However, reports indicate that GlobalLogic recently announced a policy change: contractors may only skip prompts if the AI’s response is “completely lacking information” or if it contains harmful content that necessitates special consent forms for review.
The new directive instructs contractors to evaluate portions of prompts that they are able to comprehend and to provide notes indicating their lack of specialized knowledge. One contractor expressed concern in internal communications, stating, “I thought the point of skipping was to increase accuracy by giving it to someone better,” highlighting the contentious nature of this policy change.
