As generative artificial intelligence (AI) spreads, new forms of gender-based violence such as deepfake sex crimes are increasing, yet a study finds that among AI developers, awareness of these risks is lacking. There are calls to build Korean-language datasets that reflect misogyny and gender conflict in Korean society and use them in AI development.

According to the Korea Women's Policy Institute's report released on the 4th, titled 'A Study on the Need for AI Benchmark Datasets to Prevent Digital Gender-Based Violence', the researchers analyzed that awareness among AI developers that issues of gender violence or discrimination can arise in the AI development process is quite thin. The team asked eight experts in large IT corporations, general conglomerates, and startups who are in charge of large language model (LLM) development or AI-related practical work about where ethics and safety policies stand within AI development culture.

An AI practitioner in their 30s who has worked at a major corporation for more than five years said, "It didn't naturally occur to me how (AI) could be used maliciously, so I didn't really recognize the problem," adding, "I can't think at all of what kinds of issues there could be regarding sexual harassment or sexual abuse via AI."

Even in public projects, establishing ethics and safety guidelines for AI platforms was found to be pushed down the priority list. A software developer in their 50s at a small or medium-sized enterprise working on a government agency's AI platform project said, "We're digging through materials from the National Information Society Agency (NIA) and the like, but (colleagues) aren't that interested," adding, "We still haven't been able to pay attention to (ethics guidelines)." Another technology planning manager at a large company said, "People developing at a very low level (of the stack) really dislike (safety issues)," and, "From a truly macro perspective, it's not the time to worry about data; what matters is improving completeness."

Developers agreed that the public sector, rather than the private sector, should establish safety systems to prevent gender-based violence. An employee in charge of AI-related work at a major company said, "Strengthening safety policies that explicitly address gender-based violence seems difficult for companies. It would be too much to form a large team," adding, "There's a lot of safety-related material abroad, so if what's being addressed now comes out as Korean-language (data)sets, many would use them."

Comparing the status of AI benchmark dataset construction at home and abroad, the researchers pointed out a lack of dedicated datasets to prevent gender-based violence in Korean society. A dataset (data set) is a collection of data used by artificial intelligence for learning or for evaluating performance. A benchmark dataset is a reference dataset created to compare the performance of various AI models. Only when benchmark datasets incorporate a perspective that defines and seeks to prevent gender-based violence can the social safety of AI models be secured. However, the researchers explained that current benchmark research is conducted mainly in the English-speaking world, leaving Korean-language datasets at the level of identifying general social biases such as region and age.

The researchers stressed, "There is hardly any discussion on datasets that reflect the unique patterns of gender conflict in Korean society, online community discourse, and linguistic nuances," adding, "The public sector needs to lead the creation of safe and ethical datasets and provide policy support so that they are mandatorily used in AI development and verification through an industry-academia-research-government cooperation system."