Carnegie Mellon College’s Robotics Institute has a brand new artist-in-residence.
FRIDA, a robotic arm with a paintbrush taped to it, makes use of synthetic intelligence to collaborate with people on artistic endeavors. Ask FRIDA to color an image, and it will get to work placing brush to canvas.
“There’s this one portray of a frog ballerina that I believe turned out actually properly,” mentioned Peter Schaldenbrand, a College of Laptop Science Ph.D. scholar within the Robotics Institute working with FRIDA and exploring AI and creativity. “It’s actually foolish and enjoyable, and I believe the shock of what FRIDA generated primarily based on my enter was actually enjoyable to see.”
FRIDA, named after Frida Kahlo, stands for Framework and Robotics Initiative for Growing Arts. The mission is led by Schaldenbrand with RI school members Jean Oh and Jim McCann, and has attracted college students and researchers throughout CMU.
Customers can direct FRIDA by inputting a textual content description, submitting different artistic endeavors to encourage its model, or importing {a photograph} and asking it to color a illustration of it. The crew is experimenting with different inputs as nicely, together with audio. They performed ABBA’s “Dancing Queen” and requested FRIDA to color it.
“FRIDA is a robotic portray system, however FRIDA isn’t an artist,” Schaldenbrand mentioned. “FRIDA isn’t producing the concepts to speak. FRIDA is a system that an artist might collaborate with. The artist can specify high-level targets for FRIDA after which FRIDA can execute them.”
The robotic makes use of AI fashions just like these powering instruments like OpenAI’s ChatGPT and DALL-E 2, which generate textual content or a picture, respectively, in response to a immediate. FRIDA simulates how it might paint a picture with brush strokes and makes use of machine studying to guage its progress as it really works.
FRIDA’s ultimate merchandise are impressionistic and kooky. The brushstrokes are daring. They lack the precision sought so typically in robotic endeavors. If FRIDA makes a mistake, it riffs on it, incorporating the errant splotch of paint into the top consequence.
“FRIDA is a mission exploring the intersection of human and robotic creativity,” McCann mentioned. “FRIDA is utilizing the form of AI fashions which have been developed to do issues like caption photographs and perceive scene content material and making use of it to this inventive generative downside.”
FRIDA faucets into AI and machine studying a number of instances throughout its inventive course of. First, it spends an hour or extra studying the right way to use its paintbrush. Then, it makes use of massive vision-language fashions skilled on large datasets that pair textual content and pictures scraped from the web, corresponding to OpenAI’s Contrastive Language-Picture Pre-Coaching (CLIP), to know the enter. AI methods use these fashions to generate new textual content or photographs primarily based on a immediate.
Different image-generating instruments corresponding to OpenAI’s DALL-E 2, use massive vision-language fashions to provide digital photographs. FRIDA takes {that a} step additional and makes use of its embodied robotic system to provide bodily work. One of many largest technical challenges in producing a bodily picture is decreasing the simulation-to-real hole, the distinction between what FRIDA composes in simulation and what it paints on the canvas. FRIDA makes use of an concept often known as real2sim2real. The robotic’s precise brush strokes are used to coach the simulator to replicate and mimic the bodily capabilities of the robotic and portray supplies.
FRIDA’s crew additionally seeks to handle among the limitations in present massive vision-language fashions by regularly refining those they use. The crew fed the fashions the headlines from information articles to provide it a way of what was occurring on the earth and additional skilled them on photographs and textual content extra consultant of various cultures to keep away from an American or Western bias. This multicultural collaboration effort is led by Zhixuan Liu and Beverley-Claire Okogwu, first-year RI grasp’s college students, and Youeun Shin and Youngsik Yun, visiting grasp’s college students from Dongguk College in Korea. Their efforts embrace coaching knowledge contributions from China, Japan, Korea, Mexico, Nigeria, Norway, Vietnam and different nations.
As soon as FRIDA’s human person has specified a high-level idea of the portray they need to create, the robotic makes use of machine studying to create its simulation and develop a plan to make a portray to attain the person’s targets. FRIDA shows a coloration pallet on a pc display for a human to combine and supply to the robotic. Computerized paint mixing is at the moment being developed, led by Jiaying Wei, a grasp’s scholar within the College of Structure, with Eunsu Kang, school within the Machine Studying Division.
Armed with a brush and paint, FRIDA will make its first strokes. Once in a while, the robotic makes use of an overhead digital camera to seize a picture of the portray. The picture helps FRIDA consider its progress and refine its plan, if wanted. The entire course of takes hours.
“Individuals marvel if FRIDA goes to take artists’ jobs, however the primary aim of the FRIDA mission is sort of the other. We need to actually promote human creativity via FRIDA,” Oh mentioned. “As an example, I personally needed to be an artist. Now, I can truly collaborate with FRIDA to specific my concepts in portray.”
Extra details about FRIDA is offered on its web site. The crew will current its newest analysis from the mission, “FRIDA: A Collaborative Robotic Painter With a Differentiable, Real2Sim2Real Planning Setting” on the 2023 IEEE Worldwide Convention on Robotics and Automation this Could in London. FRIDA resides within the RI’s Bot Intelligence Group (BIG) lab within the Squirrel Hill neighborhood of Pittsburgh.
