Humans and machines are in constant conversations. Intelligent machines today are not only observers of the world, but they also make creative decisions. If A.I imitates human beings to generate a symbolic system and actively communicate with us based on their own understandings of the universe, to what degrees their messages and meanings will recontextualize our coexistence?
_仓 颉 的 诗 2020
Second Episode of Cangjie
Interactive Media Art Installation
VIDEO DOCUMENTATION [ 2 minutes ]
Cangjie’s Poetry is a multimodal intelligent system that was designed as a conceptual response to the future semantic human-machine reality. Inspired by Cangjie, an ancient Chinese legendary historian (c.2650 BCE), who invented Chinese characters based on his observation of characteristics of everything on earth, we trained a neural network to learn the constructions and principles of over 9000 Chinese characters and named it Cangjie.
Cangjie first transforms surroundings perceived by the neural network through a camera into a cluster of ever-changing new symbols that constructed by Chinese strokes. Those novel symbols compose a dynamically evolving poem. In the meantime, Cangjie also creates meanings for those symbols by using computer vision systems to generate descriptive sentences of the surroundings in natural language that humans can read.
Installation Mock-Up View
Visualization I on the left side, Visualization II on the right side
Two interactive visualizations are generated by Cangjie through perceiving the real-world via a camera (located in the exhibition space) in real-time.
The first visualization presents an ever-changing poem written in the novel symbols that are generated through the lens of Cangjie, which continuously evolves and composes poetic ink imagery. We use unsupervised learning techniques trained a neural network (named Cangjie) to learn from vector stroke data of over 9000 Chinese characters. After successful training, the discriminator and the encoder/generator reach a stable state like a Nash Equilibrium. The network learns a low-dimensional latent representation of these images. Thus, when the live streaming of the real world is processed by the system, the encoder network can produce its latent representation. Then the generator network can reconstruct the image and generate novel symbols based on the given latent representation.
The second visualization is an evolving concrete poem that is written in natural language that both make meanings for those new-born symbols and communicate with audiences in English. The aesthetic of the visualization is inspired by concrete poetry in the 20th century which arranges linguistic elements topographically in the space to convey meanings. We utilized pre-trained fully convolutional localization networks to determine the arrangement of sentences and generates multiple localized descriptive sentences in the multiple regions of the image. The model also outputs a confidence score to value how much the descriptive sentence corresponds to the ground truth. In the visualization, we intentionally combined the lowest confidence output (the most inaccurate result) with the highest confident output (the most accurate result), which emphasizes the ambiguity and unexpectedness of the intelligent poetry. Therefore, we get an interesting poem that reflects how the machine reads the reality which also makes the Cangjie's poetry (write with novel symbols) understandable to audiences.
Cangjie is writing poems using its own symbolic system
The ink flow movement is interactive / determined by the data of the live streaming
Cangjie is making meanings to the new symbols by observing surroundings and describing it based on its own understanding
It is a collage of live streaming texture with text Cangjie generates
Cangjie is not only a conceptual response to the tension and fragility in the coexistence of humans and machines but also an artistic expression of a future language that reflects on ancient truths, a way to evoke enchantment in this artificial intelligence era. The interactivity of this intelligent system prioritizes ambiguity and tension that exists between the actual and the virtual, machinic vision and human perception, and past and present.
From learning over 9000 Chinese characters, Cangjie creates its own symbolic system. It observes the real-world, writes poetry using its symbolic system, and explains it in natural language to audiences.
Just like the legendary historian CangJie did nearly 5000 years ago.
It continuously writes poetry with humans collaboratively,
as long as the real world exists.
Never - The - End