pepepepebrick on Instagram: "🧱 2024/09/11 • This is a hand-tracking interactive system for sound and visuals. • In #TouchDesigner, I used the #MediaPipe plugin developed by @blankensmithing to recognize the coordinates of the user’s fingers in front of the camera. I extracted three values: the distance between the index finger and thumb of each hand, and the distance between both hands. These values are sent to the audio software and also affect the size of circular points in the visuals, as well as distort the image. • In #VCVRack, hand signals are received via the #OSC protocol and interact with the modular synthesizer. The three values are mapped to the OSC Frequencer, Drum Attack, and Liquid Filter. When both fingers close, the primary audio track is muted. • This system is an experimental project with potential future applications. For example, party DJs could perform with virtual controllers, or technicians could operate without carrying heavy hardware equipment. • 這是一個手部追蹤的聲音及影像互動系統 • 在#TouchDesigner 中，我使用了由 @blankensmithing 開發的 #MediaPipe 插件，它能夠辨識鏡頭前使用者的手指座標。我提取了左右手食指與拇指的距離，以及雙手之間的距離，並將這三個數值傳入到聲音軟體。同時，它們也會影響線段圓點的尺寸，以及對畫面產生扭曲 • 在#VCVRack 中，會透過 #OSC 格式接收手部訊號，並且與模組合成器產生聲音互動。三個數值分別代入在OSC Frequencer、Drum Attack、Liquid Filter，當雙指都關閉時，則會屏蔽掉主要音軌的聲音 • 這個系統是一項實驗性專案，我們可以想像它未來應用在更多情境，例如派對 DJ 可以用虛擬的控制器演出、控台人員可以不用帶很多的硬體設備 • 🧿 @touchdesigner 🧬 @vcvrack • #audiovisual #interactiondesign #vcvrack #modularsynth #touchdesinger #generativeart"
168K likes, 907 comments - pepepepebrick on November 23, 2024: "🧱 2024/09/11 • This is a hand-tracking interactive system for sound and visuals. • In #TouchDesigner, I used the #MediaPipe plugin developed by @blankensmithing to recognize the coordinates of the user’s fingers in front of the camera. I extracted three values: the distance between the index finger and thumb of each hand, and the distance between both hands. These values are sent to the audio software and also affect the size of circular points in the visuals, as well as distort the image. • In #VCVRack, hand signals are received via the #OSC protocol and interact with the modular synthesizer. The three values are mapped to the OSC Frequencer, Drum Attack, and Liquid Filter. When both fingers close, the primary audio track is muted. • This system is an experimental project with potential future applications. For example, party DJs could perform with virtual controllers, or technicians could operate without carrying heavy hardware equipment. • 這是一個手部追蹤的聲音及影像互動系統 • 在#TouchDesigner 中，我使用了由 @blankensmithing 開發的 #MediaPipe 插件，它能夠辨識鏡頭前使用者的手指座標。我提取了左右手食指與拇指的距離，以及雙手之間的距離，並將這三個數值傳入到聲音軟體。同時，它們也會影響線段圓點的尺寸，以及對畫面產生扭曲 • 在#VCVRack 中，會透過 #OSC 格式接收手部訊號，並且與模組合成器產生聲音互動。三個數值分別代入在OSC Frequencer、Drum Attack、Liquid Filter，當雙指都關閉時，則會屏蔽掉主要音軌的聲音 • 這個系統是一項實驗性專案，我們可以想像它未來應用在更多情境，例如派對 DJ 可以用虛擬的控制器演出、控台人員可以不用帶很多的硬體設備 • 🧿 @touchdesigner 🧬 @vcvrack • #audiovisual #interactiondesign #vcvrack #modularsynth #touchdesinger #generativeart".
https://www.instagram.com/p/DCwbZwczaER/
Seonglae Cho
