Live Input

Live input is what you do while a project runs: moving the mouse, using the keyboard, holding up a hand, standing in front of the camera. VerkLabs reads it as numbers, once for every frame, and you wire those numbers into nodes. A hand can change a shader’s color, move a filter, or play notes.

Learn it by doing. These lessons walk you through it, step by step, in the editor:

Live input only exists while someone is there. A project you share, open in viewer mode, or put on another display keeps working: the numbers are simply whatever the person at that window is doing.

Where Each Input Works

The nodes have the same names, lists and wiring in both places, so something you learn in one works in the other. The difference is the kind of wire: a shader node outputs a position or a state, and an instrument node outputs plain numbers that can drive any numeric input, such as a Filter’s Frequency.

Mouse

The Mouse node gives the position of the mouse in the Scene Viewer, from 0 to 1, with 0 at the left and at the bottom. The button output is 1 while the mouse button is down in the Scene Viewer, and 0 otherwise.

In an instrument, when the Scene Viewer is not on screen (the Notes workspace, say), the position is measured across the whole window, and a click counts anywhere that is not a button or a text field. Until the mouse has moved, an instrument’s Mouse gives its x when not live and y when not live values.

A touch screen works the same way: a finger is the mouse.

Keyboard

The Keyboard node gives 1 while a chosen key is pressed, and 0 otherwise. Pick the key in the node’s list (it is the same list in shaders and instruments). A shader’s Keyboard only knows the last key pressed; an instrument’s knows every key held, so several Keyboard nodes can be on at once. In an instrument, a capital letter counts as the same key as its lower-case letter, and keys typed into a text field are ignored.

Hand and Body

The hand and body nodes read the project’s camera. Turn it on with the camera button in the Media Bar; if an instrument or shader needs it and it is off, an Enable camera message offers to start it. The same camera and the same tracking are shared by every node, in shaders and instruments, so using both costs no more.

• Gesture Data follows a hand. hand is 0 for the first hand seen and 1 for the second. The Gesture Landmark 2D and 3D nodes read it: choose a landmark (the Wrist, a fingertip, a knuckle…) and get its position in the view, from 0 to 1 and mirrored as in a mirror, so your right hand is on the right. 3D adds z, the depth relative to the wrist, which is a small number.
• Hand Gesture outputs 1 while a hand makes a chosen gesture: Closed Fist, Open Palm, Pointing Up, Thumb Down, Thumb Up, Victory or I Love You.
• Pose Data follows a body, and Pose Landmark 2D and 3D give the position of a landmark such as the Nose, a shoulder or a wrist, in the same way.

In a shader these nodes take the camera as an input (wire a Camera node into Gesture Data or Pose Data). In an instrument they have no such input: they use the project’s camera directly.

Tracking has a delay of about a tenth of a second, so use it to shape a sound or a look, not to time notes. When nobody is in view, an instrument’s landmark nodes give their when not live values.

MIDI Controllers

A MIDI keyboard or controller plays notes, and its knobs, faders, touch strips and pads become live input too. In the project settings, under Music, MIDI, connect the device and click Learn beside one of the sixteen controls, then move a knob: it is remembered, with a name you give it, in the project. A MIDI Control node then outputs where that control is, from 0 to 1, in a shader or an instrument, and a MIDI Note node gives a pad or key as a switch. Until a control has been moved, the node gives its when not live value. See Playing from a MIDI Keyboard.

When Not Live

Every live input node in an instrument has when not live values (the x when not live of Mouse, the when not live switch of Keyboard and Hand Gesture, and so on). They are what the node gives when nobody is there to play: in a recording or an export, before the mouse has moved, and when no hand or body is in view. Because of this, a recording is the same every time, and an instrument that uses live input is always played live, never rendered ahead of time.

Playing Notes Live

Live input can also play notes, not just move values.

• Set a Music event’s Notes to Live (or right-click a Music track and choose Add Live Music), and the computer keyboard and an on-screen piano play its instrument while the event runs. The keys play whenever the Runsheet runs and the Live event plays, unless you are typing into a field.
• Give a Live event a note graph to play it from the mouse, keys, a hand or a body; the graph then decides every note that event plays (a Played Key node in it brings in what you play on the keyboard or a MIDI keyboard, one copy of the graph for each key held). A note graph ends in Note Out nodes, with Scale Note to keep the notes in a scale and Threshold to turn a moving value into a switch. A small badge at the end of such an event on the Runsheet opens its note graph.
• While the Node Editor is open on an instrument, the piano plays that instrument so you can hear it as you build it.
• The piano can be closed with its X when it is in the way (it covers part of the Piano Roll); a small Piano tab with a keyboard icon takes its place, and one click brings it back. It remembers what you chose. The computer keyboard keeps playing while it is closed. A lesson closes it while it runs, except the lessons about playing live.

Everything about these is in Instruments and Music. What you play can be recorded as notes with the piano’s Record button, then looped, layered and edited like any other part - see Recording What You Play. A Live event itself is not part of a video recording or an export.

Input from a presenter window on another display, or from another browser window, can feed the same nodes - see Presenting.