Interactive Illustrations

This is an idea I’ve been toying with for quite some time. Letting programming and painting meet somewhere in the neighborhood of illustration / animation / motion graphics / creative coding / games. There’s not much of a concept yet, it will take some trial and error, but a few central ideas are:

  • each illustration should work as a stand-alone still image
  • I want to “add magic” by animating things (the moving pictures in Harry Potter would be an ideal)
  • some interactivity should be there, otherwise it would just be a video

So here’s a first demo, where I explored how to do something like this in a web browser. This is from a painting I recently developed from an old inktober sketch, titled “Sugar Raid” for obvious reasons 🙂

Click on the image to open the interactive version in a new window:

Sugar Raid!

A few technical details

The illustration itself was done in Procreate (the iPad painting app), where each moving object resides on its own layer. The implementation of the demo required:

  • exporting the layers as image files
  • cutting and resizing the layer images, also generating a Javascript snippet with size and offset information for each layer
  • writing Javascript code (using the quite awesome three.js library) to render all the layers – essentially another implementation of how a painting app composes the layers. I use custom shaders to support blend modes such as screen, overlay, etc.
  • animating layers – pretty straightforward in three.js
  • as a boon, I added a particle system simulation (the exploding sugar cube)
  • the painful part: testing in various browsers on the desktop and smart phones

Creating videos with Processing

Other than interactive live applications, Processing can can also be used to generate cool animations from frame sequences! It‘s not so hard, but you need to consider a few things.

Randomness

If your animation uses random values, be aware that random() creates different results each time the program is run. To make sure you always get the same good-looking frame sequence of your choice, use randomSeed().

Recording the frame sequence

Set resolution and frame rate in setup():

void setup()
{
size(640, 640);
frameRate(30);
}

The frame rate doesn’t have to be the same as in the final video, you can speed up or slow down the sequence when creating the video file.

To save the first 100 animation frames, include this snippet:

if (frameCount <= 100)
{
saveFrame(“frames-###.jpg”);
print(“.”);
}

Video file creation

On a Mac, this is easy – you can use the built-in Quicktime player tool to create a video from your image sequence.

On a PC, a good option is ffmpeg. One way to install is as part of the ImageMagick suite, which is a very handy collection of image processing tools.

It’s a command line tool with a billion options. A minimal command to generate an MP4 file from an image sequence looks like this:

ffmpeg -framerate 30 -i frames-%03d.jpg -c h264 output.mp4

Video format considerations

Depending on how you want to use your video, you’ll want to make sure format, frame rate, and image size are appropriate.

For instance, to post a video on instagram, the recommendations are:

  • Aspect 1.9 to 1:1
  • a minimum of 600×315, resp 600×600 px
  • maximum width 1080 px (anything bigger is downscaled)
  • 3 sec minimum duration, 60 sec max.
  • 30 fps maximum frame rate
  • H.264 or VP8 codecs

Happy recording!

Resonance

Ever since I started drawing, trying to understand how ‘real artists’ work and think, I was often surprised to learn about something I knew from my own profession! Even between two activities so apparently far apart as pure logic and abstract thinking on one side and making colorful marks with emotional impact on the other, one can find resonance.

Here’s the shortlist of some of my obvious and not-so-obvious findings:

Problem solving

Programming is obviously a problem solving activity. It was really news to me how artists would view much of their work as problem solving, though. It may be a concept artist, who is given the task of designing a vehicle within given constraints until tomorrow, or a painter figuring out the perfect composition, shape language, lighting, color palette…

I think the analogy carries pretty far. To solve your problems, you can work methodically top/down (resp. rough to detailed), or employ trial and error. If you didn’t solve your fundamental problems properly early on, they will come and bite you later (I haven’t heard anyone talking about a ‘bug-ridden picture’ so far, but they certainly exist 🙂 ).

Attention to detail

This seems quite obvious. What is a single misplaced character breaking a program, can be one wrong line ruining your sketch. And you should absolutely know what every element in your program / picture is doing to help achieve its goal. Useless clutter, anyone?

Teams of introverts

It seems both programmers and artists generally prefer to be left alone to work on their piece for endless hours. In professional reality, it is important, though, to communicate a lot with others, as projects get bigger and bigger all the time. Feedback is also a topic here (code reviews, art critique).

Life-long learning

To me, at least, doing something that you’ve never done before is the essence of software development, always expanding your mental toolbox, so you can tackle bigger tasks the next time. There is no finish line, where you have learned everything there is and will be. The same seems to be true for passionate artists, as far as I can tell.

Creative bursts and discipline

In coding, painting, and I guess all creative endeauvours, there are spurts of energy and inspiration, followed by long stretches of filling in the gaps, where great discipline is required to get through the project. Understanding how much making art also requires discipline and quite simply showing up and getting to work helped me a great deal. You’re not a failed artist, if you aren’t inspired every day. You’re not a bad developer, if you can’t produce brilliant code like a machine.

Composition and architecture

As much as a good initial composition can decide over success or failure of a picture, the software architecture of your system can determine the fate of your project. There are functional as well as pleasing designs, concepts such as symmetry and readability in both worlds.

Looks too easy

From personal experience, I know the complete inability of, say, upper management in some companies, to understand why all of this software business is taking so long, when certainly it can’t be that complicated and the computer is doing the work, after all. And if things seem easy and fast, nobody can see it’s the result of years of training and countless hours of practice. Then again, you’ll hear ‘oh, I could never do that’ very, very often 🙂

I’ve heard the same, anecdotally, from working artists.

Style

Obvious in the arts, but there’s personal style in programming as well! Even if everyone in your team is following the same coding guidelines (where to put the braces, etc.), chances are you’ll be able to recognize who in your team wrote a piece of code after a while. Also, you’ll find people able to adapt their style to fit into whatever project they’re working on, while others value their freedom of expression higher than anything else – they might be brilliant, but substantially harder to work with.

I could elaborate on some of these topics, but this is essentially what I found so far, Anything missing? Probably 🙂

It might be worth writing about fundamental differences between making a drawing and making software, too. Have to think a bit about that.

Pocket Observatory at Make Munich

Pocket Observatory at Make Munich! Slipped in at the very last minute, and probably had the tiniest desk in the hall. But the response was great, I am still totally overwhelmed by the positive response and all the enthusiasm. A huge THANK YOU to everyone who came over to check out the app on Gear VR! Nice people, great conversations, useful feedback, and even more ideas for future development. After two days working the desk non-stop (together with my wonderful GF) I feel a little spent but very inspired 🙂

Pocket Observatory released for Gear VR

On 3/16, Pocket Observatory has been released to the Oculus App Store for Gear VR! It has taken a lot longer than expected, but in the end, the additional iterations and feedback have improved the product tremendously. Of course, this is only version 1 – there are tons of additions on my list already, and I am open to suggestions 🙂

Here’s a link to the product page in the store.

Venturing into social VR with Pocket Observatory!

The past few weeks I’ve been working away on a really exciting feature for the upcoming Gear VR version of Pocket Observatory: You will be able to invite a friend (on the Oculus platform) and start a voice chat beneath the stars! GPS coordinates are exchanged between the app instances, so players can visit each other’s GPS locations. This is currently under review, and will hopefully be up in a few weeks in the Oculus Store.

To my knowledge, this is the very first social VR astronomy app ever! I’ve been thinking about this for quite a while during initial development, but didn’t realize how easy it would be to integrate using the Oculus platform SDK. Mind you, setting up peer-to-peer networking can still be nerve-wrecking, given the unreliable nature of network communication, but still… managed to pull this off in just a few weeks. Happy!

Check out the updated page at https://pocketobservatory.com for the details. Here’s a screenshot of the chat UI: (thinking about avatars and a shared space experience, too, but that’s for later.)

Educational VR molecules

Now that I’ve gained some experience with Virtual Reality and my astronomy app, I’m thinking educational software for VR could be a worthwhile field for future projects. So I’ve started tossing ideas about, one of which involves playing with molecules in a VR environment.

Aspirin molecule

p5.js Animated Grid

So here’s a simple processing demo I did a while ago, ported to p5.js, a library allowing you to essentially write processing in JavaScript, using an HTML canvas! As you will see when running the demo, JavaScript performance is not really where you’d like it to be when doing animation 🙂 The JavaScript engine of your browser makes a huge difference here. I recommend running this in Chrome, it’s quite noticably faster than Firefox.

Open demo in new browser tab

Source code

Pocket Observatory for Google Cardboard

Just finished and submitted the iPhone / Google Cardboard version of Pocket Observatory! It really paid off to use Unity – porting from Android with the Oculus SDK to iPhone with GoogleVR turned out to be really easy.

Here are the quirks I encountered, might be useful to know if you’re embarking on a similar project:

  • In Gear VR, system messages (e.g., asking for permissions) are displayed properly and can be confirmed while you’re in VR. On the iPhone, a standard system dialog pops up. To deal with location service permissions, I trigger the message from within a special startup scene, before entering VR mode in the main scene.
  • Texture compression support has to be adjusted with the platform. On the iPhone, compression defaults to PVRTC, which requires square textures. The Unity importer stretches non-square textures to make them  compressible with PVRTC. This results in awful artifacts, so I had to go over the compressions options for all of my (non-square) textures.
  • Make sure the text for camera use permission is set in the iOS player settings – in GoogleVR, there is a UI button to allow the user to switch viewers. This will activate the phone’s camera in order to scan the QR code on the viewer. Not setting the text will result in an app crash.
  • Unfortunately, the Cardboard app doesn’t run in the simulator – there is no suitable architecture of the gvr library, so the app crashes at startup. I guess it would be possible to build the library from source, but haven’t tried that yet.

Visit https://pocketobservatory.com for details regarding app features and release plans!