The real-time / not real-time aspect of vFx

General Discussion
,
#1

I saw some amazing work going by on the facebook group, including Matt Radford’s efforts to find a process for stylizing renders. So I thought it would be amazing to take it one step further. I haven’t done any experiments yet as I’m still unpacking from a recent move, but I thought I’d share some resources I’ve come across.

There’s tons of rendering things that we do to make flipbooks, but I’m seriously intrigued by neural nets.

Neural net program for stylizing that is fairly in reach for most scripters.

Same, but exposed to twitter.
http://www.makeuseof.com/tag/create-neural-paintings-deepstyle-ubuntu/
It occurs to me that those of us who can’t code, can still send individual frames to the twitter bot until we have an animation that we can put through twixtor or slate.

Here’s a bunch of random links I use to try to get my programmer friends interested.
https://deeplearning4j.org/convolutionalnets
But if you want more basic understanding of neural nets…

Feature extraction.
http://ufldl.stanford.edu/tutorial/supervised/FeatureExtractionUsingConvolution/
oooh…nice crash course

but not completely related to photo manipulation

So my idea is to render a subset of frames for the style, but use the full frame blending vectors, like the following.
Shout out to Klemen Lozar : Klemen Lozar's Online Portfolio

The very latest from adobe labs: GitHub - luanfujun/deep-photo-styletransfer: Code and data for paper "Deep Photo Style Transfer": https://arxiv.org/abs/1703.07511

2 Likes
#2

Awesome man. I saw this posted somewhere a while back and it’s definitely interesting stuff! Is your ultimate goal to get this sort of thing to play in a real-time engine as some sort of post process effect?

#3

I don’t think it can be made to run fast enough, and I wouldn’t be the person to move the entire library onto a GPU. But what I do want to try is making flipbooks with it.

1 Like
#4

This might not be comparable to neural paintings, but this guy made some paint effect at realtime using photogrametry technic and vector painting on his assets :

And this one is a pack of interesting stylised realtime post-process on unreal4 (using only nodal scripting I believe) : Illustrious: The illustration style toolkit - Showcase - Unreal Engine Forums

So without being complex and heavy algorithm, im sure there is way in real-time to make interesting effects like neural painting. I want to see more =)

3 Likes
#5

That’s pretty damn amazing.

#6

Everything in this thread is amazing. I’m bookmarking for later.

#7

I drop this link everywhere these days…
https://twitter.com/_kzr?lang=en

2 Likes
#10

That’s great! I work on a similar technique with PopcornFx but with a lot of calculation of light (like reflection, shadow, sss,…) directly in the particles with raycast.