We Can All Be Video Game Characters With This AI

We Can All Be Video Game Characters With This AI

Dear Fellow Scholars, this is Two Minute Papers
with Károly Zsolnai-Fehér. The title of this paper is very descriptive,
it says Controllable Characters Extracted from Real-World Videos. This sounds a little like science fiction,
so let’s pick this apart. If we forget about the controllable part,
we get something that you’ve seen in this series many times – pose estimation. Pose estimation means that we have a human
character in an image or a video, have a computer program look at it, and tell us the current
position this character is taking. This is useful for medical applications, such
as detecting issues with motor functionality, fall detection, or we can also use it for
motion capture for our video games and blockbuster movies. So just performing the pose estimation part
is a great invention, but relatively old news. So what’s really new here? Why is this work interesting? How does it go beyond pose estimation? Well, as a hint, the title contains an additional
word, “controllable”, so, look at this! Woo-hoo! As you see, this technique is not only able
to identify where a character is, but we can grab a controller, and move it around! This means making this character perform novel
actions, and showing it from novel views. That’s really remarkable, because this requires
a proper understanding of the video we’re watching. And this means that we can not only watch
these real-world videos, as you see this small piece of footage used for the learning, but
by performing these actions with a controller, we can make a video game out of it. Especially given that here, the background
has also been changed. To achieve this, this work contains two key
elements. Element number one is the pose2pose network
that takes an input posture and the button we pushed on the controller, and creates the
next step of the animation. And then, element number two, the pose2frame
architecture then blends this new animation step into an already existing image. The neural network that performs this is trained
in a way where it is encouraged to create these character masks in a way that is continuous
and doesn’t contain jarring jumps between the individual frames, leading to smooth and
believable movements. Now, clearly, anyone who takes a cursory look
sees that the animations are not perfect and still contain artifacts, but just imagine
that this paper is among the first introductory works on this problem. Imagine what we’ll have two more papers
down the line. I can’t wait. Thanks for watching and for your generous
support, and I’ll see you next time!

71 thoughts on “We Can All Be Video Game Characters With This AI”

  1. hi two minutes paper! it's really unfair that an AI experiment is not covered on this channel where AI with help of lasers made the "BOSE EINSTEIN CONDENSATE"

  2. The title of the video is enough for me to hit the like button before watching it.

    EDIT : Watched it. Awesome as always.

  3. It will be very interesting when AI can take a 360 degree video of a person and turn it into a fully controllable animated character.

  4. This is one of those papers that freaks me out a bit at the sheer rate of progress we are seeing in the field.

  5. In my opinion, this is not practical for game design because of the computational expense. Consider instead generating models as shown in this article: https://news.developer.nvidia.com/transforming-paintings-and-photos-into-animations-with-ai/

  6. What if you want to control more than just movement?
    Movement is already challenging enough. The AI would have to anticipate the player's next input to choose a suitable pose for fluid and realistic motion that still reflects the player's intent.
    I imagine that this way the AI could even inform the player what movements to choose.

    But if you also want to control the tennis racket, for example, that is going to make creating suitable movements a lot more complex.

  7. It probably won't have much application in videogames outside the novel factor, but I can already see this being used in porn.

  8. In the future AI will be able to create whole brand new virtual reality worlds based on just one picture or book.

  9. hello. Dear Károly Zsolnai-Fehér

    Thanks for your great work.

    In some of the popular websites you can watch videos with other languages subtitle (like Ted and coursera) which is done by volunteers from other countries.

    I'm a fan of your website and was wondering if you have the same feature? And if so, how can I use it?

    Otherwise I can be a Persian language coordinator for you and I can bring some other coordinators as well.

    Best, reza mahmoudi

  10. Haha wow – imagine those character customisers, except they can just take a short video of you and have a character that looks very similar to yourself. Incredible

  11. Aside from transposing the footage and movement on a new scene, I'm very impressed that it actually adds shadows to the environment. Does it do this via a depth map or advanced image data, or is it just a photo/scene?

  12. Style-tranfer alone has the potential to change the traditional 3d-graphics paradigm, but even with nVidia's hardware support for it we've yet to see a single game use any of these many promising AI based graphics techniques, so presumably we'll have to wait until they're in patented corporate products 🙂 Games used to be driven by innovation…

  13. Imagine wearing AR glasses at a physical sports arena while watching the best esports players alive play the best athletes that ever lived

  14. this is just mind blowing and goes beyond deep fake and all those stuff. this just opens up the possibility to construct a near perfect real world simulator. as mentioned this is just introductory work in this field.

  15. >two more papers down the line
    >moores law has went vertical and Neo-China arrives from the future bringing with it all manner of techno-eldrich post-human abominations as nick land cackles in glee over twitch livestreme

  16. This feels like the old, old 2D Prince of Persia game. Or all those 1990s adventure games with greenscreens. Amazing!

  17. I thought about creation something like this for years and researched that topic, but was too busy with job.. and now here it is. Great news, thanx man.

  18. Holy shit, that's Lisa Rhee (a youtube) dancing to Bamm at 2:40. Never imagined I'd see two of my favorite YouTube channels combined

  19. Imagine having entire games with interactive FMV. It will be like watching a movie where you literally control the protagonist.

Leave a Reply

Your email address will not be published. Required fields are marked *