The main thing you can do is set MaxUpdateRate on the VncServerSession. It defaults to 15 (frames per second).
VncScreenFramebufferSource also has a constructor you can pass screen bounds to, if you want VNC to only show a certain program's window bounds for instance.

Beyond this, what kind of response are you hoping to improve? (Are you running out of CPU? VncFramebufferCache is *not* very efficient right now.)

I'm on a final-project with my teammates now.
Our goal is to dev a VNC program and remotely control the other computer.
and I started with yours, cause it's based on C# and working nicely:)  (thank you alot Zer!)

so I add some code to transfer mouse&keyboard command
and fortunately works well!

but I faced off a problem.

different from cmd transferring,
the screen refresh rate doesn't reach our expect.

How can I improve the overall response?
I've tried several ways to do this,
but I failed cause it's program-constructively FIRM to me
I don't know even where i should add or modify code for it.

I hope you to read this post. and Thank you a lot for your effort
It was very helpful and I promise to include credit at the end of mine!