|
|
|
|
@ -9,16 +9,52 @@ This application is composed of two parts:
|
|
|
|
|
The client is responsible to push the server to the device and start its
|
|
|
|
|
execution.
|
|
|
|
|
|
|
|
|
|
Once the client and the server are connected to each other, the server initially
|
|
|
|
|
sends device information (name and initial screen dimensions), then starts to
|
|
|
|
|
send a raw H.264 video stream of the device screen. The client decodes the video
|
|
|
|
|
frames, and display them as soon as possible, without buffering, to minimize
|
|
|
|
|
latency. The client is not aware of the device rotation (which is handled by the
|
|
|
|
|
server), it just knows the dimensions of the video frames.
|
|
|
|
|
The client and the server establish communication using separate sockets for
|
|
|
|
|
video, audio and controls. Any of them may be disabled (but not all), so
|
|
|
|
|
there are 1, 2 or 3 socket(s).
|
|
|
|
|
|
|
|
|
|
The client captures relevant keyboard and mouse events, that it transmits to the
|
|
|
|
|
server, which injects them to the device.
|
|
|
|
|
The server initially sends the device name on the first socket (it is used for
|
|
|
|
|
the scrcpy window title), then each socket is used for its own purpose. All
|
|
|
|
|
reads and writes are performed from a dedicated thread for each socket, both on
|
|
|
|
|
the client and on the server.
|
|
|
|
|
|
|
|
|
|
If video is enabled, then the server sends a raw video stream (H.264 by default)
|
|
|
|
|
of the device screen, with some additional headers for each packet. The client
|
|
|
|
|
decodes the video frames, and displays them as soon as possible, without
|
|
|
|
|
buffering (unless `--display-buffer=delay` is specified) to minimize latency.
|
|
|
|
|
The client is not aware of the device rotation (which is handled by the server),
|
|
|
|
|
it just knows the dimensions of the video frames it receives.
|
|
|
|
|
|
|
|
|
|
Similarly, if audio is enabled, then the server sends a raw audio stream (OPUS
|
|
|
|
|
by default) of the device audio output (or the microphone if
|
|
|
|
|
`--audio-source=mic` is specified), with some additional headers for each
|
|
|
|
|
packet. The client decodes the stream, attempts to keep a minimal latency by
|
|
|
|
|
maintaining an average buffering. The [blog post][scrcpy2] of the scrcpy v2.0
|
|
|
|
|
release gives more details about the audio feature.
|
|
|
|
|
|
|
|
|
|
If control is enabled, then the client captures relevant keyboard and mouse
|
|
|
|
|
events, that it transmits to the server, which injects them to the device. This
|
|
|
|
|
is the only socket which is used in both direction: input events are sent from
|
|
|
|
|
the client to the device, and when the device clipboard changes, the new content
|
|
|
|
|
is sent from the device to the client to support seamless copy-paste.
|
|
|
|
|
|
|
|
|
|
[scrcpy2]: https://blog.rom1v.com/2023/03/scrcpy-2-0-with-audio/
|
|
|
|
|
|
|
|
|
|
Note that the client-server roles are expressed at the application level:
|
|
|
|
|
|
|
|
|
|
- the server _serves_ video and audio streams, and handle requests from the
|
|
|
|
|
client,
|
|
|
|
|
- the client _controls_ the device through the server.
|
|
|
|
|
|
|
|
|
|
However, by default (when `--force-adb-forward` is not set), the roles are
|
|
|
|
|
reversed at the network level:
|
|
|
|
|
|
|
|
|
|
- the client opens a server socket and listen on a port before starting the
|
|
|
|
|
server,
|
|
|
|
|
- the server connects to the client.
|
|
|
|
|
|
|
|
|
|
This role inversion guarantees that the connection will not fail due to race
|
|
|
|
|
conditions without polling.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Server
|
|
|
|
|
@ -32,15 +68,14 @@ The server is a Java application (with a [`public static void main(String...
|
|
|
|
|
args)`][main] method), compiled against the Android framework, and executed as
|
|
|
|
|
`shell` on the Android device.
|
|
|
|
|
|
|
|
|
|
[main]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/server/src/main/java/com/genymobile/scrcpy/Server.java#L123
|
|
|
|
|
[main]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/Server.java#L193
|
|
|
|
|
|
|
|
|
|
To run such a Java application, the classes must be [_dexed_][dex] (typically,
|
|
|
|
|
to `classes.dex`). If `my.package.MainClass` is the main class, compiled to
|
|
|
|
|
`classes.dex`, pushed to the device in `/data/local/tmp`, then it can be run
|
|
|
|
|
with:
|
|
|
|
|
|
|
|
|
|
adb shell CLASSPATH=/data/local/tmp/classes.dex \
|
|
|
|
|
app_process / my.package.MainClass
|
|
|
|
|
adb shell CLASSPATH=/data/local/tmp/classes.dex app_process / my.package.MainClass
|
|
|
|
|
|
|
|
|
|
_The path `/data/local/tmp` is a good candidate to push the server, since it's
|
|
|
|
|
readable and writable by `shell`, but not world-writable, so a malicious
|
|
|
|
|
@ -49,7 +84,7 @@ application may not replace the server just before the client executes it._
|
|
|
|
|
Instead of a raw _dex_ file, `app_process` accepts a _jar_ containing
|
|
|
|
|
`classes.dex` (e.g. an [APK]). For simplicity, and to benefit from the gradle
|
|
|
|
|
build system, the server is built to an (unsigned) APK (renamed to
|
|
|
|
|
`scrcpy-server`).
|
|
|
|
|
`scrcpy-server.jar`).
|
|
|
|
|
|
|
|
|
|
[dex]: https://en.wikipedia.org/wiki/Dalvik_(software)
|
|
|
|
|
[apk]: https://en.wikipedia.org/wiki/Android_application_package
|
|
|
|
|
@ -65,42 +100,77 @@ They can be called using reflection though. The communication with hidden
|
|
|
|
|
components is provided by [_wrappers_ classes][wrappers] and [aidl].
|
|
|
|
|
|
|
|
|
|
[hidden]: https://stackoverflow.com/a/31908373/1987178
|
|
|
|
|
[wrappers]: https://github.com/Genymobile/scrcpy/tree/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/server/src/main/java/com/genymobile/scrcpy/wrappers
|
|
|
|
|
[aidl]: https://github.com/Genymobile/scrcpy/tree/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/server/src/main/aidl/android/view
|
|
|
|
|
[wrappers]: https://github.com/Genymobile/scrcpy/tree/master/server/src/main/java/com/genymobile/scrcpy/wrappers
|
|
|
|
|
[aidl]: https://github.com/Genymobile/scrcpy/tree/master/server/src/main/aidl
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Threading
|
|
|
|
|
|
|
|
|
|
The server uses 3 threads:
|
|
|
|
|
### Execution
|
|
|
|
|
|
|
|
|
|
- the **main** thread, encoding and streaming the video to the client;
|
|
|
|
|
- the **controller** thread, listening for _control messages_ (typically,
|
|
|
|
|
keyboard and mouse events) from the client;
|
|
|
|
|
- the **receiver** thread (managed by the controller), sending _device messages_
|
|
|
|
|
to the clients (currently, it is only used to send the device clipboard
|
|
|
|
|
content).
|
|
|
|
|
The server is started by the client basically by executing the following
|
|
|
|
|
commands:
|
|
|
|
|
|
|
|
|
|
Since the video encoding is typically hardware, there would be no benefit in
|
|
|
|
|
encoding and streaming in two different threads.
|
|
|
|
|
```bash
|
|
|
|
|
adb push scrcpy-server /data/local/tmp/scrcpy-server.jar
|
|
|
|
|
adb forward tcp:27183 localabstract:scrcpy
|
|
|
|
|
adb shell CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server 2.1
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The first argument (`2.1` in the example) is the client scrcpy version. The
|
|
|
|
|
server fails if the client and the server do not have the exact same version.
|
|
|
|
|
The protocol between the client and the server may change from version to
|
|
|
|
|
version (see [protocol](#protocol) below), and there is no backward or forward
|
|
|
|
|
compatibility (there is no point to use different client and server versions).
|
|
|
|
|
This check allows to detect misconfiguration (running an older or newer server
|
|
|
|
|
by mistake).
|
|
|
|
|
|
|
|
|
|
It is followed by any number of arguments, in the form of `key=value` pairs.
|
|
|
|
|
Their order is irrelevant. The possible keys and associated value types can be
|
|
|
|
|
found in the [server][server-options] and [client][client-options] code.
|
|
|
|
|
|
|
|
|
|
[server-options]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/Options.java#L181
|
|
|
|
|
[client-options]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/app/src/server.c#L226
|
|
|
|
|
|
|
|
|
|
For example, if we execute `scrcpy -m1920 --no-audio`, then the server
|
|
|
|
|
execution will look like this:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
# scid is a random number to identify different clients running on the same device
|
|
|
|
|
adb shell CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server 2.1 scid=12345678 log_level=info audio=false max_size=1920
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Components
|
|
|
|
|
|
|
|
|
|
When executed, its [`main()`][main] method is executed (on the "main" thread).
|
|
|
|
|
It parses the arguments, establishes the connection with the client and starts
|
|
|
|
|
the other "components":
|
|
|
|
|
- the **video** streamer: it captures the video screen and send encoded video
|
|
|
|
|
packets on the _video_ socket (from the _video_ thread).
|
|
|
|
|
- the **audio** streamer: it uses several threads to capture raw packets,
|
|
|
|
|
submits them to encoding and retrieve encoded packets, which it sends on the
|
|
|
|
|
_audio_ socket.
|
|
|
|
|
- the **controller**: it receives _control messages_ (typically input events)
|
|
|
|
|
on the _control_ socket from one thread, and sends _device messages_ (e.g. to
|
|
|
|
|
transmit the device clipboard content to the client) on the same _control
|
|
|
|
|
socket_ from another thread. Thus, the _control_ socket is used in both
|
|
|
|
|
directions (contrary to the _video_ and _audio_ sockets).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Screen video encoding
|
|
|
|
|
|
|
|
|
|
The encoding is managed by [`ScreenEncoder`].
|
|
|
|
|
|
|
|
|
|
The video is encoded using the [`MediaCodec`] API. The codec takes its input
|
|
|
|
|
from a [surface] associated to the display, and writes the resulting H.264
|
|
|
|
|
stream to the provided output stream (the socket connected to the client).
|
|
|
|
|
The video is encoded using the [`MediaCodec`] API. The codec encodes the content
|
|
|
|
|
of a `Surface` associated to the display, and writes the encoding packets to the
|
|
|
|
|
client (on the _video_ socket).
|
|
|
|
|
|
|
|
|
|
[`ScreenEncoder`]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
|
|
|
|
|
[`ScreenEncoder`]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
|
|
|
|
|
[`MediaCodec`]: https://developer.android.com/reference/android/media/MediaCodec.html
|
|
|
|
|
[surface]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L68-L69
|
|
|
|
|
|
|
|
|
|
On device [rotation], the codec, surface and display are reinitialized, and a
|
|
|
|
|
new video stream is produced.
|
|
|
|
|
On device rotation (or folding), the encoding session is [reset] and restarted.
|
|
|
|
|
|
|
|
|
|
New frames are produced only when changes occur on the surface. This is good
|
|
|
|
|
because it avoids to send unnecessary frames, but there are drawbacks:
|
|
|
|
|
New frames are produced only when changes occur on the surface. This avoids to
|
|
|
|
|
send unnecessary frames, but by default there might be drawbacks:
|
|
|
|
|
|
|
|
|
|
- it does not send any frame on start if the device screen does not change,
|
|
|
|
|
- after fast motion changes, the last frame may have poor quality.
|
|
|
|
|
@ -108,11 +178,24 @@ because it avoids to send unnecessary frames, but there are drawbacks:
|
|
|
|
|
Both problems are [solved][repeat] by the flag
|
|
|
|
|
[`KEY_REPEAT_PREVIOUS_FRAME_AFTER`][repeat-flag].
|
|
|
|
|
|
|
|
|
|
[reset]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L179
|
|
|
|
|
[rotation]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L90
|
|
|
|
|
[repeat]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L147-L148
|
|
|
|
|
[repeat]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L246-L247
|
|
|
|
|
[repeat-flag]: https://developer.android.com/reference/android/media/MediaFormat.html#KEY_REPEAT_PREVIOUS_FRAME_AFTER
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Audio encoding
|
|
|
|
|
|
|
|
|
|
Similarly, the audio is [captured] using an [`AudioRecord`], and [encoded] using
|
|
|
|
|
the [`MediaCodec`] asynchronous API.
|
|
|
|
|
|
|
|
|
|
More details are available on the [blog post][scrcpy2] introducing the audio feature.
|
|
|
|
|
|
|
|
|
|
[captured]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/AudioCapture.java
|
|
|
|
|
[encoded]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/AudioEncoder.java
|
|
|
|
|
[`AudioRecord`]: https://developer.android.com/reference/android/media/AudioRecord
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Input events injection
|
|
|
|
|
|
|
|
|
|
_Control messages_ are received from the client by the [`Controller`] (run in a
|
|
|
|
|
@ -124,13 +207,13 @@ separate thread). There are several types of input events:
|
|
|
|
|
- other commands (e.g. to switch the screen on or to copy the clipboard).
|
|
|
|
|
|
|
|
|
|
Some of them need to inject input events to the system. To do so, they use the
|
|
|
|
|
_hidden_ method [`InputManager.injectInputEvent`] (exposed by our
|
|
|
|
|
_hidden_ method [`InputManager.injectInputEvent()`] (exposed by the
|
|
|
|
|
[`InputManager` wrapper][inject-wrapper]).
|
|
|
|
|
|
|
|
|
|
[`Controller`]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/server/src/main/java/com/genymobile/scrcpy/Controller.java#L81
|
|
|
|
|
[`Controller`]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/Controller.java
|
|
|
|
|
[`KeyEvent`]: https://developer.android.com/reference/android/view/KeyEvent.html
|
|
|
|
|
[`MotionEvent`]: https://developer.android.com/reference/android/view/MotionEvent.html
|
|
|
|
|
[`InputManager.injectInputEvent`]: https://android.googlesource.com/platform/frameworks/base/+/oreo-release/core/java/android/hardware/input/InputManager.java#857
|
|
|
|
|
[`InputManager.injectInputEvent()`]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/wrappers/InputManager.java#L34
|
|
|
|
|
[inject-wrapper]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/server/src/main/java/com/genymobile/scrcpy/wrappers/InputManager.java#L27
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@ -140,126 +223,222 @@ _hidden_ method [`InputManager.injectInputEvent`] (exposed by our
|
|
|
|
|
The client relies on [SDL], which provides cross-platform API for UI, input
|
|
|
|
|
events, threading, etc.
|
|
|
|
|
|
|
|
|
|
The video stream is decoded by [libav] (FFmpeg).
|
|
|
|
|
The video and audio streams are decoded by [FFmpeg].
|
|
|
|
|
|
|
|
|
|
[SDL]: https://www.libsdl.org
|
|
|
|
|
[libav]: https://www.libav.org/
|
|
|
|
|
[ffmpeg]: https://ffmpeg.org/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Initialization
|
|
|
|
|
|
|
|
|
|
On startup, in addition to _libav_ and _SDL_ initialization, the client must
|
|
|
|
|
push and start the server on the device, and open two sockets (one for the video
|
|
|
|
|
stream, one for control) so that they may communicate.
|
|
|
|
|
The client parses the command line arguments, then [runs one of two code
|
|
|
|
|
paths][run]:
|
|
|
|
|
- scrcpy in "normal" mode ([`scrcpy.c`])
|
|
|
|
|
- scrcpy in [OTG mode](hid-otg.md) ([`scrcpy_otg.c`])
|
|
|
|
|
|
|
|
|
|
Note that the client-server roles are expressed at the application level:
|
|
|
|
|
[run]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/app/src/main.c#L81-L82
|
|
|
|
|
[`scrcpy.c`]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/app/src/scrcpy.c#L292-L293
|
|
|
|
|
[`scrcpy_otg.c`]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/app/src/usb/scrcpy_otg.c#L51-L52
|
|
|
|
|
|
|
|
|
|
- the server _serves_ video stream and handle requests from the client,
|
|
|
|
|
- the client _controls_ the device through the server.
|
|
|
|
|
In the remaining of this document, we assume that the "normal" mode is used
|
|
|
|
|
(read the code for the OTG mode).
|
|
|
|
|
|
|
|
|
|
However, the roles are reversed at the network level:
|
|
|
|
|
|
|
|
|
|
- the client opens a server socket and listen on a port before starting the
|
|
|
|
|
server,
|
|
|
|
|
- the server connects to the client.
|
|
|
|
|
|
|
|
|
|
This role inversion guarantees that the connection will not fail due to race
|
|
|
|
|
conditions, and avoids polling.
|
|
|
|
|
|
|
|
|
|
_(Note that over TCP/IP, the roles are not reversed, due to a bug in `adb
|
|
|
|
|
reverse`. See commit [1038bad] and [issue #5].)_
|
|
|
|
|
|
|
|
|
|
Once the server is connected, it sends the device information (name and initial
|
|
|
|
|
screen dimensions). Thus, the client may init the window and renderer, before
|
|
|
|
|
the first frame is available.
|
|
|
|
|
|
|
|
|
|
To minimize startup time, SDL initialization is performed while listening for
|
|
|
|
|
the connection from the server (see commit [90a46b4]).
|
|
|
|
|
|
|
|
|
|
[1038bad]: https://github.com/Genymobile/scrcpy/commit/1038bad3850f18717a048a4d5c0f8110e54ee172
|
|
|
|
|
[issue #5]: https://github.com/Genymobile/scrcpy/issues/5
|
|
|
|
|
[90a46b4]: https://github.com/Genymobile/scrcpy/commit/90a46b4c45637d083e877020d85ade52a9a5fa8e
|
|
|
|
|
On startup, the client:
|
|
|
|
|
- opens the _video_, _audio_ and _control_ sockets;
|
|
|
|
|
- pushes and starts the server on the device;
|
|
|
|
|
- initializes its components (demuxers, decoders, recorder…).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Threading
|
|
|
|
|
### Video and audio streams
|
|
|
|
|
|
|
|
|
|
The client uses 4 threads:
|
|
|
|
|
|
|
|
|
|
- the **main** thread, executing the SDL event loop,
|
|
|
|
|
- the **stream** thread, receiving the video and used for decoding and
|
|
|
|
|
recording,
|
|
|
|
|
- the **controller** thread, sending _control messages_ to the server,
|
|
|
|
|
- the **receiver** thread (managed by the controller), receiving _device
|
|
|
|
|
messages_ from the server.
|
|
|
|
|
|
|
|
|
|
In addition, another thread can be started if necessary to handle APK
|
|
|
|
|
installation or file push requests (via drag&drop on the main window) or to
|
|
|
|
|
print the framerate regularly in the console.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Stream
|
|
|
|
|
|
|
|
|
|
The video [stream] is received from the socket (connected to the server on the
|
|
|
|
|
device) in a separate thread.
|
|
|
|
|
|
|
|
|
|
If a [decoder] is present (i.e. `--no-display` is not set), then it uses _libav_
|
|
|
|
|
to decode the H.264 stream from the socket, and notifies the main thread when a
|
|
|
|
|
new frame is available.
|
|
|
|
|
|
|
|
|
|
There are two [frames][video_buffer] simultaneously in memory:
|
|
|
|
|
- the **decoding** frame, written by the decoder from the decoder thread,
|
|
|
|
|
- the **rendering** frame, rendered in a texture from the main thread.
|
|
|
|
|
|
|
|
|
|
When a new decoded frame is available, the decoder _swaps_ the decoding and
|
|
|
|
|
rendering frame (with proper synchronization). Thus, it immediately starts
|
|
|
|
|
to decode a new frame while the main thread renders the last one.
|
|
|
|
|
|
|
|
|
|
If a [recorder] is present (i.e. `--record` is enabled), then it muxes the raw
|
|
|
|
|
H.264 packet to the output video file.
|
|
|
|
|
|
|
|
|
|
[stream]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/app/src/stream.h
|
|
|
|
|
[decoder]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/app/src/decoder.h
|
|
|
|
|
[video_buffer]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/app/src/video_buffer.h
|
|
|
|
|
[recorder]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/app/src/recorder.h
|
|
|
|
|
Depending on the arguments passed to `scrcpy`, several components may be used.
|
|
|
|
|
Here is an overview of the video and audio components:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
+----------+ +----------+
|
|
|
|
|
---> | decoder | ---> | screen |
|
|
|
|
|
+---------+ / +----------+ +----------+
|
|
|
|
|
socket ---> | stream | ----
|
|
|
|
|
+---------+ \ +----------+
|
|
|
|
|
---> | recorder |
|
|
|
|
|
+----------+
|
|
|
|
|
V4L2 sink
|
|
|
|
|
/
|
|
|
|
|
decoder
|
|
|
|
|
/ \
|
|
|
|
|
VIDEO -------------> demuxer display
|
|
|
|
|
\
|
|
|
|
|
recorder
|
|
|
|
|
/
|
|
|
|
|
AUDIO -------------> demuxer
|
|
|
|
|
\
|
|
|
|
|
decoder --- audio player
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The _demuxer_ is responsible to extract video and audio packets (read some
|
|
|
|
|
header, split the video stream into packets at correct boundaries, etc.).
|
|
|
|
|
|
|
|
|
|
The demuxed packets may be sent to a _decoder_ (one per stream, to produce
|
|
|
|
|
frames) and to a recorder (receiving both video and audio stream to record a
|
|
|
|
|
single file). The packets are encoded on the device (by `MediaCodec`), but when
|
|
|
|
|
recording, they are _muxed_ (asynchronously) into a container (MKV or MP4) on
|
|
|
|
|
the client side.
|
|
|
|
|
|
|
|
|
|
Video frames are sent to the screen/display to be rendered in the scrcpy window.
|
|
|
|
|
They may also be sent to a [V4L2 sink](v4l2.md).
|
|
|
|
|
|
|
|
|
|
Audio "frames" (an array of decoded samples) are sent to the audio player.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Controller
|
|
|
|
|
|
|
|
|
|
The [controller] is responsible to send _control messages_ to the device. It
|
|
|
|
|
The _controller_ is responsible to send _control messages_ to the device. It
|
|
|
|
|
runs in a separate thread, to avoid I/O on the main thread.
|
|
|
|
|
|
|
|
|
|
On SDL event, received on the main thread, the [input manager][inputmanager]
|
|
|
|
|
creates appropriate [_control messages_][controlmsg]. It is responsible to
|
|
|
|
|
convert SDL events to Android events (using [convert]). It pushes the _control
|
|
|
|
|
messages_ to a queue hold by the controller. On its own thread, the controller
|
|
|
|
|
takes messages from the queue, that it serializes and sends to the client.
|
|
|
|
|
|
|
|
|
|
[controller]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/app/src/controller.h
|
|
|
|
|
[controlmsg]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/app/src/control_msg.h
|
|
|
|
|
[inputmanager]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/app/src/input_manager.h
|
|
|
|
|
[convert]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/app/src/convert.h
|
|
|
|
|
On SDL event, received on the main thread, the _input manager_ creates
|
|
|
|
|
appropriate _control messages_. It is responsible to convert SDL events to
|
|
|
|
|
Android events. It then pushes the _control messages_ to a queue hold by the
|
|
|
|
|
controller. On its own thread, the controller takes messages from the queue,
|
|
|
|
|
that it serializes and sends to the client.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### UI and event loop
|
|
|
|
|
## Protocol
|
|
|
|
|
|
|
|
|
|
Initialization, input events and rendering are all [managed][scrcpy] in the main
|
|
|
|
|
thread.
|
|
|
|
|
The protocol between the client and the server must be considered _internal_: it
|
|
|
|
|
may (and will) change at any time for any reason. Everything may change (the
|
|
|
|
|
number of sockets, the order in which the sockets must be opened, the data
|
|
|
|
|
format on the wire…) from version to version. A client must always be run with a
|
|
|
|
|
matching server version.
|
|
|
|
|
|
|
|
|
|
Events are handled in the [event loop], which either updates the [screen] or
|
|
|
|
|
delegates to the [input manager][inputmanager].
|
|
|
|
|
This section documents the current protocol in scrcpy v2.1.
|
|
|
|
|
|
|
|
|
|
[scrcpy]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/app/src/scrcpy.c
|
|
|
|
|
[event loop]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/app/src/scrcpy.c#L201
|
|
|
|
|
[screen]: https://github.com/Genymobile/scrcpy/blob/ffe0417228fb78ab45b7ee4e202fc06fc8875bf3/app/src/screen.h
|
|
|
|
|
### Connection
|
|
|
|
|
|
|
|
|
|
Firstly, the client sets up an adb tunnel:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
# By default, a reverse redirection: the computer listens, the device connects
|
|
|
|
|
adb reverse localabstract:scrcpy_<SCID> tcp:27183
|
|
|
|
|
|
|
|
|
|
# As a fallback (or if --force-adb forward is set), a forward redirection:
|
|
|
|
|
# the device listens, the computer connects
|
|
|
|
|
adb forward tcp:27183 localabstract:scrcpy_<SCID>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
(`<SCID>` is a 31-bit random number, so that it does not fail when several
|
|
|
|
|
scrcpy instances start "at the same time" for the same device.)
|
|
|
|
|
|
|
|
|
|
Then, up to 3 sockets are opened, in that order:
|
|
|
|
|
- a _video_ socket
|
|
|
|
|
- an _audio_ socket
|
|
|
|
|
- a _control_ socket
|
|
|
|
|
|
|
|
|
|
Each one may be disabled (respectively by `--no-video`, `--no-audio` and
|
|
|
|
|
`--no-control`, directly or indirectly). For example, if `--no-audio` is set,
|
|
|
|
|
then the _video_ socket is opened first, then the _control_ socket.
|
|
|
|
|
|
|
|
|
|
On the _first_ socket opened (whichever it is), if the tunnel is _forward_, then
|
|
|
|
|
a [dummy byte] is sent from the device to the client. This allows to detect a
|
|
|
|
|
connection error (the client connection does not fail as long as there is an adb
|
|
|
|
|
forward redirection, even if nothing is listening on the device side).
|
|
|
|
|
|
|
|
|
|
Still on this _first_ socket, the device sends some [metadata][device meta] to
|
|
|
|
|
the client (currently only the device name, used as the window title, but there
|
|
|
|
|
might be other fields in the future).
|
|
|
|
|
|
|
|
|
|
[dummy byte]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/DesktopConnection.java#L93
|
|
|
|
|
[device meta]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/DesktopConnection.java#L151
|
|
|
|
|
|
|
|
|
|
You can read the [client][client-connection] and [server][server-connection]
|
|
|
|
|
code for more details.
|
|
|
|
|
|
|
|
|
|
[client-connection]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/app/src/server.c#L465-L466
|
|
|
|
|
[server-connection]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/DesktopConnection.java#L63
|
|
|
|
|
|
|
|
|
|
Then each socket is used for its intended purpose.
|
|
|
|
|
|
|
|
|
|
### Video and audio
|
|
|
|
|
|
|
|
|
|
On the _video_ and _audio_ sockets, the device first sends some [codec
|
|
|
|
|
metadata]:
|
|
|
|
|
- On the _video_ socket, 12 bytes:
|
|
|
|
|
- the codec id (`u32`) (H264, H265 or AV1)
|
|
|
|
|
- the initial video width (`u32`)
|
|
|
|
|
- the initial video height (`u32`)
|
|
|
|
|
- On the _audio_ socket, 4 bytes:
|
|
|
|
|
- the codec id ('u32`) (OPUS, AAC or RAW)
|
|
|
|
|
|
|
|
|
|
[codec metadata]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/Streamer.java#L33-L51
|
|
|
|
|
|
|
|
|
|
Then each packet produced by `MediaCodec` is sent, prefixed by a 12-byte [frame
|
|
|
|
|
header]:
|
|
|
|
|
- config packet flag (`u1`)
|
|
|
|
|
- key frame flag (`u1`)
|
|
|
|
|
- PTS (`u62`)
|
|
|
|
|
- packet size (`u32`)
|
|
|
|
|
|
|
|
|
|
Here is a schema describing the frame header:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
[. . . . . . . .|. . . .]. . . . . . . . . . . . . . . ...
|
|
|
|
|
<-------------> <-----> <-----------------------------...
|
|
|
|
|
PTS packet raw packet
|
|
|
|
|
size
|
|
|
|
|
<--------------------->
|
|
|
|
|
frame header
|
|
|
|
|
|
|
|
|
|
The most significant bits of the PTS are used for packet flags:
|
|
|
|
|
|
|
|
|
|
byte 7 byte 6 byte 5 byte 4 byte 3 byte 2 byte 1 byte 0
|
|
|
|
|
CK...... ........ ........ ........ ........ ........ ........ ........
|
|
|
|
|
^^<------------------------------------------------------------------->
|
|
|
|
|
|| PTS
|
|
|
|
|
| `- key frame
|
|
|
|
|
`-- config packet
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
[frame header]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/Streamer.java#L83
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Controls
|
|
|
|
|
|
|
|
|
|
Controls messages are sent via a custom binary protocol.
|
|
|
|
|
|
|
|
|
|
The only documentation for this protocol is the set of unit tests on both sides:
|
|
|
|
|
- `ControlMessage` (from client to device): [serialization](https://github.com/Genymobile/scrcpy/blob/master/app/tests/test_control_msg_serialize.c) | [deserialization](https://github.com/Genymobile/scrcpy/blob/master/server/src/test/java/com/genymobile/scrcpy/ControlMessageReaderTest.java)
|
|
|
|
|
- `DeviceMessage` (from device to client) [serialization](https://github.com/Genymobile/scrcpy/blob/master/server/src/test/java/com/genymobile/scrcpy/DeviceMessageWriterTest.java) | [deserialization](https://github.com/Genymobile/scrcpy/blob/master/app/tests/test_device_msg_deserialize.c)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Standalone server
|
|
|
|
|
|
|
|
|
|
Although the server is designed to work for the scrcpy client, it can be used
|
|
|
|
|
with any client which uses the same protocol.
|
|
|
|
|
|
|
|
|
|
For simplicity, some [server-specific options] have been added to produce raw
|
|
|
|
|
streams easily:
|
|
|
|
|
- `send_device_meta=false`: disable the device metata (in practice, the device
|
|
|
|
|
name) sent on the _first_ socket
|
|
|
|
|
- `send_frame_meta=false`: disable the 12-byte header for each packet
|
|
|
|
|
- `send_dummy_byte`: disable the dummy byte sent on forward connections
|
|
|
|
|
- `send_codec_meta`: disable the codec information (and initial device size for
|
|
|
|
|
video)
|
|
|
|
|
- `raw_stream`: disable all the above
|
|
|
|
|
|
|
|
|
|
[server-specific options]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/Options.java#L309-L329
|
|
|
|
|
|
|
|
|
|
Concretely, here is how to expose a raw H.264 stream on a TCP socket:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
adb push scrcpy-server-v2.1 /data/local/tmp/scrcpy-server-manual.jar
|
|
|
|
|
adb forward tcp:1234 localabstract:scrcpy
|
|
|
|
|
adb shell CLASSPATH=/data/local/tmp/scrcpy-server-manual.jar \
|
|
|
|
|
app_process / com.genymobile.scrcpy.Server 2.1 \
|
|
|
|
|
tunnel_forward=true audio=false control=false cleanup=false \
|
|
|
|
|
raw_stream=true max_size=1920
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
As soon as a client connects over TCP on port 1234, the device will start
|
|
|
|
|
streaming the video. For example, VLC can play the video (although you will
|
|
|
|
|
experience a very high latency, more details [here][vlc-0latency]):
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
vlc -Idummy --demux=h264 --network-caching=0 tcp://localhost:1234
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
[vlc-0latency]: https://code.videolan.org/rom1v/vlc/-/merge_requests/20
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Hack
|
|
|
|
|
|
