Compare commits

...

37 Commits

Author SHA1 Message Date
8c154d53d6 Add support for AAC audio codec
Add option --audio-codec=aac.
2023-02-18 19:30:59 +01:00
105b482082 Add option to select audio codec
Introduce the selection mechanism. Alternative codecs will be added
later.
2023-02-18 19:30:14 +01:00
4dbb7aa685 Add --audio-bit-rate option
Add an option to configure the audio bit-rate.
2023-02-18 19:10:29 +01:00
637683ed7a Remove default bit-rate on client side
If no bit-rate is passed, let the server use the default value (8Mbps).

This avoids to define a default value on both sides, and to pass the
default bit-rate as an argument when starting the server.
2023-02-18 19:10:29 +01:00
166257b808 Disable audio if device could not capture
By default, audio is enabled (--no-audio must be explicitly passed to
disable it).

However, some devices may not support audio capture (typically devices
below Android 11, or Android 11 when the shell application is not
foreground on start).

In that case, make the server notify the client to dynamically disable
audio forwarding so that it does not wait indefinitely for an audio
stream.
2023-02-18 19:10:28 +01:00
3038bf4c3b Add record audio support
Make the recorder accept two input sources (video and audio), and mux
them into a single file.
2023-02-18 19:10:28 +01:00
b7af74b01a Rename video-specific variables in recorder
This paves the way to add audio-specific variables.
2023-02-18 19:10:28 +01:00
3f3e60dd7a Do not merge config audio packets
For video streams (at least H.264 and H.265), the config packet
containing SPS/PPS must be prepended to the next packet (the following
keyframe).

For audio streams (at least OPUS), they must not be merged.
2023-02-18 19:10:28 +01:00
d722b9de4d Add an audio demuxer
Add a demuxer which will read the stream from the audio socket.
2023-02-18 19:10:28 +01:00
6a3d311d47 Rename demuxer to video_demuxer
There will be another demuxer instance for audio.
2023-02-18 19:10:28 +01:00
a1b97fbfaa Extract OPUS extradata
For OPUS codec, FFmpeg expects the raw extradata, but MediaCodec wraps
it in some structure.

Fix the config packet to send only the raw extradata.
2023-02-18 19:10:28 +01:00
5d5a1d6888 Use a streamer to send the audio stream
Send each encoded audio packet using a streamer.
2023-02-18 19:10:28 +01:00
c19fefbcbd Encode recorded audio in OPUS on the device
For now, the encoded packets are just logged in the console.
2023-02-18 19:10:28 +01:00
bc0f51023e Capture device audio
Create an AudioRecorder to capture the audio source REMOTE_SUBMIX.

For now, the captured packets are just logged in the console.

Co-authored-by: Romain Vimont <rom@rom1v.com>
Signed-off-by: Romain Vimont <rom@rom1v.com>
2023-02-18 19:10:28 +01:00
3a8a7d5ba8 Add a new socket for audio stream
When audio is enabled, open a new socket to send the audio stream from
the device to the client.

Co-authored-by: Romain Vimont <rom@rom1v.com>
Signed-off-by: Romain Vimont <rom@rom1v.com>
2023-02-18 19:10:27 +01:00
1fe2afb4df Add --no-audio option
Audio will be enabled by default (when supported). Add an option to
disable it.

Co-authored-by: Romain Vimont <rom@rom1v.com>
Signed-off-by: Romain Vimont <rom@rom1v.com>
2023-02-18 19:10:27 +01:00
47d0c189ec Use FakeContext for Application instance
This will expose the correct package name and UID to the application
context.
2023-02-18 19:10:27 +01:00
c76264c149 Use shell package name for workarounds
For consistency.
2023-02-18 19:10:27 +01:00
be09e65fa4 Use PACKAGE_NAME from FakeContext
Remove duplicated constant.
2023-02-18 19:10:27 +01:00
de8a9a4045 Use AttributionSource from FakeContext
FakeContext already provides an AttributeSource instance.

Co-authored-by: Simon Chan <1330321+yume-chan@users.noreply.github.com>
2023-02-18 19:10:27 +01:00
4e47d78f30 Add a fake Android Context
Since scrcpy-server is not an Android application (it's a java
executable), it has no Context.

Some features will require a Context instance to get the package name
and the UID. Add a FakeContext for this purpose.

Co-authored-by: Romain Vimont <rom@rom1v.com>
Signed-off-by: Romain Vimont <rom@rom1v.com>
2023-02-18 19:10:27 +01:00
0873b08c85 Use Process.ROOT_UID
Replace ServiceManager.USER_ID by existing constant Process.ROOT_UID.
2023-02-18 19:10:27 +01:00
83c64d918b Make streamer independent of codec type
Rename VideoStreamer to Streamer, and extract a Codec interface which
will also support audio codecs.
2023-02-18 19:10:26 +01:00
aedd90a955 Pass all args to ScreenEncoder constructor
There is no good reason to pass some of them in the constructor and some
others as parameters of the streamScreen() method.
2023-02-18 19:10:26 +01:00
e153f4657c Move screen encoder initialization
This prepares further refactors.
2023-02-18 19:10:26 +01:00
dd2959b0cb Write streamer header from ScreenEncoder
The screen encoder is responsible to write data to the video streamer.
2023-02-18 19:10:26 +01:00
6118c2b4cd Use VideoStreamer directly from ScreenEncoder
The Callbacks interface notifies new packets. But in addition, the
screen encoder will need to write headers on start.

We could add a function onStart(), but for simplicity, just remove the
interface, which brings no value, and call the streamer directly.

Refs 87972e2022
2023-02-18 19:10:26 +01:00
fad8c68090 Simplify error handling on socket creation
On any error, all previously opened sockets must be closed.

Handle these errors in a single catch-block. Currently, there are only 2
sockets, but this will simplify even more with more sockets.

Note: this commit is better displayed with --ignore-space-change (-b).
2023-02-18 19:10:26 +01:00
9b5b75d800 Refactor recorder logic
Process the initial config packet (necessary to write the header)
separately.
2023-02-18 19:10:26 +01:00
1270617422 Move last packet recording
Write the last packet at the end.
2023-02-18 19:10:26 +01:00
0cbbf82917 Open recording file from the recorder thread
The recorder opened the target file from the packet sink open()
callback, called by the demuxer. Only then the recorder thread was
started.

One golden rule for the recorder is to never block the demuxer for I/O,
because it would impact mirroring. This rule is respected on recording
packets, but not for the initial recorder opening.

Therefore, start the recorder thread from sc_recorder_init(), open the
file immediately from the recorder thread, then make it wait for the
stream to start (on packet sink open()).

Now that the recorder can report errors directly (rather than making the
demuxer call fail), it is possible to report file opening error even
before the packet sink is open.
2023-02-18 19:10:26 +01:00
25b3ad844c Inline packet_sink impl in recorder
Remove useless wrappers.
2023-02-18 19:10:26 +01:00
689314578e Initialize recorder fields from init()
The recorder has two initialization phases: one to initialize the
concrete recorder object, and one to open its packet_sink trait.

Initialize mutex and condvar as part of the object initialization.

If there were several packet_sink traits, the mutex and condvar would
still be initialized only once.
2023-02-18 19:10:26 +01:00
af61c7340d Report recorder errors
Stop scrcpy on recorder errors.

It was previously indirectly stopped by the demuxer, which failed to
push packets to a recorder in error. Report it directly instead:
 - it avoids to wait for the next demuxer call;
 - it will allow to open the target file from a separate thread and stop
   immediately on any I/O error.
2023-02-18 19:10:25 +01:00
f0c991537f Move previous packet to a local variable
It is only used from run_recorder().
2023-02-18 19:10:25 +01:00
6b542c8ceb Move pts_origin to a local variable
It is only used from run_recorder().
2023-02-18 19:10:25 +01:00
01b647b910 Change PTS origin type from uint64_t to int64_t
It is initialized from AVPacket.pts, which is an int64_t.
2023-02-18 19:10:25 +01:00
29 changed files with 1642 additions and 452 deletions

View File

@ -201,10 +201,6 @@ conf.set('PORTABLE', get_option('portable'))
conf.set('DEFAULT_LOCAL_PORT_RANGE_FIRST', '27183') conf.set('DEFAULT_LOCAL_PORT_RANGE_FIRST', '27183')
conf.set('DEFAULT_LOCAL_PORT_RANGE_LAST', '27199') conf.set('DEFAULT_LOCAL_PORT_RANGE_LAST', '27199')
# the default video bitrate, in bits/second
# overridden by option --bit-rate
conf.set('DEFAULT_BIT_RATE', '8000000') # 8Mbps
# run a server debugger and wait for a client to be attached # run a server debugger and wait for a client to be attached
conf.set('SERVER_DEBUGGER', get_option('server_debugger')) conf.set('SERVER_DEBUGGER', get_option('server_debugger'))

View File

@ -19,11 +19,23 @@ provides display and control of Android devices connected on USB (or over TCP/IP
.B \-\-always\-on\-top .B \-\-always\-on\-top
Make scrcpy window always on top (above other windows). Make scrcpy window always on top (above other windows).
.TP
.BI "\-\-audio\-bit\-rate " value
Encode the audio at the given bit\-rate, expressed in bits/s. Unit suffixes are supported: '\fBK\fR' (x1000) and '\fBM\fR' (x1000000).
Default is 196K (196000).
.TP
.BI "\-\-audio\-codec " name
Select an audio codec (opus or aac).
Default is opus.
.TP .TP
.BI "\-b, \-\-bit\-rate " value .BI "\-b, \-\-bit\-rate " value
Encode the video at the given bit\-rate, expressed in bits/s. Unit suffixes are supported: '\fBK\fR' (x1000) and '\fBM\fR' (x1000000). Encode the video at the given bit\-rate, expressed in bits/s. Unit suffixes are supported: '\fBK\fR' (x1000) and '\fBM\fR' (x1000000).
Default is 8000000. Default is 8M (8000000).
.TP .TP
.BI "\-\-codec " name .BI "\-\-codec " name

View File

@ -58,6 +58,9 @@
#define OPT_PRINT_FPS 1038 #define OPT_PRINT_FPS 1038
#define OPT_NO_POWER_ON 1039 #define OPT_NO_POWER_ON 1039
#define OPT_CODEC 1040 #define OPT_CODEC 1040
#define OPT_NO_AUDIO 1041
#define OPT_AUDIO_BIT_RATE 1042
#define OPT_AUDIO_CODEC 1043
struct sc_option { struct sc_option {
char shortopt; char shortopt;
@ -98,13 +101,28 @@ static const struct sc_option options[] = {
.longopt = "always-on-top", .longopt = "always-on-top",
.text = "Make scrcpy window always on top (above other windows).", .text = "Make scrcpy window always on top (above other windows).",
}, },
{
.longopt_id = OPT_AUDIO_BIT_RATE,
.longopt = "audio-bit-rate",
.argdesc = "value",
.text = "Encode the audio at the given bit-rate, expressed in bits/s. "
"Unit suffixes are supported: 'K' (x1000) and 'M' (x1000000).\n"
"Default is 196K (196000).",
},
{
.longopt_id = OPT_AUDIO_CODEC,
.longopt = "audio-codec",
.argdesc = "name",
.text = "Select an audio codec (opus or aac).\n"
"Default is opus.",
},
{ {
.shortopt = 'b', .shortopt = 'b',
.longopt = "bit-rate", .longopt = "bit-rate",
.argdesc = "value", .argdesc = "value",
.text = "Encode the video at the given bit-rate, expressed in bits/s. " .text = "Encode the video at the given bit-rate, expressed in bits/s. "
"Unit suffixes are supported: 'K' (x1000) and 'M' (x1000000).\n" "Unit suffixes are supported: 'K' (x1000) and 'M' (x1000000).\n"
"Default is " STR(DEFAULT_BIT_RATE) ".", "Default is 8M (8000000).",
}, },
{ {
.longopt_id = OPT_CODEC, .longopt_id = OPT_CODEC,
@ -299,6 +317,11 @@ static const struct sc_option options[] = {
.text = "Do not display device (only when screen recording or V4L2 " .text = "Do not display device (only when screen recording or V4L2 "
"sink is enabled).", "sink is enabled).",
}, },
{
.longopt_id = OPT_NO_AUDIO,
.longopt = "no-audio",
.text = "Disable audio forwarding.",
},
{ {
.longopt_id = OPT_NO_KEY_REPEAT, .longopt_id = OPT_NO_KEY_REPEAT,
.longopt = "no-key-repeat", .longopt = "no-key-repeat",
@ -1403,6 +1426,20 @@ parse_codec(const char *optarg, enum sc_codec *codec) {
return false; return false;
} }
static bool
parse_audio_codec(const char *optarg, enum sc_codec *codec) {
if (!strcmp(optarg, "opus")) {
*codec = SC_CODEC_OPUS;
return true;
}
if (!strcmp(optarg, "aac")) {
*codec = SC_CODEC_AAC;
return true;
}
LOGE("Unsupported audio codec: %s (expected opus)", optarg);
return false;
}
static bool static bool
parse_args_with_getopt(struct scrcpy_cli_args *args, int argc, char *argv[], parse_args_with_getopt(struct scrcpy_cli_args *args, int argc, char *argv[],
const char *optstring, const struct option *longopts) { const char *optstring, const struct option *longopts) {
@ -1418,6 +1455,11 @@ parse_args_with_getopt(struct scrcpy_cli_args *args, int argc, char *argv[],
return false; return false;
} }
break; break;
case OPT_AUDIO_BIT_RATE:
if (!parse_bit_rate(optarg, &opts->audio_bit_rate)) {
return false;
}
break;
case OPT_CROP: case OPT_CROP:
opts->crop = optarg; opts->crop = optarg;
break; break;
@ -1627,6 +1669,9 @@ parse_args_with_getopt(struct scrcpy_cli_args *args, int argc, char *argv[],
case OPT_NO_DOWNSIZE_ON_ERROR: case OPT_NO_DOWNSIZE_ON_ERROR:
opts->downsize_on_error = false; opts->downsize_on_error = false;
break; break;
case OPT_NO_AUDIO:
opts->audio = false;
break;
case OPT_NO_CLEANUP: case OPT_NO_CLEANUP:
opts->cleanup = false; opts->cleanup = false;
break; break;
@ -1641,6 +1686,11 @@ parse_args_with_getopt(struct scrcpy_cli_args *args, int argc, char *argv[],
return false; return false;
} }
break; break;
case OPT_AUDIO_CODEC:
if (!parse_audio_codec(optarg, &opts->audio_codec)) {
return false;
}
break;
case OPT_OTG: case OPT_OTG:
#ifdef HAVE_USB #ifdef HAVE_USB
opts->otg = true; opts->otg = true;

View File

@ -23,6 +23,8 @@ sc_demuxer_to_avcodec_id(uint32_t codec_id) {
#define SC_CODEC_ID_H264 UINT32_C(0x68323634) // "h264" in ASCII #define SC_CODEC_ID_H264 UINT32_C(0x68323634) // "h264" in ASCII
#define SC_CODEC_ID_H265 UINT32_C(0x68323635) // "h265" in ASCII #define SC_CODEC_ID_H265 UINT32_C(0x68323635) // "h265" in ASCII
#define SC_CODEC_ID_AV1 UINT32_C(0x00617631) // "av1" in ASCII #define SC_CODEC_ID_AV1 UINT32_C(0x00617631) // "av1" in ASCII
#define SC_CODEC_ID_OPUS UINT32_C(0x6f707573) // "opus" in ASCII
#define SC_CODEC_ID_AAC UINT32_C(0x00616163) // "aac in ASCII"
switch (codec_id) { switch (codec_id) {
case SC_CODEC_ID_H264: case SC_CODEC_ID_H264:
return AV_CODEC_ID_H264; return AV_CODEC_ID_H264;
@ -30,6 +32,10 @@ sc_demuxer_to_avcodec_id(uint32_t codec_id) {
return AV_CODEC_ID_HEVC; return AV_CODEC_ID_HEVC;
case SC_CODEC_ID_AV1: case SC_CODEC_ID_AV1:
return AV_CODEC_ID_AV1; return AV_CODEC_ID_AV1;
case SC_CODEC_ID_OPUS:
return AV_CODEC_ID_OPUS;
case SC_CODEC_ID_AAC:
return AV_CODEC_ID_AAC;
default: default:
LOGE("Unknown codec id 0x%08" PRIx32, codec_id); LOGE("Unknown codec id 0x%08" PRIx32, codec_id);
return AV_CODEC_ID_NONE; return AV_CODEC_ID_NONE;
@ -157,6 +163,16 @@ sc_demuxer_open_sinks(struct sc_demuxer *demuxer, const AVCodec *codec) {
return true; return true;
} }
static void
sc_demuxer_disable_sinks(struct sc_demuxer *demuxer) {
for (unsigned i = 0; i < demuxer->sink_count; ++i) {
struct sc_packet_sink *sink = demuxer->sinks[i];
if (sink->ops->disable) {
sink->ops->disable(sink);
}
}
}
static int static int
run_demuxer(void *data) { run_demuxer(void *data) {
struct sc_demuxer *demuxer = data; struct sc_demuxer *demuxer = data;
@ -171,6 +187,13 @@ run_demuxer(void *data) {
goto end; goto end;
} }
if (raw_codec_id == 0) {
// Stream explicitly disabled by the device
sc_demuxer_disable_sinks(demuxer);
eos = true;
goto end;
}
enum AVCodecID codec_id = sc_demuxer_to_avcodec_id(raw_codec_id); enum AVCodecID codec_id = sc_demuxer_to_avcodec_id(raw_codec_id);
if (codec_id == AV_CODEC_ID_NONE) { if (codec_id == AV_CODEC_ID_NONE) {
// Error already logged // Error already logged
@ -188,8 +211,15 @@ run_demuxer(void *data) {
goto end; goto end;
} }
// Config packets must be merged with the next non-config packet only for
// video streams
bool must_merge_config_packet = codec->type == AVMEDIA_TYPE_VIDEO;
struct sc_packet_merger merger; struct sc_packet_merger merger;
sc_packet_merger_init(&merger);
if (must_merge_config_packet) {
sc_packet_merger_init(&merger);
}
AVPacket *packet = av_packet_alloc(); AVPacket *packet = av_packet_alloc();
if (!packet) { if (!packet) {
@ -205,11 +235,13 @@ run_demuxer(void *data) {
break; break;
} }
// Prepend any config packet to the next media packet if (must_merge_config_packet) {
ok = sc_packet_merger_merge(&merger, packet); // Prepend any config packet to the next media packet
if (!ok) { ok = sc_packet_merger_merge(&merger, packet);
av_packet_unref(packet); if (!ok) {
break; av_packet_unref(packet);
break;
}
} }
ok = sc_demuxer_push_packet(demuxer, packet); ok = sc_demuxer_push_packet(demuxer, packet);
@ -222,7 +254,9 @@ run_demuxer(void *data) {
LOGD("End of frames"); LOGD("End of frames");
sc_packet_merger_destroy(&merger); if (must_merge_config_packet) {
sc_packet_merger_destroy(&merger);
}
av_packet_free(&packet); av_packet_free(&packet);
finally_close_sinks: finally_close_sinks:
@ -236,6 +270,8 @@ end:
void void
sc_demuxer_init(struct sc_demuxer *demuxer, sc_socket socket, sc_demuxer_init(struct sc_demuxer *demuxer, sc_socket socket,
const struct sc_demuxer_callbacks *cbs, void *cbs_userdata) { const struct sc_demuxer_callbacks *cbs, void *cbs_userdata) {
assert(socket != SC_SOCKET_NONE);
demuxer->socket = socket; demuxer->socket = socket;
demuxer->sink_count = 0; demuxer->sink_count = 0;

View File

@ -4,3 +4,4 @@
#define SC_EVENT_SERVER_CONNECTED (SDL_USEREVENT + 3) #define SC_EVENT_SERVER_CONNECTED (SDL_USEREVENT + 3)
#define SC_EVENT_USB_DEVICE_DISCONNECTED (SDL_USEREVENT + 4) #define SC_EVENT_USB_DEVICE_DISCONNECTED (SDL_USEREVENT + 4)
#define SC_EVENT_DEMUXER_ERROR (SDL_USEREVENT + 5) #define SC_EVENT_DEMUXER_ERROR (SDL_USEREVENT + 5)
#define SC_EVENT_RECORDER_ERROR (SDL_USEREVENT + 6)

View File

@ -14,6 +14,7 @@ const struct scrcpy_options scrcpy_options_default = {
#endif #endif
.log_level = SC_LOG_LEVEL_INFO, .log_level = SC_LOG_LEVEL_INFO,
.codec = SC_CODEC_H264, .codec = SC_CODEC_H264,
.audio_codec = SC_CODEC_OPUS,
.record_format = SC_RECORD_FORMAT_AUTO, .record_format = SC_RECORD_FORMAT_AUTO,
.keyboard_input_mode = SC_KEYBOARD_INPUT_MODE_INJECT, .keyboard_input_mode = SC_KEYBOARD_INPUT_MODE_INJECT,
.port_range = { .port_range = {
@ -27,7 +28,8 @@ const struct scrcpy_options scrcpy_options_default = {
.count = 2, .count = 2,
}, },
.max_size = 0, .max_size = 0,
.bit_rate = DEFAULT_BIT_RATE, .bit_rate = 0,
.audio_bit_rate = 0,
.max_fps = 0, .max_fps = 0,
.lock_video_orientation = SC_LOCK_VIDEO_ORIENTATION_UNLOCKED, .lock_video_orientation = SC_LOCK_VIDEO_ORIENTATION_UNLOCKED,
.rotation = 0, .rotation = 0,
@ -66,4 +68,5 @@ const struct scrcpy_options scrcpy_options_default = {
.cleanup = true, .cleanup = true,
.start_fps_counter = false, .start_fps_counter = false,
.power_on = true, .power_on = true,
.audio = true,
}; };

View File

@ -27,6 +27,8 @@ enum sc_codec {
SC_CODEC_H264, SC_CODEC_H264,
SC_CODEC_H265, SC_CODEC_H265,
SC_CODEC_AV1, SC_CODEC_AV1,
SC_CODEC_OPUS,
SC_CODEC_AAC,
}; };
enum sc_lock_video_orientation { enum sc_lock_video_orientation {
@ -100,6 +102,7 @@ struct scrcpy_options {
#endif #endif
enum sc_log_level log_level; enum sc_log_level log_level;
enum sc_codec codec; enum sc_codec codec;
enum sc_codec audio_codec;
enum sc_record_format record_format; enum sc_record_format record_format;
enum sc_keyboard_input_mode keyboard_input_mode; enum sc_keyboard_input_mode keyboard_input_mode;
enum sc_mouse_input_mode mouse_input_mode; enum sc_mouse_input_mode mouse_input_mode;
@ -109,6 +112,7 @@ struct scrcpy_options {
struct sc_shortcut_mods shortcut_mods; struct sc_shortcut_mods shortcut_mods;
uint16_t max_size; uint16_t max_size;
uint32_t bit_rate; uint32_t bit_rate;
uint32_t audio_bit_rate;
uint16_t max_fps; uint16_t max_fps;
enum sc_lock_video_orientation lock_video_orientation; enum sc_lock_video_orientation lock_video_orientation;
uint8_t rotation; uint8_t rotation;
@ -147,6 +151,7 @@ struct scrcpy_options {
bool cleanup; bool cleanup;
bool start_fps_counter; bool start_fps_counter;
bool power_on; bool power_on;
bool audio;
}; };
extern const struct scrcpy_options scrcpy_options_default; extern const struct scrcpy_options scrcpy_options_default;

View File

@ -8,10 +8,11 @@
#include "util/log.h" #include "util/log.h"
#include "util/str.h" #include "util/str.h"
/** Downcast packet_sink to recorder */ /** Downcast packet sinks to recorder */
#define DOWNCAST(SINK) container_of(SINK, struct sc_recorder, packet_sink) #define DOWNCAST_VIDEO(SINK) \
container_of(SINK, struct sc_recorder, video_packet_sink)
#define SC_PTS_ORIGIN_NONE UINT64_C(-1) #define DOWNCAST_AUDIO(SINK) \
container_of(SINK, struct sc_recorder, audio_packet_sink)
static const AVRational SCRCPY_TIME_BASE = {1, 1000000}; // timestamps in us static const AVRational SCRCPY_TIME_BASE = {1, 1000000}; // timestamps in us
@ -80,9 +81,7 @@ sc_recorder_get_format_name(enum sc_record_format format) {
} }
static bool static bool
sc_recorder_write_header(struct sc_recorder *recorder, const AVPacket *packet) { sc_recorder_set_extradata(AVStream *ostream, const AVPacket *packet) {
AVStream *ostream = recorder->ctx->streams[0];
uint8_t *extradata = av_malloc(packet->size * sizeof(uint8_t)); uint8_t *extradata = av_malloc(packet->size * sizeof(uint8_t));
if (!extradata) { if (!extradata) {
LOG_OOM(); LOG_OOM();
@ -94,183 +93,56 @@ sc_recorder_write_header(struct sc_recorder *recorder, const AVPacket *packet) {
ostream->codecpar->extradata = extradata; ostream->codecpar->extradata = extradata;
ostream->codecpar->extradata_size = packet->size; ostream->codecpar->extradata_size = packet->size;
int ret = avformat_write_header(recorder->ctx, NULL);
if (ret < 0) {
LOGE("Failed to write header to %s", recorder->filename);
return false;
}
return true; return true;
} }
static void static inline void
sc_recorder_rescale_packet(struct sc_recorder *recorder, AVPacket *packet) { sc_recorder_rescale_packet(AVStream *stream, AVPacket *packet) {
AVStream *ostream = recorder->ctx->streams[0]; av_packet_rescale_ts(packet, SCRCPY_TIME_BASE, stream->time_base);
av_packet_rescale_ts(packet, SCRCPY_TIME_BASE, ostream->time_base);
} }
static bool static bool
sc_recorder_write(struct sc_recorder *recorder, AVPacket *packet) { sc_recorder_write_stream(struct sc_recorder *recorder, int stream_index,
if (!recorder->header_written) { AVPacket *packet) {
if (packet->pts != AV_NOPTS_VALUE) { AVStream *stream = recorder->ctx->streams[stream_index];
LOGE("The first packet is not a config packet"); sc_recorder_rescale_packet(stream, packet);
return false; return av_interleaved_write_frame(recorder->ctx, packet) >= 0;
}
bool ok = sc_recorder_write_header(recorder, packet);
if (!ok) {
return false;
}
recorder->header_written = true;
return true;
}
if (packet->pts == AV_NOPTS_VALUE) {
// ignore config packets
return true;
}
sc_recorder_rescale_packet(recorder, packet);
return av_write_frame(recorder->ctx, packet) >= 0;
} }
static int static inline bool
run_recorder(void *data) { sc_recorder_write_video(struct sc_recorder *recorder, AVPacket *packet) {
struct sc_recorder *recorder = data; return sc_recorder_write_stream(recorder, recorder->video_stream_index,
packet);
}
for (;;) { static inline bool
sc_mutex_lock(&recorder->mutex); sc_recorder_write_audio(struct sc_recorder *recorder, AVPacket *packet) {
return sc_recorder_write_stream(recorder, recorder->audio_stream_index,
while (!recorder->stopped && sc_queue_is_empty(&recorder->queue)) { packet);
sc_cond_wait(&recorder->queue_cond, &recorder->mutex);
}
// if stopped is set, continue to process the remaining events (to
// finish the recording) before actually stopping
if (recorder->stopped && sc_queue_is_empty(&recorder->queue)) {
sc_mutex_unlock(&recorder->mutex);
struct sc_record_packet *last = recorder->previous;
if (last) {
// assign an arbitrary duration to the last packet
last->packet->duration = 100000;
bool ok = sc_recorder_write(recorder, last->packet);
if (!ok) {
// failing to write the last frame is not very serious, no
// future frame may depend on it, so the resulting file
// will still be valid
LOGW("Could not record last packet");
}
sc_record_packet_delete(last);
}
break;
}
struct sc_record_packet *rec;
sc_queue_take(&recorder->queue, next, &rec);
sc_mutex_unlock(&recorder->mutex);
if (recorder->pts_origin == SC_PTS_ORIGIN_NONE
&& rec->packet->pts != AV_NOPTS_VALUE) {
// First PTS received
recorder->pts_origin = rec->packet->pts;
}
if (rec->packet->pts != AV_NOPTS_VALUE) {
// Set PTS relatve to the origin
rec->packet->pts -= recorder->pts_origin;
rec->packet->dts = rec->packet->pts;
}
// recorder->previous is only written from this thread, no need to lock
struct sc_record_packet *previous = recorder->previous;
recorder->previous = rec;
if (!previous) {
// we just received the first packet
continue;
}
// config packets have no PTS, we must ignore them
if (rec->packet->pts != AV_NOPTS_VALUE
&& previous->packet->pts != AV_NOPTS_VALUE) {
// we now know the duration of the previous packet
previous->packet->duration =
rec->packet->pts - previous->packet->pts;
}
bool ok = sc_recorder_write(recorder, previous->packet);
sc_record_packet_delete(previous);
if (!ok) {
LOGE("Could not record packet");
sc_mutex_lock(&recorder->mutex);
recorder->failed = true;
// discard pending packets
sc_recorder_queue_clear(&recorder->queue);
sc_mutex_unlock(&recorder->mutex);
break;
}
}
if (!recorder->failed) {
if (recorder->header_written) {
int ret = av_write_trailer(recorder->ctx);
if (ret < 0) {
LOGE("Failed to write trailer to %s", recorder->filename);
recorder->failed = true;
}
} else {
// the recorded file is empty
recorder->failed = true;
}
}
if (recorder->failed) {
LOGE("Recording failed to %s", recorder->filename);
} else {
const char *format_name = sc_recorder_get_format_name(recorder->format);
LOGI("Recording complete to %s file: %s", format_name,
recorder->filename);
}
LOGD("Recorder thread ended");
return 0;
} }
static bool static bool
sc_recorder_open(struct sc_recorder *recorder, const AVCodec *input_codec) { sc_recorder_open_output_file(struct sc_recorder *recorder) {
bool ok = sc_mutex_init(&recorder->mutex);
if (!ok) {
return false;
}
ok = sc_cond_init(&recorder->queue_cond);
if (!ok) {
goto error_mutex_destroy;
}
sc_queue_init(&recorder->queue);
recorder->stopped = false;
recorder->failed = false;
recorder->header_written = false;
recorder->previous = NULL;
recorder->pts_origin = SC_PTS_ORIGIN_NONE;
const char *format_name = sc_recorder_get_format_name(recorder->format); const char *format_name = sc_recorder_get_format_name(recorder->format);
assert(format_name); assert(format_name);
const AVOutputFormat *format = find_muxer(format_name); const AVOutputFormat *format = find_muxer(format_name);
if (!format) { if (!format) {
LOGE("Could not find muxer"); LOGE("Could not find muxer");
goto error_cond_destroy; return false;
} }
recorder->ctx = avformat_alloc_context(); recorder->ctx = avformat_alloc_context();
if (!recorder->ctx) { if (!recorder->ctx) {
LOG_OOM(); LOG_OOM();
goto error_cond_destroy; return false;
}
int ret = avio_open(&recorder->ctx->pb, recorder->filename,
AVIO_FLAG_WRITE);
if (ret < 0) {
LOGE("Failed to open output file: %s", recorder->filename);
avformat_free_context(recorder->ctx);
return false;
} }
// contrary to the deprecated API (av_oformat_next()), av_muxer_iterate() // contrary to the deprecated API (av_oformat_next()), av_muxer_iterate()
@ -282,71 +154,417 @@ sc_recorder_open(struct sc_recorder *recorder, const AVCodec *input_codec) {
av_dict_set(&recorder->ctx->metadata, "comment", av_dict_set(&recorder->ctx->metadata, "comment",
"Recorded by scrcpy " SCRCPY_VERSION, 0); "Recorded by scrcpy " SCRCPY_VERSION, 0);
AVStream *ostream = avformat_new_stream(recorder->ctx, input_codec);
if (!ostream) {
goto error_avformat_free_context;
}
ostream->codecpar->codec_type = AVMEDIA_TYPE_VIDEO;
ostream->codecpar->codec_id = input_codec->id;
ostream->codecpar->format = AV_PIX_FMT_YUV420P;
ostream->codecpar->width = recorder->declared_frame_size.width;
ostream->codecpar->height = recorder->declared_frame_size.height;
int ret = avio_open(&recorder->ctx->pb, recorder->filename,
AVIO_FLAG_WRITE);
if (ret < 0) {
LOGE("Failed to open output file: %s", recorder->filename);
// ostream will be cleaned up during context cleaning
goto error_avformat_free_context;
}
LOGD("Starting recorder thread");
ok = sc_thread_create(&recorder->thread, run_recorder, "scrcpy-recorder",
recorder);
if (!ok) {
LOGE("Could not start recorder thread");
goto error_avio_close;
}
LOGI("Recording started to %s file: %s", format_name, recorder->filename); LOGI("Recording started to %s file: %s", format_name, recorder->filename);
return true; return true;
error_avio_close:
avio_close(recorder->ctx->pb);
error_avformat_free_context:
avformat_free_context(recorder->ctx);
error_cond_destroy:
sc_cond_destroy(&recorder->queue_cond);
error_mutex_destroy:
sc_mutex_destroy(&recorder->mutex);
return false;
} }
static void static void
sc_recorder_close(struct sc_recorder *recorder) { sc_recorder_close_output_file(struct sc_recorder *recorder) {
sc_mutex_lock(&recorder->mutex);
recorder->stopped = true;
sc_cond_signal(&recorder->queue_cond);
sc_mutex_unlock(&recorder->mutex);
sc_thread_join(&recorder->thread, NULL);
avio_close(recorder->ctx->pb); avio_close(recorder->ctx->pb);
avformat_free_context(recorder->ctx); avformat_free_context(recorder->ctx);
sc_cond_destroy(&recorder->queue_cond);
sc_mutex_destroy(&recorder->mutex);
} }
static bool static bool
sc_recorder_push(struct sc_recorder *recorder, const AVPacket *packet) { sc_recorder_wait_video_stream(struct sc_recorder *recorder) {
sc_mutex_lock(&recorder->mutex); sc_mutex_lock(&recorder->mutex);
assert(!recorder->stopped); while (!recorder->video_codec && !recorder->stopped) {
sc_cond_wait(&recorder->stream_cond, &recorder->mutex);
}
const AVCodec *codec = recorder->video_codec;
sc_mutex_unlock(&recorder->mutex);
if (recorder->failed) { if (codec) {
// reject any new packet (this will stop the stream) AVStream *stream = avformat_new_stream(recorder->ctx, codec);
if (!stream) {
return false;
}
stream->codecpar->codec_type = AVMEDIA_TYPE_VIDEO;
stream->codecpar->codec_id = codec->id;
stream->codecpar->format = AV_PIX_FMT_YUV420P;
stream->codecpar->width = recorder->declared_frame_size.width;
stream->codecpar->height = recorder->declared_frame_size.height;
recorder->video_stream_index = stream->index;
}
return true;
}
static bool
sc_recorder_wait_audio_stream(struct sc_recorder *recorder) {
sc_mutex_lock(&recorder->mutex);
while (!recorder->audio_codec && !recorder->audio_disabled
&& !recorder->stopped) {
sc_cond_wait(&recorder->stream_cond, &recorder->mutex);
}
if (recorder->audio_disabled) {
// Reset audio flag. From there, the recorder thread may access this
// flag without any mutex.
recorder->audio = false;
}
const AVCodec *codec = recorder->audio_codec;
sc_mutex_unlock(&recorder->mutex);
if (codec) {
AVStream *stream = avformat_new_stream(recorder->ctx, codec);
if (!stream) {
return false;
}
stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO;
stream->codecpar->codec_id = codec->id;
stream->codecpar->ch_layout.nb_channels = 2;
stream->codecpar->sample_rate = 48000;
recorder->audio_stream_index = stream->index;
}
return true;
}
static inline bool
sc_recorder_has_empty_queues(struct sc_recorder *recorder) {
if (sc_queue_is_empty(&recorder->video_queue)) {
// The video queue is empty
return true;
}
if (recorder->audio && sc_queue_is_empty(&recorder->audio_queue)) {
// The audio queue is empty (when audio is enabled)
return true;
}
// No queue is empty
return false;
}
static bool
sc_recorder_process_header(struct sc_recorder *recorder) {
sc_mutex_lock(&recorder->mutex);
while (!recorder->stopped && sc_recorder_has_empty_queues(recorder)) {
sc_cond_wait(&recorder->queue_cond, &recorder->mutex);
}
if (recorder->stopped && sc_recorder_has_empty_queues(recorder)) {
sc_mutex_unlock(&recorder->mutex);
return false;
}
struct sc_record_packet *video_pkt;
sc_queue_take(&recorder->video_queue, next, &video_pkt);
struct sc_record_packet *audio_pkt;
if (recorder->audio) {
sc_queue_take(&recorder->audio_queue, next, &audio_pkt);
}
sc_mutex_unlock(&recorder->mutex);
int ret = false;
if (video_pkt->packet->pts != AV_NOPTS_VALUE) {
LOGE("The first video packet is not a config packet");
goto end;
}
assert(recorder->video_stream_index >= 0);
AVStream *video_stream =
recorder->ctx->streams[recorder->video_stream_index];
bool ok = sc_recorder_set_extradata(video_stream, video_pkt->packet);
if (!ok) {
goto end;
}
if (recorder->audio) {
if (audio_pkt->packet->pts != AV_NOPTS_VALUE) {
LOGE("The first audio packet is not a config packet");
goto end;
}
assert(recorder->audio_stream_index >= 0);
AVStream *audio_stream =
recorder->ctx->streams[recorder->audio_stream_index];
ok = sc_recorder_set_extradata(audio_stream, audio_pkt->packet);
if (!ok) {
goto end;
}
}
ok = avformat_write_header(recorder->ctx, NULL) >= 0;
if (!ok) {
LOGE("Failed to write header to %s", recorder->filename);
goto end;
}
ret = true;
end:
sc_record_packet_delete(video_pkt);
if (recorder->audio) {
sc_record_packet_delete(audio_pkt);
}
return ret;
}
static bool
sc_recorder_process_packets(struct sc_recorder *recorder) {
int64_t pts_origin = AV_NOPTS_VALUE;
bool header_written = sc_recorder_process_header(recorder);
if (!header_written) {
return false;
}
struct sc_record_packet *video_pkt = NULL;
struct sc_record_packet *audio_pkt = NULL;
// We can write a video packet only once we received the next one so that
// we can set its duration (next_pts - current_pts)
struct sc_record_packet *video_pkt_previous = NULL;
bool error = false;
for (;;) {
sc_mutex_lock(&recorder->mutex);
while (!recorder->stopped) {
if (!video_pkt && !sc_queue_is_empty(&recorder->video_queue)) {
// A new packet may be assigned to video_pkt and be processed
break;
}
if (recorder->audio && !audio_pkt
&& !sc_queue_is_empty(&recorder->audio_queue)) {
// A new packet may be assigned to audio_pkt and be processed
break;
}
sc_cond_wait(&recorder->queue_cond, &recorder->mutex);
}
// If stopped is set, continue to process the remaining events (to
// finish the recording) before actually stopping.
// If there is no audio, then the audio_queue will remain empty forever
// and audio_pkt will always be NULL.
assert(recorder->audio
|| (!audio_pkt && sc_queue_is_empty(&recorder->audio_queue)));
if (recorder->stopped && sc_queue_is_empty(&recorder->video_queue)
&& sc_queue_is_empty(&recorder->audio_queue)) {
sc_mutex_unlock(&recorder->mutex);
break;
}
if (!video_pkt && !sc_queue_is_empty(&recorder->video_queue)) {
sc_queue_take(&recorder->video_queue, next, &video_pkt);
}
if (!audio_pkt && !sc_queue_is_empty(&recorder->audio_queue)) {
sc_queue_take(&recorder->audio_queue, next, &audio_pkt);
}
assert(video_pkt || audio_pkt); // at least one
sc_mutex_unlock(&recorder->mutex);
// Ignore further config packets (e.g. on device orientation
// change). The next non-config packet will have the config packet
// data prepended.
if (video_pkt && video_pkt->packet->pts == AV_NOPTS_VALUE) {
sc_record_packet_delete(video_pkt);
video_pkt = NULL;
}
if (audio_pkt && audio_pkt->packet->pts == AV_NOPTS_VALUE) {
sc_record_packet_delete(audio_pkt);
audio_pkt= NULL;
}
if (pts_origin == AV_NOPTS_VALUE) {
if (!recorder->audio) {
assert(video_pkt);
pts_origin = video_pkt->packet->pts;
} else if (video_pkt && audio_pkt) {
pts_origin =
MIN(video_pkt->packet->pts, audio_pkt->packet->pts);
} else {
// We need both video and audio packets to initialize pts_origin
continue;
}
}
assert(pts_origin != AV_NOPTS_VALUE);
if (video_pkt) {
video_pkt->packet->pts -= pts_origin;
video_pkt->packet->dts = video_pkt->packet->pts;
if (video_pkt_previous) {
// we now know the duration of the previous packet
video_pkt_previous->packet->duration =
video_pkt->packet->pts - video_pkt_previous->packet->pts;
bool ok = sc_recorder_write_video(recorder,
video_pkt_previous->packet);
sc_record_packet_delete(video_pkt_previous);
if (!ok) {
LOGE("Could not record video packet");
// TODO cleanup
return false;
}
}
video_pkt_previous = video_pkt;
video_pkt = NULL;
}
if (audio_pkt) {
audio_pkt->packet->pts -= pts_origin;
audio_pkt->packet->dts = audio_pkt->packet->pts;
bool ok = sc_recorder_write_audio(recorder, audio_pkt->packet);
sc_record_packet_delete(audio_pkt);
if (!ok) {
LOGE("Could not record audio packet");
error = true;
goto end;
}
audio_pkt = NULL;
}
}
// Write the last video packet
struct sc_record_packet *last = video_pkt_previous;
if (last) {
// assign an arbitrary duration to the last packet
last->packet->duration = 100000;
bool ok = sc_recorder_write_video(recorder, last->packet);
if (!ok) {
// failing to write the last frame is not very serious, no
// future frame may depend on it, so the resulting file
// will still be valid
LOGW("Could not record last packet");
}
sc_record_packet_delete(last);
}
int ret = av_write_trailer(recorder->ctx);
if (ret < 0) {
LOGE("Failed to write trailer to %s", recorder->filename);
error = false;
}
end:
if (video_pkt) {
sc_record_packet_delete(video_pkt);
}
if (audio_pkt) {
sc_record_packet_delete(audio_pkt);
}
return !error;
}
static bool
sc_recorder_record(struct sc_recorder *recorder) {
bool ok = sc_recorder_open_output_file(recorder);
if (!ok) {
return false;
}
ok = sc_recorder_wait_video_stream(recorder);
if (!ok) {
sc_recorder_close_output_file(recorder);
return false;
}
if (recorder->audio) {
ok = sc_recorder_wait_audio_stream(recorder);
if (!ok) {
sc_recorder_close_output_file(recorder);
return false;
}
}
// If recorder->stopped, process any queued packet anyway
ok = sc_recorder_process_packets(recorder);
sc_recorder_close_output_file(recorder);
return ok;
}
static int
run_recorder(void *data) {
struct sc_recorder *recorder = data;
bool success = sc_recorder_record(recorder);
sc_mutex_lock(&recorder->mutex);
// Prevent the producer to push any new packet
recorder->stopped = true;
// Discard pending packets
sc_recorder_queue_clear(&recorder->video_queue);
sc_mutex_unlock(&recorder->mutex);
if (success) {
const char *format_name = sc_recorder_get_format_name(recorder->format);
LOGI("Recording complete to %s file: %s", format_name,
recorder->filename);
} else {
LOGE("Recording failed to %s", recorder->filename);
}
LOGD("Recorder thread ended");
recorder->cbs->on_ended(recorder, success, recorder->cbs_userdata);
return 0;
}
static bool
sc_recorder_video_packet_sink_open(struct sc_packet_sink *sink,
const AVCodec *codec) {
struct sc_recorder *recorder = DOWNCAST_VIDEO(sink);
assert(codec);
sc_mutex_lock(&recorder->mutex);
if (recorder->stopped) {
sc_mutex_unlock(&recorder->mutex);
return false;
}
recorder->video_codec = codec;
sc_cond_signal(&recorder->stream_cond);
sc_mutex_unlock(&recorder->mutex);
return true;
}
static void
sc_recorder_video_packet_sink_close(struct sc_packet_sink *sink) {
struct sc_recorder *recorder = DOWNCAST_VIDEO(sink);
sc_mutex_lock(&recorder->mutex);
// EOS also stops the recorder
recorder->stopped = true;
sc_cond_signal(&recorder->queue_cond);
sc_mutex_unlock(&recorder->mutex);
}
static bool
sc_recorder_video_packet_sink_push(struct sc_packet_sink *sink,
const AVPacket *packet) {
struct sc_recorder *recorder = DOWNCAST_VIDEO(sink);
sc_mutex_lock(&recorder->mutex);
if (recorder->stopped) {
// reject any new packet
sc_mutex_unlock(&recorder->mutex); sc_mutex_unlock(&recorder->mutex);
return false; return false;
} }
@ -358,7 +576,9 @@ sc_recorder_push(struct sc_recorder *recorder, const AVPacket *packet) {
return false; return false;
} }
sc_queue_push(&recorder->queue, next, rec); rec->packet->stream_index = 0;
sc_queue_push(&recorder->video_queue, next, rec);
sc_cond_signal(&recorder->queue_cond); sc_cond_signal(&recorder->queue_cond);
sc_mutex_unlock(&recorder->mutex); sc_mutex_unlock(&recorder->mutex);
@ -366,51 +586,190 @@ sc_recorder_push(struct sc_recorder *recorder, const AVPacket *packet) {
} }
static bool static bool
sc_recorder_packet_sink_open(struct sc_packet_sink *sink, sc_recorder_audio_packet_sink_open(struct sc_packet_sink *sink,
const AVCodec *codec) { const AVCodec *codec) {
struct sc_recorder *recorder = DOWNCAST(sink); struct sc_recorder *recorder = DOWNCAST_AUDIO(sink);
return sc_recorder_open(recorder, codec); assert(recorder->audio);
// only written from this thread, no need to lock
assert(!recorder->audio_disabled);
assert(codec);
sc_mutex_lock(&recorder->mutex);
recorder->audio_codec = codec;
sc_cond_signal(&recorder->stream_cond);
sc_mutex_unlock(&recorder->mutex);
return true;
} }
static void static void
sc_recorder_packet_sink_close(struct sc_packet_sink *sink) { sc_recorder_audio_packet_sink_close(struct sc_packet_sink *sink) {
struct sc_recorder *recorder = DOWNCAST(sink); struct sc_recorder *recorder = DOWNCAST_AUDIO(sink);
sc_recorder_close(recorder); assert(recorder->audio);
// only written from this thread, no need to lock
assert(!recorder->audio_disabled);
sc_mutex_lock(&recorder->mutex);
// EOS also stops the recorder
// TODO must stop only once both video and audio stream are complete
recorder->stopped = true;
sc_cond_signal(&recorder->queue_cond);
sc_mutex_unlock(&recorder->mutex);
} }
static bool static bool
sc_recorder_packet_sink_push(struct sc_packet_sink *sink, sc_recorder_audio_packet_sink_push(struct sc_packet_sink *sink,
const AVPacket *packet) { const AVPacket *packet) {
struct sc_recorder *recorder = DOWNCAST(sink); struct sc_recorder *recorder = DOWNCAST_AUDIO(sink);
return sc_recorder_push(recorder, packet); assert(recorder->audio);
// only written from this thread, no need to lock
assert(!recorder->audio_disabled);
sc_mutex_lock(&recorder->mutex);
if (recorder->stopped) {
// reject any new packet
sc_mutex_unlock(&recorder->mutex);
return false;
}
struct sc_record_packet *rec = sc_record_packet_new(packet);
if (!rec) {
LOG_OOM();
sc_mutex_unlock(&recorder->mutex);
return false;
}
rec->packet->stream_index = 1;
sc_queue_push(&recorder->audio_queue, next, rec);
sc_cond_signal(&recorder->queue_cond);
sc_mutex_unlock(&recorder->mutex);
return true;
}
static void
sc_recorder_audio_packet_sink_disable(struct sc_packet_sink *sink) {
struct sc_recorder *recorder = DOWNCAST_AUDIO(sink);
assert(recorder->audio);
// only written from this thread, no need to lock
assert(!recorder->audio_disabled);
assert(!recorder->audio_codec);
LOGW("Audio stream disabled: it could not be captured by the device");
sc_mutex_lock(&recorder->mutex);
recorder->audio_disabled = true;
sc_cond_signal(&recorder->stream_cond);
sc_mutex_unlock(&recorder->mutex);
} }
bool bool
sc_recorder_init(struct sc_recorder *recorder, sc_recorder_init(struct sc_recorder *recorder, const char *filename,
const char *filename, enum sc_record_format format, bool audio,
enum sc_record_format format, struct sc_size declared_frame_size,
struct sc_size declared_frame_size) { const struct sc_recorder_callbacks *cbs, void *cbs_userdata) {
recorder->filename = strdup(filename); recorder->filename = strdup(filename);
if (!recorder->filename) { if (!recorder->filename) {
LOG_OOM(); LOG_OOM();
return false; return false;
} }
bool ok = sc_mutex_init(&recorder->mutex);
if (!ok) {
goto error_free_filename;
}
ok = sc_cond_init(&recorder->queue_cond);
if (!ok) {
goto error_mutex_destroy;
}
ok = sc_cond_init(&recorder->stream_cond);
if (!ok) {
goto error_queue_cond_destroy;
}
recorder->audio = audio;
sc_queue_init(&recorder->video_queue);
sc_queue_init(&recorder->audio_queue);
recorder->stopped = false;
recorder->video_codec = NULL;
recorder->audio_codec = NULL;
recorder->audio_disabled = false;
recorder->video_stream_index = -1;
recorder->audio_stream_index = -1;
recorder->format = format; recorder->format = format;
recorder->declared_frame_size = declared_frame_size; recorder->declared_frame_size = declared_frame_size;
static const struct sc_packet_sink_ops ops = { assert(cbs && cbs->on_ended);
.open = sc_recorder_packet_sink_open,
.close = sc_recorder_packet_sink_close, recorder->cbs = cbs;
.push = sc_recorder_packet_sink_push, recorder->cbs_userdata = cbs_userdata;
static const struct sc_packet_sink_ops video_ops = {
.open = sc_recorder_video_packet_sink_open,
.close = sc_recorder_video_packet_sink_close,
.push = sc_recorder_video_packet_sink_push,
}; };
recorder->packet_sink.ops = &ops; recorder->video_packet_sink.ops = &video_ops;
if (audio) {
static const struct sc_packet_sink_ops audio_ops = {
.open = sc_recorder_audio_packet_sink_open,
.close = sc_recorder_audio_packet_sink_close,
.push = sc_recorder_audio_packet_sink_push,
.disable = sc_recorder_audio_packet_sink_disable,
};
recorder->audio_packet_sink.ops = &audio_ops;
}
ok = sc_thread_create(&recorder->thread, run_recorder, "scrcpy-recorder",
recorder);
if (!ok) {
LOGE("Could not start recorder thread");
goto error_stream_cond_destroy;
}
return true; return true;
error_stream_cond_destroy:
sc_cond_destroy(&recorder->stream_cond);
error_queue_cond_destroy:
sc_cond_destroy(&recorder->queue_cond);
error_mutex_destroy:
sc_mutex_destroy(&recorder->mutex);
error_free_filename:
free(recorder->filename);
return false;
}
void
sc_recorder_stop(struct sc_recorder *recorder) {
sc_mutex_lock(&recorder->mutex);
recorder->stopped = true;
sc_cond_signal(&recorder->queue_cond);
sc_cond_signal(&recorder->stream_cond);
sc_mutex_unlock(&recorder->mutex);
}
void
sc_recorder_join(struct sc_recorder *recorder) {
sc_thread_join(&recorder->thread, NULL);
} }
void void
sc_recorder_destroy(struct sc_recorder *recorder) { sc_recorder_destroy(struct sc_recorder *recorder) {
sc_cond_destroy(&recorder->stream_cond);
sc_cond_destroy(&recorder->queue_cond);
sc_mutex_destroy(&recorder->mutex);
free(recorder->filename); free(recorder->filename);
} }

View File

@ -20,34 +20,63 @@ struct sc_record_packet {
struct sc_recorder_queue SC_QUEUE(struct sc_record_packet); struct sc_recorder_queue SC_QUEUE(struct sc_record_packet);
struct sc_recorder { struct sc_recorder {
struct sc_packet_sink packet_sink; // packet sink trait struct sc_packet_sink video_packet_sink;
struct sc_packet_sink audio_packet_sink;
/* The audio flag is unprotected:
* - it is initialized from sc_recorder_init() from the main thread;
* - it may be reset once from the recorder thread if the audio is
* disabled dynamically.
*
* Therefore, once the recorder thread is started, only the recorder thread
* may access it without data races.
*/
bool audio;
char *filename; char *filename;
enum sc_record_format format; enum sc_record_format format;
AVFormatContext *ctx; AVFormatContext *ctx;
struct sc_size declared_frame_size; struct sc_size declared_frame_size;
bool header_written;
uint64_t pts_origin;
sc_thread thread; sc_thread thread;
sc_mutex mutex; sc_mutex mutex;
sc_cond queue_cond; sc_cond queue_cond;
bool stopped; // set on recorder_close() // set on sc_recorder_stop(), packet_sink close or recording failure
bool failed; // set on packet write failure bool stopped;
struct sc_recorder_queue queue; struct sc_recorder_queue video_queue;
struct sc_recorder_queue audio_queue;
// we can write a packet only once we received the next one so that we can // wake up the recorder thread once the video or audio codec is known
// set its duration (next_pts - current_pts) sc_cond stream_cond;
// "previous" is only accessed from the recorder thread, so it does not const AVCodec *video_codec;
// need to be protected by the mutex const AVCodec *audio_codec;
struct sc_record_packet *previous; // Instead of providing an audio_codec, the demuxer may notify that the
// stream is disabled if the device could not capture audio
bool audio_disabled;
int video_stream_index;
int audio_stream_index;
const struct sc_recorder_callbacks *cbs;
void *cbs_userdata;
};
struct sc_recorder_callbacks {
void (*on_ended)(struct sc_recorder *recorder, bool success,
void *userdata);
}; };
bool bool
sc_recorder_init(struct sc_recorder *recorder, const char *filename, sc_recorder_init(struct sc_recorder *recorder, const char *filename,
enum sc_record_format format, enum sc_record_format format, bool audio,
struct sc_size declared_frame_size); struct sc_size declared_frame_size,
const struct sc_recorder_callbacks *cbs, void *cbs_userdata);
void
sc_recorder_stop(struct sc_recorder *recorder);
void
sc_recorder_join(struct sc_recorder *recorder);
void void
sc_recorder_destroy(struct sc_recorder *recorder); sc_recorder_destroy(struct sc_recorder *recorder);

View File

@ -40,7 +40,8 @@
struct scrcpy { struct scrcpy {
struct sc_server server; struct sc_server server;
struct sc_screen screen; struct sc_screen screen;
struct sc_demuxer demuxer; struct sc_demuxer video_demuxer;
struct sc_demuxer audio_demuxer;
struct sc_decoder decoder; struct sc_decoder decoder;
struct sc_recorder recorder; struct sc_recorder recorder;
#ifdef HAVE_V4L2 #ifdef HAVE_V4L2
@ -161,6 +162,9 @@ event_loop(struct scrcpy *s) {
case SC_EVENT_DEMUXER_ERROR: case SC_EVENT_DEMUXER_ERROR:
LOGE("Demuxer error"); LOGE("Demuxer error");
return SCRCPY_EXIT_FAILURE; return SCRCPY_EXIT_FAILURE;
case SC_EVENT_RECORDER_ERROR:
LOGE("Recorder error");
return SCRCPY_EXIT_FAILURE;
case SDL_QUIT: case SDL_QUIT:
LOGD("User requested to quit"); LOGD("User requested to quit");
return SCRCPY_EXIT_SUCCESS; return SCRCPY_EXIT_SUCCESS;
@ -236,7 +240,19 @@ av_log_callback(void *avcl, int level, const char *fmt, va_list vl) {
} }
static void static void
sc_demuxer_on_ended(struct sc_demuxer *demuxer, bool eos, void *userdata) { sc_recorder_on_ended(struct sc_recorder *recorder, bool success,
void *userdata) {
(void) recorder;
(void) userdata;
if (!success) {
PUSH_EVENT(SC_EVENT_RECORDER_ERROR);
}
}
static void
sc_video_demuxer_on_ended(struct sc_demuxer *demuxer, bool eos,
void *userdata) {
(void) demuxer; (void) demuxer;
(void) userdata; (void) userdata;
@ -247,6 +263,16 @@ sc_demuxer_on_ended(struct sc_demuxer *demuxer, bool eos, void *userdata) {
} }
} }
static void
sc_audio_demuxer_on_ended(struct sc_demuxer *demuxer, bool eos,
void *userdata) {
(void) demuxer;
(void) eos;
(void) userdata;
// Contrary to the video demuxer, keep mirroring if only the audio fails
}
static void static void
sc_server_on_connection_failed(struct sc_server *server, void *userdata) { sc_server_on_connection_failed(struct sc_server *server, void *userdata) {
(void) server; (void) server;
@ -303,7 +329,8 @@ scrcpy(struct scrcpy_options *options) {
#ifdef HAVE_V4L2 #ifdef HAVE_V4L2
bool v4l2_sink_initialized = false; bool v4l2_sink_initialized = false;
#endif #endif
bool demuxer_started = false; bool video_demuxer_started = false;
bool audio_demuxer_started = false;
#ifdef HAVE_USB #ifdef HAVE_USB
bool aoa_hid_initialized = false; bool aoa_hid_initialized = false;
bool hid_keyboard_initialized = false; bool hid_keyboard_initialized = false;
@ -324,16 +351,19 @@ scrcpy(struct scrcpy_options *options) {
.select_tcpip = options->select_tcpip, .select_tcpip = options->select_tcpip,
.log_level = options->log_level, .log_level = options->log_level,
.codec = options->codec, .codec = options->codec,
.audio_codec = options->audio_codec,
.crop = options->crop, .crop = options->crop,
.port_range = options->port_range, .port_range = options->port_range,
.tunnel_host = options->tunnel_host, .tunnel_host = options->tunnel_host,
.tunnel_port = options->tunnel_port, .tunnel_port = options->tunnel_port,
.max_size = options->max_size, .max_size = options->max_size,
.bit_rate = options->bit_rate, .bit_rate = options->bit_rate,
.audio_bit_rate = options->audio_bit_rate,
.max_fps = options->max_fps, .max_fps = options->max_fps,
.lock_video_orientation = options->lock_video_orientation, .lock_video_orientation = options->lock_video_orientation,
.control = options->control, .control = options->control,
.display_id = options->display_id, .display_id = options->display_id,
.audio = options->audio,
.show_touches = options->show_touches, .show_touches = options->show_touches,
.stay_awake = options->stay_awake, .stay_awake = options->stay_awake,
.codec_options = options->codec_options, .codec_options = options->codec_options,
@ -416,10 +446,12 @@ scrcpy(struct scrcpy_options *options) {
struct sc_recorder *rec = NULL; struct sc_recorder *rec = NULL;
if (options->record_filename) { if (options->record_filename) {
if (!sc_recorder_init(&s->recorder, static const struct sc_recorder_callbacks recorder_cbs = {
options->record_filename, .on_ended = sc_recorder_on_ended,
options->record_format, };
info->frame_size)) { if (!sc_recorder_init(&s->recorder, options->record_filename,
options->record_format, options->audio,
info->frame_size, &recorder_cbs, NULL)) {
goto end; goto end;
} }
rec = &s->recorder; rec = &s->recorder;
@ -428,17 +460,29 @@ scrcpy(struct scrcpy_options *options) {
av_log_set_callback(av_log_callback); av_log_set_callback(av_log_callback);
static const struct sc_demuxer_callbacks demuxer_cbs = { static const struct sc_demuxer_callbacks video_demuxer_cbs = {
.on_ended = sc_demuxer_on_ended, .on_ended = sc_video_demuxer_on_ended,
}; };
sc_demuxer_init(&s->demuxer, s->server.video_socket, &demuxer_cbs, NULL); sc_demuxer_init(&s->video_demuxer, s->server.video_socket,
&video_demuxer_cbs, NULL);
if (options->audio) {
static const struct sc_demuxer_callbacks audio_demuxer_cbs = {
.on_ended = sc_audio_demuxer_on_ended,
};
sc_demuxer_init(&s->audio_demuxer, s->server.audio_socket,
&audio_demuxer_cbs, NULL);
}
if (dec) { if (dec) {
sc_demuxer_add_sink(&s->demuxer, &dec->packet_sink); sc_demuxer_add_sink(&s->video_demuxer, &dec->packet_sink);
} }
if (rec) { if (rec) {
sc_demuxer_add_sink(&s->demuxer, &rec->packet_sink); sc_demuxer_add_sink(&s->video_demuxer, &rec->video_packet_sink);
if (options->audio) {
sc_demuxer_add_sink(&s->audio_demuxer, &rec->audio_packet_sink);
}
} }
struct sc_controller *controller = NULL; struct sc_controller *controller = NULL;
@ -646,17 +690,24 @@ aoa_hid_end:
#endif #endif
// now we consumed the header values, the socket receives the video stream // now we consumed the header values, the socket receives the video stream
// start the demuxer // start the video demuxer
if (!sc_demuxer_start(&s->demuxer)) { if (!sc_demuxer_start(&s->video_demuxer)) {
goto end; goto end;
} }
demuxer_started = true; video_demuxer_started = true;
if (options->audio) {
if (!sc_demuxer_start(&s->audio_demuxer)) {
goto end;
}
audio_demuxer_started = true;
}
ret = event_loop(s); ret = event_loop(s);
LOGD("quit..."); LOGD("quit...");
// Close the window immediately on closing, because screen_destroy() may // Close the window immediately on closing, because screen_destroy() may
// only be called once the demuxer thread is joined (it may take time) // only be called once the video demuxer thread is joined (it may take time)
sc_screen_hide_window(&s->screen); sc_screen_hide_window(&s->screen);
end: end:
@ -683,6 +734,9 @@ end:
if (file_pusher_initialized) { if (file_pusher_initialized) {
sc_file_pusher_stop(&s->file_pusher); sc_file_pusher_stop(&s->file_pusher);
} }
if (recorder_initialized) {
sc_recorder_stop(&s->recorder);
}
if (screen_initialized) { if (screen_initialized) {
sc_screen_interrupt(&s->screen); sc_screen_interrupt(&s->screen);
} }
@ -694,8 +748,12 @@ end:
// now that the sockets are shutdown, the demuxer and controller are // now that the sockets are shutdown, the demuxer and controller are
// interrupted, we can join them // interrupted, we can join them
if (demuxer_started) { if (video_demuxer_started) {
sc_demuxer_join(&s->demuxer); sc_demuxer_join(&s->video_demuxer);
}
if (audio_demuxer_started) {
sc_demuxer_join(&s->audio_demuxer);
} }
#ifdef HAVE_V4L2 #ifdef HAVE_V4L2
@ -714,8 +772,9 @@ end:
} }
#endif #endif
// Destroy the screen only after the demuxer is guaranteed to be finished, // Destroy the screen only after the video demuxer is guaranteed to be
// because otherwise the screen could receive new frames after destruction // finished, because otherwise the screen could receive new frames after
// destruction
if (screen_initialized) { if (screen_initialized) {
sc_screen_join(&s->screen); sc_screen_join(&s->screen);
sc_screen_destroy(&s->screen); sc_screen_destroy(&s->screen);
@ -729,6 +788,7 @@ end:
} }
if (recorder_initialized) { if (recorder_initialized) {
sc_recorder_join(&s->recorder);
sc_recorder_destroy(&s->recorder); sc_recorder_destroy(&s->recorder);
} }

View File

@ -165,6 +165,10 @@ sc_server_get_codec_name(enum sc_codec codec) {
return "h265"; return "h265";
case SC_CODEC_AV1: case SC_CODEC_AV1:
return "av1"; return "av1";
case SC_CODEC_OPUS:
return "opus";
case SC_CODEC_AAC:
return "aac";
default: default:
return NULL; return NULL;
} }
@ -215,11 +219,22 @@ execute_server(struct sc_server *server,
ADD_PARAM("scid=%08x", params->scid); ADD_PARAM("scid=%08x", params->scid);
ADD_PARAM("log_level=%s", log_level_to_server_string(params->log_level)); ADD_PARAM("log_level=%s", log_level_to_server_string(params->log_level));
ADD_PARAM("bit_rate=%" PRIu32, params->bit_rate);
if (params->bit_rate) {
ADD_PARAM("bit_rate=%" PRIu32, params->bit_rate);
}
if (!params->audio) {
ADD_PARAM("audio=false");
} else if (params->audio_bit_rate) {
ADD_PARAM("audio_bit_rate=%" PRIu32, params->audio_bit_rate);
}
if (params->codec != SC_CODEC_H264) { if (params->codec != SC_CODEC_H264) {
ADD_PARAM("codec=%s", sc_server_get_codec_name(params->codec)); ADD_PARAM("codec=%s", sc_server_get_codec_name(params->codec));
} }
if (params->audio_codec != SC_CODEC_OPUS) {
ADD_PARAM("audio_codec=%s",
sc_server_get_codec_name(params->audio_codec));
}
if (params->max_size) { if (params->max_size) {
ADD_PARAM("max_size=%" PRIu16, params->max_size); ADD_PARAM("max_size=%" PRIu16, params->max_size);
} }
@ -388,6 +403,7 @@ sc_server_init(struct sc_server *server, const struct sc_server_params *params,
server->stopped = false; server->stopped = false;
server->video_socket = SC_SOCKET_NONE; server->video_socket = SC_SOCKET_NONE;
server->audio_socket = SC_SOCKET_NONE;
server->control_socket = SC_SOCKET_NONE; server->control_socket = SC_SOCKET_NONE;
sc_adb_tunnel_init(&server->tunnel); sc_adb_tunnel_init(&server->tunnel);
@ -431,9 +447,11 @@ sc_server_connect_to(struct sc_server *server, struct sc_server_info *info) {
const char *serial = server->serial; const char *serial = server->serial;
assert(serial); assert(serial);
bool audio = server->params.audio;
bool control = server->params.control; bool control = server->params.control;
sc_socket video_socket = SC_SOCKET_NONE; sc_socket video_socket = SC_SOCKET_NONE;
sc_socket audio_socket = SC_SOCKET_NONE;
sc_socket control_socket = SC_SOCKET_NONE; sc_socket control_socket = SC_SOCKET_NONE;
if (!tunnel->forward) { if (!tunnel->forward) {
video_socket = net_accept_intr(&server->intr, tunnel->server_socket); video_socket = net_accept_intr(&server->intr, tunnel->server_socket);
@ -441,6 +459,14 @@ sc_server_connect_to(struct sc_server *server, struct sc_server_info *info) {
goto fail; goto fail;
} }
if (audio) {
audio_socket =
net_accept_intr(&server->intr, tunnel->server_socket);
if (audio_socket == SC_SOCKET_NONE) {
goto fail;
}
}
if (control) { if (control) {
control_socket = control_socket =
net_accept_intr(&server->intr, tunnel->server_socket); net_accept_intr(&server->intr, tunnel->server_socket);
@ -467,6 +493,18 @@ sc_server_connect_to(struct sc_server *server, struct sc_server_info *info) {
goto fail; goto fail;
} }
if (audio) {
audio_socket = net_socket();
if (audio_socket == SC_SOCKET_NONE) {
goto fail;
}
bool ok = net_connect_intr(&server->intr, audio_socket, tunnel_host,
tunnel_port);
if (!ok) {
goto fail;
}
}
if (control) { if (control) {
// we know that the device is listening, we don't need several // we know that the device is listening, we don't need several
// attempts // attempts
@ -493,9 +531,11 @@ sc_server_connect_to(struct sc_server *server, struct sc_server_info *info) {
} }
assert(video_socket != SC_SOCKET_NONE); assert(video_socket != SC_SOCKET_NONE);
assert(!audio || audio_socket != SC_SOCKET_NONE);
assert(!control || control_socket != SC_SOCKET_NONE); assert(!control || control_socket != SC_SOCKET_NONE);
server->video_socket = video_socket; server->video_socket = video_socket;
server->audio_socket = audio_socket;
server->control_socket = control_socket; server->control_socket = control_socket;
return true; return true;
@ -507,6 +547,12 @@ fail:
} }
} }
if (audio_socket != SC_SOCKET_NONE) {
if (!net_close(audio_socket)) {
LOGW("Could not close audio socket");
}
}
if (control_socket != SC_SOCKET_NONE) { if (control_socket != SC_SOCKET_NONE) {
if (!net_close(control_socket)) { if (!net_close(control_socket)) {
LOGW("Could not close control socket"); LOGW("Could not close control socket");
@ -852,6 +898,11 @@ run_server(void *data) {
assert(server->video_socket != SC_SOCKET_NONE); assert(server->video_socket != SC_SOCKET_NONE);
net_interrupt(server->video_socket); net_interrupt(server->video_socket);
if (server->audio_socket != SC_SOCKET_NONE) {
// There is no audio_socket if --no-audio is set
net_interrupt(server->audio_socket);
}
if (server->control_socket != SC_SOCKET_NONE) { if (server->control_socket != SC_SOCKET_NONE) {
// There is no control_socket if --no-control is set // There is no control_socket if --no-control is set
net_interrupt(server->control_socket); net_interrupt(server->control_socket);
@ -913,6 +964,9 @@ sc_server_destroy(struct sc_server *server) {
if (server->video_socket != SC_SOCKET_NONE) { if (server->video_socket != SC_SOCKET_NONE) {
net_close(server->video_socket); net_close(server->video_socket);
} }
if (server->audio_socket != SC_SOCKET_NONE) {
net_close(server->audio_socket);
}
if (server->control_socket != SC_SOCKET_NONE) { if (server->control_socket != SC_SOCKET_NONE) {
net_close(server->control_socket); net_close(server->control_socket);
} }

View File

@ -26,6 +26,7 @@ struct sc_server_params {
const char *req_serial; const char *req_serial;
enum sc_log_level log_level; enum sc_log_level log_level;
enum sc_codec codec; enum sc_codec codec;
enum sc_codec audio_codec;
const char *crop; const char *crop;
const char *codec_options; const char *codec_options;
const char *encoder_name; const char *encoder_name;
@ -34,10 +35,12 @@ struct sc_server_params {
uint16_t tunnel_port; uint16_t tunnel_port;
uint16_t max_size; uint16_t max_size;
uint32_t bit_rate; uint32_t bit_rate;
uint32_t audio_bit_rate;
uint16_t max_fps; uint16_t max_fps;
int8_t lock_video_orientation; int8_t lock_video_orientation;
bool control; bool control;
uint32_t display_id; uint32_t display_id;
bool audio;
bool show_touches; bool show_touches;
bool stay_awake; bool stay_awake;
bool force_adb_forward; bool force_adb_forward;
@ -69,6 +72,7 @@ struct sc_server {
struct sc_adb_tunnel tunnel; struct sc_adb_tunnel tunnel;
sc_socket video_socket; sc_socket video_socket;
sc_socket audio_socket;
sc_socket control_socket; sc_socket control_socket;
const struct sc_server_callbacks *cbs; const struct sc_server_callbacks *cbs;

View File

@ -19,9 +19,20 @@ struct sc_packet_sink {
}; };
struct sc_packet_sink_ops { struct sc_packet_sink_ops {
/* The codec instance is static, it is valid until the end of the program */
bool (*open)(struct sc_packet_sink *sink, const AVCodec *codec); bool (*open)(struct sc_packet_sink *sink, const AVCodec *codec);
void (*close)(struct sc_packet_sink *sink); void (*close)(struct sc_packet_sink *sink);
bool (*push)(struct sc_packet_sink *sink, const AVPacket *packet); bool (*push)(struct sc_packet_sink *sink, const AVPacket *packet);
/*/
* Called when the input stream has been disabled at runtime.
*
* If it is called, then open(), close() and push() will never be called.
*
* It is useful to notify the recorder that the requested audio stream has
* finally been disabled because the device could not capture it.
*/
void (*disable)(struct sc_packet_sink *sink);
}; };
#endif #endif

View File

@ -0,0 +1,47 @@
package com.genymobile.scrcpy;
import android.media.MediaFormat;
public enum AudioCodec implements Codec {
OPUS(0x6f_70_75_73, "opus", MediaFormat.MIMETYPE_AUDIO_OPUS),
AAC(0x00_61_61_63, "aac", MediaFormat.MIMETYPE_AUDIO_AAC);
private final int id; // 4-byte ASCII representation of the name
private final String name;
private final String mimeType;
AudioCodec(int id, String name, String mimeType) {
this.id = id;
this.name = name;
this.mimeType = mimeType;
}
@Override
public Type getType() {
return Type.AUDIO;
}
@Override
public int getId() {
return id;
}
@Override
public String getName() {
return name;
}
@Override
public String getMimeType() {
return mimeType;
}
public static AudioCodec findByName(String name) {
for (AudioCodec codec : values()) {
if (codec.name.equals(name)) {
return codec;
}
}
return null;
}
}

View File

@ -0,0 +1,320 @@
package com.genymobile.scrcpy;
import android.annotation.SuppressLint;
import android.annotation.TargetApi;
import android.media.AudioFormat;
import android.media.AudioRecord;
import android.media.AudioTimestamp;
import android.media.MediaCodec;
import android.media.MediaFormat;
import android.media.MediaRecorder;
import android.os.Build;
import android.os.Handler;
import android.os.HandlerThread;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
public final class AudioEncoder {
private static class InputTask {
final int index;
InputTask(int index) {
this.index = index;
}
}
private static class OutputTask {
final int index;
final MediaCodec.BufferInfo bufferInfo;
OutputTask(int index, MediaCodec.BufferInfo bufferInfo) {
this.index = index;
this.bufferInfo = bufferInfo;
}
}
private static final int SAMPLE_RATE = 48000;
private static final int CHANNELS = 2;
private static final int BUFFER_MS = 15; // milliseconds
private static final int BUFFER_SIZE = SAMPLE_RATE * CHANNELS * BUFFER_MS / 1000;
private final Streamer streamer;
private final int bitRate;
private AudioRecord recorder;
private MediaCodec mediaCodec;
// Capacity of 64 is in practice "infinite" (it is limited by the number of available MediaCodec buffers, typically 4).
// So many pending tasks would lead to an unacceptable delay anyway.
private final BlockingQueue<InputTask> inputTasks = new ArrayBlockingQueue<>(64);
private final BlockingQueue<OutputTask> outputTasks = new ArrayBlockingQueue<>(64);
private Thread thread;
private HandlerThread mediaCodecThread;
private Thread inputThread;
private Thread outputThread;
private boolean ended;
public AudioEncoder(Streamer streamer, int bitRate) {
this.streamer = streamer;
this.bitRate = bitRate;
}
private static AudioFormat createAudioFormat() {
AudioFormat.Builder builder = new AudioFormat.Builder();
builder.setEncoding(AudioFormat.ENCODING_PCM_16BIT);
builder.setSampleRate(SAMPLE_RATE);
builder.setChannelMask(CHANNELS == 2 ? AudioFormat.CHANNEL_IN_STEREO : AudioFormat.CHANNEL_IN_MONO);
return builder.build();
}
@TargetApi(Build.VERSION_CODES.M)
@SuppressLint({"WrongConstant", "MissingPermission"})
private static AudioRecord createAudioRecord() {
AudioRecord.Builder builder = new AudioRecord.Builder();
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {
// On older APIs, Workarounds.fillAppInfo() must be called beforehand
builder.setContext(FakeContext.get());
}
builder.setAudioSource(MediaRecorder.AudioSource.REMOTE_SUBMIX);
builder.setAudioFormat(createAudioFormat());
builder.setBufferSizeInBytes(1024 * 1024);
return builder.build();
}
private static MediaFormat createFormat(String mimeType, int bitRate) {
MediaFormat format = new MediaFormat();
format.setString(MediaFormat.KEY_MIME, mimeType);
format.setInteger(MediaFormat.KEY_BIT_RATE, bitRate);
format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, CHANNELS);
format.setInteger(MediaFormat.KEY_SAMPLE_RATE, SAMPLE_RATE);
return format;
}
@TargetApi(Build.VERSION_CODES.N)
private void inputThread() throws IOException, InterruptedException {
final AudioTimestamp timestamp = new AudioTimestamp();
long previousPts = 0;
long nextPts = 0;
while (!Thread.currentThread().isInterrupted()) {
InputTask task = inputTasks.take();
ByteBuffer buffer = mediaCodec.getInputBuffer(task.index);
int r = recorder.read(buffer, BUFFER_SIZE);
if (r < 0) {
throw new IOException("Could not read audio: " + r);
}
long pts;
int ret = recorder.getTimestamp(timestamp, AudioTimestamp.TIMEBASE_MONOTONIC);
if (ret == AudioRecord.SUCCESS) {
pts = timestamp.nanoTime / 1000;
} else {
if (nextPts == 0) {
Ln.w("Could not get any audio timestamp");
}
// compute from previous timestamp and packet size
pts = nextPts;
}
long durationMs = r * 1000 / CHANNELS / SAMPLE_RATE;
nextPts = pts + durationMs;
if (previousPts != 0 && pts < previousPts) {
// Audio PTS may come from two sources:
// - recorder.getTimestamp() if the call works;
// - an estimation from the previous PTS and the packet size as a fallback.
//
// Therefore, the property that PTS are monotonically increasing is no guaranteed in corner cases, so enforce it.
pts = previousPts + 1;
}
mediaCodec.queueInputBuffer(task.index, 0, r, pts, 0);
previousPts = pts;
}
}
private void outputThread() throws IOException, InterruptedException {
streamer.writeHeader();
while (!Thread.currentThread().isInterrupted()) {
OutputTask task = outputTasks.take();
ByteBuffer buffer = mediaCodec.getOutputBuffer(task.index);
try {
streamer.writePacket(buffer, task.bufferInfo);
} finally {
mediaCodec.releaseOutputBuffer(task.index, false);
}
}
}
public void start() {
thread = new Thread(() -> {
try {
encode();
} catch (IOException e) {
Ln.e("Audio encoding error", e);
} finally {
Ln.d("Audio encoder stopped");
}
});
thread.start();
}
public void stop() {
if (thread != null) {
// Just wake up the blocking wait from the thread, so that it properly releases all its resources and terminates
end();
}
}
public void join() throws InterruptedException {
if (thread != null) {
thread.join();
}
}
private synchronized void end() {
ended = true;
notify();
}
private synchronized void waitEnded() {
try {
while (!ended) {
wait();
}
} catch (InterruptedException e) {
// ignore
}
}
@TargetApi(Build.VERSION_CODES.M)
public void encode() throws IOException {
try {
try {
String mimeType = streamer.getCodec().getMimeType();
mediaCodec = MediaCodec.createEncoderByType(mimeType); // may throw IOException
recorder = createAudioRecord();
mediaCodecThread = new HandlerThread("AudioEncoder");
mediaCodecThread.start();
MediaFormat format = createFormat(mimeType, bitRate);
mediaCodec.setCallback(new EncoderCallback(), new Handler(mediaCodecThread.getLooper()));
mediaCodec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
recorder.startRecording();
} catch (Throwable e) {
// Notify the client that the audio could not be captured
streamer.writeDisableStream();
throw e;
}
inputThread = new Thread(() -> {
try {
inputThread();
} catch (IOException | InterruptedException e) {
Ln.e("Audio capture error", e);
} finally {
end();
}
});
outputThread = new Thread(() -> {
try {
outputThread();
} catch (InterruptedException e) {
// this is expected on close
} catch (IOException e) {
// Broken pipe is expected on close, because the socket is closed by the client
if (!IO.isBrokenPipe(e)) {
Ln.e("Audio encoding error", e);
}
} finally {
end();
}
});
mediaCodec.start();
inputThread.start();
outputThread.start();
} catch (Throwable e) {
if (mediaCodec != null) {
mediaCodec.release();
}
if (recorder != null) {
recorder.release();
}
throw e;
}
try {
waitEnded();
} finally {
cleanUp();
}
}
private void cleanUp() {
mediaCodecThread.getLooper().quit();
inputThread.interrupt();
outputThread.interrupt();
try {
mediaCodecThread.join();
inputThread.join();
outputThread.join();
} catch (InterruptedException e) {
// Should never happen
throw new AssertionError(e);
}
mediaCodec.stop();
mediaCodec.release();
recorder.stop();
recorder.release();
}
private class EncoderCallback extends MediaCodec.Callback {
@TargetApi(Build.VERSION_CODES.N)
@Override
public void onInputBufferAvailable(MediaCodec codec, int index) {
try {
inputTasks.put(new InputTask(index));
} catch (InterruptedException e) {
end();
}
}
@Override
public void onOutputBufferAvailable(MediaCodec codec, int index, MediaCodec.BufferInfo bufferInfo) {
try {
outputTasks.put(new OutputTask(index, bufferInfo));
} catch (InterruptedException e) {
end();
}
}
@Override
public void onError(MediaCodec codec, MediaCodec.CodecException e) {
Ln.e("MediaCodec error", e);
end();
}
@Override
public void onOutputFormatChanged(MediaCodec codec, MediaFormat format) {
// ignore
}
}
}

View File

@ -0,0 +1,14 @@
package com.genymobile.scrcpy;
public interface Codec {
enum Type {VIDEO, AUDIO}
Type getType();
int getId();
String getName();
String getMimeType();
}

View File

@ -20,6 +20,9 @@ public final class DesktopConnection implements Closeable {
private final LocalSocket videoSocket; private final LocalSocket videoSocket;
private final FileDescriptor videoFd; private final FileDescriptor videoFd;
private final LocalSocket audioSocket;
private final FileDescriptor audioFd;
private final LocalSocket controlSocket; private final LocalSocket controlSocket;
private final InputStream controlInputStream; private final InputStream controlInputStream;
private final OutputStream controlOutputStream; private final OutputStream controlOutputStream;
@ -27,9 +30,10 @@ public final class DesktopConnection implements Closeable {
private final ControlMessageReader reader = new ControlMessageReader(); private final ControlMessageReader reader = new ControlMessageReader();
private final DeviceMessageWriter writer = new DeviceMessageWriter(); private final DeviceMessageWriter writer = new DeviceMessageWriter();
private DesktopConnection(LocalSocket videoSocket, LocalSocket controlSocket) throws IOException { private DesktopConnection(LocalSocket videoSocket, LocalSocket audioSocket, LocalSocket controlSocket) throws IOException {
this.videoSocket = videoSocket; this.videoSocket = videoSocket;
this.controlSocket = controlSocket; this.controlSocket = controlSocket;
this.audioSocket = audioSocket;
if (controlSocket != null) { if (controlSocket != null) {
controlInputStream = controlSocket.getInputStream(); controlInputStream = controlSocket.getInputStream();
controlOutputStream = controlSocket.getOutputStream(); controlOutputStream = controlSocket.getOutputStream();
@ -38,6 +42,7 @@ public final class DesktopConnection implements Closeable {
controlOutputStream = null; controlOutputStream = null;
} }
videoFd = videoSocket.getFileDescriptor(); videoFd = videoSocket.getFileDescriptor();
audioFd = audioSocket != null ? audioSocket.getFileDescriptor() : null;
} }
private static LocalSocket connect(String abstractName) throws IOException { private static LocalSocket connect(String abstractName) throws IOException {
@ -55,40 +60,50 @@ public final class DesktopConnection implements Closeable {
return SOCKET_NAME_PREFIX + String.format("_%08x", scid); return SOCKET_NAME_PREFIX + String.format("_%08x", scid);
} }
public static DesktopConnection open(int scid, boolean tunnelForward, boolean control, boolean sendDummyByte) throws IOException { public static DesktopConnection open(int scid, boolean tunnelForward, boolean audio, boolean control, boolean sendDummyByte) throws IOException {
String socketName = getSocketName(scid); String socketName = getSocketName(scid);
LocalSocket videoSocket; LocalSocket videoSocket = null;
LocalSocket audioSocket = null;
LocalSocket controlSocket = null; LocalSocket controlSocket = null;
if (tunnelForward) { try {
try (LocalServerSocket localServerSocket = new LocalServerSocket(socketName)) { if (tunnelForward) {
videoSocket = localServerSocket.accept(); try (LocalServerSocket localServerSocket = new LocalServerSocket(socketName)) {
if (sendDummyByte) { videoSocket = localServerSocket.accept();
// send one byte so the client may read() to detect a connection error if (sendDummyByte) {
videoSocket.getOutputStream().write(0); // send one byte so the client may read() to detect a connection error
} videoSocket.getOutputStream().write(0);
if (control) { }
try { if (audio) {
audioSocket = localServerSocket.accept();
}
if (control) {
controlSocket = localServerSocket.accept(); controlSocket = localServerSocket.accept();
} catch (IOException | RuntimeException e) {
videoSocket.close();
throw e;
} }
} }
} } else {
} else { videoSocket = connect(socketName);
videoSocket = connect(socketName); if (audio) {
if (control) { audioSocket = connect(socketName);
try { }
if (control) {
controlSocket = connect(socketName); controlSocket = connect(socketName);
} catch (IOException | RuntimeException e) {
videoSocket.close();
throw e;
} }
} }
} catch (IOException | RuntimeException e) {
if (videoSocket != null) {
videoSocket.close();
}
if (audioSocket != null) {
audioSocket.close();
}
if (controlSocket != null) {
controlSocket.close();
}
throw e;
} }
return new DesktopConnection(videoSocket, controlSocket); return new DesktopConnection(videoSocket, audioSocket, controlSocket);
} }
public void close() throws IOException { public void close() throws IOException {
@ -121,6 +136,10 @@ public final class DesktopConnection implements Closeable {
return videoFd; return videoFd;
} }
public FileDescriptor getAudioFd() {
return audioFd;
}
public ControlMessage receiveControlMessage() throws IOException { public ControlMessage receiveControlMessage() throws IOException {
ControlMessage msg = reader.next(); ControlMessage msg = reader.next();
while (msg == null) { while (msg == null) {

View File

@ -0,0 +1,40 @@
package com.genymobile.scrcpy;
import android.annotation.TargetApi;
import android.content.AttributionSource;
import android.content.ContextWrapper;
import android.os.Build;
import android.os.Process;
public final class FakeContext extends ContextWrapper {
public static final String PACKAGE_NAME = "com.android.shell";
private static final FakeContext INSTANCE = new FakeContext();
public static FakeContext get() {
return INSTANCE;
}
private FakeContext() {
super(null);
}
@Override
public String getPackageName() {
return PACKAGE_NAME;
}
@Override
public String getOpPackageName() {
return PACKAGE_NAME;
}
@TargetApi(Build.VERSION_CODES.S)
@Override
public AttributionSource getAttributionSource() {
AttributionSource.Builder builder = new AttributionSource.Builder(Process.SHELL_UID);
builder.setPackageName(PACKAGE_NAME);
return builder.build();
}
}

View File

@ -8,9 +8,12 @@ public class Options {
private Ln.Level logLevel = Ln.Level.DEBUG; private Ln.Level logLevel = Ln.Level.DEBUG;
private int scid = -1; // 31-bit non-negative value, or -1 private int scid = -1; // 31-bit non-negative value, or -1
private boolean audio = true;
private int maxSize; private int maxSize;
private VideoCodec codec = VideoCodec.H264; private VideoCodec codec = VideoCodec.H264;
private AudioCodec audioCodec = AudioCodec.OPUS;
private int bitRate = 8000000; private int bitRate = 8000000;
private int audioBitRate = 196000;
private int maxFps; private int maxFps;
private int lockVideoOrientation = -1; private int lockVideoOrientation = -1;
private boolean tunnelForward; private boolean tunnelForward;
@ -49,6 +52,14 @@ public class Options {
this.scid = scid; this.scid = scid;
} }
public boolean getAudio() {
return audio;
}
public void setAudio(boolean audio) {
this.audio = audio;
}
public int getMaxSize() { public int getMaxSize() {
return maxSize; return maxSize;
} }
@ -65,6 +76,14 @@ public class Options {
this.codec = codec; this.codec = codec;
} }
public AudioCodec getAudioCodec() {
return audioCodec;
}
public void setAudioCodec(AudioCodec audioCodec) {
this.audioCodec = audioCodec;
}
public int getBitRate() { public int getBitRate() {
return bitRate; return bitRate;
} }
@ -73,6 +92,14 @@ public class Options {
this.bitRate = bitRate; this.bitRate = bitRate;
} }
public int getAudioBitRate() {
return audioBitRate;
}
public void setAudioBitRate(int audioBitRate) {
this.audioBitRate = audioBitRate;
}
public int getMaxFps() { public int getMaxFps() {
return maxFps; return maxFps;
} }

View File

@ -21,10 +21,6 @@ import java.util.concurrent.atomic.AtomicBoolean;
public class ScreenEncoder implements Device.RotationListener { public class ScreenEncoder implements Device.RotationListener {
public interface Callbacks {
void onPacket(ByteBuffer codecBuffer, MediaCodec.BufferInfo bufferInfo) throws IOException;
}
private static final int DEFAULT_I_FRAME_INTERVAL = 10; // seconds private static final int DEFAULT_I_FRAME_INTERVAL = 10; // seconds
private static final int REPEAT_FRAME_DELAY_US = 100_000; // repeat after 100ms private static final int REPEAT_FRAME_DELAY_US = 100_000; // repeat after 100ms
private static final String KEY_MAX_FPS_TO_ENCODER = "max-fps-to-encoder"; private static final String KEY_MAX_FPS_TO_ENCODER = "max-fps-to-encoder";
@ -35,7 +31,8 @@ public class ScreenEncoder implements Device.RotationListener {
private final AtomicBoolean rotationChanged = new AtomicBoolean(); private final AtomicBoolean rotationChanged = new AtomicBoolean();
private final String videoMimeType; private final Device device;
private final Streamer streamer;
private final String encoderName; private final String encoderName;
private final List<CodecOption> codecOptions; private final List<CodecOption> codecOptions;
private final int bitRate; private final int bitRate;
@ -45,8 +42,10 @@ public class ScreenEncoder implements Device.RotationListener {
private boolean firstFrameSent; private boolean firstFrameSent;
private int consecutiveErrors; private int consecutiveErrors;
public ScreenEncoder(String videoMimeType, int bitRate, int maxFps, List<CodecOption> codecOptions, String encoderName, boolean downsizeOnError) { public ScreenEncoder(Device device, Streamer streamer, int bitRate, int maxFps, List<CodecOption> codecOptions, String encoderName,
this.videoMimeType = videoMimeType; boolean downsizeOnError) {
this.device = device;
this.streamer = streamer;
this.bitRate = bitRate; this.bitRate = bitRate;
this.maxFps = maxFps; this.maxFps = maxFps;
this.codecOptions = codecOptions; this.codecOptions = codecOptions;
@ -63,11 +62,15 @@ public class ScreenEncoder implements Device.RotationListener {
return rotationChanged.getAndSet(false); return rotationChanged.getAndSet(false);
} }
public void streamScreen(Device device, Callbacks callbacks) throws IOException { public void streamScreen() throws IOException {
String videoMimeType = streamer.getCodec().getMimeType();
MediaCodec codec = createCodec(videoMimeType, encoderName); MediaCodec codec = createCodec(videoMimeType, encoderName);
MediaFormat format = createFormat(videoMimeType, bitRate, maxFps, codecOptions); MediaFormat format = createFormat(videoMimeType, bitRate, maxFps, codecOptions);
IBinder display = createDisplay(); IBinder display = createDisplay();
device.setRotationListener(this); device.setRotationListener(this);
streamer.writeHeader();
boolean alive; boolean alive;
try { try {
do { do {
@ -92,7 +95,7 @@ public class ScreenEncoder implements Device.RotationListener {
codec.start(); codec.start();
alive = encode(codec, callbacks); alive = encode(codec, streamer);
// do not call stop() on exception, it would trigger an IllegalStateException // do not call stop() on exception, it would trigger an IllegalStateException
codec.stop(); codec.stop();
} catch (IllegalStateException | IllegalArgumentException e) { } catch (IllegalStateException | IllegalArgumentException e) {
@ -161,7 +164,7 @@ public class ScreenEncoder implements Device.RotationListener {
return 0; return 0;
} }
private boolean encode(MediaCodec codec, Callbacks callbacks) throws IOException { private boolean encode(MediaCodec codec, Streamer streamer) throws IOException {
boolean eof = false; boolean eof = false;
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo(); MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
@ -184,7 +187,7 @@ public class ScreenEncoder implements Device.RotationListener {
consecutiveErrors = 0; consecutiveErrors = 0;
} }
callbacks.onPacket(codecBuffer, bufferInfo); streamer.writePacket(codecBuffer, bufferInfo);
} }
} finally { } finally {
if (outputBufferId >= 0) { if (outputBufferId >= 0) {

View File

@ -69,29 +69,36 @@ public final class Server {
int scid = options.getScid(); int scid = options.getScid();
boolean tunnelForward = options.isTunnelForward(); boolean tunnelForward = options.isTunnelForward();
boolean control = options.getControl(); boolean control = options.getControl();
boolean audio = options.getAudio();
boolean sendDummyByte = options.getSendDummyByte(); boolean sendDummyByte = options.getSendDummyByte();
Workarounds.prepareMainLooper(); Workarounds.prepareMainLooper();
if (Build.BRAND.equalsIgnoreCase("meizu")) {
// Workarounds must be applied for Meizu phones: // Workarounds must be applied for Meizu phones:
// - <https://github.com/Genymobile/scrcpy/issues/240> // - <https://github.com/Genymobile/scrcpy/issues/240>
// - <https://github.com/Genymobile/scrcpy/issues/365> // - <https://github.com/Genymobile/scrcpy/issues/365>
// - <https://github.com/Genymobile/scrcpy/issues/2656> // - <https://github.com/Genymobile/scrcpy/issues/2656>
// //
// But only apply when strictly necessary, since workarounds can cause other issues: // But only apply when strictly necessary, since workarounds can cause other issues:
// - <https://github.com/Genymobile/scrcpy/issues/940> // - <https://github.com/Genymobile/scrcpy/issues/940>
// - <https://github.com/Genymobile/scrcpy/issues/994> // - <https://github.com/Genymobile/scrcpy/issues/994>
boolean mustFillAppInfo = Build.BRAND.equalsIgnoreCase("meizu");
// Before Android 11, audio is not supported.
// Since Android 12, we can properly set a context on the AudioRecord.
// Only on Android 11 we must fill app info for the AudioRecord to work.
mustFillAppInfo |= audio && Build.VERSION.SDK_INT == Build.VERSION_CODES.R;
if (mustFillAppInfo) {
Workarounds.fillAppInfo(); Workarounds.fillAppInfo();
} }
try (DesktopConnection connection = DesktopConnection.open(scid, tunnelForward, control, sendDummyByte)) { try (DesktopConnection connection = DesktopConnection.open(scid, tunnelForward, audio, control, sendDummyByte)) {
VideoCodec codec = options.getCodec(); VideoCodec codec = options.getCodec();
if (options.getSendDeviceMeta()) { if (options.getSendDeviceMeta()) {
Size videoSize = device.getScreenInfo().getVideoSize(); Size videoSize = device.getScreenInfo().getVideoSize();
connection.sendDeviceMeta(Device.getDeviceName(), videoSize.getWidth(), videoSize.getHeight()); connection.sendDeviceMeta(Device.getDeviceName(), videoSize.getWidth(), videoSize.getHeight());
} }
ScreenEncoder screenEncoder = new ScreenEncoder(codec.getMimeType(), options.getBitRate(), options.getMaxFps(), codecOptions,
options.getEncoderName(), options.getDownsizeOnError());
Controller controller = null; Controller controller = null;
if (control) { if (control) {
@ -102,13 +109,20 @@ public final class Server {
device.setClipboardListener(text -> controllerRef.getSender().pushClipboardText(text)); device.setClipboardListener(text -> controllerRef.getSender().pushClipboardText(text));
} }
AudioEncoder audioEncoder = null;
if (audio) {
Streamer audioStreamer = new Streamer(connection.getAudioFd(), options.getAudioCodec(), options.getSendCodecId(),
options.getSendFrameMeta());
audioEncoder = new AudioEncoder(audioStreamer, options.getAudioBitRate());
audioEncoder.start();
}
Streamer videoStreamer = new Streamer(connection.getVideoFd(), codec, options.getSendCodecId(), options.getSendFrameMeta());
ScreenEncoder screenEncoder = new ScreenEncoder(device, videoStreamer, options.getBitRate(), options.getMaxFps(),
codecOptions, options.getEncoderName(), options.getDownsizeOnError());
try { try {
// synchronous // synchronous
VideoStreamer videoStreamer = new VideoStreamer(connection.getVideoFd(), options.getSendFrameMeta()); screenEncoder.streamScreen();
if (options.getSendCodecId()) {
videoStreamer.writeHeader(codec.getId());
}
screenEncoder.streamScreen(device, videoStreamer);
} catch (IOException e) { } catch (IOException e) {
// Broken pipe is expected on close, because the socket is closed by the client // Broken pipe is expected on close, because the socket is closed by the client
if (!IO.isBrokenPipe(e)) { if (!IO.isBrokenPipe(e)) {
@ -117,12 +131,18 @@ public final class Server {
} finally { } finally {
Ln.d("Screen streaming stopped"); Ln.d("Screen streaming stopped");
initThread.interrupt(); initThread.interrupt();
if (audioEncoder != null) {
audioEncoder.stop();
}
if (controller != null) { if (controller != null) {
controller.stop(); controller.stop();
} }
try { try {
initThread.join(); initThread.join();
if (audioEncoder != null) {
audioEncoder.join();
}
if (controller != null) { if (controller != null) {
controller.join(); controller.join();
} }
@ -172,6 +192,10 @@ public final class Server {
Ln.Level level = Ln.Level.valueOf(value.toUpperCase(Locale.ENGLISH)); Ln.Level level = Ln.Level.valueOf(value.toUpperCase(Locale.ENGLISH));
options.setLogLevel(level); options.setLogLevel(level);
break; break;
case "audio":
boolean audio = Boolean.parseBoolean(value);
options.setAudio(audio);
break;
case "codec": case "codec":
VideoCodec codec = VideoCodec.findByName(value); VideoCodec codec = VideoCodec.findByName(value);
if (codec == null) { if (codec == null) {
@ -179,6 +203,13 @@ public final class Server {
} }
options.setCodec(codec); options.setCodec(codec);
break; break;
case "audio_codec":
AudioCodec audioCodec = AudioCodec.findByName(value);
if (audioCodec == null) {
throw new IllegalArgumentException("Audio codec " + value + " not supported");
}
options.setAudioCodec(audioCodec);
break;
case "max_size": case "max_size":
int maxSize = Integer.parseInt(value) & ~7; // multiple of 8 int maxSize = Integer.parseInt(value) & ~7; // multiple of 8
options.setMaxSize(maxSize); options.setMaxSize(maxSize);
@ -187,6 +218,10 @@ public final class Server {
int bitRate = Integer.parseInt(value); int bitRate = Integer.parseInt(value);
options.setBitRate(bitRate); options.setBitRate(bitRate);
break; break;
case "audio_bit_rate":
int audioBitRate = Integer.parseInt(value);
options.setAudioBitRate(audioBitRate);
break;
case "max_fps": case "max_fps":
int maxFps = Integer.parseInt(value); int maxFps = Integer.parseInt(value);
options.setMaxFps(maxFps); options.setMaxFps(maxFps);

View File

@ -0,0 +1,121 @@
package com.genymobile.scrcpy;
import android.media.MediaCodec;
import java.io.FileDescriptor;
import java.io.IOException;
import java.nio.ByteBuffer;
public final class Streamer {
private static final long PACKET_FLAG_CONFIG = 1L << 63;
private static final long PACKET_FLAG_KEY_FRAME = 1L << 62;
private static final long AOPUSHDR = 0x5244485355504F41L; // "AOPUSHDR" in ASCII (little-endian)
private final FileDescriptor fd;
private final Codec codec;
private final boolean sendCodecId;
private final boolean sendFrameMeta;
private final ByteBuffer headerBuffer = ByteBuffer.allocate(12);
public Streamer(FileDescriptor fd, Codec codec, boolean sendCodecId, boolean sendFrameMeta) {
this.fd = fd;
this.codec = codec;
this.sendCodecId = sendCodecId;
this.sendFrameMeta = sendFrameMeta;
}
public Codec getCodec() {
return codec;
}
public void writeHeader() throws IOException {
if (sendCodecId) {
ByteBuffer buffer = ByteBuffer.allocate(4);
buffer.putInt(codec.getId());
buffer.flip();
IO.writeFully(fd, buffer);
}
}
public void writeDisableStream() throws IOException {
// Writing 0 (32-bit) as codec-id means that the device disables the stream (because it could not capture)
byte[] zeros = new byte[4];
IO.writeFully(fd, zeros, 0, zeros.length);
}
public void writePacket(ByteBuffer codecBuffer, MediaCodec.BufferInfo bufferInfo) throws IOException {
if (codec == AudioCodec.OPUS) {
fixOpusConfigPacket(codecBuffer, bufferInfo);
}
if (sendFrameMeta) {
writeFrameMeta(fd, bufferInfo, codecBuffer.remaining());
}
IO.writeFully(fd, codecBuffer);
}
private void writeFrameMeta(FileDescriptor fd, MediaCodec.BufferInfo bufferInfo, int packetSize) throws IOException {
headerBuffer.clear();
long ptsAndFlags;
if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
ptsAndFlags = PACKET_FLAG_CONFIG; // non-media data packet
} else {
ptsAndFlags = bufferInfo.presentationTimeUs;
if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_KEY_FRAME) != 0) {
ptsAndFlags |= PACKET_FLAG_KEY_FRAME;
}
}
headerBuffer.putLong(ptsAndFlags);
headerBuffer.putInt(packetSize);
headerBuffer.flip();
IO.writeFully(fd, headerBuffer);
}
private static void fixOpusConfigPacket(ByteBuffer buffer, MediaCodec.BufferInfo bufferInfo) throws IOException {
// Here is an example of the config packet received for an OPUS stream:
//
// 00000000 41 4f 50 55 53 48 44 52 13 00 00 00 00 00 00 00 |AOPUSHDR........|
// -------------- BELOW IS THE PART WE MUST PUT AS EXTRADATA -------------------
// 00000010 4f 70 75 73 48 65 61 64 01 01 38 01 80 bb 00 00 |OpusHead..8.....|
// 00000020 00 00 00 |... |
// ------------------------------------------------------------------------------
// 00000020 41 4f 50 55 53 44 4c 59 08 00 00 00 00 | AOPUSDLY.....|
// 00000030 00 00 00 a0 2e 63 00 00 00 00 00 41 4f 50 55 53 |.....c.....AOPUS|
// 00000040 50 52 4c 08 00 00 00 00 00 00 00 00 b4 c4 04 00 |PRL.............|
// 00000050 00 00 00 |...|
//
// Each "section" is prefixed by a 64-bit ID and a 64-bit length.
boolean isConfig = (bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0;
if (!isConfig) {
return;
}
while (buffer.remaining() >= 16) {
long id = buffer.getLong();
long sizeLong = buffer.getLong();
if (sizeLong < 0 || sizeLong >= 0x7FFFFFFF) {
throw new IOException("Invalid block size in OPUS header: " + sizeLong);
}
int size = (int) sizeLong;
if (id == AOPUSHDR) {
if (buffer.remaining() < size) {
throw new IOException("Not enough data in OPUS header (invalid size: " + size + ")");
}
// Set the buffer to point to the OPUS header slice
buffer.limit(buffer.position() + size);
return;
}
buffer.position(buffer.position() + size);
}
throw new IOException("OPUS header not found");
}
}

View File

@ -2,7 +2,7 @@ package com.genymobile.scrcpy;
import android.media.MediaFormat; import android.media.MediaFormat;
public enum VideoCodec { public enum VideoCodec implements Codec {
H264(0x68_32_36_34, "h264", MediaFormat.MIMETYPE_VIDEO_AVC), H264(0x68_32_36_34, "h264", MediaFormat.MIMETYPE_VIDEO_AVC),
H265(0x68_32_36_35, "h265", MediaFormat.MIMETYPE_VIDEO_HEVC), H265(0x68_32_36_35, "h265", MediaFormat.MIMETYPE_VIDEO_HEVC),
AV1(0x00_61_76_31, "av1", MediaFormat.MIMETYPE_VIDEO_AV1); AV1(0x00_61_76_31, "av1", MediaFormat.MIMETYPE_VIDEO_AV1);
@ -17,10 +17,22 @@ public enum VideoCodec {
this.mimeType = mimeType; this.mimeType = mimeType;
} }
@Override
public Type getType() {
return Type.VIDEO;
}
@Override
public int getId() { public int getId() {
return id; return id;
} }
@Override
public String getName() {
return name;
}
@Override
public String getMimeType() { public String getMimeType() {
return mimeType; return mimeType;
} }

View File

@ -1,58 +0,0 @@
package com.genymobile.scrcpy;
import android.media.MediaCodec;
import java.io.FileDescriptor;
import java.io.IOException;
import java.nio.ByteBuffer;
public final class VideoStreamer implements ScreenEncoder.Callbacks {
private static final long PACKET_FLAG_CONFIG = 1L << 63;
private static final long PACKET_FLAG_KEY_FRAME = 1L << 62;
private final FileDescriptor fd;
private final boolean sendFrameMeta;
private final ByteBuffer headerBuffer = ByteBuffer.allocate(12);
public VideoStreamer(FileDescriptor fd, boolean sendFrameMeta) {
this.fd = fd;
this.sendFrameMeta = sendFrameMeta;
}
public void writeHeader(int codecId) throws IOException {
ByteBuffer buffer = ByteBuffer.allocate(4);
buffer.putInt(codecId);
buffer.flip();
IO.writeFully(fd, buffer);
}
@Override
public void onPacket(ByteBuffer codecBuffer, MediaCodec.BufferInfo bufferInfo) throws IOException {
if (sendFrameMeta) {
writeFrameMeta(fd, bufferInfo, codecBuffer.remaining());
}
IO.writeFully(fd, codecBuffer);
}
private void writeFrameMeta(FileDescriptor fd, MediaCodec.BufferInfo bufferInfo, int packetSize) throws IOException {
headerBuffer.clear();
long ptsAndFlags;
if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
ptsAndFlags = PACKET_FLAG_CONFIG; // non-media data packet
} else {
ptsAndFlags = bufferInfo.presentationTimeUs;
if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_KEY_FRAME) != 0) {
ptsAndFlags |= PACKET_FLAG_KEY_FRAME;
}
}
headerBuffer.putLong(ptsAndFlags);
headerBuffer.putInt(packetSize);
headerBuffer.flip();
IO.writeFully(fd, headerBuffer);
}
}

View File

@ -3,13 +3,12 @@ package com.genymobile.scrcpy;
import android.annotation.SuppressLint; import android.annotation.SuppressLint;
import android.app.Application; import android.app.Application;
import android.app.Instrumentation; import android.app.Instrumentation;
import android.content.Context; import android.content.ContextWrapper;
import android.content.pm.ApplicationInfo; import android.content.pm.ApplicationInfo;
import android.os.Looper; import android.os.Looper;
import java.lang.reflect.Constructor; import java.lang.reflect.Constructor;
import java.lang.reflect.Field; import java.lang.reflect.Field;
import java.lang.reflect.Method;
public final class Workarounds { public final class Workarounds {
private Workarounds() { private Workarounds() {
@ -50,7 +49,7 @@ public final class Workarounds {
Object appBindData = appBindDataConstructor.newInstance(); Object appBindData = appBindDataConstructor.newInstance();
ApplicationInfo applicationInfo = new ApplicationInfo(); ApplicationInfo applicationInfo = new ApplicationInfo();
applicationInfo.packageName = "com.genymobile.scrcpy"; applicationInfo.packageName = FakeContext.PACKAGE_NAME;
// appBindData.appInfo = applicationInfo; // appBindData.appInfo = applicationInfo;
Field appInfoField = appBindDataClass.getDeclaredField("appInfo"); Field appInfoField = appBindDataClass.getDeclaredField("appInfo");
@ -62,11 +61,10 @@ public final class Workarounds {
mBoundApplicationField.setAccessible(true); mBoundApplicationField.setAccessible(true);
mBoundApplicationField.set(activityThread, appBindData); mBoundApplicationField.set(activityThread, appBindData);
// Context ctx = activityThread.getSystemContext(); Application app = Application.class.newInstance();
Method getSystemContextMethod = activityThreadClass.getDeclaredMethod("getSystemContext"); Field baseField = ContextWrapper.class.getDeclaredField("mBase");
Context ctx = (Context) getSystemContextMethod.invoke(activityThread); baseField.setAccessible(true);
baseField.set(app, FakeContext.get());
Application app = Instrumentation.newApplication(Application.class, ctx);
// activityThread.mInitialApplication = app; // activityThread.mInitialApplication = app;
Field mInitialApplicationField = activityThreadClass.getDeclaredField("mInitialApplication"); Field mInitialApplicationField = activityThreadClass.getDeclaredField("mInitialApplication");

View File

@ -5,6 +5,7 @@ import com.genymobile.scrcpy.Ln;
import android.os.Binder; import android.os.Binder;
import android.os.IBinder; import android.os.IBinder;
import android.os.IInterface; import android.os.IInterface;
import android.os.Process;
import java.lang.reflect.Field; import java.lang.reflect.Field;
import java.lang.reflect.InvocationTargetException; import java.lang.reflect.InvocationTargetException;
@ -48,10 +49,10 @@ public class ActivityManager {
Object[] args; Object[] args;
if (getContentProviderExternalMethodNewVersion) { if (getContentProviderExternalMethodNewVersion) {
// new version // new version
args = new Object[]{name, ServiceManager.USER_ID, token, null}; args = new Object[]{name, Process.ROOT_UID, token, null};
} else { } else {
// old version // old version
args = new Object[]{name, ServiceManager.USER_ID, token}; args = new Object[]{name, Process.ROOT_UID, token};
} }
// ContentProviderHolder providerHolder = getContentProviderExternal(...); // ContentProviderHolder providerHolder = getContentProviderExternal(...);
Object providerHolder = method.invoke(manager, args); Object providerHolder = method.invoke(manager, args);

View File

@ -1,11 +1,13 @@
package com.genymobile.scrcpy.wrappers; package com.genymobile.scrcpy.wrappers;
import com.genymobile.scrcpy.FakeContext;
import com.genymobile.scrcpy.Ln; import com.genymobile.scrcpy.Ln;
import android.content.ClipData; import android.content.ClipData;
import android.content.IOnPrimaryClipChangedListener; import android.content.IOnPrimaryClipChangedListener;
import android.os.Build; import android.os.Build;
import android.os.IInterface; import android.os.IInterface;
import android.os.Process;
import java.lang.reflect.InvocationTargetException; import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method; import java.lang.reflect.Method;
@ -58,22 +60,22 @@ public class ClipboardManager {
private static ClipData getPrimaryClip(Method method, boolean alternativeMethod, IInterface manager) private static ClipData getPrimaryClip(Method method, boolean alternativeMethod, IInterface manager)
throws InvocationTargetException, IllegalAccessException { throws InvocationTargetException, IllegalAccessException {
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) { if (Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) {
return (ClipData) method.invoke(manager, ServiceManager.PACKAGE_NAME); return (ClipData) method.invoke(manager, FakeContext.PACKAGE_NAME);
} }
if (alternativeMethod) { if (alternativeMethod) {
return (ClipData) method.invoke(manager, ServiceManager.PACKAGE_NAME, null, ServiceManager.USER_ID); return (ClipData) method.invoke(manager, FakeContext.PACKAGE_NAME, null, Process.ROOT_UID);
} }
return (ClipData) method.invoke(manager, ServiceManager.PACKAGE_NAME, ServiceManager.USER_ID); return (ClipData) method.invoke(manager, FakeContext.PACKAGE_NAME, Process.ROOT_UID);
} }
private static void setPrimaryClip(Method method, boolean alternativeMethod, IInterface manager, ClipData clipData) private static void setPrimaryClip(Method method, boolean alternativeMethod, IInterface manager, ClipData clipData)
throws InvocationTargetException, IllegalAccessException { throws InvocationTargetException, IllegalAccessException {
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) { if (Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) {
method.invoke(manager, clipData, ServiceManager.PACKAGE_NAME); method.invoke(manager, clipData, FakeContext.PACKAGE_NAME);
} else if (alternativeMethod) { } else if (alternativeMethod) {
method.invoke(manager, clipData, ServiceManager.PACKAGE_NAME, null, ServiceManager.USER_ID); method.invoke(manager, clipData, FakeContext.PACKAGE_NAME, null, Process.ROOT_UID);
} else { } else {
method.invoke(manager, clipData, ServiceManager.PACKAGE_NAME, ServiceManager.USER_ID); method.invoke(manager, clipData, FakeContext.PACKAGE_NAME, Process.ROOT_UID);
} }
} }
@ -106,11 +108,11 @@ public class ClipboardManager {
private static void addPrimaryClipChangedListener(Method method, boolean alternativeMethod, IInterface manager, private static void addPrimaryClipChangedListener(Method method, boolean alternativeMethod, IInterface manager,
IOnPrimaryClipChangedListener listener) throws InvocationTargetException, IllegalAccessException { IOnPrimaryClipChangedListener listener) throws InvocationTargetException, IllegalAccessException {
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) { if (Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) {
method.invoke(manager, listener, ServiceManager.PACKAGE_NAME); method.invoke(manager, listener, FakeContext.PACKAGE_NAME);
} else if (alternativeMethod) { } else if (alternativeMethod) {
method.invoke(manager, listener, ServiceManager.PACKAGE_NAME, null, ServiceManager.USER_ID); method.invoke(manager, listener, FakeContext.PACKAGE_NAME, null, Process.ROOT_UID);
} else { } else {
method.invoke(manager, listener, ServiceManager.PACKAGE_NAME, ServiceManager.USER_ID); method.invoke(manager, listener, FakeContext.PACKAGE_NAME, Process.ROOT_UID);
} }
} }

View File

@ -1,11 +1,15 @@
package com.genymobile.scrcpy.wrappers; package com.genymobile.scrcpy.wrappers;
import com.genymobile.scrcpy.FakeContext;
import com.genymobile.scrcpy.Ln; import com.genymobile.scrcpy.Ln;
import com.genymobile.scrcpy.SettingsException; import com.genymobile.scrcpy.SettingsException;
import android.annotation.SuppressLint; import android.annotation.SuppressLint;
import android.content.AttributionSource;
import android.os.Build;
import android.os.Bundle; import android.os.Bundle;
import android.os.IBinder; import android.os.IBinder;
import android.os.Process;
import java.io.Closeable; import java.io.Closeable;
import java.lang.reflect.InvocationTargetException; import java.lang.reflect.InvocationTargetException;
@ -51,11 +55,10 @@ public class ContentProvider implements Closeable {
@SuppressLint("PrivateApi") @SuppressLint("PrivateApi")
private Method getCallMethod() throws NoSuchMethodException { private Method getCallMethod() throws NoSuchMethodException {
if (callMethod == null) { if (callMethod == null) {
try { if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {
Class<?> attributionSourceClass = Class.forName("android.content.AttributionSource"); callMethod = provider.getClass().getMethod("call", AttributionSource.class, String.class, String.class, String.class, Bundle.class);
callMethod = provider.getClass().getMethod("call", attributionSourceClass, String.class, String.class, String.class, Bundle.class);
callMethodVersion = 0; callMethodVersion = 0;
} catch (NoSuchMethodException | ClassNotFoundException e0) { } else {
// old versions // old versions
try { try {
callMethod = provider.getClass() callMethod = provider.getClass()
@ -75,40 +78,29 @@ public class ContentProvider implements Closeable {
return callMethod; return callMethod;
} }
@SuppressLint("PrivateApi")
private Object getAttributionSource()
throws ClassNotFoundException, NoSuchMethodException, IllegalAccessException, InvocationTargetException, InstantiationException {
if (attributionSource == null) {
Class<?> cl = Class.forName("android.content.AttributionSource$Builder");
Object builder = cl.getConstructor(int.class).newInstance(ServiceManager.USER_ID);
cl.getDeclaredMethod("setPackageName", String.class).invoke(builder, ServiceManager.PACKAGE_NAME);
attributionSource = cl.getDeclaredMethod("build").invoke(builder);
}
return attributionSource;
}
private Bundle call(String callMethod, String arg, Bundle extras) private Bundle call(String callMethod, String arg, Bundle extras)
throws ClassNotFoundException, NoSuchMethodException, InvocationTargetException, InstantiationException, IllegalAccessException { throws NoSuchMethodException, InvocationTargetException, IllegalAccessException {
try { try {
Method method = getCallMethod(); Method method = getCallMethod();
Object[] args; Object[] args;
switch (callMethodVersion) {
case 0: if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S && callMethodVersion == 0) {
args = new Object[]{getAttributionSource(), "settings", callMethod, arg, extras}; args = new Object[]{FakeContext.get().getAttributionSource(), "settings", callMethod, arg, extras};
break; } else {
case 1: switch (callMethodVersion) {
args = new Object[]{ServiceManager.PACKAGE_NAME, null, "settings", callMethod, arg, extras}; case 1:
break; args = new Object[]{FakeContext.PACKAGE_NAME, null, "settings", callMethod, arg, extras};
case 2: break;
args = new Object[]{ServiceManager.PACKAGE_NAME, "settings", callMethod, arg, extras}; case 2:
break; args = new Object[]{FakeContext.PACKAGE_NAME, "settings", callMethod, arg, extras};
default: break;
args = new Object[]{ServiceManager.PACKAGE_NAME, callMethod, arg, extras}; default:
break; args = new Object[]{FakeContext.PACKAGE_NAME, callMethod, arg, extras};
break;
}
} }
return (Bundle) method.invoke(provider, args); return (Bundle) method.invoke(provider, args);
} catch (InvocationTargetException | IllegalAccessException | NoSuchMethodException | ClassNotFoundException | InstantiationException e) { } catch (InvocationTargetException | IllegalAccessException | NoSuchMethodException e) {
Ln.e("Could not invoke method", e); Ln.e("Could not invoke method", e);
throw e; throw e;
} }
@ -147,7 +139,7 @@ public class ContentProvider implements Closeable {
public String getValue(String table, String key) throws SettingsException { public String getValue(String table, String key) throws SettingsException {
String method = getGetMethod(table); String method = getGetMethod(table);
Bundle arg = new Bundle(); Bundle arg = new Bundle();
arg.putInt(CALL_METHOD_USER_KEY, ServiceManager.USER_ID); arg.putInt(CALL_METHOD_USER_KEY, Process.ROOT_UID);
try { try {
Bundle bundle = call(method, key, arg); Bundle bundle = call(method, key, arg);
if (bundle == null) { if (bundle == null) {
@ -163,7 +155,7 @@ public class ContentProvider implements Closeable {
public void putValue(String table, String key, String value) throws SettingsException { public void putValue(String table, String key, String value) throws SettingsException {
String method = getPutMethod(table); String method = getPutMethod(table);
Bundle arg = new Bundle(); Bundle arg = new Bundle();
arg.putInt(CALL_METHOD_USER_KEY, ServiceManager.USER_ID); arg.putInt(CALL_METHOD_USER_KEY, Process.ROOT_UID);
arg.putString(NAME_VALUE_TABLE_VALUE, value); arg.putString(NAME_VALUE_TABLE_VALUE, value);
try { try {
call(method, key, arg); call(method, key, arg);

View File

@ -10,9 +10,6 @@ import java.lang.reflect.Method;
@SuppressLint("PrivateApi,DiscouragedPrivateApi") @SuppressLint("PrivateApi,DiscouragedPrivateApi")
public final class ServiceManager { public final class ServiceManager {
public static final String PACKAGE_NAME = "com.android.shell";
public static final int USER_ID = 0;
private static final Method GET_SERVICE_METHOD; private static final Method GET_SERVICE_METHOD;
static { static {
try { try {