62 Commits

Author SHA1 Message Date
TreeHugger Robot
ab34199e9c Merge "Found out it is not possible to exec procrank as non-root/shell. But it is ok to disable procrank since the information can be obtain via dumpsys meminfo" into pi-dev 2018-03-28 19:37:46 +00:00
Yi Jin
c36e91dd12 Found out it is not possible to exec procrank as non-root/shell.
But it is ok to disable procrank since the information can be obtain via
dumpsys meminfo

Optimize the code style a bit.

Bug: 72510307
Test: atest incidentd_test
Change-Id: If06201aa708461b569d9fb007f20d1e51342d5c2
2018-03-27 15:01:20 -07:00
Yi Jin
a9635e442b Fix apct test.
Bug: 76208934
Test: atest incidentd_test
Change-Id: Ibc101aa3c523398b1e9dd6cff3d6735b4d704667
2018-03-23 12:05:32 -07:00
Yi Jin
83fb1d58b5 Fix the test failure of LogSection, don't consider read log msg error as
a failure since it is used to signal EOF.

Additionally tag this message as auto so user_id won't become explicit.

Bug: 75017304
Test: atest incidentd_test
Change-Id: I151bab5a72a532e7c9f54ae0686561001730bdeb
2018-03-16 16:03:18 -07:00
Yi Jin
6355d2f3ab Wrap fd with unique_fd so it won't leak.
Bug: 74021345
Test: manual and atest incidentd_test
Change-Id: Ib1000bfe6917c3d5cae7b9edce5b67d50897e10d
2018-03-14 16:26:28 -07:00
Yi Jin
86dce413f8 Optimize incidentd memory usage
1. Remove dependency of libprotobuf-cpp-lite, saves .so mmap ~200KB
2. Don't use auto except iterator for readability.

Bug: 74254200
Test: adb shell dumpsys meminfo `pid incidentd`
Change-Id: If6198521c3b80929d6ea3f7ed466b5195991ccfd
2018-03-07 18:25:44 -08:00
Kweku Adams
eadd123d68 Dumping stack traces to proto.
Bug: 72177715
Test: flash device and check incident.proto output
Change-Id: Id2a15e0fc62b66efe875949af97f0eb651c7e322
(cherry picked from commit 5c804e2b9893c7892900148229cf19fa6268e7dc)
2018-03-07 21:50:12 +00:00
Yi Jin
5ee0787024 Use uint64_t instead of long long as API type for consistent reason.
Bug: 74118023
Test: manual
Change-Id: Icd5f506c76d3a008a79cb6c9d2061962ca7fdd40
2018-03-05 18:18:27 -08:00
Yi Jin
06ebd1af8e Fix heap buffer overflow
Bug: 74000767
Test: manual
Change-Id: Id57674c0ae527da055a06acf4f458c440328c5c3
2018-02-28 11:25:58 -08:00
TreeHugger Robot
16b989fc0e Merge "Add incidentd to readproc group." 2018-02-28 04:13:20 +00:00
Kweku Adams
391284396b Add incidentd to readproc group.
This allows it to get cpu info and ps data.

Bug: 72384374
Test: flash device and check incident.proto output
Change-Id: I09f6318861fbedbf4fae1a4325e6a7d12b32b10e
2018-02-27 12:26:13 -08:00
Yi Jin
1a11fa1097 Implement a new section to attach LAST_KMSG to incident report
This section simply gzip a large file and stores result in GZippedFileProto

This greatly improves the size, before gzip, the last kmsg size ~500KB,
after gzip the proto size is ~60KB.

Bug: 73354384
Test: atest incidentd_test and manual on device test
Change-Id: I9bfc2cf07384487671edbffb5f0bd8495608fea6
2018-02-27 11:36:29 -08:00
Yi Jin
4e843106a6 Throttler for incidentd based on size putting into dropbox.
The incidentd will accumulate the total size put into dropbox and once
it exceeds a threshold (currently 20MB) daily, it will stop further
requests. It allows collection again 24 hours later.

Bug: 64219725
Test: atest incidentd_test and manually flashed incidentd and test.
Change-Id: Iea21fbae40d5d01108797b190231d73e74eff213
2018-02-20 13:11:22 -08:00
Yi Jin
b592e3bc31 This cl formats incidentd and makes it easier for debugging.
Bug: 72755317
Test: clang-format -type=file -i <files>
Change-Id: Ide91227f26c6b1db6d2e5fe8117ca5cc4cf77fd3
2018-02-15 16:53:22 -08:00
TreeHugger Robot
9ed8ec9b49 Merge "Put metadata or stats into each dropbox incident report." 2018-02-15 20:42:00 +00:00
Yi Jin
329130b7e4 Put metadata or stats into each dropbox incident report.
Bug: 65451198
Test: atest incidentd_test
Change-Id: Ib406b177ad7f1b4bda7fef2e606fc66a9836e060
2018-02-14 16:12:57 -08:00
Yi Jin
4bad04eda4 Add owners file for statsd/incidentd
Test: N/A
Change-Id: I5536a239e0fefbc05a0db3b88de6d4a6ccddfdee
2018-02-14 15:47:39 -08:00
Yi Jin
7f9e63b632 Add clear method to ProtoOutputStream so it is reusable by just
rewinding its internal pointer.

Bug: 72319159
Test: atest incidentd_test and atest statsd_test
Change-Id: Id046b987a7460d68300e5d4d46ae56aec1eba2d0
2018-02-09 14:20:48 -08:00
Yi Jin
afb36062d5 Allow statsd_config be able to set destination value AUTO/EXPLICIT.
Also move incidentd integration to its own files.
clang-format -style=file -i src/anomaly/* and src/subscribers/Incidentd*

Bug: 70239380
Test: manual
Change-Id: I7bfe14d704d9e86d925365a8a21ffed726723e60
2018-02-06 11:15:09 -08:00
Yi Jin
3ec5cc792e Modify SystemApi so it can be used by CTS to trigger incident report
Bug: 72502621
Test: Cts/Gts tests covered, see the cls from the same topic
Change-Id: Id0c1cc0fc0054e620de1349dab66513e554b1caa
2018-01-29 21:01:09 -08:00
Yi Jin
b8344dc72f Fix command line tool to set the right privacy enum.
Test: manual
Change-Id: I3f0845ea32b46f1a028f8b6d3d4180a90a47beb5
2018-01-24 18:25:33 -08:00
Yi Jin
974e56f141 Enable incidentd.rc also add dump and usage_stats permission to statsd
so it bypass incidentd's permission check

Test: manual
Change-Id: I65b501fe46f66f9f62fedfcfc75aa17f29fc1076
2018-01-22 14:41:19 -08:00
Yi Jin
437aa6e8ad Add more information to incident header. Especially add config keys
to check if the report is uploadable.

Move incidentheader.proto to libincident so statds is able to include a lite
proto class for incident header.

Change IncidentReportArgs to add the proto object instead of serialized
bytes to prevent caller gives meaningless data.

Bug: 70241842
Test: push config to statsd and verify incidentd generate the report
with correct header.

Change-Id: If95b655be71047b019b229e5903a08f3c21a1f29
2018-01-21 22:36:01 -08:00
Yi Jin
4bab3a191a Fix permissions problems of incidentd.
Test: manual
Change-Id: I4ee0d1f2349ee1a25a422cabf1b5b87c612710d2
2018-01-17 19:16:49 -08:00
Yi Jin
30789db28c make incidentd run in its own uid under log group for reading logs.
Test: manual
Change-Id: I7106614c0bed1e862fbc80e42a87b43efc063145
2018-01-10 13:20:14 -08:00
TreeHugger Robot
c99dea0a4b Merge "Revert "Enable incidentd, it will just be idle while no one triggers it."" 2018-01-09 03:28:11 +00:00
Yi Jin
44ed76eb9f Revert "Enable incidentd, it will just be idle while no one triggers it."
This reverts commit d612df381e7867780266f0a1161b817e089c3bb6.

Reason for revert: Selinux denials, see b/71591707 for more details.

Change-Id: I864cd74e7ba57d67e2ed8328a7df9c5ff335508c
2018-01-09 01:44:47 +00:00
Yi Jin
3c034c987e Implement a new type of section which reads from logd and dumps proto.
And implement a file section which reads from event-log-tags for
decoding binary logs.

Bug: 70936599
Test: atest incidentd_test && atest incident_helper_test and flush on
device and test log sections and event_log_tag_map
Change-Id: Ib3d35e317f355de69f01ded012482486e9a43da6
2018-01-06 20:33:14 -08:00
TreeHugger Robot
e85fac24db Merge "Enable incidentd, it will just be idle while no one triggers it." 2018-01-04 01:40:41 +00:00
Yi Jin
d612df381e Enable incidentd, it will just be idle while no one triggers it.
Test: N/A
Change-Id: I169108f0d61fc1fffea629eb2e862bfa7aad65f9
2018-01-03 15:22:18 -08:00
Kweku Adams
f5cc5759d5 incidentd: parsing ps dump into proto.
Also changing from execv to execvp so that we don't have to specify the full command path.

Bug: 65750831
Test: atest incident_helper_test
Change-Id: I92191afff4e7f9a6d08ea22ecfc2de5623d3bde5
2018-01-02 15:49:23 -08:00
Yi Jin
5e4ce2c941 Add AndroidTest.xml to incident_helper and incidentd
This change depends on fix of b/69802886:
https://android-review.googlesource.com/#/c/platform/build/+/555203/

Bug: 65754586
Test: atest incident_helper_test && atest incidentd_test
Change-Id: I64d793c8abe4381114e5b38fa12d085b1d808cd1
2017-12-07 18:06:47 +00:00
Yi Jin
bdf58942c8 1. Implement Privacy Policy Tag Inheritance
2. Expose Dest values in IIncidentReportArgs.h

Please see bug for detail descriptions.

Bug: 67869340
Test: unit tested and on-device test. Ran command:
$ out/host/linux-x86/bin/incident-section-gen incidentd
to test various of privacy taggings.
Change-Id: I12af3d7da4c7c72bd63598893fe2c1f70b09baa4
2017-12-01 13:37:13 -08:00
Yi Jin
0eb223496c Implement Cpu Freq Section
When poll from sysfs, revents return POLLERR by default, handles
this edge case in this cl.

Bug: 68774444
Test: unit tested + on device tests
Change-Id: I23540299c026d3e7676497f56690e9f8646a47bd
2017-11-14 13:22:25 -08:00
Wei Wang
254102d858 Remove unnecessary parameter WriteStringToFile
By default, WriteStringToFile will not follow link, the 'false'
parameter is unnecessary.

Test: build
Change-Id: Ied6dcbf2a85533108dc2f6d71a4571b44684e800
2017-11-10 17:32:25 +00:00
Yi Jin
be6de3017b Enable Window Dumpsys Section
Protobuf defines classes, which can have self recursive message
definitions and cause a bug in generating privacy flags.
Solve the problem here. The details is in incident_section_gen/main.cpp.
The logic is a bit complicated to address more than one level of self
recursion proto message definition.

Also solve a bug when PrivacyBuffer strips fields.
Modify PRIVACY_POLICY_LIST to be type Privacy** in order to allow
initialization by a method.

Bug: 68162512
Test: unit tested and on device tests
Change-Id: I1d0b79f6813e5fd66c4cf5823d0fa17efc57bb1d
2017-11-01 18:24:57 -07:00
Yi Jin
e08333067a Fix several nits
1. Change the APIs of ProtoOutputStream to be able to write bytes
2. Fix the tests in incidentd, stdout is closed so can't capture the
value, use temporaryFile instead.

Test: N/A
Change-Id: Ibc31f2efd068afc6c06188d92f57ca5a754c3683
2017-10-23 16:36:06 -07:00
Yi Jin
22769e0123 Fixed several bugs found for incidentd
1. Add logging for proto can't be parsed by incident_report
2. Close opened file descriptor of incident report request!
3. Fix invalid syntax of auto-gen section_list.cpp

Bug: 67849582
Test: manually run incident and checks /proc/<its pid>/fd
Change-Id: I38e6ac28de09aca6243ad536ed41d8059e08ba24
2017-10-18 13:17:32 -07:00
Yi Jin
42711a0b4f Use ProtoOutputStream to strip Pii sensetive fields in incidentd
The in-place algorithm in ProtoOutputStream will have at most 2 copies
of possible nested message values incidentd is getting which makes it
more ram-efficient, and minimize allocations.

Bug: 65641021
Test: the units pass
Change-Id: Ic3fe99b7e27895bd613a129ba0f12ccfca4af317
2017-10-13 15:44:26 -07:00
Yi Jin
974a9c2885 Implement go/streaming-proto native libs, part 2
Implement ProtoOutputStream.h to write protobuf format data.

Usage of dumping proto:
ProtoOutputStream proto(fd);

proto.write(fieldId, value1); // dump a single value

// start to dump a message:
long long token = proto.start(messageFieldId);
proto.write(nestedField1, nestedValue1);
...
proto.end(token);

fieldId will be generated by protoc-cpp plugin(TBD). It is an encoded uint64_t
value, with 0 - 32 bits as its proto number, 33 - 40 bits reserved for
field type, int32, bool, string, message, etc. and 41 - 43 bits for
single, repeated or packed type. Currently packed field is not
supported.

Bug: 65641021
Test: N/A, need to wait for protoc-cpp plugin and will test in
incident_helper
Change-Id: Ic188615b950235aae0edeee4876b78d31feb5619
2017-10-10 17:16:34 -07:00
Yi Jin
667ded2fc9 Merge "Implement c++ native lib for streaming proto, part 1" 2017-09-30 00:08:44 +00:00
Yi Jin
a5c5e8a99b Enable Diskstats Section
Bug: 67019205
Test: manually tested in incident_report tool
Change-Id: I291d4b0b0c01385e57c6ad83a12678c0f8d8746b
2017-09-28 14:09:02 -07:00
Yi Jin
c23fad2f90 Implement c++ native lib for streaming proto, part 1
Extract protobuf class out and creates EncodedBuffer class
which holds protobuf data.
Next step is to create a ProtoOutputStream and let incident helper
adapt the change as well.
please see frameworks/base/core/java/android/util/proto

Bug: 65641021
Test: unit tested
Change-Id: I0dd343b2e62d60f091c8f857fae3452ec8da6b96
2017-09-27 12:23:51 -07:00
George Burgess IV
bbe04e87e9 incidentd: fix memory leaks in a test
The static analyzer found many memory leaks in this file. Converting
everything to use unique_ptr became really ugly, and involved lots of
copying from lists of unique_ptr<Privacy> to lists of Privacy*. Since
these tests really don't seem to allocate much, freeing everything en
masse at the end should be good enough.

Bug: 27101951
Test: mma; memory leak warnings are gone. Running
/data/nativetest64/incidentd_test/incidentd_test shows no failures.
Change-Id: I151b7ef4e02ee736e4e833924c0500e8dbedcc49
2017-09-19 16:42:01 -07:00
Max Bires
2eba753d38 Merge "Adding extra rc commands to fix incidentd with respect to selinux" 2017-09-15 15:56:11 +00:00
Max Bires
e6b224920e Adding extra rc commands to fix incidentd with respect to selinux
Incidentd has sepolicy rules to allow it to write to
/data/misc/incidents/ but it needs the directory to already be created.
This modification will set the service to be UID/GID system and create
the directory under system permissions.

Test: no selinux denials related to incidentd on bootup
Bug: 64275854
Change-Id: I0f4db90f4eeb6dae5019e5bc1df590bd781a9507
2017-09-14 23:26:55 +00:00
Yi Jin
7e0b4e5880 Fixes regarding to comments in ag/2839267
1. use struct {} to instantiate privacy fields for efficiency reason
2. use vector<uint8_t>* instead of vector<uint8_t>& to indicate the
caller knows the value gets changed.
3. binary search privay policy for sections

Bug: 65595927
Test: unit test covers
Change-Id: Ic58c2f607465d1a7f10352b9a38c3d8b1a5cf352
2017-09-13 14:32:42 -07:00
Yi Jin
edfd5bb788 Extract IncidentHeaderProto to a separated file for statsd to consume
Create a HeaderSection to deal with header protos which is more testable

Bug: 65422268
Test: unit tested
Change-Id: Icdcbeded8bc963940a8f9e503cb65a9a247ca5b2
2017-09-07 17:33:07 -07:00
Yi Jin
0f0471623e Implement Pii Stripper Part 3
The incident request args sets privacy spec. Strip action is optimized
to run once for each type of spec and ready for flush multiple times.
Incident command is updated to take -p option to specify privacy spec.

Bug: 64687253
Test: unit tests written, manually run incident command to test as well
Change-Id: I6753df117f76dc1a5f4d2152baa3fbbf56b490e4
2017-09-07 10:53:51 -07:00
Yi Jin
99c248feb2 Implement PII Stripper, part 2
Implement EncodedBuffer that strip pii based on given privacy request.
The reason to implement another buffer is the length-delimited field's
size could change when its submessage gets stripped. It also intends to
keep the orignal data around for other requests to consume it.

In addition, the section implementation has adapted EncodedBuffer so
write out to each request's fd could be request-specific. The next step
is allow requests to set its privacy spec.

Notice the current design set the privacy spec of dropbox to AUTOMATIC,
this behavior might change in the future.

Bug: 64687253
Test: unit tests are writtern, see README.md for how to run unit tests.
Change-Id: I7ac236b8265ba9289dc6e17a8a5bf7f67ffb6bf5
2017-09-07 10:53:51 -07:00