739 Commits

Author SHA1 Message Date
Josh Gao
3330d1208f statsd: fix double close.
Previously, fdopen was being called on a file descriptor that was owned
by a unique_fd without releasing. This leads to a double close, since
both fclose and the unique_fd destructor will try to close the fd.

Bug: http://b/113880863
Test: treehugger
Change-Id: I6f6f48d304861b5e4d7efee0d3ad0e30178a95a4
2018-09-04 11:12:04 -07:00
Yao Chen
38b92986df Merge "Disable statsd and make StatsLog no-op if ro.statsd.enable=false" into pi-dev 2018-06-07 21:10:16 +00:00
Yao Chen
cf3829a696 Disable statsd and make StatsLog no-op if ro.statsd.enable=false
Bug: 79324611
Test: manually tested on marlin_svelte
Change-Id: Ic7fbf86d7c295cb6b2f20e7f9a957d1ed74c42c0
2018-06-06 17:25:08 -07:00
Yangster-mac
1c58f04cd3 Add a field in config to disable/enable the string hashing in metric report.
Statsd hashes (using its own hashing function) raw strings to reduce the
upload data size when there are duplicate strings in the report. And in cloud,
the clearcut translator would backfill the strings.

In a few droidfood users, we find the translator was unable to do that. While
debugging the root cause, we first decided to provide an option to disable
the hashing from the cloud.

Test: statsd unit test, CTS test, tested manually

BUG: b/79943763
Change-Id: If0359c8cf3f3cf83a2938db9ebf95ea7906f0b0c
2018-05-18 10:39:50 -07:00
Chenjie Yu
021e25307d ValueMetric pushed events should check condition
+ fix unit test flakiness

Bug: 79873404
Change-Id: I15b52a79b18c05603640781e4450e7b62fac24ba
Fix: 79873404
Test: unit test
2018-05-16 14:50:11 -07:00
TreeHugger Robot
c9076fac9b Merge "Fixes Value metrics in statsd and app upgrades." into pi-dev 2018-05-16 02:27:37 +00:00
David Chen
092a5a9b85 Fixes Value metrics in statsd and app upgrades.
Pulled value metrics with conditions had a subtle bug that caused
us to leave the condition on even if it should've been false.

Bug: 79778783
Test: Added unit-test and verified on marlin-eng.
Change-Id: I31f34791118319b3471f7a6ea8a024e2d511cfe7
2018-05-15 17:51:47 -07:00
Chenjie Yu
47234644ca Configurable data error action in value metric
Right now in value metric, if a later pull produces a smaller number
than the previous one, we use absolute value of the current value.
This is not correct for some atoms as listed in the CL, which should
just take 0.
For some other atoms, this is unexpected error and should just dump
stale data.

Test: manual test
Bug: 79265262
Change-Id: I59fbfd96cbb57be22cd8d21cb57a7c60ca6856ee
2018-05-14 13:50:25 -07:00
TreeHugger Robot
67cee97754 Merge "Guardrail the max gauge atoms per dimension in one bucket." into pi-dev 2018-05-12 03:03:01 +00:00
Yangster-mac
50b0c9a608 Guardrail the max gauge atoms per dimension in one bucket.
Test: statsd
BUG: b/78603347
Change-Id: Ide77cc8ff0cde6cf6cc9e56034727ed27dd7be88
2018-05-11 17:15:44 -07:00
David Chen
56ae0d9a48 Fixes statsd reports missing strings and SCS.
Reports written to disk don't contain the strings used, which will
make this report unusable if there are strings that don't show up
again. We should always include the strings, so this option is
removed entirely.

Also, we hard-coded the wrong number of fields when pulling
ModemActivityInfo. There are actually 10 fields, not 6.

Bug: 79601503
Test: Tested unit-tests pass on marlin-eng.
Change-Id: I6834b096ced77418a9cc2ddd79b08d1c9c447fae
2018-05-11 17:04:56 -07:00
TreeHugger Robot
3b592910c5 Merge "Fix the gauge metric data size estimation." into pi-dev 2018-05-10 02:43:20 +00:00
Yang Lu
c2f44a19b2 Merge "Add a sampling method to pull gauge when condition changes to true." into pi-dev 2018-05-10 02:37:49 +00:00
TreeHugger Robot
3f57b16deb Merge "Skip writing metrics to disk if it's entirely composed of no_report_metric" into pi-dev 2018-05-10 02:02:37 +00:00
TreeHugger Robot
a159842161 Merge "Fix the flaky gauge/value e2e test due to cached events." into pi-dev 2018-05-10 01:02:10 +00:00
Yangster-mac
b2532daf4d Fix the gauge metric data size estimation.
Test: statsd test
BUG: b/77906846
Change-Id: Ie5ab54771acadde1e825289fe5ec491cc2225458
2018-05-09 23:14:26 +00:00
yro
028091cb15 Skip writing metrics to disk if it's entirely composed of
no_report_metric

Test: unit test, cts
Bug: 79488249
Change-Id: I3e13a6271cc36665a43d0f09d8663e5996224477
2018-05-09 16:03:27 -07:00
Yangster
ec3c7a3e58 Add a sampling method to pull gauge when condition changes to true.
Test: statsd test

BUG: b/79490890
Change-Id: I82ed6681151f8a217e74bd0d855f817b4fa0619c
2018-05-09 16:02:35 -07:00
android-build-team Robot
c3d0798455 Merge "Fix partial bucket unit tests." into pi-dev 2018-05-09 18:07:24 +00:00
Chenjie Yu
9743fae2d9 Merge "fix cts error on x86 simulator" into pi-dev 2018-05-09 00:53:09 +00:00
android-build-team Robot
0d3a6dee7f Merge "Fix statsd returning uidmap with empty reports." into pi-dev 2018-05-09 00:25:31 +00:00
Yangster-mac
58e609e339 Fix the flaky gauge/value e2e test due to cached events.
Test: statsd test
BUG: b/79265262
Change-Id: I4d67f1c2edb6215a3cea23f8c7b2e8d5099c4aac
2018-05-08 16:19:48 -07:00
Chenjie Yu
3b3adcd8b6 fix cts error on x86 simulator
Bug: 78242568
Fix: 78242568

Test: cts test
Change-Id: I235991c7c5b7c5a0d660ebaaff1356cc69fd1854
2018-05-08 15:40:01 -07:00
Yangster
613a7e2a7d Should not use the absolute value when the value does not change.
Test: statsd test

BUG: b/79265262
Change-Id: Ic654e1a07a505ec0e073a41fb41bcd38a29b125e
2018-05-08 15:12:30 -07:00
David Chen
9e6dbbdadf Fix statsd returning uidmap with empty reports.
We notice devices uploading a bunch of bytes for the uidmap even if
the device is running an empty config, so there are no actual metrics
to report. This hardcodes some logic to skip the inclusion of the
uidmap if there are exactly 0 metrics.

Bug: 79381210
Test: Tested unit-tests on marlin-eng
Change-Id: I96348235341a7faf15ff57d4d1eccac635a3a999
2018-05-07 18:07:19 -07:00
Yang Lu
cdb093e080 Merge "Fixes statsd returning too much data at once." into pi-dev 2018-05-07 23:49:54 +00:00
Yao Chen
cc884dfc94 Fix partial bucket unit tests.
Bug: 79347749
Test: statsd_test
Change-Id: I69eee7172d6fe4ce895530f089193eb08653e269
2018-05-07 10:34:31 -07:00
yro
be6d7f90e8 Re-enable the logging of APP_BREADCRUMB_REPORTED atom
Bug: 78613419
Test: manual, cts, unit tests
Change-Id: I279158c8031eda3ee648053ae6a0d13fde7f1176
2018-05-06 22:25:05 -07:00
David Chen
48944901f7 Fixes statsd returning too much data at once.
We observe a single ConfigMetricsReportList can be greater than the
safe size for the binder transaction buffer since we only check the
size of the current metrics in progress, but we also return the
previous reports stored on disk.

This change will attempt to send another ConfigMetricsReportList
as soon as possible if there's already a report on disk.

Also fixes a bug when trying to trigger data fetch before the client
has registered the corresponding dataFetchOperation.

Bug: 79201869
Test: Tested manually on marlin-eng
Change-Id: I2d3677162804a27e7a7a95d482d80c46bd994a67
2018-05-04 17:09:16 -07:00
Yao Chen
a62ae51ba9 Merge "Add cmd to let statsd print all logs it received for debugging." into pi-dev 2018-05-04 20:21:18 +00:00
android-build-team Robot
ec41a069fd Merge "Reset statsd and correctly record the dump reason when system server restarts/crashes." into pi-dev 2018-05-04 02:43:03 +00:00
Yangster-mac
892f3d3229 Reset statsd and correctly record the dump reason when system
server restarts/crashes.

Test: statsd test
BUG: b/79161505
Change-Id: I0646c764964f6eafde91f9ae0179a1c837af320d
2018-05-03 17:05:24 -07:00
Yangster-mac
754e29edd7 Turns DEBUG to false in statsd.
Test: statsd test
BUG: b/79161505
Change-Id: Ic6eee527d625b10aa86b2beb4b4c4fc05b051c7d
2018-05-03 21:19:39 +00:00
Yao Chen
876889cb76 Add cmd to let statsd print all logs it received for debugging.
It only works on eng build. And all code is behind a build flag, so the
code will be stripped out in production builds.

Bug: 78239479
Test: manual
Change-Id: I20ee51822d18e6c77ca324a5327712cbed09593e
2018-05-03 10:53:27 -07:00
Yang Lu
493bb2b119 Merge "Lock the pulling alarm handler." into pi-dev 2018-05-02 18:25:18 +00:00
android-build-team Robot
b85e5616df Merge "Reduce statsd log data size." into pi-dev 2018-05-02 00:45:51 +00:00
android-build-team Robot
e442db3e40 Merge "Increase the bytes per config threshold to avoid too frequent uploading." into pi-dev 2018-05-01 21:02:32 +00:00
Yangster-mac
9def8e3995 Reduce statsd log data size.
1. Hash the strings in metric dimensions.
2. Optimize the timestamp encoding in bucket.
   Use bucket num for full bucket and millis for
   partial bucket.
3. Encode the dimension path per metric and avoid
   deduping it across dimensons.

Test: statsd test
Change-Id: I18f69654de85edb21a9c835c73edead756295e05
BUG: b/77813755
2018-04-26 04:30:18 -07:00
Yangster-mac
a802d73da6 Clear the past buckets for no-report metrics
Test: statsd tests.
BUG: b/78503732
Change-Id: I3de855929ad05677769cffaae4fea2db0d2f32da
2018-04-24 08:07:20 -07:00
Yao Chen
49954cd681 Add socket listener to statsd. And remove Davey atom.
+ The socket listener is behind a flag. It's disabled until we get sepolicy changes in.
+ Data parsing code is from logd, because we use the same format.

+ Removed Davey from JankTracker because it violates our new sepolicy

Test: manually
Bug: 78239479

Change-Id: Ib17729fbc362cdb13385f780e2d636a95adf9bc3
2018-04-20 17:41:25 -07:00
Yao Chen
f7bc6ab7fa Allow StatsLog to directly write to statsd's socket.
+ Reuse the log_event_list from liblog. StatsLog's binary format remains unchanged
+ Copied socket write code from liblog, including the retry logic.
+ Added build flags to control the StatsLog channel (logd, statsd, or both for debugging)

Bug: 78239479
Test: locally tested and saw logs being written to statsd

Change-Id: I7b1f0069ead00bbf3c29e4bd5b7f363a7ce26abe
2018-04-20 14:52:01 -07:00
Yangster-mac
db185c94cc Increase the bytes per config threshold to avoid
too frequent uploading.

BUG: b/77813755
Test: statsd test
Change-Id: I2b7b9292edf2c7d607f1ae6af4d52b104cd8443a
2018-04-19 11:31:03 -07:00
TreeHugger Robot
a194a6bee3 Merge "add dump report reason to reports" into pi-dev 2018-04-18 20:35:29 +00:00
Chenjie Yu
e36018b272 add dump report reason to reports
+ also change uidmapping version numbers to int64_t

Bug: 78132855
Change-Id: Iac7ea93e4bf651bd65bd03383e7ab4971af4fc29
Fix: 78132855
Test: gts test
2018-04-18 20:19:21 +00:00
TreeHugger Robot
2abcea2b8a Merge "Fix merge isolated uid to host uid" into pi-dev 2018-04-18 17:07:13 +00:00
Chenjie Yu
728d52a6db Fix merge isolated uid to host uid
Bug: 78190783
Fix: 78190783
Test: unit test
Change-Id: I71b5c575bf3c226509ebf088cb5dd961acd93459
2018-04-17 15:24:52 -07:00
TreeHugger Robot
bbc057444f Merge "Adds option to drop small buckets for statsd." into pi-dev 2018-04-17 19:12:33 +00:00
David Chen
81245fd53a Adds option to drop small buckets for statsd.
We notice that some of the pulled metrics have a ton of data, and
during app upgrades, we're forming partial buckets that represent
small periods of time but require many bytes of data. We now have an
option to drop these buckets that are too short. Note that we still
have to pull the data to keep the metrics for the next bucket
correct. We include a new field in the value and gauge metric outputs
so that it's easy to tell when a bucket was dropped.

We drop the partial buckets also from anomaly detection since we
should be computing anomalies from the same data that is reported.

Test: Added unit-tests for value and gauge metrics.
Bug: 77925710
Change-Id: Ic370496377c6afd380e02278a6c1ed8b521a2731
2018-04-16 18:42:14 -07:00
Tej Singh
abdf6e8039 Merge "Enforce dump permission in stats service" into pi-dev 2018-04-17 01:34:47 +00:00
Jeff Sharkey
936c0868f8 Merge "Protect usage data with OP_GET_USAGE_STATS." into pi-dev 2018-04-17 00:04:53 +00:00