128 Commits

Author SHA1 Message Date
Yao Chen
876889cb76 Add cmd to let statsd print all logs it received for debugging.
It only works on eng build. And all code is behind a build flag, so the
code will be stripped out in production builds.

Bug: 78239479
Test: manual
Change-Id: I20ee51822d18e6c77ca324a5327712cbed09593e
2018-05-03 10:53:27 -07:00
Yangster-mac
9def8e3995 Reduce statsd log data size.
1. Hash the strings in metric dimensions.
2. Optimize the timestamp encoding in bucket.
   Use bucket num for full bucket and millis for
   partial bucket.
3. Encode the dimension path per metric and avoid
   deduping it across dimensons.

Test: statsd test
Change-Id: I18f69654de85edb21a9c835c73edead756295e05
BUG: b/77813755
2018-04-26 04:30:18 -07:00
Yao Chen
49954cd681 Add socket listener to statsd. And remove Davey atom.
+ The socket listener is behind a flag. It's disabled until we get sepolicy changes in.
+ Data parsing code is from logd, because we use the same format.

+ Removed Davey from JankTracker because it violates our new sepolicy

Test: manually
Bug: 78239479

Change-Id: Ib17729fbc362cdb13385f780e2d636a95adf9bc3
2018-04-20 17:41:25 -07:00
Yangster-mac
15f6bbc24f Flush the bucket when creating the metric producer.
Use int64 for value field.
E2e test for gauge/value metric.

BUG: b/74445671

Test: statsd test.
Change-Id: I823a0bade8f89834bdfb9cf48864852a47d7b63b
2018-04-10 20:25:13 -07:00
David Chen
bd12527c90 Fix uid map to be simpler and fix partial bucket.
The previous scheme captured periodic snapshots for each config with
complex logic that's unnecessary and wasted memory. We actually don't
need to store any snapshots since we just convert the current state
into a snapshot and also include the deltas (change events) since the
previous report until now.

To make the system more robust, we also include up to 100 of the
deleted apps in the uid map.

Also, fix the wiring of the partial buckets so the metric producers
form partial buckets on both app upgrade and removal, but not on
installation of a new app.

Also, we update StatsCompanionService to also include disabled apps.

Bug: 77607583
Test: Verified unit-tests pass and added new e2e tests.
Change-Id: I98e1f544d6e6571545ae1581c4cebab807596f51
2018-04-05 16:15:01 -07:00
Yangster-mac
b142cc8add Statsd config TTL
Roughly check the config every hour to see whether the ttl expired.
If so, read the config from disk and recreate the metric manager.

Test: statsd test

BUG: b/77274363

Change-Id: I16838afe5bbe966c3a0f638869751f9b59a5a259
2018-04-04 15:59:43 +00:00
Yangster-mac
684d195227 E2e test for periodic alarm.
Test: new test

BUG: b/76281156
Change-Id: I60cb28baaeec6996e946a7cb3358ec8e0aca80e5
2018-03-27 13:26:20 -07:00
TreeHugger Robot
cdc9d9008b Merge "Flush the past buckets in anomaly tracker when time jumps forward" into pi-dev 2018-03-27 07:21:21 +00:00
Yangster-mac
be10ddfe46 Flush the past buckets in anomaly tracker when time jumps forward
E2e test for count/duration anomaly trackers.

Test: new statsd tests.

BUG: b/74446029

Change-Id: Ia9be0240ba5021d44c1e1f096d67563e9138bb59
2018-03-26 18:44:01 -07:00
TreeHugger Robot
c8ab224b2a Merge "Fix StatsCompanionService sometimes can be null" into pi-dev 2018-03-24 16:28:01 +00:00
Chenjie Yu
aa5b201015 Fix StatsCompanionService sometimes can be null
Bug: 75970648
Test: will add cts test for puller alarms
Change-Id: I51b7d13f855d3c8ded8325d7cf0f614531eceea5
2018-03-23 14:33:58 -07:00
Yao Chen
461a75f269 Remove unsigned-integer-overflow signed-integer-overflow from sanitizer.
Sanitizer is picking on the protobuf generated code. Need to figure out why
Disable them to unblock development first.

Bug: 76117312
Test: manual
Change-Id: I3f3a39dfe933f4c1b1cf4063d621392b66f0adea
2018-03-21 16:38:26 -07:00
TreeHugger Robot
dc5677336c Merge "Enable sanitizer on statsd eng build." into pi-dev 2018-03-21 20:09:10 +00:00
Yao Chen
cd7f90053d Enable sanitizer on statsd eng build.
Bug: 74946342

Test: manual
Change-Id: I287def435d077670814db50e4d8398ecee539b74
2018-03-21 11:04:37 -07:00
David Chen
9fdd40302e Allow statsd to be given empty config.
Statsd clients may want to set an empty config temporarily, so it's
more convenient to allow them to set an empty config instead of
having to use the removeConfig and then having to remember to call
StatsManager#setDataFetchOperation.

Test: Added unit-tests and check they pass on marlin-eng.
Bug: 74997752
Change-Id: I2e762e5ec01e5a2c9a3469fb330b53fefbd734d6
2018-03-20 15:56:11 -07:00
David Chen
f384b90049 Removes stats_log proto from uid map in statsd.
We don't need to parse the proto of uid map, so we use the
ProtoOutputStreame class to generate the binary form of the proto
output that's needed for parsing the uid map data.

Test: Verified unit-tests still pass.
Bug: 74010813
Change-Id: Ia2f7572f3b78bb6f7b60e8b14cf5d65428469ab6
2018-03-15 13:33:04 -07:00
TreeHugger Robot
c7ce1a0978 Merge "Add wall clock timestamp for ConfigMetricsReport and gauge atoms." into pi-dev 2018-03-14 19:03:06 +00:00
Yangster-mac
3fa5d7fb23 Add wall clock timestamp for ConfigMetricsReport and gauge atoms.
Fix the bug when serializing multiple atoms in gauge metric

BUG: b/74159560

Test: new test for ALL_CONDITION_CHANGES sampling method.
Change-Id: I6d33c1efbac92b6e13be2d64c323e090cb1f84aa
2018-03-10 22:25:28 -08:00
Yao Chen
b19425e011 Remove unused shared lib from statsd
Test: build statsd, and statsd_test

Bug: 72129300
Change-Id: I0ebff977dabe796799a0d41c64adb2c2f2e9035e
2018-03-08 14:38:12 -08:00
Yangster
13fb7e4eea Statsd cpu optimizaton
- cache the changed dimensions in condition tracker.
- avoid query condition wizard when unnecessary.
- avoid copy dimension keys in condition key generation.

Test: statsd tests.

BUG: b/73959649
Change-Id: I17d68e2a82643de3f421309841e75f84c6fd8f43
2018-03-07 17:36:07 -08:00
Chenjie Yu
ec67661bdf Refactor cpu stats pullers
use same cpu stats readers with BatteryStats so that
1) both throttle to avoid too frequent pulls
2) cached value is served within throttle interval to avoid double
pulling by both statsd and BatteryStats

To run unit tests:
bit
FrameworksCoreTests:com.android.internal.os.KernelUidCpuFreqTimeReaderTest
bit
FrameworksCoreTests:com.android.internal.os.KernelUidCpuClusterTimeReaderTest
bit
FrameworksCoreTests:com.android.internal.os.KernelUidCpuActiveTimeReaderTest

make -j56 statsd_test && adb sync data && adb shell
/data/nativetest64/statsd_test/statsd_test

Test: cts test, unit test
Bug: 73745189
Bug: 73780619
Bug: 73360959

Merged-In: I10a9bc91ca67fa812f4cd71c4fbd73c1a5ba580e

Change-Id: I10a9bc91ca67fa812f4cd71c4fbd73c1a5ba580e
2018-03-07 21:43:00 +00:00
TreeHugger Robot
644e15d5bd Merge "Reduce statsd binary size from 730KB-> 664KB" 2018-03-02 19:41:56 +00:00
Yao Chen
580ea321b1 Add StateTracker.
StateTracker is a special condition tracker that's based on a state atom.
State atoms are annotated in atoms.proto.

The rules for StateTracker:
 1. must not have "stop". must have "dimension"
 2. must be based on a state atom.
 3. it must have the all primary fields and the exclusive state field in its dimension.

 For example UidProcessStateTracker, will have output dimension {uid, state}.

Test: unit tests added.
Change-Id: I6b77e58e9fabe61f7326daf929577d8b2cfbf27b
2018-02-28 16:57:03 -08:00
Yao Chen
20e9e6231a Reduce statsd binary size from 730KB-> 664KB
1. StatsdStats does not use the proto object in memory anymore.
2. lite_static -> lite
3. don't use sstream

Bug: 72129300
Test: statsd_test

Change-Id: I8a5adaf222d4d5034e8bf115215fb6dd5f042cac
2018-02-28 16:20:54 -08:00
TreeHugger Robot
03b91d77c4 Merge "Alarm: wakes up statsd and notifies the subscribers." 2018-02-27 23:08:31 +00:00
Yangster-mac
932ececa16 Alarm: wakes up statsd and notifies the subscribers.
Test: manually tested it.
Change-Id: Id796a68976aeb1611183023ba4e9c6a8b8c44bb8
2018-02-27 13:30:48 -08:00
TreeHugger Robot
dcd6ae8bc8 Merge "Atom: TemperatureReported" 2018-02-27 07:00:45 +00:00
Tej Singh
4029831e36 Atom: TemperatureReported
Makes the temperature reported atom pulled, and adds CPU, GPU, and SKIN
temperatures. Pulls information from the thermal hal.

Test: CTS test on cl in this topic
Change-Id: I0a8e2d1135bdd77e1cc510f24ff5214ce9e14ead
2018-02-26 20:55:26 -08:00
TreeHugger Robot
3b3d38d996 Merge "Avoid creating temporary objects in FiterValue()." 2018-02-24 10:13:19 +00:00
Yangster-mac
f520492279 Avoid creating temporary objects in FiterValue().
It reduces the cpu time from 1000ns to 750ns

Test: statsd test.
Change-Id: Ifa7e98e3368f8d55f85c7b09d05a6c416482981d
2018-02-23 20:53:23 -08:00
Yao Chen
9c1debe330 Add annotation to atoms that represent a state change in atoms.proto
+ A state change atom can have one exclusive state field, and any
  number of primary key fields.

  When there is primary key in the atom, it means the state belongs to the primary key.
  For example,
  message UidProcessStateChanged {
    optional int32 uid = 1 [(stateFieldOption).option = PRIMARY];
    optional android.app.ProcessStateEnum state = 2 [(stateFieldOption).option = EXCLUSIVE];
  }

  When there is no primary key fields in the atom, the state is global.
  For example,
  message ScreenStateChanged {
     optional android.view.DisplayStateEnum state = 1 [(stateFieldOption).option = EXCLUSIVE];
  }

+ The annotation is consumed by stats_log_api_gen to generate a static map from the state
  atoms to its primary fields, and exclusive fields

+ stats_log.proto is splitted into 2 proto files, because statsd needs proto lite, and c++
  lite proto library cannot properly ignore the field options which requires full proto.

 This CL doesn't change any logic in the statsd yet. A separate CL will use the field option
 information to correctly track the state.

Test: added unit tests in stats_log_api_gen_test. and statsd_test pases.
Change-Id: I9e8a979fe81ba60efd4d854bb7087ce4b2b147ec
2018-02-21 16:46:56 -08:00
Yao Chen
4c959cb99e Remove unused variables in statsd, and make more warnings show.
Test: statsd_test

Change-Id: I2c7b674cb615f22c5de90c2de5f2d58108ab2e7f
2018-02-13 15:31:22 -08:00
TreeHugger Robot
e8cf6110ae Merge "Statsd CPU optimization." 2018-02-12 19:53:56 +00:00
Yao Chen
8a8d16ceea Statsd CPU optimization.
The key change is to revamp how we parse/store/match a log event, especially how we match repeated
field and attribution nodes, and how we construct dimensions and compare them.

+ We use a integer to encode the field of a log element. And also encode the FieldMatcher into an
integer and a bit mask. The log matching becomes 2 integer operations.

+ Dimension is stored as encoded field and value pair. Checking if 2 dimensions are equal is then
  becoming checking if the underlying integers are equal. The integers are stored contiguously
  in memory, so it's much faster than previous tree structure.

Start review from FieldValue.h

Test: statsd_test + new unit tests

Bug: 72659059

Change-Id: Iec8daeacdd3f39ab297c10ab9cd7b710a9c42e86
2018-02-12 10:38:45 -08:00
TreeHugger Robot
8a1bacaf91 Merge "Benchmark: stats_write" 2018-02-10 03:33:20 +00:00
Chenjie Yu
80f9112aec Pullers merge isolated process with host
Test: unit test
Change-Id: Idcb4d19e1f5182554b503c5a35fbde4da146835c
Fix: 72230210
2018-02-07 22:16:50 +00:00
Yi Jin
afb36062d5 Allow statsd_config be able to set destination value AUTO/EXPLICIT.
Also move incidentd integration to its own files.
clang-format -style=file -i src/anomaly/* and src/subscribers/Incidentd*

Bug: 70239380
Test: manual
Change-Id: I7bfe14d704d9e86d925365a8a21ffed726723e60
2018-02-06 11:15:09 -08:00
Tej Singh
3410634f2c Benchmark: stats_write
benchmarks how long stats_write takes for boot sequence atom
results: I ran it a few times, and the times ranged from ~18-20us on
marlin

Test: ran the benchmark test
Change-Id: I900ef26ce219301a6d43999fe7be5e4875ae5b8a
2018-02-05 11:27:23 -08:00
TreeHugger Robot
ad73f41329 Merge "Add command to clear puller cache" 2018-02-01 23:58:36 +00:00
TreeHugger Robot
4d41f29127 Merge "Atom: Remaining and full battery capacity" 2018-02-01 22:32:06 +00:00
Chenjie Yu
e72252b688 Add command to clear puller cache
Test: manual test
Change-Id: Id1fa48b5a6f2200d22f4a746763f65d0c8d65f72
2018-02-01 13:22:27 -08:00
Tej Singh
bf972d99a9 Atom: Remaining and full battery capacity
Add pulled remaining battery capacity and full battery capacity atoms.

Test: cts test accompanies
Change-Id: I0f09e9459b56e25cc3ac21e1a5e781daadea0a01
2018-01-31 17:27:50 -08:00
Yangster-mac
9369446f0b Support dimension in condition in metric producers.
Test: added e2e tests for count/duration metrics sliced by fields in condition and with/without links.

Change-Id: Ie34deba68e6780abdde458be3f0ce5284e76a1a2
2018-01-30 13:33:57 -08:00
yro
9b97976e76 Re-enable statsd from droidfooders by default
Test: the team-wide bug bash to capture any issues in advance but did
not find any statsd crash

Change-Id: I253e0067a710c37bf10667e3a8dbde99d0b289c7
2018-01-29 16:23:13 -08:00
yro
181c835f81 Disable statsd that was accidentally enabled in ag/3480570
Test: manual test

Change-Id: I57176203a1ca7d0b5639119642335e3e9a1e4e4c
2018-01-25 10:55:57 -08:00
TreeHugger Robot
d0c260ff41 Merge "Adding guardrails on writing to disk from statsd" 2018-01-25 06:47:29 +00:00
TreeHugger Robot
73a331f524 Merge "Statsd broadcast subscriber" 2018-01-25 06:23:36 +00:00
Howard Ro
540aa206e2 Merge "Revert "Revert "Revert "Enable init.rc for statsd to start statsd by default in boot time"""" 2018-01-25 04:32:50 +00:00
Howard Ro
62ac686168 Revert "Revert "Revert "Enable init.rc for statsd to start statsd by default in boot time"""
This reverts commit b00cce84d9ddba04a3ebd5ed45457c1e154be535.

Reason for revert: b/72459151 

Change-Id: I71593b4b323d69edb548dcb1bb0e1f4eb7cddbc9
2018-01-25 02:00:35 +00:00
Bookatz
c697797d43 Statsd broadcast subscriber
Allows a uid that uploads a statsd config to additionally
register a BroadcastSubscriber with statsd. If statsd
detects an anomaly (according to the config's Alert),
statsd can inform a BroadcastSubscriber provided in the config.
The config uses a subscriberId (just an int) to identify the
BroadcastSubscriber. It then uses StatsManager.setBroadcastSubscriber
to associate that subscriberId with a given PendingIntent.
Then, when the anomaly is detected, statsd sends a broadcast
using that PendingIntent, alerting whoever was specified by
the config/setBroadcastSubscriber.

Bug: 70356901
Test: cts-tradefed run cts-dev -m CtsStatsdHostTestCases -t android.cts.statsd.alert.BroadcastSubscriberTests
Change-Id: I4d9ea9a6c8a85e61fadfd99c1513c55abbadd5e9
2018-01-24 13:08:19 -08:00