732 Commits

Author SHA1 Message Date
Yangster-mac
be10ddfe46 Flush the past buckets in anomaly tracker when time jumps forward
E2e test for count/duration anomaly trackers.

Test: new statsd tests.

BUG: b/74446029

Change-Id: Ia9be0240ba5021d44c1e1f096d67563e9138bb59
2018-03-26 18:44:01 -07:00
TreeHugger Robot
dafbeecaf7 Merge "Up the dimension key limit for CpuTimePerUidFreq" into pi-dev 2018-03-26 23:28:30 +00:00
TreeHugger Robot
d41fad71a6 Merge "Fix uidmap in statsd." into pi-dev 2018-03-26 22:38:25 +00:00
Tej Singh
d89137ed24 Temperature atom: Use int instead of float.
Changes the temperature atom to use an int instead of a float because
metrics and anomaly detection assume the data will be an int or a long.

Bug: b/74011562
Test: statsd temperature CTS test
Change-Id: I6e5845a40c28aefd30c2be9709c6f9de1783cf02
2018-03-26 14:51:40 -07:00
Chenjie Yu
c58750578f Up the dimension key limit for CpuTimePerUidFreq
This atom logs time spent on cpu per frequency per uid.
On marlin, there are 27 frequency steps per uid (should be 50+ if
flatten to both cores)
On walleye, there are 52 frequency steps per uid.
So it easily goes to 6k data.

Soft limit set to 6k now.
Hard limit set to 10k to accommodate future processors

Memory impact: on marlin, start memory no config is 2,346K
after using this atom in one guage metric, it is 3,067K

Bug: 72505991
Test: cts test
Change-Id: I067a32e54e4a457fdf9a25911aa16030e893ef4d
2018-03-26 10:29:38 -07:00
David Chen
35045cbc34 Fix uidmap in statsd.
Previously tried an optimization that results in corrupted proto
output. This changes to a safer approach of storing the snapshot data
in memory and only converting to proto output when the
ProtoOutputStream is provided.

Also fixes a security issue when trying to invoke triggerUidSnapshot
since we forgot to use SCS' permissions.

Test: Added a unit-test to verify output of StatsLogProcessor.
Bug: 76231867
Change-Id: Id410ce3505fda9d71caa71942ef3068b55872c66
2018-03-24 15:01:04 -07:00
TreeHugger Robot
c8ab224b2a Merge "Fix StatsCompanionService sometimes can be null" into pi-dev 2018-03-24 16:28:01 +00:00
Chenjie Yu
aa5b201015 Fix StatsCompanionService sometimes can be null
Bug: 75970648
Test: will add cts test for puller alarms
Change-Id: I51b7d13f855d3c8ded8325d7cf0f614531eceea5
2018-03-23 14:33:58 -07:00
TreeHugger Robot
3991138f3a Merge "rename neq_all_string to neq_any_string in statsd_config" into pi-dev 2018-03-23 21:15:02 +00:00
Yao Chen
d50f2ae487 rename neq_all_string to neq_any_string in statsd_config
Bug: 73897465
Test: statsd_test
Change-Id: I1d020de873fc26fbb502f0b3487b85fdb3896753
2018-03-23 11:10:13 -07:00
TreeHugger Robot
dc2dda5c49 Merge "Fix statsd dropping metrics data." into pi-dev 2018-03-23 17:40:18 +00:00
David Chen
4c6d97a1e4 Fix statsd dropping metrics data.
We can increase the buffer of metrics we store in statsd memory, but
we still request the clients to call getData when the metrics memory
exceeds 128 KB (previously was 90% of 128 KB).

Bug: 76171061
Test: Test that unit-tests still pass.
Change-Id: I901545b364ed313af8c033ce9b40d3cfadb93213
2018-03-22 16:46:54 -07:00
Yao Chen
5f8939eecd Merge "Fix StorageManager stack use after scope." into pi-dev 2018-03-22 23:19:21 +00:00
TreeHugger Robot
e8582d0ca7 Merge "Fix the heap-after-use in DurationAnomalyTracker" into pi-dev 2018-03-22 21:41:35 +00:00
TreeHugger Robot
23ceeb3a44 Merge "Fix UidMap proto output error. And fix a bug in MaxDurationTracker." into pi-dev 2018-03-22 21:36:50 +00:00
Yao Chen
c219348186 Merge "Remove unsigned-integer-overflow signed-integer-overflow from sanitizer." into pi-dev 2018-03-22 20:24:41 +00:00
Yao Chen
59cc0a2237 Fix UidMap proto output error. And fix a bug in MaxDurationTracker.
Bug: 76118312
Bug: 76145379

Test: adb shell cmd stats dump-report
      statsd_test
Change-Id: I9a987525df5b37fcefd5f28700bfa1c3952f6692
2018-03-22 09:07:03 -07:00
TreeHugger Robot
dcccfd9742 Merge "ScheduledJobStateChanged: add stop reason" into pi-dev 2018-03-22 01:30:52 +00:00
Yao Chen
1aa00024fa Fix the heap-after-use in DurationAnomalyTracker
Bug: 74531081
Test: cts
Change-Id: I21a50944a119b7576c76fd25fae23b9c2d8d96ab
2018-03-21 17:04:34 -07:00
Yao Chen
461a75f269 Remove unsigned-integer-overflow signed-integer-overflow from sanitizer.
Sanitizer is picking on the protobuf generated code. Need to figure out why
Disable them to unblock development first.

Bug: 76117312
Test: manual
Change-Id: I3f3a39dfe933f4c1b1cf4063d621392b66f0adea
2018-03-21 16:38:26 -07:00
Yao Chen
fef48cb64c Fix StorageManager stack use after scope.
Bug: 75968524
Test: CTS
Change-Id: I2fd81a21452b26d8c9239d7aeddd4907b9618179
2018-03-21 14:44:34 -07:00
TreeHugger Robot
dc5677336c Merge "Enable sanitizer on statsd eng build." into pi-dev 2018-03-21 20:09:10 +00:00
Yao Chen
cd7f90053d Enable sanitizer on statsd eng build.
Bug: 74946342

Test: manual
Change-Id: I287def435d077670814db50e4d8398ecee539b74
2018-03-21 11:04:37 -07:00
TreeHugger Robot
8512630f8b Merge "Allow statsd to be given empty config." into pi-dev 2018-03-21 05:34:16 +00:00
Howard Ro
e51af37475 Merge "Fix recovery of stats data from previous input while using ProtoOutputStream" into pi-dev 2018-03-21 00:31:23 +00:00
TreeHugger Robot
0421c2cf8f Merge "Fix pulling interval may become 0 in cts" into pi-dev 2018-03-20 23:31:24 +00:00
David Chen
9fdd40302e Allow statsd to be given empty config.
Statsd clients may want to set an empty config temporarily, so it's
more convenient to allow them to set an empty config instead of
having to use the removeConfig and then having to remember to call
StatsManager#setDataFetchOperation.

Test: Added unit-tests and check they pass on marlin-eng.
Bug: 74997752
Change-Id: I2e762e5ec01e5a2c9a3469fb330b53fefbd734d6
2018-03-20 15:56:11 -07:00
yro
4beccbe3de Fix recovery of stats data from previous input while using
ProtoOutputStream

- Specify the length of message to avoid libprotoutil from thinking that
we are trying to write bool
- We only attach the previous dump file to the upload file where config
key matches
- Store ConfigMetricsReport (instead of ConfigMetricsReportList) onto
disk
- Stop use stack after scope in StorageManager
- Migrate UidMap to use ProtoOutputStream and renaming variables to
prevent confusion

Bug: 74021554
Bug: 75968524
Test: manual test, statsd_test, CTS tests
Change-Id: Iedf52633d7f5b985f5a934a3fb5a0c3c3b2e7fd1
2018-03-20 15:00:59 -07:00
TreeHugger Robot
039deb4390 Merge "Moves the settings changed logging for statsd." into pi-dev 2018-03-20 20:30:31 +00:00
Tej Singh
33a412b52d ScheduledJobStateChanged: add stop reason
Adds stop reason to scheduled job state changed

Bug: b/71755130
Test: cts-tradefed run cts-dev -m CtsStatsdHostTestCases -t android.cts.statsd.atom.UidAtomTests#testScheduledJobState
Test: cts-tradefed run cts-dev -m CtsIncidentHostTestCases -t com.android.server.cts.JobSchedulerIncidentTest
Change-Id: Ic29a5e9160ac258140a52d31bc6210300949545d
2018-03-20 11:20:58 -07:00
TreeHugger Robot
aed0fed078 Merge "Add uid field annotation in atoms.proto and statd memory usage optimization." into pi-dev 2018-03-20 17:12:25 +00:00
David Chen
bd78991bc6 Moves the settings changed logging for statsd.
Previously, we wrote a log entry regardless of permission checks, so
the logging could be misleading. Now we only send the log to statsd
after verifying that this setting mutation is valid.

Test: Flashed onto marlin-eng and verified stats-log as expected.
Bug: 73493944
Change-Id: I2a8b052aa8c380ffc5d15caec089fffcdc5823f4
2018-03-19 17:11:03 -07:00
TreeHugger Robot
9859c29a8f Merge "Fix a UidMap crash." into pi-dev 2018-03-19 23:07:06 +00:00
TreeHugger Robot
dba47fcd09 Merge "Atoms: Mobile Connection" into pi-dev 2018-03-19 20:45:55 +00:00
Yao Chen
34900c3b4d Fix a UidMap crash.
Bug: 75968717

Test: CTS with sanitizer
Change-Id: I9516d3b59f72816159c9fc1fe75ca64bfe21d799
2018-03-19 13:43:29 -07:00
TreeHugger Robot
f5de606f51 Merge "Deletes default allowed_log_sources in statsd." into pi-dev 2018-03-19 18:14:18 +00:00
Chenjie Yu
83baaa1212 Fix pulling interval may become 0 in cts
If the bucket size is lower than 1 min, the interval should be at least
1 min.

Bug: 74533948
Test: cts
Change-Id: Icb72579367cd12d3f475c7e2935b98c0ebc66419
2018-03-19 10:41:35 -07:00
Tej Singh
eee317b7b6 Atoms: Mobile Connection
Atom definitions for MobileConnectionStateChanged and
MobileRadioTechnologyChanged

Also cleans up batterystats.

Bug: b/72320589
Test: verified logs appear in adb logcat -b stats
Change-Id: I9feb258cf6dd4a8c8bf1cffc9566b5d0a851a9fa
2018-03-19 09:48:22 -07:00
TreeHugger Robot
790befca1d Merge "Delete TODO about renumbering statd atoms" into pi-dev 2018-03-19 05:54:22 +00:00
TreeHugger Robot
ad121b9fdc Merge changes I3ffb6e97,I689df136,Ia67a8eb6 into pi-dev
* changes:
  Statsd: remove DurationAnomalyTracker.resetStorage
  Statsd AnomalyDetection stopAlarm also checks old
  Statsd AnomalyDetection improvements
2018-03-19 05:49:27 +00:00
Bookatz
555ad97637 Delete TODO about renumbering statd atoms
Test: none
Bug: 72553942
Change-Id: If59ee407dffaed713a3a2f7543314aef3e70ef2c
2018-03-16 15:47:57 -07:00
Bookatz
6a1d3af7f5 Statsd: remove DurationAnomalyTracker.resetStorage
The explicit resetStorage function in the subclass isn't needed.
It does exactly what its superclass does.
The previous check for !mAlarms.empty() was actually not needed, since
there can be legit cases where !mAlarms.empty() (namely, when the
mMostRecentBucket is moved forward initially - we may call reset, but we
still want the old alarms to remain).

Test: still pass
Change-Id: I3ffb6e97c02b75170c246f57d55ccf59cd368b9c
Fixes: 74607818
2018-03-16 15:25:49 -07:00
Yao Chen
c40a19d2e4 Add uid field annotation in atoms.proto and statd memory usage optimization.
[memory]
  statsd binary size from 664k -> 600k
  memory usage 1978k -> 1813k (with no configs)
  + Avoid initialize any static map in statslog.h to avoid many copies of the map in each include.
    - Do it in cpp so that it is initialized only in places that use them

[Uid annotation]
+ Uid annotation is needed for extracting uid from dimension for UidCpuPuller.
+ After the change, stand-alone uids don't need to be in field 1 anymore.
+ Also added exclusive bit annotation in AppDied
+ Currently only allow one uid field in an Atom. This is to keep things simple until
  we find an exception.

Test: statsd_test
Bug: 73958484
Bug: 72129300

Change-Id: I8a916d5c00d5930e24ae7e0825a57dea19c0e744
2018-03-16 13:56:38 -07:00
Bookatz
3e8cd35b55 Statsd AnomalyDetection stopAlarm also checks old
Every time stopAlarm() is called, it should also,
right then, check to see if the alarm should actually
have already fired (but didn't due to AlarmManager lag).
Right now, the client needs to do this check separately,
but they always go together. Indeed, MaxDurationTracker
forgot to do the check, which is a bug. It would make
much more sense if the stopAlarm takes care of it for
them, to prevent such mistakes.

Bug: 75273733
Test: make statsd_test && adb sync data && adb shell data/nativetest64/statsd_test/statsd_test
Test: cts-tradefed run cts-dev -m CtsStatsdHostTestCases -t android.cts.statsd.alert
Change-Id: I689df13690df822090ac34b1171e948be1ad0d9f
2018-03-16 13:23:19 -07:00
Bookatz
6bf9825b15 Statsd AnomalyDetection improvements
Various fixes and improvements to statsd's anomaly detection.

Bug: 74607818
Test: make statsd_test && adb sync data && adb shell data/nativetest64/statsd_test/statsd_test

Change-Id: Ia67a8eb6da0ea9293f698949e1565f7f024a7cb9
2018-03-16 11:34:21 -07:00
David Chen
8faaa01489 Deletes default allowed_log_sources in statsd.
We should fail right away if someone forgets to set this field
instead of setting default values since this may lead to hard to
explain bugs in the future (eg, why isn't systemui logging an atom
to a config).

Bug: 74608359
Test: Verified unit-tests still pass on marlin-eng.
Change-Id: Ibd8a6ccbc5cc8d2dfb8a1577c64bf9b49822a2c3
2018-03-15 17:45:58 -07:00
David Chen
f384b90049 Removes stats_log proto from uid map in statsd.
We don't need to parse the proto of uid map, so we use the
ProtoOutputStreame class to generate the binary form of the proto
output that's needed for parsing the uid map data.

Test: Verified unit-tests still pass.
Bug: 74010813
Change-Id: Ia2f7572f3b78bb6f7b60e8b14cf5d65428469ab6
2018-03-15 13:33:04 -07:00
TreeHugger Robot
a825c8ee5d Merge "Support slicing by chain." into pi-dev 2018-03-15 02:27:15 +00:00
TreeHugger Robot
8568931d6f Merge "Add stats from storage manager to dumpsys result" into pi-dev 2018-03-14 23:05:33 +00:00
Yangster-mac
e06cfd777a Support slicing by chain.
BUG: b/73975181

Test: statsd test
Change-Id: I913ae0f68ff21ed0703bb5da9c60d3eaa3bf5981
2018-03-14 15:36:24 -07:00