20 Commits

Author SHA1 Message Date
Ruchir Rastogi
e449b0c185 Move statsd (and tests) to libbinder_ndk
Major changes include:
    - Removing unused permission checks within StatsService. These
      include ENFORCE_DUMP_AND_USAGE_STATS, checkDumpAndUsageStats,
      kOpUsage, and kPermissionUsage.
    - Converting from sp to shared_ptr
    - Using libbinder_ndk functions instead of libbinder functions
      (e.g. for installing death recipients, getting calling uids, etc.)
        - New death recipients were added in StatsService,
          ConfigManager, and SubscriberReporter.
    - Using a unique token (timestamp) to identify shell subscribers
      instead of IResultReceiver because IResultReceiver is not exposed by
      libbinder_ndk. Currently, statsd cannot detect if perfd dies; we
      will fix that later.

Bug: 145232107
Bug: 148609603
Test: m statsd
Test: m statsd_test
Test: bit stastd_test:*
Test: atest GtsStatsdHostTestCases
Change-Id: Ia1fda7280c22320bc4ebc8371acaadbe8eabcbd2
2020-02-14 18:07:37 -08:00
Yao Chen
4ce07298e0 Add more alert details to incidentd header.
+ Add metric_id, dimension, metric_value in the incident report header
  to make it easier to identify an anomaly.
+ In case there is a uid in the dimension, put the its package info in the header too

Test: statsd_test && tested locally with a config

Change-Id: I762a1a4b0563fb3104e0d32b616b4a2367485e48
2019-02-13 17:21:42 -08:00
Chih-Hung Hsieh
2491af24a8 Merge "Fix performance-for-range-copy warnings" am: bc29242288 am: 7adb73f332
am: 2296036a40

Change-Id: Ia99ea6c9633986e68340bc88983d06baa7a1c597
2018-12-17 14:31:14 -08:00
Chih-Hung Hsieh
a1b644e88c Fix performance-for-range-copy warnings
Bug: 30413223
Test: make with WITH_TIDY=1 DEFAULT_GLOBAL_TIDY_CHECKS=-*,performance*
Change-Id: Ie481e88025a7a1f3abde8ff63420d5ccd8577e52
2018-12-17 20:00:55 +00:00
Bookatz
d27ab45ad3 Remove TODO in statsd AnomalyTracker_test
The underlying item the TODO is referencing had already been resolved
so the test line can be properly added, per the TODO.

Change-Id: I5c16e7ea319bd16e37475381def656b38f39d17f
Fixes: 80095149
Test: make statsd_test && adb sync data && adb shell data/nativetest64/statsd_test/statsd_test
2018-05-24 10:35:02 -07:00
Yangster-mac
c04feba805 Move forward the alarm timestamp when config is added to statsd.
Test: statsd test
BUG: b/77344187

Change-Id: Ieacffaa29422829b8956f2b3fcb2c647c8c3eed9
2018-04-02 18:12:36 -07:00
Bookatz
6bf9825b15 Statsd AnomalyDetection improvements
Various fixes and improvements to statsd's anomaly detection.

Bug: 74607818
Test: make statsd_test && adb sync data && adb shell data/nativetest64/statsd_test/statsd_test

Change-Id: Ia67a8eb6da0ea9293f698949e1565f7f024a7cb9
2018-03-16 11:34:21 -07:00
Yangster-mac
932ececa16 Alarm: wakes up statsd and notifies the subscribers.
Test: manually tested it.
Change-Id: Id796a68976aeb1611183023ba4e9c6a8b8c44bb8
2018-02-27 13:30:48 -08:00
Yao Chen
8a8d16ceea Statsd CPU optimization.
The key change is to revamp how we parse/store/match a log event, especially how we match repeated
field and attribution nodes, and how we construct dimensions and compare them.

+ We use a integer to encode the field of a log element. And also encode the FieldMatcher into an
integer and a bit mask. The log matching becomes 2 integer operations.

+ Dimension is stored as encoded field and value pair. Checking if 2 dimensions are equal is then
  becoming checking if the underlying integers are equal. The integers are stored contiguously
  in memory, so it's much faster than previous tree structure.

Start review from FieldValue.h

Test: statsd_test + new unit tests

Bug: 72659059

Change-Id: Iec8daeacdd3f39ab297c10ab9cd7b710a9c42e86
2018-02-12 10:38:45 -08:00
Yangster-mac
9369446f0b Support dimension in condition in metric producers.
Test: added e2e tests for count/duration metrics sliced by fields in condition and with/without links.

Change-Id: Ie34deba68e6780abdde458be3f0ce5284e76a1a2
2018-01-30 13:33:57 -08:00
Bookatz
1bf94382d0 Anomaly detection is per dimension
Anomalies are now detected and declared per dimension. This means that
declareAnomaly now gets access to the key that is responsible for the
anomaly. Moreover, the refractory period is per dimension, not overall.
So a second anomaly for the same metric but a different dimension can
fire within the first dimension's refractory period. Thus, if app A
misbehaves and app B misbehaves shortly thereafter, they will both be
detected.

Eventually this key will be passed to the subscribers, although this cl
doesn't do anything with it.

Test: adb shell data/nativetest64/statsd_test/statsd_test
Change-Id: Id76856dc44fe9ecf91ac81a423e84f97c81d30ab
2018-01-09 11:29:41 -08:00
Yangster-mac
a7fb12d2d2 1/ Use FieldMatcher to specify the value fields in value metric.
2/ rename number_of_buckets as num_buckets
3/ use double for the Alert's threshold

Test: statsd unit tests passed.
Change-Id: Id1f55f14d3712eddee561681e3cd77343f086c7a
2018-01-03 23:49:36 -08:00
Yangster-mac
94e197cceb 1/ Change all "name" to id in statsD.
2/ Handle Subscription for alert.
3/ Support no_report_metric

Bug: 69522276
Test: all statsd unit tests passed.
Change-Id: I851b235f2d149b8602b0cad632d5bf541962f40a
2018-01-03 15:34:00 -08:00
Yangster-mac
2087716f2b 1/ Support nested message and repeated fields in statsd.
2/ Filter gauge fields by FieldMatcher.
3/ Wire up wakelock attribution chain.
4/ e2e test: wakelock duration metric with aggregated predicate dimensions.
5/ e2e test: count metric with multiple metric condition links for 2 predicates and 1 non-sliced predicate.

Test: statsd unit test passed.

Change-Id: I89db31cb068184a54e0a892fad710966d3127bc9
2018-01-01 10:01:36 -08:00
Bookatz
857aaa5208 Splits AnomalyTracker into two files
Splits out DurationAnomalyTracker-specific functions into their own
subclass.

Test: the unit tests and CTS tests
Change-Id: Id6eb74d232b4a9c3a932d805d1ba3f0ba43a88b1
2017-12-22 15:05:31 -08:00
Yao Chen
d5aa01b371 Remove the hacky HashableDimensionKey.
+ Add a real HashableDimensionKey as a wrapper of the dimension.
So we can get rid of the maps that we kept.

Pay down technical debt and reduce memory usage.

Test: statsd_test & manual
Change-Id: I233280cf1e2ce93da6a8cd4e8514abb066f4016d
2017-12-20 18:45:43 -08:00
Bookatz
8f2f3d8205 Anomaly Alert declarations in StatsdStats
StatsdStats now tracks the number of times an anomaly is detected (per
config, per alert name).

Also adds a configKey to AnomalyTracker, which is needed, not only for
statsdstats, but also (in the future) for reporting the header
information to incidentd.

Bug: 67978682
Test: adb shell data/nativetest64/statsd_test/statsd_test
Change-Id: Ib254db7e1edb4f0f193f4772d17f14934cdf7e30
2017-12-07 18:00:44 -08:00
Bookatz
cc5adef2d0 Statsd anomaly detection - fixes
Fixes a few items in AnomalyTracker, especially to do with what happens
when an anomaly alarm fires.

Test: unit tests still pass
Change-Id: Ia89bd617442e952e587336b890c3ca67430b5e21
2017-11-27 15:35:40 -08:00
Yangster-mac
e2cd6d509b 1/ Duration anomaly tracker with alarm.
2/ Init anomaly from config based on the public language.
3/ Unit tests for anomaly detection in count/gauge producer.
4/ Revisit the duration tracker logic.

Test: unit test passed.
Change-Id: I2423c0e0f05b1e37626954de9e749303423963f2
2017-11-20 15:37:24 -08:00
Yang Lu
3eba621865 Update CountAnomalyTracker to handle slicing.
Re-added CountAnomalyTracker (under the name DiscreteAnomalyDetector).
It is now able to handle dimensions.

Test: unit test passed.

Change-Id: I133fcf6db99b9e305756f7ee5891c25f0f276348
2017-11-09 12:32:13 -08:00