28 Commits

Author SHA1 Message Date
Chenjie Yu
5caaa9d854 Clean up atoms.proto
changes are:
1) for pushed atoms, use attribution node in place of uid when
appropriate
2) name changes to be more consistent

Bug: 73823969
Test: manual test
Change-Id: Iacf7186dbd7a2282f7fe481f43dbbf92e1165b47
2018-03-30 10:11:03 -07:00
Yangster-mac
be10ddfe46 Flush the past buckets in anomaly tracker when time jumps forward
E2e test for count/duration anomaly trackers.

Test: new statsd tests.

BUG: b/74446029

Change-Id: Ia9be0240ba5021d44c1e1f096d67563e9138bb59
2018-03-26 18:44:01 -07:00
David Chen
8faaa01489 Deletes default allowed_log_sources in statsd.
We should fail right away if someone forgets to set this field
instead of setting default values since this may lead to hard to
explain bugs in the future (eg, why isn't systemui logging an atom
to a config).

Bug: 74608359
Test: Verified unit-tests still pass on marlin-eng.
Change-Id: Ibd8a6ccbc5cc8d2dfb8a1577c64bf9b49822a2c3
2018-03-15 17:45:58 -07:00
TreeHugger Robot
a825c8ee5d Merge "Support slicing by chain." into pi-dev 2018-03-15 02:27:15 +00:00
Yangster-mac
e06cfd777a Support slicing by chain.
BUG: b/73975181

Test: statsd test
Change-Id: I913ae0f68ff21ed0703bb5da9c60d3eaa3bf5981
2018-03-14 15:36:24 -07:00
Yangster-mac
3fa5d7fb23 Add wall clock timestamp for ConfigMetricsReport and gauge atoms.
Fix the bug when serializing multiple atoms in gauge metric

BUG: b/74159560

Test: new test for ALL_CONDITION_CHANGES sampling method.
Change-Id: I6d33c1efbac92b6e13be2d64c323e090cb1f84aa
2018-03-10 22:25:28 -08:00
Yangster
13fb7e4eea Statsd cpu optimizaton
- cache the changed dimensions in condition tracker.
- avoid query condition wizard when unnecessary.
- avoid copy dimension keys in condition key generation.

Test: statsd tests.

BUG: b/73959649
Change-Id: I17d68e2a82643de3f421309841e75f84c6fd8f43
2018-03-07 17:36:07 -08:00
Yao Chen
9c1debe330 Add annotation to atoms that represent a state change in atoms.proto
+ A state change atom can have one exclusive state field, and any
  number of primary key fields.

  When there is primary key in the atom, it means the state belongs to the primary key.
  For example,
  message UidProcessStateChanged {
    optional int32 uid = 1 [(stateFieldOption).option = PRIMARY];
    optional android.app.ProcessStateEnum state = 2 [(stateFieldOption).option = EXCLUSIVE];
  }

  When there is no primary key fields in the atom, the state is global.
  For example,
  message ScreenStateChanged {
     optional android.view.DisplayStateEnum state = 1 [(stateFieldOption).option = EXCLUSIVE];
  }

+ The annotation is consumed by stats_log_api_gen to generate a static map from the state
  atoms to its primary fields, and exclusive fields

+ stats_log.proto is splitted into 2 proto files, because statsd needs proto lite, and c++
  lite proto library cannot properly ignore the field options which requires full proto.

 This CL doesn't change any logic in the statsd yet. A separate CL will use the field option
 information to correctly track the state.

Test: added unit tests in stats_log_api_gen_test. and statsd_test pases.
Change-Id: I9e8a979fe81ba60efd4d854bb7087ce4b2b147ec
2018-02-21 16:46:56 -08:00
Howard Ro
db15052ae2 Merge "Add a guardrail to limit minimum bucket duration to be 5 minutes except when configured through adb command" 2018-02-15 00:18:22 +00:00
TreeHugger Robot
2f5b9c9b03 Merge "Updates statsd atoms.proto with small changes." 2018-02-14 19:18:39 +00:00
Yangster-mac
330af58f2b Use elapsed realtime instead of times based on wall clock, which can jump around and go backwards.
Test: statsd unit test passed

Change-Id: Ib541df99231e171b3be2a24f75632693e36da90e
2018-02-13 23:30:39 -08:00
yro
59cc24dbfd Add a guardrail to limit minimum bucket duration to be 5 minutes except
when configured through adb command

Bug: 73287251
Test: statsd_test
Change-Id: Iee51fedbaabb2c1f534a6edef3c564da88ef658b
2018-02-13 20:26:31 -08:00
David Chen
0b5c90cd8c Updates statsd atoms.proto with small changes.
Renames the apphook atom to be more descriptive. Renames time
suffixes to match more clear convention (ms and msec are ambiguous).

Adds a field to CPU active time that's required for the metrics
to be usable.

Test: Test that statsd can still build.
Change-Id: I94866510738db994e8d757260f30e599ba995dbd
2018-02-13 16:17:25 -08:00
Yao Chen
8a8d16ceea Statsd CPU optimization.
The key change is to revamp how we parse/store/match a log event, especially how we match repeated
field and attribution nodes, and how we construct dimensions and compare them.

+ We use a integer to encode the field of a log element. And also encode the FieldMatcher into an
integer and a bit mask. The log matching becomes 2 integer operations.

+ Dimension is stored as encoded field and value pair. Checking if 2 dimensions are equal is then
  becoming checking if the underlying integers are equal. The integers are stored contiguously
  in memory, so it's much faster than previous tree structure.

Start review from FieldValue.h

Test: statsd_test + new unit tests

Bug: 72659059

Change-Id: Iec8daeacdd3f39ab297c10ab9cd7b710a9c42e86
2018-02-12 10:38:45 -08:00
David Chen
27785a8a4a Partial buckets on app upgrade and fix duration.
Statsd will create partial buckets in all metrics producers when an
app is upgraded so that we can separate metrics between different
versions of an app. By looking at the uid map changes, we can tell
which app versions belong to a bucket; for metrics that are not
affected by an app version, we can instead join the buckets together.

To simplify the logic, the ends of the full buckets are always
aligned to when the metric producers were created. These boundaries
are computed on the fly by using the bucket number and the metric
producers' start times.

We keep the anomaly trackers to only be given full buckets; we buffer
the partial buckets within each metric producer.

Duration metric's MAX_SPARSE is fixed to be implemented as such. In
addition, after further discussion, we find anomaly detection on
MAX_SPARSE to be unnecessary, so this functionality is removed.

Test: Unit-tests added and modified, passed on marlin-eng.
Change-Id: I5ff7a9c7f05c406e9faf400c6a39162970ded102
2018-02-02 13:30:10 -08:00
Yangster-mac
9369446f0b Support dimension in condition in metric producers.
Test: added e2e tests for count/duration metrics sliced by fields in condition and with/without links.

Change-Id: Ie34deba68e6780abdde458be3f0ce5284e76a1a2
2018-01-30 13:33:57 -08:00
Yangster-mac
34ea1103a0 Extend gauge metric to support memory metric.
Test: statd unit test passed.

Test: statsd unit test passed
Change-Id: I2e3f26563678ae77d44afe168454b6d1ea449f3a
2018-01-30 09:14:54 -08:00
Bookatz
1a1b0464cb Statsd and frameworks reference proto enums
For frameworks constants that don't have intrinsic meaning (i.e. their actual
value and order don't matter), so that it is unlikely that their values
will be changed:
This cl introduces proto enums representing some constants found in
the Android codebase, and connects the two.

By using the Proto enum as the source-of-truth, it means that Java and
proto can be kept in sync. Otherwise, when the Java frameworks code
changes, it silently breaks the protos from working properly, since the enums
are wrong. By having the Java code reference the proto enums, it ensures
that everything is in sync. The values of the constants are unchanged.
But future changes to these constants will need to be done in the proto
file, which the Java file merely references.

The protos are necessary for incidentd and statsd and, in the future,
possibly dumpsys. In this way, the logging mechanism is much less likely
to get broken when new constants are added, and we can be ensured that
the logging accurately reflects the underlying codebase.

Bug: 69478930
Test: cts-tradefed run cts-dev -m CtsStatsdHostTestCases
Test: cts-tradefed run cts-dev -m CtsIncidentHostTestCases
Change-Id: If79032c34b2799db1e3e70cb47b1312fd72092b9
2018-01-25 15:47:57 -08:00
TreeHugger Robot
5e1f854620 Merge "Rename the dimension_in_what/condition as fields_in_what/condition in MetricConditionLink proto." 2018-01-19 17:39:57 +00:00
Yangster-mac
2c6dc474ac Rename the dimension_in_what/condition as fields_in_what/condition in MetricConditionLink proto.
Test: statsd cts tests passed.
Change-Id: I4c21b0da3d153c8d0c368a78647c44699a3161a5
2018-01-18 16:17:43 -08:00
Yangster-mac
5503f5caae condition dimension query.
Test: Modified the metric link test to query a subset of the predicate/condition dimensions.

Change-Id: I693f14e0b11693fc8ee0bf4fc550977c20d31f71
2018-01-18 14:51:51 -08:00
Yangster-mac
468ff04fd4 Proto change for dimension from condition.
Test: all statsd unit tests and cts tests passed.

Change-Id: I955143aca7c76e2e22a10d7ba42eb797bd32e147
2018-01-17 21:19:45 -08:00
Yangster-mac
87718e283a Gauge metric e2e test
Test: new test passed
Change-Id: I6a8bceb43fbb6e7b82f47951e71b5620779f2ceb
2018-01-15 20:41:08 -08:00
Yangster-mac
d40053eb8b Map isolated uid to host uid when processing log event in statsD.
Test: added test case for isolated uid in Attribution e2e test.
Change-Id: I63d16ebee3e611b1ef0c910e5154cf27766cb330
2018-01-09 21:45:46 -08:00
Yangster-mac
b5bc74122f E2e test: attribution matching and slicing by attribution.
Test: statsd unit tests passed.
Change-Id: I05875b80a9a1ab5f87a478d1f63543f9798254a5
2018-01-06 23:23:19 -08:00
Yangster-mac
b814481ad1 Use TimeUnit enum to specify the bucket size.
Test: all statsd unit test passed

Change-Id: I4f6b80ba2f8c984b06e46e6de6df3e546e99a968
2018-01-04 13:01:14 -08:00
Yangster-mac
94e197cceb 1/ Change all "name" to id in statsD.
2/ Handle Subscription for alert.
3/ Support no_report_metric

Bug: 69522276
Test: all statsd unit tests passed.
Change-Id: I851b235f2d149b8602b0cad632d5bf541962f40a
2018-01-03 15:34:00 -08:00
Yangster-mac
2087716f2b 1/ Support nested message and repeated fields in statsd.
2/ Filter gauge fields by FieldMatcher.
3/ Wire up wakelock attribution chain.
4/ e2e test: wakelock duration metric with aggregated predicate dimensions.
5/ e2e test: count metric with multiple metric condition links for 2 predicates and 1 non-sliced predicate.

Test: statsd unit test passed.

Change-Id: I89db31cb068184a54e0a892fad710966d3127bc9
2018-01-01 10:01:36 -08:00