The key change is to revamp how we parse/store/match a log event, especially how we match repeated
field and attribution nodes, and how we construct dimensions and compare them.
+ We use a integer to encode the field of a log element. And also encode the FieldMatcher into an
integer and a bit mask. The log matching becomes 2 integer operations.
+ Dimension is stored as encoded field and value pair. Checking if 2 dimensions are equal is then
becoming checking if the underlying integers are equal. The integers are stored contiguously
in memory, so it's much faster than previous tree structure.
Start review from FieldValue.h
Test: statsd_test + new unit tests
Bug: 72659059
Change-Id: Iec8daeacdd3f39ab297c10ab9cd7b710a9c42e86
Statsd will create partial buckets in all metrics producers when an
app is upgraded so that we can separate metrics between different
versions of an app. By looking at the uid map changes, we can tell
which app versions belong to a bucket; for metrics that are not
affected by an app version, we can instead join the buckets together.
To simplify the logic, the ends of the full buckets are always
aligned to when the metric producers were created. These boundaries
are computed on the fly by using the bucket number and the metric
producers' start times.
We keep the anomaly trackers to only be given full buckets; we buffer
the partial buckets within each metric producer.
Duration metric's MAX_SPARSE is fixed to be implemented as such. In
addition, after further discussion, we find anomaly detection on
MAX_SPARSE to be unnecessary, so this functionality is removed.
Test: Unit-tests added and modified, passed on marlin-eng.
Change-Id: I5ff7a9c7f05c406e9faf400c6a39162970ded102
Test: added e2e tests for count/duration metrics sliced by fields in condition and with/without links.
Change-Id: Ie34deba68e6780abdde458be3f0ce5284e76a1a2
1/ Avoid unnecessary field/dimension proto construction.
2/ use unordered_map for slicing.
3/ Use dimension fields to compare dimension keys.
Test: all statsd tests passed.
Change-Id: I2f74f78589b7f6ecd0803a2ead822b8d0399f334
For frameworks constants that don't have intrinsic meaning (i.e. their actual
value and order don't matter), so that it is unlikely that their values
will be changed:
This cl introduces proto enums representing some constants found in
the Android codebase, and connects the two.
By using the Proto enum as the source-of-truth, it means that Java and
proto can be kept in sync. Otherwise, when the Java frameworks code
changes, it silently breaks the protos from working properly, since the enums
are wrong. By having the Java code reference the proto enums, it ensures
that everything is in sync. The values of the constants are unchanged.
But future changes to these constants will need to be done in the proto
file, which the Java file merely references.
The protos are necessary for incidentd and statsd and, in the future,
possibly dumpsys. In this way, the logging mechanism is much less likely
to get broken when new constants are added, and we can be ensured that
the logging accurately reflects the underlying codebase.
Bug: 69478930
Test: cts-tradefed run cts-dev -m CtsStatsdHostTestCases
Test: cts-tradefed run cts-dev -m CtsIncidentHostTestCases
Change-Id: If79032c34b2799db1e3e70cb47b1312fd72092b9
Statsd will contain at least one snapshot of the uid map. The
previous design was not very robust in case a snapshot was missing.
Also fixes subtle bug with updating the isolated uid mapping since
this should always be kept up to date even if there are no metrics
being used (since metrics may be added later after the isolated uid
was created).
Test: Checked that unit-tests pass on marlin-eng.
Change-Id: I99754ed9016d980564e409b0946a46b398fd12b7
- Update configName to configID in StorageManager reflecting ag/3410772
- Fix a bug of using strtok destructivelyand correctly parse file name
- Fix a unit test to call StartupForTest of config manager
Bug: 70667694
Test: statsd, statsd_test
Change-Id: If32c02982d1f0c509bdd4e6f445924257bbc074b
Anomalies are now detected and declared per dimension. This means that
declareAnomaly now gets access to the key that is responsible for the
anomaly. Moreover, the refractory period is per dimension, not overall.
So a second anomaly for the same metric but a different dimension can
fire within the first dimension's refractory period. Thus, if app A
misbehaves and app B misbehaves shortly thereafter, they will both be
detected.
Eventually this key will be passed to the subscribers, although this cl
doesn't do anything with it.
Test: adb shell data/nativetest64/statsd_test/statsd_test
Change-Id: Id76856dc44fe9ecf91ac81a423e84f97c81d30ab
+ Don't need to keep the condition key if the condition is not sliced.
+ Whiten the hash before returning.
Test: statsd_test
Change-Id: Ie0bcb14b3adc4a15b79ba3be050431a672a68c28
startup for unit test to avoid reading configs for tests
Bug: 70667694
Test: manual testing conducted, statsd, statsd_test
Change-Id: I49cb82d2b1d7c178b7a4c30c089d8d93e7902faa
2/ rename number_of_buckets as num_buckets
3/ use double for the Alert's threshold
Test: statsd unit tests passed.
Change-Id: Id1f55f14d3712eddee561681e3cd77343f086c7a
2/ Handle Subscription for alert.
3/ Support no_report_metric
Bug: 69522276
Test: all statsd unit tests passed.
Change-Id: I851b235f2d149b8602b0cad632d5bf541962f40a
Splits out DurationAnomalyTracker-specific functions into their own
subclass.
Test: the unit tests and CTS tests
Change-Id: Id6eb74d232b4a9c3a932d805d1ba3f0ba43a88b1
+ Add a real HashableDimensionKey as a wrapper of the dimension.
So we can get rid of the maps that we kept.
Pay down technical debt and reduce memory usage.
Test: statsd_test & manual
Change-Id: I233280cf1e2ce93da6a8cd4e8514abb066f4016d
+ Add log source whitelist in StatsdConfig
+ Some changes in UidMap API. Listener needs to be wp instead of sp.
+ Update dogfood app config to have log source
+ Increase the stats service thread pool size to 10 (9+1).
TODO: add unit tests(b/70805664). This unit test takes some time to write.
Test: statsd_test & manual
Change-Id: I129b1cc13db5114db7417580962bd7cc4438519d
bug fix: GaugeMetricProducer now works better with pulled events.
unit test also includes GaugeMetricProducer_test
Test: unit test
Change-Id: Ic60f09342d14cfb107be2130d445b323a56909e0
We use one alarm clock for all pulled atoms.
If metrics from different configs are not aligned,
the clock will be set to repeat at higher and higher
frequency, and consume a lot of battery.
Current implementation assumes a 5min minimum bucket
size. New metric start time is set to be aligned to
the start time of statsd in the next 5min.
So it will ignore events up to 5min.
align puller alarm to minute bundary
Test: unit test
Change-Id: I77ffa3c13de363c780b1000181b9a9b780dd0846
Add anomly detection to Value metric in statsd.
Test: make statsd_test && adb sync data && adb shell data/nativetest64/statsd_test/statsd_test
Test: run cts-dev -m CtsStatsdHostTestCases -t android.cts.statsd.HostAtomTests#testValueAnomalyDetection
Fixes: 70240042
Change-Id: I05cf36495cdfd0ac7aa1a922f0e253a60fda1787
The exception is EventMetricProducer. Each EventMetricProducer will still have a ProtoOutputStream
Because LogEvent comes as a fixed 4K, it's more memory efficient to have an 8k ProtoOutputStream for
storing the events.
Also removed finish() api in MetricProducer, which was intended to use with Dropbox.
Test: statsd_test & dogfood app
Bug: 70393808
Change-Id: I2efe4ecc76a88060a9aa5eb49d1fa6ea60bc5da8
Since there is a separate guardrail for memory used by uid map, we
no longer add the memory from uid map with the memory per each
config's metrics. We also prevent the byte size check from happening
too frequently. In order to mock the MetricsManager, we refactor
some of the existing methods.
Test: Added unit-tests and verified they all pass on marlin.
Change-Id: I15cf105f7d95f4016fdb0443b0a33eebe862cafb
StatsdStats now tracks the number of times an anomaly is detected (per
config, per alert name).
Also adds a configKey to AnomalyTracker, which is needed, not only for
statsdstats, but also (in the future) for reporting the header
information to incidentd.
Bug: 67978682
Test: adb shell data/nativetest64/statsd_test/statsd_test
Change-Id: Ib254db7e1edb4f0f193f4772d17f14934cdf7e30
