54 Commits

Author SHA1 Message Date
Philip P. Moltmann
e0cf4e6087 Add PermissionGrantRequestResultReported atom
Also add a prototype for a future system-api accessible metrics file
that will be auto-generated from atoms.proto

Test: - ./out/host/linux-x86/bin/statsd_testdrive -p com.google.android.permissoncontroller 170
      - triggered permission request
Bug: 123594188, 123663448
Change-Id: Icede6ff1f12ca79ebad6267c045a4fb3a9955402
2019-02-01 10:22:37 -08:00
Chiachang Wang
d09275059c Add NetworkStack to allowed log sources in statsd_testdrive
Test: - atest statsd_test
      - manual test with statsd_testdrive
Bug: 123260425
Change-Id: Id23162efbdee6979cfa1da5862ff874eed825926
2019-01-23 15:48:34 +08:00
arangelov
fd9cdcbdee Add ManagedProvisioning to allowed log sources in statsd_testdrive.
Test: manual
Bug: 122346015
Change-Id: I3f4ec145bb331493efb850574a18a50bd893954c
2019-01-04 12:25:43 +00:00
Ben Murdoch
7b58df821a Refactor statsd_testdrive to support recording N atoms at the same time.
Found this useful in my testing.

Test: make and run statsd_testdrive

Change-Id: I3a9ff22119e79e89890c59be514f9f778aaced73
2018-12-20 19:06:12 +00:00
Ben Murdoch
d36cbf9548 Allow events from AID_LMKD in statsd_testdrive.
Test: ran statsd_testdrive for atom 51 (lmk_kill_occurred), saw events
Change-Id: I2b9c9817813dcf4c0c7fd69a984726c6ef24db20
2018-12-20 11:45:15 +00:00
Bookatz
8d349f6f64 LocalDrive fails fast on P devices
Makes LocalDrive quit for devices running P or earlier; the
statsd of such devices lack the --keep_data flag and therefore will
fail anyway, so we may as well the user why and fail fast.

Test: Manual testing on userdebug and userbuild, P and Q
Change-Id: Ia26528383c9d35732ffd819ed2e2ac6f3bb41b34
2018-12-17 17:31:02 -08:00
Bookatz
17fe8de6b3 Allow statsd_testdrive to work in root
LocalDrive needs to handle the situation whereby the caller does 'adb
root' in between calls, and therefore runs everything as shell.

TestDrive is a single-call, so it can just run as the caller. That's
what it used to do, but a recent refactoring made some of its calls via
shell, causes an error. This fixes that.

Test: manual confirmation that it worked for both shell and root
Change-Id: I5e31cdd59d61290a480cb6fae107170616daabc0
2018-12-13 11:13:24 -08:00
Bookatz
3e90658294 statsd local tool
Adds a tool for local usage of statsd. The tool can:
-upload a config from a file
-get the report data from statsd
Both the config and the report can be either in binary or human-readable
format, as specified.

Usage:
make statsd_localdrive
./out/host/linux-x86/bin/statsd_localdrive

Also, adds the ability to specify whether dump-report should also erase
the data when it returns it. A test for this is added.

Test: make -j8 statsd_test && adb sync data && adb shell data/nativetest64/statsd_test/statsd_test
Test: make statsd_localdrive && ./out/host/linux-x86/bin/statsd_localdrive <commands>
Bug: 77909781
Change-Id: I9a38964988e90c4158a555f41879534267aadd32
2018-12-12 11:42:39 -08:00
Colin Cross
bc2dc31c4d Convert statsd and incidentd to Android.bp
See build/soong/README.md for more information.

Fixes: 110563449
Test: m checkbuild
Change-Id: I8c80878cec46c9299234c6f4c1e0e19efc0a221e
2018-10-30 13:30:23 -07:00
Chenjie Yu
328432d887 make testdrive log message less verbose
1. Use custom TestDriveFormatter for log output.
It should be easy to fine tune it from here.
2. Fine tune logging level

Now the output looks like this:

./out/host/linux-x86/bin/statsd_testdrive 10
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.protobuf.Utf8$UnsafeProcessor (file:/android2/master/out/host/linux-x86/framework/statsd_testdrive.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of com.google.protobuf.Utf8$UnsafeProcessor
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Pushed the following config to statsd:
id: 12345
event_metric {
  id: 1111
  what: 1234567
}
atom_matcher {
  id: 1234567
  simple_atom_matcher {
    atom_id: 10
  }
}

......

hash_strings_in_metric_report: false

Now please play with the device to trigger the event. All events should be dumped after 1 min ...
Success!
Got following metric data dump:
metric_id: 1111
event_metrics {
  data {

.....

        state: RELEASE
      }
    }
  }
}

Test: manual test
Change-Id: I9fcb09bbd1663f0eae85d428ed45a3b139f62782
2018-10-24 16:26:05 -07:00
Chenjie Yu
dd7dc4f7de local test drive tool for statsd
example usage:

$./out/host/linux-x86/bin/statsd_testdrive 10009
Uploading the following config to statsd:
id: 56789
gauge_metric {
  id: 2222
  what: 1234567
  gauge_fields_filter {
    include_all: true
  }
  bucket: ONE_MINUTE
}
atom_matcher {
  id: 1234567
  simple_atom_matcher {
    atom_id: 10009
  }
}
allowed_log_source: "AID_GRAPHICS"
allowed_log_source: "AID_INCIDENTD"
allowed_log_source: "AID_STATSD"
allowed_log_source: "AID_RADIO"
allowed_log_source: "com.android.systemui"
allowed_log_source: "com.android.vending"
allowed_log_source: "AID_SYSTEM"
allowed_log_source: "AID_ROOT"
allowed_log_source: "AID_BLUETOOTH"
hash_strings_in_metric_report: false

Success!

Success!

Now wait for 2 minutes ...
Success!

Got following metric data:
metric_id: 2222
gauge_metrics {
  data {
    bucket_info {
      atom {
        cpu_time_per_uid {
          uid: 0
          user_time_millis: 581844000
          sys_time_millis: 135242354000
        }
      }
      elapsed_timestamp_nanos: 610841423012521
      start_bucket_elapsed_millis: 610841423
      end_bucket_elapsed_millis: 610880290
    }
    bucket_info {
      atom {
        cpu_time_per_uid {
          uid: 0
          user_time_millis: 581904000
          sys_time_millis: 135251114000
        }
      }
      elapsed_timestamp_nanos: 610880295723373
      bucket_num: 65
    }
    bucket_info {
      atom {
        cpu_time_per_uid {
          uid: 0
          user_time_millis: 581944000
          sys_time_millis: 135264674000
        }
      }
      elapsed_timestamp_nanos: 610940293803881
      start_bucket_elapsed_millis: 610940290
      end_bucket_elapsed_millis: 610961579
    }
  }
}
time_base_elapsed_nano_seconds: 606980290711787
bucket_size_nano_seconds: 60000000000

Success!

Test: manual test
Change-Id: I8ea5249807cf3cf6971e3bbc8696ff33025ba8d7
2018-10-15 15:47:17 -07:00
Yangster-mac
a8a304458b Fix the typos and naming convention in atoms.proto
Test: cts,statsd
BUG: b/117708491

Change-Id: Ib381ef66ae9925938e1f70b9a8869ef008e3d335
2018-10-14 22:04:52 -07:00
Bookatz
4f71629002 StatsManager throws exceptions
When StatsManager fails to connect to statsd, it now throws an exception
for the caller to catch. It also throws an exception of the config being
added is of an unreadable format.

Due to backwards compatibility issues, the old APIs could not be
changed, so new ones were made to replace the old ones. The old ones are
now temporary and will be removed when the compatibility issue is
resolved.

Bug: 77648233
Test: gts-tradefed run gts-dev --module GtsStatsdHostTestCases
Change-Id: Ibea05883a29b9b3ef9927d2f8fe295eb99832ab7
2018-04-10 19:07:32 -07:00
Anton Hansson
d137c872cd frameworks/base: Set LOCAL_SDK_VERSION where possible.
This change sets LOCAL_SDK_VERSION for all packages where
this is possible without breaking the build, and
LOCAL_PRIVATE_PLATFORM_APIS := true otherwise.

Setting one of these two will be made required soon, and this
is a change in preparation for that. Not setting LOCAL_SDK_VERSION
makes the app implicitly depend on the bootclasspath, which is
often not required. This change effectively makes depending on
private apis opt-in rather than opt-out.

Test: make relevant packages
Bug: 73535841
Exempt-From-Owner-Approval: Global cleanup
Change-Id: I26458e41ecb84de91ac9a356a5d4bafb44f463c1
2018-03-07 11:42:04 +00:00
Yao Chen
20e9e6231a Reduce statsd binary size from 730KB-> 664KB
1. StatsdStats does not use the proto object in memory anymore.
2. lite_static -> lite
3. don't use sstream

Bug: 72129300
Test: statsd_test

Change-Id: I8a5adaf222d4d5034e8bf115215fb6dd5f042cac
2018-02-28 16:20:54 -08:00
Yao Chen
9c1debe330 Add annotation to atoms that represent a state change in atoms.proto
+ A state change atom can have one exclusive state field, and any
  number of primary key fields.

  When there is primary key in the atom, it means the state belongs to the primary key.
  For example,
  message UidProcessStateChanged {
    optional int32 uid = 1 [(stateFieldOption).option = PRIMARY];
    optional android.app.ProcessStateEnum state = 2 [(stateFieldOption).option = EXCLUSIVE];
  }

  When there is no primary key fields in the atom, the state is global.
  For example,
  message ScreenStateChanged {
     optional android.view.DisplayStateEnum state = 1 [(stateFieldOption).option = EXCLUSIVE];
  }

+ The annotation is consumed by stats_log_api_gen to generate a static map from the state
  atoms to its primary fields, and exclusive fields

+ stats_log.proto is splitted into 2 proto files, because statsd needs proto lite, and c++
  lite proto library cannot properly ignore the field options which requires full proto.

 This CL doesn't change any logic in the statsd yet. A separate CL will use the field option
 information to correctly track the state.

Test: added unit tests in stats_log_api_gen_test. and statsd_test pases.
Change-Id: I9e8a979fe81ba60efd4d854bb7087ce4b2b147ec
2018-02-21 16:46:56 -08:00
Yangster-mac
330af58f2b Use elapsed realtime instead of times based on wall clock, which can jump around and go backwards.
Test: statsd unit test passed

Change-Id: Ib541df99231e171b3be2a24f75632693e36da90e
2018-02-13 23:30:39 -08:00
TreeHugger Robot
959b80f191 Merge "Modifies statsd output for start and end times." 2018-02-08 20:24:31 +00:00
David Chen
1604957800 Modifies statsd output for start and end times.
We include the start of when the last dump occurred and the current
timestamp. These timestamps are shared across all metrics, so
there's no advantage in duplicating these numbers across all metrics.

Also, we should use elapsed realtime instead of times based on wall
clock, which can jump around and go backwards.

Test: Test that statsd can still build and
adb shell cmd stats dump-report doesn't crash.
Change-Id: I819e5643cee75dfa3e78a58f94c9d61ededa78d7
2018-02-08 09:59:45 -08:00
David Chen
661f791a25 Statsd notifies listener with pendingintent.
Previously, statsd would inform interested listeners that it's time
to collect data via a protected broadcast. However, the preferred
solution is to pass a PendingIntent via a separate setter. Whenever
statsd wants the listener to call getData, StatsCompanionService
will trigger the pending intent.

Test: Tested in marlin-eng that functionality works as expected with
dogfood app.
Bug: 72562867
Change-Id: Ibcfcd5072a1a78947f8a7cbcd0bc429b54351da3
2018-02-07 16:16:23 -08:00
Bookatz
9086762d56 Use statsd constants instead of literals
Instead of writing StatsLog.write(StatsLog.NAME, 1), we replace the 1
with the appropriate StatsLog constant.

Bug: 72749863
Test: still compiles

Change-Id: I68c8206de49df7e77ee66333dd633e4b70c7b226
2018-01-31 17:02:20 -08:00
Yao Chen
6d6619a44e Make statsd dogfood app to be a system app.
And update the baseline config.

Bug: 72710440
Test: manual
Change-Id: Icc37d1d688d3788e9c9e98a594b9532889ffeea9
2018-01-30 15:53:15 -08:00
Bookatz
1a1b0464cb Statsd and frameworks reference proto enums
For frameworks constants that don't have intrinsic meaning (i.e. their actual
value and order don't matter), so that it is unlikely that their values
will be changed:
This cl introduces proto enums representing some constants found in
the Android codebase, and connects the two.

By using the Proto enum as the source-of-truth, it means that Java and
proto can be kept in sync. Otherwise, when the Java frameworks code
changes, it silently breaks the protos from working properly, since the enums
are wrong. By having the Java code reference the proto enums, it ensures
that everything is in sync. The values of the constants are unchanged.
But future changes to these constants will need to be done in the proto
file, which the Java file merely references.

The protos are necessary for incidentd and statsd and, in the future,
possibly dumpsys. In this way, the logging mechanism is much less likely
to get broken when new constants are added, and we can be ensured that
the logging accurately reflects the underlying codebase.

Bug: 69478930
Test: cts-tradefed run cts-dev -m CtsStatsdHostTestCases
Test: cts-tradefed run cts-dev -m CtsIncidentHostTestCases
Change-Id: If79032c34b2799db1e3e70cb47b1312fd72092b9
2018-01-25 15:47:57 -08:00
Bookatz
c697797d43 Statsd broadcast subscriber
Allows a uid that uploads a statsd config to additionally
register a BroadcastSubscriber with statsd. If statsd
detects an anomaly (according to the config's Alert),
statsd can inform a BroadcastSubscriber provided in the config.
The config uses a subscriberId (just an int) to identify the
BroadcastSubscriber. It then uses StatsManager.setBroadcastSubscriber
to associate that subscriberId with a given PendingIntent.
Then, when the anomaly is detected, statsd sends a broadcast
using that PendingIntent, alerting whoever was specified by
the config/setBroadcastSubscriber.

Bug: 70356901
Test: cts-tradefed run cts-dev -m CtsStatsdHostTestCases -t android.cts.statsd.alert.BroadcastSubscriberTests
Change-Id: I4d9ea9a6c8a85e61fadfd99c1513c55abbadd5e9
2018-01-24 13:08:19 -08:00
Yangster
b84e8b1fcf Sync statsd google3 protos to aosp.
Test: statsd unit tests passed
Change-Id: I66b05478b7c5b0f9fdd6386b2e2ba7c38a3f9b88
2018-01-23 16:42:24 +00:00
Yao Chen
9bcda3e072 Make StatsdDogfood build again.
Test: manual
Change-Id: I14614c060cf0bfb49809315b3ef0482d8a64a1a5
2018-01-21 13:01:17 -08:00
Primiano Tucci
e4d44919aa Statsd -> Perfetto integration
This CL invokes the perfetto client utility to start the
collection of a trace when an anomaly that subscribed to
Perfetto is detected. The code simply spawns the
/system/bin/perfetto client and passes the trace config via
stdin. The client takes care of the dropbox upload.
The CollectPerfettoTraceAndUploadToDropbox() function does
NOT wait for the full trace collection (in order to avoid
blocking statsd) and instead returns immediately after having
spawned perfetto.

Change-Id: I4f02067bad7a46ede7b6e4841cdcf381c1a4e2a7
Bug: 71795552
2018-01-19 23:50:07 +00:00
Yangster-mac
468ff04fd4 Proto change for dimension from condition.
Test: all statsd unit tests and cts tests passed.

Change-Id: I955143aca7c76e2e22a10d7ba42eb797bd32e147
2018-01-17 21:19:45 -08:00
Stefan Lafon
47c186ca32 More adjustments to the loadtest app.
- Add ability to set the replication factor by intent.
- Add shell script to automate the loadtesting.

Test: Loadtest app works.

Change-Id: I7263a3c23049f3ed20144e4e15b6a54ac5fddcc8
2018-01-10 09:27:38 -08:00
Stefan Lafon
c6f2fa2700 Fix a few things with the loadtest app.
Test: Loadtest app is running fine.

Change-Id: I834f6c618f5a6b0b17ecd5592e2ab2209ce7b94a
2018-01-05 17:02:56 -08:00
TreeHugger Robot
efc331b287 Merge "Use TimeUnit enum to specify the bucket size." 2018-01-04 22:53:29 +00:00
Yangster-mac
b814481ad1 Use TimeUnit enum to specify the bucket size.
Test: all statsd unit test passed

Change-Id: I4f6b80ba2f8c984b06e46e6de6df3e546e99a968
2018-01-04 13:01:14 -08:00
Yao Chen
147ce60278 use only string type in the log source whitelist.
+ predefined "AID_X" will be provided as string type to statsd, and we will translate
  to integer uid using the static map.

Test: statsd_test

Change-Id: Ie47d8481e0c456457e6881ebb9cb4ce008e772b8
2018-01-04 09:57:03 -08:00
Yangster-mac
94e197cceb 1/ Change all "name" to id in statsD.
2/ Handle Subscription for alert.
3/ Support no_report_metric

Bug: 69522276
Test: all statsd unit tests passed.
Change-Id: I851b235f2d149b8602b0cad632d5bf541962f40a
2018-01-03 15:34:00 -08:00
Yangster-mac
2087716f2b 1/ Support nested message and repeated fields in statsd.
2/ Filter gauge fields by FieldMatcher.
3/ Wire up wakelock attribution chain.
4/ e2e test: wakelock duration metric with aggregated predicate dimensions.
5/ e2e test: count metric with multiple metric condition links for 2 predicates and 1 non-sliced predicate.

Test: statsd unit test passed.

Change-Id: I89db31cb068184a54e0a892fad710966d3127bc9
2018-01-01 10:01:36 -08:00
Yao Chen
0d7d4d4f30 Modify Loadtest app to be able to exclude certain metric types.
Test: manual
Change-Id: Iba025e4d6cd7c15a231550d382768aa7661ccc86
2017-12-22 09:13:37 -08:00
Yao Chen
d10f7b1c7b Add log source filtering in statsd to filter out spams.
+ Add log source whitelist in StatsdConfig
+ Some changes in UidMap API. Listener needs to be wp instead of sp.
+ Update dogfood app config to have log source
+ Increase the stats service thread pool size to 10 (9+1).

TODO: add unit tests(b/70805664). This unit test takes some time to write.

Test: statsd_test & manual

Change-Id: I129b1cc13db5114db7417580962bd7cc4438519d
2017-12-20 18:45:43 -08:00
TreeHugger Robot
e6abcbed99 Merge "Turn off verbose logging, remove the fake config." 2017-12-17 17:01:36 +00:00
Yao Chen
3c0b95ceb5 Turn off verbose logging, remove the fake config.
Test: statsd_test & manual
Change-Id: I43068c8db006cbbeccacaa9d8426ab4d0394313a
2017-12-16 14:34:20 -08:00
David Chen
0a368b2c39 Adds API for apps to push events to statsd.
This API allows app to construct custom metrics based on labels
chosen by the app developers. Also added some buttons to manually
test this functionality in the dogfood app.

Test: Verified that Android can be built and tested with custom app.
Bug: 69522276
Change-Id: Ifb7abea4c1d62fb435a9cb6f32df12bc2234d82f
2017-12-14 16:19:29 -08:00
Joe Onorato
c9f1a9e3dd Merge "Fix how we build the statsd protos." 2017-12-14 05:33:33 +00:00
Joe Onorato
62c220b20b Fix how we build the statsd protos.
This lets us include frameworks protos, and use the constants
and messages from them.

Change-Id: I609d6e524f780e6a5beea543a68561bede47813e
Test: make
2017-12-13 18:26:45 -08:00
Stefan Lafon
14e076e676 Improvements to loadtest app.
- Support for statsd stats in loadtest app.
- Skeleton for correctness validation.
- New config with pulled atoms in gauge metrics.

Test: Ran statsd.

Change-Id: I0f062d55b8827452cc0421093fc35242dc3afb23
2017-12-11 10:54:22 -08:00
Tej Singh
c542320b4f Copy atoms.proto changes to atoms_copy.proto
Also changed load test app to use enum.ToString instead of big switch
case.

Test: built & ran statsd & cts test. Also built & ran load test app.
Change-Id: I975c75e04100df3c380c2e3a46d3737b81f7e21a
2017-12-07 14:11:23 -08:00
Stefan Lafon
9d25218985 Merge "Track PSS in loadtest." 2017-12-07 20:23:37 +00:00
Stefan Lafon
9709fa26f1 Track PSS in loadtest.
Test: Ran the loadtest. Not changing statsd.

Change-Id: Idc43cba59ec2c9d4213e20b395a083fdda58e8c4
2017-12-07 10:10:48 -08:00
Stefan Lafon
a5b5191351 Rename EventConditionLink as MetricConditionLink in statsd_config.proto.
Test: Ran statsd_test and cts tests.
Change-Id: Ia9a38892854443aa4f17d800a075860fcf874652
2017-12-05 21:51:46 -08:00
Stefan Lafon
12d01fa7c1 Rename more things in statsd_config.proto.
SimpleCondition -> SimplePredicate
simple_condition -> simple_predicate
Condition -> Predicate
condition -> predicate

Test: Ran statsd and statsd_test.

Change-Id: I3115fb51a4dc3cbe5f23535dfc448d5755b9ed7f
2017-12-05 09:20:28 -08:00
Stefan Lafon
b8c9aa8c9c Rename a few things in statsd_config.proto.
This for frameworks/base/

SimpleLogEntryMatcher -> SimpleAtomMatcher.
simple_log_entry_matcher -> simple_atom_matcher
LogEntryMatcher -> AtomMatcher
log_entry_matcher -> atom_matcher

Test: Ran statsd_test.

Change-Id: I7398758e3fab86235897c162884f5ceab3002510
2017-12-04 10:59:23 -08:00
TreeHugger Robot
e1e892d4a0 Merge "Create statsd loadtest app." 2017-12-03 19:47:25 +00:00