We now keep track of all the threads that are stopped, and
print stacks for all of them. Also more threads are now adding
themselves to the watchdog.
Unfortunately the stack we get from threads is far less useful
than the stacks from the ANR report, because these don't include
any information about the lock the thread is blocked on and what
thread is holding that lock. For example, here is a test of the
log output from causing a hang in the system process:
W/Watchdog( 5205): *** WATCHDOG KILLING SYSTEM PROCESS: com.android.server.am.ActivityManagerService, main thread
W/Watchdog( 5205): foreground thread stack trace:
W/Watchdog( 5205): at com.android.server.am.ActivityManagerService.monitor(ActivityManagerService.java:14333)
W/Watchdog( 5205): at com.android.server.Watchdog$HandlerChecker.run(Watchdog.java:142)
W/Watchdog( 5205): at android.os.Handler.handleCallback(Handler.java:730)
W/Watchdog( 5205): at android.os.Handler.dispatchMessage(Handler.java:92)
W/Watchdog( 5205): at android.os.Looper.loop(Looper.java:137)
W/Watchdog( 5205): at android.os.HandlerThread.run(HandlerThread.java:61)
W/Watchdog( 5205): main thread stack trace:
W/Watchdog( 5205): at com.android.server.am.ActivityManagerService.broadcastIntent(ActivityManagerService.java:12252)
W/Watchdog( 5205): at android.app.ContextImpl.sendBroadcastAsUser(ContextImpl.java:1158)
W/Watchdog( 5205): at com.android.server.DropBoxManagerService$3.handleMessage(DropBoxManagerService.java:161)
W/Watchdog( 5205): at android.os.Handler.dispatchMessage(Handler.java:99)
W/Watchdog( 5205): at android.os.Looper.loop(Looper.java:137)
W/Watchdog( 5205): at com.android.server.ServerThread.initAndLoop(SystemServer.java:1050)
W/Watchdog( 5205): at com.android.server.SystemServer.init2(SystemServer.java:1125)
W/Watchdog( 5205): at com.android.server.SystemServer.init1(Native Method)
W/Watchdog( 5205): at com.android.server.SystemServer.main(SystemServer.java:1116)
W/Watchdog( 5205): at java.lang.reflect.Method.invokeNative(Native Method)
W/Watchdog( 5205): at java.lang.reflect.Method.invoke(Method.java:525)
W/Watchdog( 5205): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:774)
W/Watchdog( 5205): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:590)
W/Watchdog( 5205): at dalvik.system.NativeStart.main(Native Method)
I/Process ( 5205): Sending signal. PID: 5205 SIG: 9
Change-Id: I8ff9892d8d072d8dc599a73de4bdb75e3b1a6e97
This was implemented for 1.0 out of paranoia about the possibility
of needing to schedule reboots of the system after it went out in
the field, which I am happy to say was never ever used.
Let's get rid of it.
A small stub is left, to still service the API that allows people
to send a reboot broadcast to have the system immediately reboot.
Change-Id: I6731b24a28340e50c8015f8cb28e48f74f69f9b7
Adds an optimization for checking whether a looper is stuck,
with a new Looper method to see if its thread is currently
idle. This will allow us to put a large number of loopers
in the monitor efficiently, since we generally won't have to
do a context switch on each of them (since most looper threads
spend most of their time idle waiting for work).
Also change things so the system process's main thread
is actually running on the main thread. Because Jeff
asked for this, and who am I to argue? :)
Change-Id: I12999e6f9c4b056c22dd652cb78c2453c391061f
IActivityController has a new callback which the Watchdog calls
when it detects that the system process is hung. This may be
use full monkey. All hail the monkey!
Also add a new private feature to Binder to be able to turn off
all incoming dump() calls to a process. The watchdog uses this
when it reports it is hung, so that if someone, say, wants to
collect a bug report at this point they won't get stuck waiting
for things that are all busted.
Change-Id: Ib514d97451cf3b93f29e194c1954e29f948c13b1
This introduces four generic thread that services can
use in the system process:
- Background: part of the framework for all processes, for
work that is purely background (no timing constraint).
- UI: for time-critical display of UI.
- Foreground: normal foreground work.
- IO: performing IO operations.
I went through and moved services into these threads in the
places I felt relatively comfortable about understanding what
they are doing. There are still a bunch more we need to look
at -- lots of networking stuff left, 3 or so different native
daemon connectors which I didn't know how much would block,
audio stuff, etc.
Also updated Watchdog to be aware of and check these new
threads, with a new API for other threads to also participate
in this checking.
Change-Id: Ie2f11061cebde5f018d7383b3a910fbbd11d5e11
A race condition leading to false positive detections might occur
when the monitoring thread executes very fast and terminates before
the sending thread entered the synchronized section.
Change-Id: I6fe686f8f12393e11fa18326508af5b73738f9d7
Author: Jean-Christophe PINCE <jean-christophe.pince@intel.com>
Signed-off-by: Jean-Christophe PINCE <jean-christophe.pince@intel.com>
Signed-off-by: Shuo Gao <shuo.gao@intel.com>
Signed-off-by: Bruce Beare <bruce.j.beare@intel.com>
Signed-off-by: Jack Ren <jack.ren@intel.com>
Author-tracking-BZ: 81644
The HeartbeatHandler for the System Server Watchdog has been running
on the wrong thread due to a race condition in initialization. It's
designed to run on ServerThread, so that it can catch lockups in the
main looper of the System Server. It has been running on
ActivityManagerThread instead, so it does not detect lockups on the
ServerThread as it should.
ActivityManagerService is calling Watchdog.getInstance() before
ServerThread calls Watchdog.getInstance().init(), so the handler is
being bound to the ActivityManagerThread instead of the ServerThread.
Explicitly bind HeartbeatHandler to ServerThread, so that the Watchdog
catches lockups on this critical thread.
Change-Id: Iccb184ac3adb817feb86ed4ee0e50e443bf74636
Bug b/7638530 may be caused by a kernel deadlock when killing
processes under low memory conditions. Write to /proc/sysrq-trigger
to get a kernel log of blocked tasks before killing the system server.
Bug: 7638530
Change-Id: I60df324ad4affdadbf13650099dc4dfb38722420
7267494 Calendar is not syncing
Check for whether a content provider is dead before returning
it. This is kind-of a band-aid, but probably the right thing
to do; I'm just not sure exactly the full details of why this
problem is happening. Hopefully this "fixes" it, though I don't
have a way to repro to tell.
7212347 System power off dialog is only visible to user 0
Make it visible. Also turn on some battery debugging stuff and
clean it up so we can just keep it.
Change-Id: I5add25bf2a763c8dfe1df23bc5c753a9ea5d157a
Some of the BatteryService state was being locked
sometimes and it wasn't at all consistent.
Bug: 7158734
Change-Id: I46e75f66fde92c5a577a80a6bd99c9573066f3c1
If the watchdog detects a problem the system server process
is killed, that is followed by a crash. Because the crash is
done after the system server process is killed, the crash
don't contain info about the system server.
This improvement will make sure that the system is crashed
before the system server process is killed.
Behavior is only changed for eng and userdebug builds.
Change-Id: I9f1c8fd8b03d0114032ed44fb582705ad0b49733
The major goal of this rewrite is to make it easier to implement
power management policies correctly. According, the new
implementation primarily uses state-based rather than event-based
triggers for applying changes to the current power state.
For example, when an application requests that the proximity
sensor be used to manage the screen state (by way of a wake lock),
the power manager makes note of the fact that the set of
wake locks changed. Then it executes a common update function
that recalculates the entire state, first looking at wake locks,
then considering user activity, and eventually determining whether
the screen should be turned on or off. At this point it may
make a request to a component called the DisplayPowerController
to asynchronously update the display's powe state. Likewise,
DisplayPowerController makes note of the updated power request
and schedules its own update function to figure out what needs
to be changed.
The big benefit of this approach is that it's easy to mutate
multiple properties of the power state simultaneously then
apply their joint effects together all at once. Transitions
between states are detected and resolved by the update in
a consistent manner.
The new power manager service has is implemented as a set of
loosely coupled components. For the most part, information
only flows one way through these components (by issuing a
request to that component) although some components support
sending a message back to indicate when the work has been
completed. For example, the DisplayPowerController posts
a callback runnable asynchronously to tell the PowerManagerService
when the display is ready. An important feature of this
approach is that each component neatly encapsulates its
state and maintains its own invariants. Moreover, we do
not need to worry about deadlocks or awkward mutual exclusion
semantics because most of the requests are asynchronous.
The benefits of this design are especially apparent in
the implementation of the screen on / off and brightness
control animations which are able to take advantage of
framework features like properties, ObjectAnimator
and Choreographer.
The screen on / off animation is now the responsibility
of the power manager (instead of surface flinger). This change
makes it much easier to ensure that the animation is properly
coordinated with other power state changes and eliminates
the cause of race conditions in the older implementation.
The because of the userActivity() function has been changed
so that it never wakes the device from sleep. This change
removes ambiguity around forcing or disabling user activity
for various purposes. To wake the device, use wakeUp().
To put it to sleep, use goToSleep(). Simple.
The power manager service interface and API has been significantly
simplified and consolidated. Also fixed some inconsistencies
related to how the minimum and maximum screen brightness setting
was presented in brightness control widgets and enforced behind
the scenes.
At present the following features are implemented:
- Wake locks.
- User activity.
- Wake up / go to sleep.
- Power state broadcasts.
- Battery stats and event log notifications.
- Dreams.
- Proximity screen off.
- Animated screen on / off transitions.
- Auto-dimming.
- Auto-brightness control for the screen backlight with
different timeouts for ramping up versus ramping down.
- Auto-on when plugged or unplugged.
- Stay on when plugged.
- Device administration maximum user activity timeout.
- Application controlled brightness via window manager.
The following features are not yet implemented:
- Reduced user activity timeout for the key guard.
- Reduced user activity timeout for the phone application.
- Coordinating screen on barriers with the window manager.
- Preventing auto-rotation during power state changes.
- Auto-brightness adjustment setting (feature was disabled
in previous version of the power manager service pending
an improved UI design so leaving it out for now).
- Interpolated brightness control (a proposed new scheme
for more compactly specifying auto-brightness levels
in config.xml).
- Button / keyboard backlight control.
- Change window manager to associated WorkSource with
KEEP_SCREEN_ON_FLAG wake lock instead of talking
directly to the battery stats service.
- Optionally support animating screen brightness when
turning on/off instead of playing electron beam animation
(config_animateScreenLights).
Change-Id: I1d7a52e98f0449f76d70bf421f6a7f245957d1d7
- Collect data at better times.
- Collect per-thread CPU usage as soon as possible after the ANR, and print
in log.
- Based on new per-thread CPU usage, limit the number of processes we
collect stacks from to not include inactive not interesting procs.
- Improve the way ProcessStats compute and reports its data.
Change-Id: I12b17fb47d593d175be69bb792c1f57179bf4fdf
This was originally written as an in-case-we-need-it facility, but was
never actually used in production. It also soaked up a surprising amount
of cpu on occasion, as well as doing sketchy things like demoting the
system_server's primary looper thread to the background cgroup at times.
Change-Id: I9a81a8d1e9caea9e0a1277d97785fe96add438d7
The kernel threads are appended to the usual /data/anr/traces.txt file
and dropboxed along with the usual Dalvik stack dumps.
Change-Id: I120f1f5ee54c965efe9ac0c7f40fdef56385f1fa
NOTE: this change depends on the kernel publishing /proc/$PID/stack
This gives us a snapshot of what the system process was doing after 30 seconds
of apparent inactivity as well as after 1 minute, to help distinguishing actual
deadlocks from too-slow progress, livelock, etc.
Change-Id: I19758861d1b25f298e88788e8f1c7ec7bf828823
Use new observer before rebooting and shutting down.
Add some unit tests for unmount and shutdown code paths
Fix registering/unregistering part in MountService
Use ShutdownThread in PowerManager.reboot()
Add reboot support to ShutdownThread.
Remove MountService code from PowerManagerService.java and Power.java.
Clean shutdown/reboot is handled exclusively by ShutdownThread now.
Change-Id: Iefb157451d3d9c426cb431707b870a873c09123d
Capture stack traces from the system process using the same
mechanism as ANRs (which will initialize traces.txt, etc).
Also record the watchdog reset in the dropbox for uploading.
Bug: 2475557
We can now locate event log tag definitions in individual packages
(and java constants for the tag numbers get auto-generated), so move
all the tags used by the system server into the package.
Applications can now declare that they support small, normal, or
large screens. Resource selection can also be done based on these
sizes. By default, pre-Donut apps are false for small and large,
and Donut or later apps are assumed to support all sizes. In either
case they can use <supports-screens> in their manifest to declare
what they actually support.
