1. The helper query bridge service did not have the now capability
to query the screen content.
2. Fixing the build.
bug:8633951
Change-Id: Ief6a3387793710a83b83c18cc6c53b51dcda9bdf
We have APIs that allow an accessibility service to filter key events. The
service has to declare the capability to toggle event filtering in its
manifest and then it can set a flag to toggle the feature at runtime. The
problem was that even if no accessibility service was handling key events
these events were not fed back to the input system.
This change adds a new feature flag in the accessibility input filter that
is set only if at least one service can and wants to filter key events. If
the feature flag is set then the filter will deliver events to services and
,if they are not handled, to the system. This change also cleaned the logic
for updating the input filter.
bug:8713422
Change-Id: I4bc0c1348676569d1b76e9024708d1ed43ceb26a
Since the enable touch exploration capability is dynamically granted by
the user for apps targeting pre-JellybeanMR2 API level, we have to properly
update the accessibility service info for that service and also avoid
caching copies of the service info.
bug:8633951
Change-Id: I83dd1c852706ec55d40cda7209ad842889fb970a
1. UiAutomation#executeAndWaitForEvent method was invoking the passed
runnable while holding the lock which may lead to a deadlock. For
example: a runnable that calls getActivity() gets us into a state
like this.
2. UI automation services did not get all capabilities such a
service can have. Now a UI test service gets all of them.
3. When UiAutomation was exiting for event fired as a result of a
performed action, it was checking whether the received evnet time
is strictly before the time of executing the command that should
fire the event. However, if the execution is fast enough, i.e.
less than one millisecond, then the event time and the execution
time are the same. This was leading to a missed signal in rare
cases.
4. AccessibilityNodeInfoCache was not clearing the relevant state
for accessibility focus clearing event.
5. Accessibility text traversal in TextView was partially using text
and partially content description - broken. Now we are using the
text since for text view and content desc for other views. In other
words, we are using the most precise text we have.
6. AccessibilityManagerService was not granting capabilities of a
UiAutomation service - plainly wrong.
CTS change:https://googleplex-android-review.googlesource.com/#/c/300693/
bug:8695422
bug:8657560
Change-Id: I9afc5c3c69eb51f1c01930959232f44681b15e86
Accessibility services can perform special operations such as retrieve
the screen content, enable explore by touch, etc. To ensure the user
is aware that the service will perform special operations we were using
permissions. However, the special operations cannot be performed unless
the service is really enabled by the user and it is at this point that
we want to notify the user about the service capabilities.
This change adds capability attributes to the accessibility service's
meta-data XML file. The service has to declare the capability and when
it is enabled we show the user the capabilities in the warining dialog.
bug:8633951
Change-Id: Id3442dc71dad018e606888afdc40834682fdb037
This is a regression in which the input filter of the accessibility
manager service is not set if magnification is enabled but accessibility
is not - i.e. no accessibility serivces are enabled. Fixed the logic to
install the input filter if magnification is on but services are not
enabled in addition to services being enabled.
bug:8652765
Change-Id: Ia73e1064035f95ba0f246f4cabcc42d58c12a11f
When something that affects the state of accessibility in the sysytem
changes, we run a reolve method that reloads all relevant information and
if it changed we call a method that makes everyting right. One of the
interesting properties we read is the isntalled accessibliity services.
We are using equals to figure out whether these services have changed
but this is not correct since AccessibilityServiceInfo does not use all
internal members for equals and using all memthis is not reasible since
some of these internal members do not support equals propertly, for
example ResolveInfo.
Therefore, when a package is reinstalled we remove all installed services
from the list of ones we know about which forces them to be reloaded,
thus capturing the current state of a reinstalled package.
bug:8621960
Change-Id: Ie1ef4bf1036d8d6e033cd9528ea2292ce24e5320
It is possible that an accessibility service's package was force stopped
during whose handling the death recipient is unlinked and still get a call
on binderDied since the call was made before we unlink but was waiting on
the lock we held during the force stop handling. Added a check whether the
service is already disconnected and if so do nothing.
bug:8600388
Change-Id: I4a9ca305b9863d986b930a7c1ec8f9006b16a333
On eng builds we have an event consistency verifier to log any
inconsistent event stream states due to mishandling of intercepted
events by an accessibility service. On non-eng builds this verifier
is null and a null check was lacking.
bug:8616711
Change-Id: Ib083a405dfa8340025090a65e50155eb10526a90
If the connected service is not entirely setup when calling the method for
handling a change in the current user state we get a potential NPE since
the management method may have discarded the service, thus nullifying the
connection to it. Now the service is fully configured before calling the
state change management method.
bug:8600489
Change-Id: Ib0bf7c6d575e15c620da419d43ece22f4187fd34
Now that we have gestures which are detected by the system and
interpreted by an accessibility service, there is an inconsistent
behavior between using the gestures and the keyboard. Some devices
have both. Therefore, an accessibility service should be able to
interpret keys in addition to gestures to provide consistent user
experience. Now an accessibility service can expose shortcuts for
each gestural action.
This change adds APIs for an accessibility service to observe and
intercept at will key events before they are dispatched to the
rest of the system. The service can return true or false from its
onKeyEvent to either consume the event or to let it be delivered
to the rest of the system. However, the service will *not* be
able to inject key events or modify the observed ones.
Previous ideas of allowing the service to say it "tracks" the event
so the latter is not delivered to the system until a subsequent
event is either "handled" or "not handled" will not work. If the
service tracks a key but no other key is pressed essentially this
key is not delivered to the app and at potentially much later point
this stashed event will be delivered in maybe a completely different
context.The correct way of implementing shortcuts is a combination
of modifier keys plus some other key/key sequence. Key events already
contain information about which modifier keys are down as well as
the service can track them as well.
bug:8088812
Change-Id: I81ba9a7de9f19ca6662661f27fdc852323e38c00
If no accessibility services are enabled, we disable the
accessibility event firing to save resources. When the last
such services is disabled the system was not unbinding. As
a result the user was seeing the touch exploration enable
dialog when the service that requested it is disabled. Also
there is one service the system is bound to that is not used.
bug:8439191
Change-Id: I6f37f2573a815bfb29870298aa0abbb1fa105588
UiAutomation registers a fake accessibility service to introspect
the screen. Upon the death of the shell process that started an
instrumentation in which a UiAutomation resides the connection
between the UiAutomation and the system is kept alive allowing
the instrumentation to introspect the screen even after the death
of the shell process.
bug:8285905
Change-Id: I1a16d78abbea032116c4baed175cfc0d5dedbf0c
If an accessibility service is connected but already removed
from the list of connecting services we get a NPE since the
call to set the service connection is made over a null
remove interface. Note that service connection is asynchronous.
bug:8229877
Change-Id: I7b05f219dd0c1da6286ee4ec90b1ef310908545d
When an accessibility service connects we get a callback in
which we either add the service, if this service is in the list
of connecting services (we still want the service to connect),
or we unbind and clear the state, if the service is no longer in
the list of connecting services (we do not want this service to
connect because something change between the bind request and
the connection callback).
The problem is that when the service connects and it is not in
the list of connecting services on service connected we called
the clean up code before the connection was complete. However,
the clean up code expects fully configured services. Now we
fully connect the service and in case there is a problem -
disconnect it.
bug:8232627
Change-Id: I939e544e31ffc1406035265a012c180f2ca95d7c
On user switch the transient state of the old user was not cleared
which means that when we switch back to this user the operational
state such as which event types were dispatched, what state was sent
to local managers, etc is stale. This leads to semi-updated state
and broken behavior. Now if the user becomes inactive, we are clearing
all transient state which will be recreated when the user becomes
active.
bug:8196652
Change-Id: Ie9e0d712b6d567e5074b328f1bb87afaa5395c06
The UI test automation service was not removed from the list of
enabled and installed service where it was explicitly added on
registration. This was leaving the accessibility manager service
in an inconsistent state.
bug:8185435
Change-Id: Ice17cdef361fe98ce34f8dd01ec11dbad6c4d0c2
1. The accessibility manager service updates its internal state
based on which settings are enabled, what accessibility services
are installed and what features are requested by the enabled
services. It was trying to do the minimal amount of work to
react to contextual changes like these which resulted in missed
cases and complex code. Now there is a single method that reads
the contextual information and single method that reacts to
contextual changes. This makes the code much easier to maintain.
2. The accessibility manager service was not updating its internal
state when requested features from accessibility services change.
It was relying on changing system settings and reacting to the
settings change. This is problematic since the internal state is
not updated atomically which leads to race condition bugs. For
example, if touch exploration is enabled and a service requests
it is disabled, the internal state will not be updated but a
request for a settings change will be made. Now while the settings
change is propagating another request form the same service
comes to enable touch exploration but the system incorrectly
thinks touch exploration is enabled. At the end the feature is
disabled even though it was requested.
3. Fixed a potential NPE if the accessibility input filter's event
handler was nullified between processing two event batches.
4. Fixed a bug where, if magnification is enabled, it does not work
on the settings screen since the magnified bounds are not pushed
from the window manager to the accessibility manager.
Change-Id: Idf629a06480e12f0d88372762df6c024fe0d7856
We use an input filter to manipulate the event stream in accessibility
mode. Some input events, i.e. touch and hover events, are delivered
to a touch explorer, if touch exploration is enabled, and to a magnifier,
if screen magnification is enabled. It is possible that at the moment
each of these features is enabled we are in the middle of a touch or
hover gesture. The touch explorer and screen magnifier expect to receive
an event stream that starts with an event that denotes the stream start.
This change ensures that hover or touch events are dispatched to the
touch explorer and the magnifier only after the start of the first
well-formed hover or touch sequence.
Change-Id: I8cd0ad8e1844c59fd55cf1dfacfb79af6a8916df
Currently text editing is pretty hard (certain operations even
impossible) for a blind person. To address the issue this change
adds APIs that enable an accessibility service to perform basic
text editing operations such as copy, paste, cut, set selection,
extend selection while moving at a given granularity.
The new APIs enable an accessibility service to expose a gesture
driven efficient text editing facility.
bug:8098384
Change-Id: I82b200138a3fdf4c0c316b774fc08a096ced29d0
Currently we have an "enhance web accessibility" setting that has to be
enabled to make sure web content is accessible. We added the setting to
get user consent because we are injecting JavaScript-based screen-reader
pulled from the Google infrastructure. However, many users do not know
that and (as expected) do not read the user documentation, resulting in
critique for lacking accessibility support in WebViews with JavaScript
enabled (Browser, Gmail, etc).
To smoothen the user experience now "enhance web accessibility" is a
feature an accessibility plug-in can request, similarly to explore by
touch. Now a user does not need to know that she has to explicitly
enable the setting and web accessibility will work out-of-the-box.
Before we were showing a dialog when a plug-in tries to put the device
in a touch exploration mode. However, now that we have one more feature
a plug-in can request, showing two dialogs (assume a plug-in wants both
features) will mean that a user should potentially deal with three
dialogs, one for enabling the service, and one for each feature. We
could merge the dialogs but still the user has to poke two dialogs.
It seems that the permission mechanism is a perfect fit for getting
user permission for an app to do something, in this case to enable
an accessibility feature. We need a separate permission for explore
by touch and enhance web accessibility since the former changes the
interaction model and the latter injects JavaScript in web pages. It
is critical to get user consent for the script injection part so we
need a well-documented permission rather a vague umbrella permission
for poking accessibility features. To allow better grouping of the
accessibility permissions this patch adds a permission group as well.
bug:8089372
Change-Id: Ic125514c34f191aea0416a469e4b3481ab3200b9
This change adds APIs support for implementing UI tests. Such tests do
not rely on internal application structure and can span across application
boundaries. UI automation APIs are encapsulated in the UiAutomation object
that is provided by an Instrumentation object. It is initialized by the
system and can be used for both introspecting the screen and performing
interactions simulating a user. UI test are normal instrumentation tests
and are executed on the device.
UiAutomation uses the accessibility APIs to introspect the screen and
a special delegate object to perform privileged operations such as
injecting input events. Since instrumentation tests are invoked by a shell
command, the shell program launching the tests creates a delegate object and
passes it as an argument to started instrumentation. This delegate
allows the APK that runs the tests to access some privileged operations
protected by a signature level permissions which are explicitly granted
to the shell user.
The UiAutomation object also supports running tests in the legacy way
where the tests are run as a Java shell program. This enables existing
UiAutomator tests to keep working while the new ones should be implemented
using the new APIs. The UiAutomation object exposes lower level APIs which
allow simulation of arbitrary user interactions and writing complete UI test
cases. Clients, such as UiAutomator, are encouraged to implement higher-
level APIs which minimize development effort and can be used as a helper
library by the test developer.
The benefit of this change is decoupling UiAutomator from the system
since the former was calling hidden APIs which required that it is
bundled in the system image. This prevented UiAutomator from being
evolved separately from the system. Also UiAutomator was creating
additional API surface in the system image. Another benefit of the new
design is that now test cases have access to a context and can use
public platform APIs in addition to the UiAutomator ones. Further,
third-parties can develop their own higher level test APIs on top
of the lower level ones exposes by UiAutomation.
bug:8028258
Also this change adds the fully qualified resource name of the view's
id in the emitted AccessibilityNodeInfo if a special flag is set while
configuring the accessibility service. Also added is API for looking
up node infos by this id. The id resource name is relatively more stable
compared to the generaed id number which may change from one build to
another. This API facilitate reuing the already defined ids for UI
automation.
bug:7678973
Change-Id: I589ad14790320dec8a33095953926c2a2dd0228b
1. This patch takes care of the case where a magnified window is covering an unmagnigied
one. One example is a dialog that covers the IME window.
bug:7634430
2. Ensuring that the UI automator tool can connect and correctly dump the screen.
bug:7694696
3. Removed the partial implementation for multi display magnification. It adds
unnecessary complexity since it cannot be implemented without support for
input from multiple screens. We will revisit when necessary.
4. Moved the magnified border window as a surface in the window manager.
5. Moved the mediator APIs on the window manager and the policy methods on the
WindowManagerPolicy.
6. Implemented batch event processing for the accessibility input filter.
Change-Id: I4ebf68b94fb07201e124794f69611ece388ec116
1. The screen magnification feature was implemented entirely as a part of the accessibility
manager. To achieve that the window manager had to implement a bunch of hooks for an
external client to observe its internal state. This was problematic since it dilutes
the window manager interface and allows code that is deeply coupled with the window
manager to reside outside of it. Also the observer callbacks were IPCs which cannot
be called with the window manager's lock held. To avoid that the window manager had
to post messages requesting notification of interested parties which makes the code
consuming the callbacks to run asynchronously of the window manager. This causes timing
issues and adds unnecessary complexity.
Now the magnification logic is split in two halves. The first half that is responsible
to track the magnified portion of the screen and serve as a policy which windows can be
magnified and it is a part of the window manager. This part exposes higher level APIs
allowing interested parties with the right permissions to control the magnification
of a given display. The APIs also allow a client to be registered for callbacks on
interesting changes such as resize of the magnified region, etc. This part servers
as a mediator between magnification controllers and the window manager.
The second half is a controller that is responsible to drive the magnification
state based on touch interactions. It also presents a highlight when magnified to
suggest the magnified potion of the screen. The controller is responsible for auto
zooming out in case the user context changes - rotation, new actitivity. The controller
also auto pans if a dialog appears and it does not interesect the magnified frame.
bug:7410464
2. By design screen magnification and touch exploration work separately and together. If
magnification is enabled the user sees a larger version of the widgets and a sub section
of the screen content. Accessibility services use the introspection APIs to "see" what
is on the screen so they can speak it, navigate to the next item in response to a
gesture, etc. Hence, the information returned to accessibility services has to reflect
what a sighted user would see on the screen. Therefore, if the screen is magnified
we need to adjust the bounds and position of the infos describing views in a magnified
window such that the info bounds are equivalent to what the user sees.
To improve performance we keep accessibility node info caches in the client process.
However, when magnification state changes we have to clear these caches since the
bounds of the cached infos no longer reflect the screen content which just got smaller
or larger.
This patch propagates not only the window scale as before but also the X/Y pan and the
bounds of the magnified portion of the screen to the introspected app. This information
is used to adjust the bounds of the node infos coming from this window such that the
reported bounds are the same as the user sees not as the app thinks they are. Note that
if magnification is enabled we zoom the content and pan it along the X and Y axis. Also
recomputed is the isVisibleToUser property of the reported info since in a magnified
state the user sees a subset of the window content and the views not in the magnified
viewport should be reported as not visible to the user.
bug:7344059
Change-Id: I6f7832c7a6a65c5368b390eb1f1518d0c7afd7d2
1. In touch exploration mode the system clicks in the center of the
accessibility focus rectangle. However, if this rectangle is only
partially shown on the window or on the screen the system may not
be able to perform the click, if the accessibility focus center
is not on the screen, or click on the wrong window, if the access
focus center is outside of the window.
This change clips the rectangle to the window bounds which and the
display bounds. This will ensure no clicks are sent to the wrong
window and no clicks are sent outside of the screen.
bug:7453839
Change-Id: I79f98971e7ebcbb391c37284467dc76076172c5f
1. The accessibility layer has a back door for a UI test automation code running
from the shell to attach. The unregister code does an incorrect identity check
and as a result the register UI test automation service is not disconnected
until its process is killed. The fix is super safe and simple.
bug:7409261
Change-Id: I4b1da18be6c5619dadd4a58fca6724529bc59dea
1. We cache some events to see if the user wants to trigger magnification. If no
magnification is triggered we inject these events with adjusted time and down
time to prevent subsequent transformations being confused by stale events.
After the cached events, which always have a down, are injected we need to also
update the down time of all subsequent non cached events.
bug:7379388
Change-Id: I41d8b831cc1016a0ee8f9c5ef5f42eb60a6f64d9
1. When magnifier, if a dialog that popped up is wider than the scree we pan to its upper
left corner. We now show the upper right corner if the locale direction is RTL.
2. Keyguard dialogs are not centered since they are used as a sign to recompute the
magnified area but an unnecessary else statement prevents such dialogs from being
properly show via a pan.
bug:7374331
Change-Id: I285e46b822a29f0082c502cb642f9da32dabfe6a
1. In the magnifier we are caching the touch events until we figure
out whether the user is triple tapping to enable magnification.
If the user is not trying to engage magnification we deliver the
stashed events. However, these events are stale and the subsequent
transformations such as the touch explorer get confused when trying
to detect a tap since the delay is longer than the tap slop.
This change compensates for the time the events were cached
before sending them to the next transformation in the chain.
bug:7362365
Change-Id: Idd8539ffed7ba4892c5a916bd34910fd2ef50f75
1. Sometimes unlocking the device when the IME is up and triple tapping on the keyboard
toggles screen magnification. The core reason is that when the kayguard window is
shown we hide all other windows and when it is hidden we show these windows. We did
not notify the screen magnifier for windows being shown and hidden. Also when the
windows are shown we may reassign layers to put the IME or the wallpaper in the
right Z order. The screen magnifier is now notified upon such layer reassignment
since window layers are used when computing the magnified region.
bug:7351531
Change-Id: I0931f4ba6cfa565d8eb1e3c432268ba1818feea6
1. We auto pan when certain type of window pop up to make sure the user
knows about the context change. This does not happen however for
fragment dialog since its window type is not in the list of one
we auto pan for. Updating the window type list.
bug:7332090
Change-Id: I9b097c57df929d2e4e807a948c3a0540f4092a76
1. If a bad magnification scale is persisted, i.e. it is
not between the min and max, the screen magnifier gets
into a bad state which even a reboot does not fix since
the scale is persisted in settings.
This change ensures that only valid scales are presisted.
In general a bad value should not be attempted to be
persisted but at this point this is the safest change.
bug:7288239
Change-Id: I3e9c7c091772fa64128ab8403c2127ce65cb94b8
