1. The window manager was not notifying a window when the latter
has been moved. This was causing incorrect coordinates of the
nodes reported to accessibility services. To workaround that
we have carried the correct window location when making a
call from the accessibility layer into a window. Now the
window manager notifies the window when it is moved and the
workaround is no longer needed. This change takes it out.
2. The left and right in the attach info were not updated properly
after a report that the window has moved.
3. The accessibility manager service was calling directly methods
on the window manager service without going through the interface
of the latter. This leads to unnecessary coupling and in the
long rung increases system complexity and reduces maintability.
bug:6623031
Change-Id: Iacb734b1bf337a47fad02c827ece45bb2f53a79d
1. There was a misspelled duplicate member in the accessibility service
class which was causing inconsistent behavior because one field was
updated and another checked.
2. When the set of services that can put the device in explore by touch
mode changes we were disconnecting and reconnecting all services
and this is not correct. Now only the state of explore by touch is
updated appropriately.
bug:6798860
Change-Id: Ib3c119cef8e71c3458d56e4ce6fbde2c2f750dcd
This fix adjusts the sensitivity of the gesture recognizer by
eliminating gesture rotation in the recognition process.
Bug:6697119
Change-Id: Ic767f513c05210b27e583338c4f0adcaa1c4c625
1. Accessibility allows querying only of the active window.
The active window is the one that has input focus or the
one the user is touching. Hence, if the user is touching
a window that does not have input focus this window is
the active one and as soon as the user stops touching
it the active window becomes the one that has input
focus. Currently the active window is not updated properly
when the user lifts his finger. This leads to a scenario
of traversal actions sent to the wrong window and the user
being stuck.
The reason is that the last touch explored event that is
used to determine where to click is cleared when accessibility
focus moves but this event is also used to determine when to
send the hover exit and touch exploration gesture end events.
The problem is that the last hover event is cleared before
it is used for sending the right exit events, thus the event
stream is inconsistent and the accessibility manager service
relies on this stream to update the active window. Now we
are keeping separate copies of the last touch event - one
for clicking and one for determining the which events to
inject to ensure consistent stream.
bug:6666041
Change-Id: Ie9961e562a42ef8a9463afacfff2246adcb66303
1. We are deciding whether the user is performing a gesture or an exploration based
on the gesture velocity. If we are detecting gesture we do the recognition at the
gesture end which is when the finger goes up. This is better than having a mode
toggle gesture for exploring and gestures detection. However, it is possible that
the user really wanted to perform an exploration but was moving too fast and
unless he lifts his finger the device is in gesture detection mode. This is
frustrating since the user has no feedback and assumes exploration does not
work.
We want to perform gesture detection only for a maximal time frame and if the
user did not lift his finger we transition into touch exploration state.
bug:6663173
Change-Id: I954ff937cca902e31b51325d1e1dfce84d239624
1. The touch explorer was notified for accessibility events from
a binder thread which was poking the internal state of the
latter which by design is not tread safe. Since the touch
explorer is expected to be running only on the main thread
the accessibility manager service delivers the accessibility
events to the explorer on that thread.
bug:6635496
Change-Id: Ifdc5329e4be8e485d7f77f0fb472184494fa0d15
1. AccessibilityInput filter was not checking whether the touch
explorer instance is not null before passing it an accessibility
event. If the accessibility event is dispatched before the input
filter is installed but after it is created we runt into this
case.
2. Added a missing null check in accessibility node info.
bug:6635089
Change-Id: Ia389dc1f427427eb73794f6331ccb870e0b44c55
1. When typing into an auto completion edit field a list of completions pops up and if
the user touch explores the list and tries to double tap to select the touched
completion the latter is not selected.
The auto completion is a popup that does not take input focus and is overlaid on
top of the window that has input focus. The touch explorer was clicking on the
location of the accessibility focus if the last touch explored location is within
the bounds of the active window. In this case this was the window with the edit
text into which the user is typing. The check performed by the touch explorer
was missing the case when the last touch explored location was within the bounds
of the active window but it actually was deloverd to another overlaid window.
Now we are poking on the accessibility focus location if the last explored
location is within the active window and was delivered to it.
bug:6629535
Change-Id: Ie66d5bb81ab021f2bb0414339b7de26d96826191
1. If the last touch explored location is within the active window we
used to click on exact location if it is within the accessibility
focus otherwise in the accessibility focus center. If the last touch
explored location is not within the active window we used to just
click there. This breaks in the case were one has touch explored
at a given place in the current window and now a dialog opens *not*
covering the touch explored location. If one uses swipes to move
accessibility focus i.e. to traverse the dialog without touching
it one cannot activate anything because the touch explorer is using
the last touch explored location that is outside of the active
window e.g the dialog.
The solution is to clear the last touch explored location when a
window opens or accessibility focus moves. If the last touch
explored location is null we are clicking in the accessibility
focus location.
bug:6620911
2. There is a bug in the window manager that does not notify a
window that its location has changed (bug:6623031). This breaks
accessibility interaction with dialogs that have input because
when the IME is up the dialog is moved but not notified. Now
the accessibility layer gets incorrect location for the
accessibility focus and the window bounds.
The soluion is when the accessibility manager service calls
into the remove thress to obtain some accessibility node infos
it passes the window left and top which it gets from the
window manager. These values are used to update the attach info
window left and top so all accessibility node infos emitted
from that window had correct bounds in screen coordinates.
bug:6620796
Change-Id: I18914f2095c55cfc826acf5277bd94b776bda0c8
1. Now after setting the content description on a view we mark is as
important for accessibility of the current important for accessibility
mode of that view is auto.
2. Minor tweak to a touch explorer coefficient to make performing double
tapping easier.
bug:6615353
Change-Id: I3b477f533a3ebde85d425caf32ace5e851240f88
1. The global action to open recent apps shows the old dialog style rent apps
panel. Apparently the key code to open recent apps is not opening the new
UI so the AccessibilityManagerService is calling directly the method on
the IStatusBarSerivce to do so.
bug:6607664
Change-Id: I94c1963b07947776bf1c2448903b26f3603f9a59
1. Touch exploration gestures are demarcated by start and end
events. Due to a bug in the AccessibilityManagerService
the gesture end event was not dispatched. This caused the
AccessibilityNodeInfoCache to be off sync since it relies
on getting such events not to mention that the clients were
not getting the end but only the start event. The issue
was that the notified service types variable was not reset
after every event so when the manager sends the last hover
exit it flags that the service type is already notified
resulting in dropping on the floor the following gesture
end event.
bug:6539306
Change-Id: I2b96bcecea3b2240199d67f01afa6a033afce1de
1. If the runnable for performing a long press is not
removed when all pointers are up and it is executed
the explorer gets into delegating mode with no pointer
down and the next down crashes the explorer. Added
code to remove the long press runnable in a few places
it was missing and also added a safety in the runnable
to avoid executing it in case there are no active pointers.
bug:6557183
Change-Id: I9dab3de88fd08d8e2b38af18249ac551837c0736
1. The long press routine was using the coordintates of the
accessibility focused item in the input focused window.
As a result double tap and hold did not work in a window
that does not take input focus such as the system bar.
Now the routine is using the last touch explored location
if it cannot find accessibility focus in the last touched
window.
bug:6584438
Change-Id: Ifd43adb20a066f389a9d4bd5716dd7ad834dd574
1. Now we are asking the user to grant permission to the service to enable
touch exploration only the first time this service is enabled. If the
service was uninstalled and then later installed we ask the user again.
This avoids the scenario in which rebooting the device or upgrading an
accessibility service leaves the device in a state in which the user
cannot interact with.
bug:6582088
Change-Id: I51d24e4892b3b48c9fb11dfb09ec1118502ba526
1. If a UI test automation accessibility service is connected to the
system we pospone state updates in the AccessibilityManagerService
for the moment the UI automations service dies or is disconnected.
bug:6540522
Change-Id: I48ddf603b53d2158a00edcf8ad05cfe2575d4d75
1. We are passing the interrogating process id in the remote
accessibility requests to catch the query from the same
thread. While all other methods were doing this correctly
somehow the perform action is using the incorrect process id.
bug:6534935
Change-Id: Icef50833903c562758d51ef316b60c53c7a336c0
1. The internal service instance created by AccessibilityManagerService
was getting the looper of the current thread when created. This works
for real accessibility services but since UI automation service is
registered via an IPC the binder thread has no looper. Now we explicitly
get the correct looper.
bug:6535435
Change-Id: I63a2ada1b65c4b3c71c3d1e6deb3dfdeb7a3d6d6
1. Now the user have to double tap to activate the last
item. If the last touched window is not active because
it does not take input focus the click on the last
touch explored location. Othewise the click is on the
accessibility focus location.
bug:5932640
Change-Id: Ibb7b97262a7c5f2f94abef429e02790fdc91a8dd
1. Every accessibility services targeting JellyBean or higher has
to request a special permission for the system to bind to it.
Change-Id: I6e579326bdf3597f148d6c67317455701ec8af68
1. During gesture detection we are not injecting the events we receive
since we do not want the accessibility focus to move as a result of
the hover event of the gesture. Because of that it was possible that
we consume all events since the user performs only gesture to navigate
resulting in the screen being off while the user is actively interacting
with the device. Now we are poking the user activity in the power
manager to keep the screen on.
bug:6485171
Change-Id: I06a09c5297f01bef5e20d471cee76fa7aae0c4fe
1. Since the API version has been finalized this change
updates the SDk version checks to use the JellyBean
verson number.
bug:5947249
Change-Id: Ie22fa7e18a7ea7b0c7077d80246a26c17f327ceb
1. The initial design was to have some accessibility gestures
being handled by the system if the gesture handling access
service does not consume the gesture. However, we are not
sure what a good default is and once we add a default handler
we cannot remove it since people may rely on it. Thus, we
take the simples approach and let the accessibility service
handle the gestures. If no gestures are handled the system
will work in explore by touch as before.
bug:5932640
Change-Id: I865a83549fa03b0141d27ce9713e9b7bb45a57b4
1. Scrolling actions are crucial for enabling a gesture based
traversal of the UI and specifically scrollable containers
especially lists and anything backed by an adapter. Since
accessibility focus can land only attached views, it cannot
visit views for adapter items not shown on the screen.
Auto scrolling the list as a result of putting access focus
ot a list item does not work well since the user may get
trapped in a long list. Adding an accessibility node provider
to emit virtual views for one view before the first and one
after the last is complex and suffers the limitation of trapping
the user. Accessibility service need an explicit scroll actions
which may be performed upon an explicit user action. Hence,
the user is informed for the start/end of the visible part of
the list and he makes a deliberate choice to scroll. This will
benefit also people developing Braille devices since they can
scroll the content without telling the user to stop using the
Braille controller and take the device out of his pocket to scroll
and go back to the Braille controller.
NOTE: Without these action large portions of the screen will be
hard to access since users will have to touch and explore to
find and scroll the list.
Change-Id: Iafcf54d4967893205872b3649025a4e347a299ed
1. Delegating activation gestures has several issues that we should
decide how to handle if possible before allowing an accessibility
service to take over them:
A) It is needed that every view than can be clicked or long pressed on
reacts to such as a response to calling performClick and performLongPress
which is not necessary true since the view may watch the touch
events and do its own click long click detection. As a result it may
be possible that there are view a user cannot interact with in
touch exploration mode but can if not in that mode.
B) Clicking or long pressing on a different location in a view may yield
different results, for example NumberPicker. Ideally such views have
to implement AccessibilityNodeProvide which provider handles correctly
the request for click long press on virtual nodes. Some apps however
just fire different hover accessibility events when the user is over
a specific semantic portion of the view but do not provide virtual
nodes. Hence, a user will not be able to interact with such semantic
regions but the system can achieve that by sending the click/long click
at the precise location in the view that was last touch explored.
2. Adding a flag on accessibility service info to request explore by touch
mode. There is no need to put the device in this mode if node of the currently
enabled accessibility services supports it. Now the problem is inverted and
the service has to explicitly state its capability.
3. Fixing a bug where includeImportantViews was ignored for automation
services.
Change-Id: I3b29a19f24ab5e26ee29f974bbac2197614c9e2a
This problem was introduced in I74df9c24. The intention of the
change was still let UiTestAutomationBridge see the
non-important views, but there were bugs in the implementation:
1. AccessibilityManagerService was not really updating
mIncludeNotImportantViews when mIsAutomation is true
2. Wrong constant is used to set the flag
Change-Id: Ia0a2e9ed9720bd0ea3a563e0b492e870a6ec1586
1. Since we are using a stateless proxy accessibility service to
perform default accessibility gesture handling it shuld not
operate against not important views.
bug:6422069
Change-Id: I74df9c2415ab3b164d9ac5873f7004c0459e2bfa
1. Changed all references to granularity to movement
granularity. BTW, to be more precise it should be
text movement granularity.
bug:6435232
Change-Id: If6366b002ca3390f74918995b342baff2cbcfd01
1. The event of setting an accessibility focus on a view should not
make the host window the currently active one.
bug:6400648
Change-Id: Ib45c255f441c38489ee9d4ab5f284550ac5f6b01
1. The checks for action arguments are not needed since they
may cause trouble for developers if we add more args to
an action.
bug:6414006
Change-Id: Ia4212b52be183b1ef1cfd2561ce618cef2b015e4
1. The granularities for traversing the text content of an accessibility
node info are now predefined constants and custom ones will not be
supported. This is the simplest solution - we can always add namespaced
user defined ones (unlikely).
2. Added actions for traversing web content. These actions can be used by
an accessibility service to transparently drive the JavaScript based
screen reader that is used for handling web content.
3. Added a new accessibility event type for traversing the content of a
view. This event is needed to announce to the user what is the next
element, i.e. the one next to the cursor, after the view's text was
traversed.
bug:5932640
bug:6389591
Change-Id: I144647da55bc4005c64f89865ef333af8359e145
1. The accessibility focus directions are not needed since an
accessibility service just get the root, first child, next
sibling, previous sibling and call execute the action to
give it accessibility focus. Now the accessibility node
info tree is properly ordered taking into account layout
manager directions for both layout manager that we report
and ones that we have determined as not important for
accessibility. Also the position of a node info are ordered
properly based on their coordinates after all transformations
as opposed to child index.
bug:5932640
Change-Id: I994a8297cb1e57c829ecbac73a937c2bcbe0bac7
1. A view that creates an accessibility node info may add to the info
a list of granularity labels. These are granularities by which the
source view can iterate over its content. For example a text view
may support character, word link while a web view may additionally
support buttons, tables, etc. There are actions on accessibility
node info to go to the next/previous at a given granularity which
is passesed as an argument.
2. Added Bundle argument to the APIs for performing accessibility
actions. This is generic and extensible.
bug:5932640
Change-Id: I328cbbb4cddfdee082ab2a8b7ff1bd7477d8d6f9
1. The gesture dispatcher thread was not waiting in a loop
that check for complete initialization. Therefore is was
susceptible to missed signals and unexpected interrupts.
2. In the gesture processing message handle the interaction id
was reading the wrong message argument.
bug:5932640
Change-Id: Ic65ecc01a7fe7d43929c6c07d0759ae9001cf515
1. An accessibility service has to explicitly opt in to be notified
for gestures by the system. There is only one accessibility service
that handles gestures and in case it does not handle a gesture
the system performs default handling. This default handling ensures
that we have gesture navigation even if no accessibility service
would like to participate/customize the interaction model.
bug:5932640
Change-Id: Id8194293bd94097b455e9388b68134a45dc3b8fa
1. Added more gesture for accessibility. After a meeting
with the access-eng team we have decided that the current
set of gestures may be smaller than needed considering
that we will use four gestures for home, back, recents,
and notifications.
2. Adding actions for going back, home, opening the recents,
and opening the notifications.
3. Added preliminary mapping from some of the new gestures
to the new actions.
4. Fixed a bug in the accessibility interaction controller
which was trying to create a handled on the main looper
thread which may be null if the queried UI is in the
system process. Now the context looper of the root view
is used.
5. Fixed a bug of using an incorrect constant.
6. Added a missing locking in a couple of places.
7. Fixed view comparison for accessibilityt since it was
not anisymmetric.
bug:5932640
bug:5605641
Change-Id: Icc983bf4eafefa42b65920b3782ed8a25518e94f
Usefulness: Keep track of the current user location in the screen when
traversing the it. Enabling structural and directional
navigation over all elements on the screen. This enables
blind users that know the application layout to efficiently
locate desired elements as opposed to try touch exploring the
region where the the element should be - very tedious.
Rationale: There are two ways to implement accessibility focus One is
to let accessibility services keep track of it since they
have access to the screen content, and another to let the view
hierarchy keep track of it. While the first approach would
require almost no work on our part it poses several challenges
which make it a sub-optimal choice. Having the accessibility focus
in the accessibility service would require that service to scrape
the window content every time it changes to sync the view tree
state and the accessibility focus location. Pretty much the service
will have to keep an off screen model of the screen content. This
could be quite challenging to get right and would incur performance
cost for the multiple IPCs to repeatedly fetch the screen content.
Further, keeping virtual accessibility focus (i.e. in the service)
would require sync of the input and accessibility focus. This could
be challenging to implement right as well. Also, having an unlimited
number of accessibility services we cannot guarantee that they will
have a proper implementation, if any, to allow users to perform structural
navigation of the screen content. Assuming two accessibility
services implement structural navigation via accessibility focus,
there is not guarantee that they will behave similarly by default,
i.e. provide some standard way to navigate the screen content.
Also feedback from experienced accessibility researchers, specifically
T.V Raman, provides evidence that having virtual accessibility focus
creates many issues and it is very hard to get right.
Therefore, keeping accessibility focus in the system will avoid
keeping an off-screen model in accessibility services, it will always
be in sync with the state of the view hierarchy and the input focus.
Also this will allow having a default behavior for traversing the
screen via this accessibility focus that is consistent in all
accessibility services. We provide accessibility services with APIs to
override this behavior but all of them will perform screen traversal
in a consistent way by default.
Behavior: If accessibility is enabled the accessibility focus is the leading one
and the input follows it. Putting accessibility focus on a view moves
the input focus there. Clearing the accessibility focus of a view, clears
the input focus of this view. If accessibility focus is on a view that
cannot take input focus, then no other view should have input focus.
In accessibility mode we initially give accessibility focus to the topmost
view and no view has input focus. This ensures consistent behavior accross
all apps. Note that accessibility focus can move hierarchically in the
view tree and having it at the root is better than putting it where the
input focus would be - at the first input focusable which could be at
an arbitrary depth in the view tree. By default not all views are reported
for accessibility, only the important ones. A view may be explicitly labeled
as important or not for accessibility, or the system determines which one
is such - default. Important views for accessibility are all views that are
not dumb layout managers used only to arrange their chidren. Since the same
content arrangement can be obtained via different combintation of layout
managers, such managers cannot be used to reliably determine the application
structure. For example, a user should see a list as a list view with several
list items and each list item as a text view and a button as opposed to seeing
all the layout managers used to arrange the list item's content.
By default only important for accessibility views are regared for accessibility
purposes. View not regarded for accessibility neither fire accessibility events,
nor are reported being on the screen. An accessibility service may request the
system to regard all views. If the target SDK of an accessibility services is
less than JellyBean, then all views are regarded for accessibility.
Note that an accessibility service that requires all view to be ragarded for
accessibility may put accessibility focus on any view. Hence, it may implement
any navigational paradigm if desired. Especially considering the fact that
the system is detecting some standard gestures and delegates their processing
to an accessibility service. The default implementation of an accessibility
services performs the defualt navigation.
bug:5932640
bug:5605641
Change-Id: Ieac461d480579d706a847b9325720cb254736ebe
