UIAutomator2.0和AccessibilityService实现分析

0x00 前言

UiAutomator是Android 4.1以上提供的一个UI自动化测试工具,4.3升级到了UiAutomator2.0,实现方式也从UiTestAutomationBridge变成了UiAutomation。

0x01 UiAutomation实现分析

UiAutomation类位于android.app包下面,是API18新增的类。

public final class UiAutomation {
    private final IAccessibilityServiceClient mClient;
    private final IUiAutomationConnection mUiAutomationConnection;

    public void connect();
    public void disconnect();
    public final boolean performGlobalAction(int action);
    public final AccessibilityServiceInfo getServiceInfo();
    public AccessibilityNodeInfo getRootInActiveWindow();
    public boolean injectInputEvent(InputEvent event, boolean sync);
    public boolean setRotation(int rotation);
    public AccessibilityEvent executeAndWaitForEvent(Runnable command, AccessibilityEventFilter filter, long timeoutMillis);
    public void waitForIdle(long idleTimeoutMillis, long globalTimeoutMillis);
    public Bitmap takeScreenshot();
    public void setRunAsMonkey(boolean enable);
}

以上是UiAutomation类中最重要的成员变量和函数,所有的接口基本都是通过调用mClientmUiAutomationConnection这两个对象实现的。

public UiAutomation(Looper looper, IUiAutomationConnection connection) {
    if (looper == null) {
        throw new IllegalArgumentException("Looper cannot be null!");
    }
    if (connection == null) {
        throw new IllegalArgumentException("Connection cannot be null!");
    }
    mUiAutomationConnection = connection;
    mClient = new IAccessibilityServiceClientImpl(looper);
}

interface IUiAutomationConnection {
    void connect(IAccessibilityServiceClient client);
    void disconnect();
    boolean injectInputEvent(in InputEvent event, boolean sync);
    boolean setRotation(int rotation);
    Bitmap takeScreenshot(int width, int height);
    boolean clearWindowContentFrameStats(int windowId);
    WindowContentFrameStats getWindowContentFrameStats(int windowId);
    void clearWindowAnimationFrameStats();
    WindowAnimationFrameStats getWindowAnimationFrameStats();
    void executeShellCommand(String command, in ParcelFileDescriptor fd);
    void grantRuntimePermission(String packageName, String permission, int userId);    
    void revokeRuntimePermission(String packageName, String permission, int userId);

    // Called from the system process.
    oneway void shutdown();
}

UiAutomation构造函数需要传入两个参数,一个是线程Looper对象,用于发送消息,一个是IUiAutomationConnection接口实例。

public void connect() {
    if (mHandlerThread.isAlive()) {
        throw new IllegalStateException("Already connected!");
    }
    mHandlerThread.start();
    mUiAutomation = new UiAutomation(mHandlerThread.getLooper(),
            new UiAutomationConnection());
    mUiAutomation.connect();
}

这是UiAutomationShellWrapper类中的一个方法,正好可以看到UiAutomation如何初始化。构造函数的第二个参数实际使用的是UiAutomationConnection类实例,这个类也是在android.app包下面,正是继承自IUiAutomationConnection.Stub类。

UiAutomation中一个重要的成员变量是mClient,它的类型是IAccessibilityServiceClient

oneway interface IAccessibilityServiceClient {

    void init(in IAccessibilityServiceConnection connection, int connectionId, IBinder windowToken);

    void onAccessibilityEvent(in AccessibilityEvent event);

    void onInterrupt();

    void onGesture(int gesture);

    void clearAccessibilityCache();

    void onKeyEvent(in KeyEvent event, int sequence);
}

IAccessibilityServiceClient是一个AIDL接口定义,在抽象类AccessibilityService中实现了一个该接口的实现类IAccessibilityServiceClientWrapper。UiAutomation中定义了一个IAccessibilityServiceClientWrapper的子类IAccessibilityServiceClientImpl,主要是重写了onAccessibilityEvent方法,用于获取AccessibilityEvent事件。UiAutomation的构造函数中实例化的正是IAccessibilityServiceClientImpl实例。

UiAutomation的初始化过程主要是在connect方法中。

/**
    * Connects this UiAutomation to the accessibility introspection APIs.
    *
    * @hide
    */
public void connect() {
    synchronized (mLock) {
        throwIfConnectedLocked();
        if (mIsConnecting) {
            return;
        }
        mIsConnecting = true;
    }

    try {
        // Calling out without a lock held.
        mUiAutomationConnection.connect(mClient);
    } catch (RemoteException re) {
        throw new RuntimeException("Error while connecting UiAutomation", re);
    }

    synchronized (mLock) {
        final long startTimeMillis = SystemClock.uptimeMillis();
        try {
            while (true) {
                if (isConnectedLocked()) {
                    break;
                }
                final long elapsedTimeMillis = SystemClock.uptimeMillis() - startTimeMillis;
                final long remainingTimeMillis = CONNECT_TIMEOUT_MILLIS - elapsedTimeMillis;
                if (remainingTimeMillis <= 0) {
                    throw new RuntimeException("Error while connecting UiAutomation");
                }
                try {
                    mLock.wait(remainingTimeMillis);
                } catch (InterruptedException ie) {
                    /* ignore */
                }
            }
        } finally {
            mIsConnecting = false;
        }
    }
}

这里主要是调用了mUiAutomationConnection.connect(mClient)。

public void connect(IAccessibilityServiceClient client) {
    if (client == null) {
        throw new IllegalArgumentException("Client cannot be null!");
    }
    synchronized (mLock) {
        throwIfShutdownLocked();
        if (isConnectedLocked()) {
            throw new IllegalStateException("Already connected.");
        }
        mOwningUid = Binder.getCallingUid();
        registerUiTestAutomationServiceLocked(client);
        storeRotationStateLocked();
    }
}

private void registerUiTestAutomationServiceLocked(IAccessibilityServiceClient client) {
    IAccessibilityManager manager = IAccessibilityManager.Stub.asInterface(
            ServiceManager.getService(Context.ACCESSIBILITY_SERVICE));
    AccessibilityServiceInfo info = new AccessibilityServiceInfo();
    info.eventTypes = AccessibilityEvent.TYPES_ALL_MASK;
    info.feedbackType = AccessibilityServiceInfo.FEEDBACK_GENERIC;
    info.flags |= AccessibilityServiceInfo.FLAG_INCLUDE_NOT_IMPORTANT_VIEWS
            | AccessibilityServiceInfo.FLAG_REPORT_VIEW_IDS;
    info.setCapabilities(AccessibilityServiceInfo.CAPABILITY_CAN_RETRIEVE_WINDOW_CONTENT
            | AccessibilityServiceInfo.CAPABILITY_CAN_REQUEST_TOUCH_EXPLORATION
            | AccessibilityServiceInfo.CAPABILITY_CAN_REQUEST_ENHANCED_WEB_ACCESSIBILITY
            | AccessibilityServiceInfo.CAPABILITY_CAN_REQUEST_FILTER_KEY_EVENTS);
    try {
        // Calling out with a lock held is fine since if the system
        // process is gone the client calling in will be killed.
        manager.registerUiTestAutomationService(mToken, client, info);
        mClient = client;
    } catch (RemoteException re) {
        throw new IllegalStateException("Error while registering UiTestAutomationService.", re);
    }
}

UiAutomationConnection.connect函数接着调用了registerUiTestAutomationServiceLocked(client),而registerUiTestAutomationServiceLocked主要是调用了IAccessibilityManagerregisterUiTestAutomationService注册了一个Accessibility服务。

由此可见,UiAutomation最终也是使用了AccessibilityManagerService。

0x02 如何使用UiAutomation

UiAutomator的常见使用方式是调用uiautomator命令,或是将uiautomator.jar导入到自己的工程中。为了更加自由地使用UiAutomation提供的能力,可以考虑直接创建UiAutomation对象实例使用。由于UiAutomation的构造函数以及其它一些重要方法设置了@hide,因此无法直接使用,需要用反射的方式获取。

Object connection = null;
HandlerThread mHandlerThread = new HandlerThread("UiAutomationThread");
mHandlerThread.start();
try{
    Class<?> UiAutomationConnection = Class.forName("android.app.UiAutomationConnection");
    Constructor<?> newInstance = UiAutomationConnection.getDeclaredConstructor();
    newInstance.setAccessible(true);
    connection = newInstance.newInstance();
    Class<?> IUiAutomationConnection = Class.forName("android.app.IUiAutomationConnection");
    Constructor<?> newUiAutomation = UiAutomation.class.getDeclaredConstructor(Looper.class, IUiAutomationConnection);
    UiAutomation mUiAutomation = (UiAutomation)newUiAutomation.newInstance(mHandlerThread.getLooper(), connection);
    Method connect = UiAutomation.class.getDeclaredMethod("connect");
    connect.invoke(mUiAutomation);
    Log.i(TAG, ""+mUiAutomation);
    mUiAutomation.waitForIdle(1000, 1000 * 10);
    AccessibilityNodeInfo nodeInfo = mUiAutomation.getRootInActiveWindow();
    Log.i(TAG, ""+nodeInfo);
}catch(Exception e){
    e.printStackTrace();
    return;
}

这是一段最简单的使用代码,编译到apk里面,运行后报错了。

Caused by: java.lang.SecurityException: You do not have android.permission.RETRIEVE\_WINDOW_CONTENT required to call registerUiTestAutomationService from pid=3676, uid=10040  
     at android.os.Parcel.readException(Parcel.java:1599)  
     at android.os.Parcel.readException(Parcel.java:1552)  
     at android.view.accessibility.IAccessibilityManager$Stub$Proxy.registerUiTestAutomationService  (IAccessibilityManager.java:352)  
     at android.app.UiAutomationConnection.registerUiTestAutomationServiceLocked(UiAutomationConnection.java:337)  
     at android.app.UiAutomationConnection.connect(UiAutomationConnection.java:89)  
     at android.app.UiAutomation.connect(UiAutomation.java:197)

看来是需要android.permission.RETRIEVE_WINDOW_CONTENT权限,尝试将该权限添加到AndroidManifest.xml中,但是提示这个是系统权限,普通应用不能申请。因此,应用中不能使用UiAutomation的,同时发现,编译为jar包后使用shell权限执行,是可以正常跑过的。这也是为什么uiautomator工具可以正常运行的原因。

0x03 AccessibilityManagerService

如前所述,UiAutomation底层使用的还是AccessibilityManagerService,这里简单分析一下AccessibilityManagerService的实现。

该类内部包含了一个Service类:

/**
    * This class represents an accessibility service. It stores all per service
    * data required for the service management, provides API for starting/stopping the
    * service and is responsible for adding/removing the service in the data structures
    * for service management. The class also exposes configuration interface that is
    * passed to the service it represents as soon it is bound. It also serves as the
    * connection for the service.
    */
class Service extends IAccessibilityServiceConnection.Stub implements ServiceConnection, DeathRecipient;

从描述上看,每个Service实例代表了一个Accessibility服务,同时它实现了ServiceConnection接口,因此它也是一个Service客户端。

Service类中定义了一个bool类型的变量mIsAutomation,表示当前服务是否是UiAutomation。在AccessibilityManagerService的registerUiTestAutomationService方法中,将当前服务的组件名称设置为sFakeAccessibilityServiceComponentName,而mIsAutomation就是通过组件名称来判断是否是UiAutomation。

accessibilityServiceInfo.setComponentName(sFakeAccessibilityServiceComponentName);

mIsAutomation = (sFakeAccessibilityServiceComponentName.equals(componentName));

由于UiAutomation与AccessibilityService实现方式不同,因此,AccessibilityManagerService在很多地方都会根据mIsAutomation执行不同的逻辑。

根据前面的分析,在UiAutomation发起connect请求后,会进入AccessibilityManagerService的registerUiTestAutomationService。然后经过一系列的函数调用,进到Service的bindLocked函数。

/**
    * Binds to the accessibility service.
    *
    * @return True if binding is successful.
    */
public boolean bindLocked() {
    UserState userState = getUserStateLocked(mUserId);
    if (!mIsAutomation) {
        if (mService == null && mContext.bindServiceAsUser(
                mIntent, this,
                Context.BIND_AUTO_CREATE | Context.BIND_FOREGROUND_SERVICE_WHILE_AWAKE,
                new UserHandle(mUserId))) {
            userState.mBindingServices.add(mComponentName);
        }
    } else {
        userState.mBindingServices.add(mComponentName);
        mService = userState.mUiAutomationServiceClient.asBinder();
        mMainHandler.post(new Runnable() {
            @Override
            public void run() {
                // Simulate asynchronous connection since in onServiceConnected
                // we may modify the state data in case of an error but bind is
                // called while iterating over the data and bad things can happen.
                onServiceConnected(mComponentName, mService);
            }
        });
        userState.mUiAutomationService = this;
    }
    return false;
}

bindLocked会发送一个异步消息,调到onServiceConnected回调。

@Override
public void onServiceConnected(ComponentName componentName, IBinder service) {
    synchronized (mLock) {
        mService = service;
        mServiceInterface = IAccessibilityServiceClient.Stub.asInterface(service);
        UserState userState = getUserStateLocked(mUserId);
        addServiceLocked(this, userState);
        if (userState.mBindingServices.contains(mComponentName) || mWasConnectedAndDied) {
            userState.mBindingServices.remove(mComponentName);
            mWasConnectedAndDied = false;
            try {
                mServiceInterface.init(this, mId, mOverlayWindowToken);
                onUserStateChangedLocked(userState);
            } catch (RemoteException re) {
                Slog.w(LOG_TAG, "Error while setting connection for service: "
                        + service, re);
                binderDied();
            }
        } else {
            binderDied();
        }
    }
}

这里的mServiceInterface其实就是UiAutomation构造函数中实例化的mClient的远程对象。接着,调用了IAccessibilityServiceClient的init函数,该函数的实现位于AccessibilityService.IAccessibilityServiceClientWrapper类中。

public void init(IAccessibilityServiceConnection connection, int connectionId,
        IBinder windowToken) {
    Message message = mCaller.obtainMessageIOO(DO_INIT, connectionId,
            connection, windowToken);
    mCaller.sendMessage(message);
}

case DO_INIT: {
    mConnectionId = message.arg1;
    SomeArgs args = (SomeArgs) message.obj;
    IAccessibilityServiceConnection connection =
            (IAccessibilityServiceConnection) args.arg1;
    IBinder windowToken = (IBinder) args.arg2;
    args.recycle();
    if (connection != null) {
        AccessibilityInteractionClient.getInstance().addConnection(mConnectionId,
                connection);
        mCallback.init(mConnectionId, windowToken);
        mCallback.onServiceConnected();
    } else {
        AccessibilityInteractionClient.getInstance().removeConnection(
                mConnectionId);
        mConnectionId = AccessibilityInteractionClient.NO_ID;
        AccessibilityInteractionClient.getInstance().clearCache();
        mCallback.init(AccessibilityInteractionClient.NO_ID, null);
    }
} return;

这样,初始化完成后,AccessibilityInteractionClient实例中保存了IAccessibilityServiceConnection实例,而AccessibilityManagerService中对应的Service对象中也保存了IAccessibilityServiceClient实例,从而建立起双向的Binder通信。

在发生Accessibility事件后,AccessibilityManagerService会通过IAccessibilityServiceConnection的onAccessibilityEvent方法将事件通知给UiAutomation。

0x04 UiAutomator与AccessibilityService

AccessibilityService是一个继承自Service的抽象服务类,用户在使用时需要实现一个自己的子类。该类很大程度上依赖于AccessibilityInteractionClient类提供的接口,AccessibilityInteractionClient类内部保存了一个静态的LongSparseArray<AccessibilityInteractionClient> sClients对象,用来实现线程相关的单例对象。

/**
    * @return The client for the current thread.
    */
public static AccessibilityInteractionClient getInstance() {
    final long threadId = Thread.currentThread().getId();
    return getInstanceForThread(threadId);
}

/**
    * <strong>Note:</strong> We keep one instance per interrogating thread since
    * the instance contains state which can lead to undesired thread interleavings.
    * We do not have a thread local variable since other threads should be able to
    * look up the correct client knowing a thread id. See ViewRootImpl for details.
    *
    * @return The client for a given <code>threadId</code>.
    */
public static AccessibilityInteractionClient getInstanceForThread(long threadId) {
    synchronized (sStaticLock) {
        AccessibilityInteractionClient client = sClients.get(threadId);
        if (client == null) {
            client = new AccessibilityInteractionClient();
            sClients.put(threadId, client);
        }
        return client;
    }
}

前面提到,AccessibilityInteractionClient对象中保存了IAccessibilityServiceConnection实例,因此,可以调用该接口提供的功能。

/**
    * Interface given to an AccessibilitySerivce to talk to the AccessibilityManagerService.
    *
    * @hide
    */
interface IAccessibilityServiceConnection {

    void setServiceInfo(in AccessibilityServiceInfo info);

    boolean findAccessibilityNodeInfoByAccessibilityId(int accessibilityWindowId,
        long accessibilityNodeId, int interactionId,
        IAccessibilityInteractionConnectionCallback callback, int flags, long threadId);

    boolean findAccessibilityNodeInfosByText(int accessibilityWindowId, long accessibilityNodeId,
        String text, int interactionId, IAccessibilityInteractionConnectionCallback callback,
        long threadId);

    boolean findAccessibilityNodeInfosByViewId(int accessibilityWindowId,
        long accessibilityNodeId, String viewId, int interactionId,
        IAccessibilityInteractionConnectionCallback callback, long threadId);

    boolean findFocus(int accessibilityWindowId, long accessibilityNodeId, int focusType,
        int interactionId, IAccessibilityInteractionConnectionCallback callback, long threadId);

    boolean focusSearch(int accessibilityWindowId, long accessibilityNodeId, int direction,
        int interactionId, IAccessibilityInteractionConnectionCallback callback, long threadId);

    boolean performAccessibilityAction(int accessibilityWindowId, long accessibilityNodeId,
        int action, in Bundle arguments, int interactionId,
        IAccessibilityInteractionConnectionCallback callback, long threadId);

    AccessibilityWindowInfo getWindow(int windowId);

    List<AccessibilityWindowInfo> getWindows();

    AccessibilityServiceInfo getServiceInfo();

    boolean performGlobalAction(int action);

    oneway void setOnKeyEventResult(boolean handled, int sequence);
}

与UiAutomation不同的是,AccessibilityService的初始化是和普通的Service一致的。由于AccessibilityService比较特殊的地方在于需要在设置的辅助功能里开启对应的服务,点击开启后,会执行到BindService逻辑,进而执行到AccessibilityService的onBind回调,并触发AccessibilityManagerService中Service的onServiceConnected回调。

/**
    * Implement to return the implementation of the internal accessibility
    * service interface.
    */
@Override
public final IBinder onBind(Intent intent) {
    return new IAccessibilityServiceClientWrapper(this, getMainLooper(), new Callbacks() {
        @Override
        public void onServiceConnected() {
            AccessibilityService.this.onServiceConnected();
        }

        @Override
        public void onInterrupt() {
            AccessibilityService.this.onInterrupt();
        }

        @Override
        public void onAccessibilityEvent(AccessibilityEvent event) {
            AccessibilityService.this.onAccessibilityEvent(event);
        }

        @Override
        public void init(int connectionId, IBinder windowToken) {
            mConnectionId = connectionId;
            mWindowToken = windowToken;

            // The client may have already obtained the window manager, so
            // update the default token on whatever manager we gave them.
            final WindowManagerImpl wm = (WindowManagerImpl) getSystemService(WINDOW_SERVICE);
            wm.setDefaultToken(windowToken);
        }

        @Override
        public boolean onGesture(int gestureId) {
            return AccessibilityService.this.onGesture(gestureId);
        }

        @Override
        public boolean onKeyEvent(KeyEvent event) {
            return AccessibilityService.this.onKeyEvent(event);
        }
    });
}

相对于UiAutomation只能在shell环境中执行,AccessibilityService是可以运行在app环境中的,但是需要用户手动开启服务会略显麻烦。

0x05 总结

UiAutomator和AccessibilityService作为两种不同的实现形式,拥有各自的优缺点,这两年流行的抢红包工具基本也是基于这两种方式实现的。在自动化中使用它们也能起到一些辅助作用。

分享