There are several possible causes for the above mentioned error when scanning a MS Excel file. For example, the Excel file might be opened in a different application while the scan is being performed. So please ensure that the migration-center job server is the only application that accesses the file.
Another cause might be a too high wrapper.java.maxmemory setting in the wrapper.conf file of migration-center's job server. If you set this value to its maximum of 1536, the "System resource exceeded" error occurs when scanning an Excel file. In this case, reducing the value to 1280 solves the issue.
If none of the above solutions solve your problem, please search in the Microsofts forums in the Internet for possible causes and their solution.