You can set Magento category specific layout XML in the following location as follows:
<CATEGORY_10> <!-- 10 is category ID. you can get this id in your magento admin -->
<!-- set category specific layout XML here -->
Note: 1) <CATEGORY_10> is one of the layout handle which is applied before custom layout XML specified in Admin Panel.
2) The layout handle <catalog_category_default> does not work in custom layout XML for specific category.