you would need to create a recordset to lookup the selected category and the subcategory name that is entered to see if it already exists, using Server Validation from form toolkit or CSS Form builder, you can validate that the reocrdset is empty.
in the server validation server behavior, Set the server variable to "$totalRows_RecordsetName" where RecordsetName is the name of the recordset.
Use number validation. set the minimum to -1 and the maximum to 0.