You can only delete a user who has no claims associated with their record. If the user has existing claims, clicking XX Remove will deactivate their account instead of deleting it. This is to preserve the audit trail.
Go to Setup > People.
Locate the user and select the checkbox next to their name. An Options panel appears on the right.
Click Delete. The user is deleted.