He is saying that the human conscience makes one believe something is either good or bad and if one does something good or bad it may way heavily on the conscience. I do agree with Hamlet's theory because it is only natural for someone to think that every decision they have made is a good one or bad one.
Hamlet is telling his friends that he is only slightly mad, but not mad enough not to know the difference between a hawk and a handsaw. It is difficult to distinguish whether he is mad or acting, but most of the time we can assume it is just a mixture of both.