by the method of elimination.  If one or other of the factors is missing and we observe a slight diminution in the corresponding psychological reaction ("hesitation” in our case), we may conclude that the factor is in other uses genuinely positive.  The second of our four factors applies only to the interrogative use of whom, the fourth factor applies with more force to the interrogative than to the relative.  We can therefore understand why a sentence like Is he the man whom you referred to? though not as idiomatic as Is he the man (that) you referred to? (remember that it sins against counts one and three), is still not as difficult to reconcile with our innate feeling for English expression as Whom did you see? If we eliminate the fourth factor from the interrogative usage,[138] say in Whom are you looking at? where the vowel following whom relieves this word of its phonetic weight, we can observe, if I am not mistaken, a lesser reluctance to use the whom. Who are you looking at? might even sound slightly offensive to ears that welcome Who did you see?

[Footnote 136:  Students of language cannot be entirely normal in their attitude towards their own speech.  Perhaps it would be better to say “naive” than “normal.”]

[Footnote 137:  It is probably this variability of value in the significant compounds of a general linguistic drift that is responsible for the rise of dialectic variations.  Each dialect continues the general drift of the common parent, but has not been able to hold fast to constant values for each component of the drift.  Deviations as to the drift itself, at first slight, later cumulative, are therefore unavoidable.]

[Footnote 138:  Most sentences beginning with interrogative whom are likely to be followed by did or does, do.  Yet not all.]

We may set up a scale of “hesitation values” somewhat after this fashion: 

Value 1:  factors 1, 3.  “The man whom I referred to.” 
Value 2:  factors 1, 3, 4.  “The man whom they referred to.” 
Value 3:  factors 1, 2, 3.  “Whom are you looking at?”
Value 4:  factors 1, 2, 3, 4.  “Whom did you see?”

We may venture to surmise that while whom will ultimately disappear from English speech, locutions of the type Whom did you see? will be obsolete when phrases like The man whom I referred to are still in lingering use.  It is impossible to be certain, however, for we can never tell if we have isolated all the determinants of a drift.  In our particular case we have ignored what may well prove to be a controlling factor in the history of who and whom in the relative sense.  This is the unconscious desire to leave these words to their interrogative function and to concentrate on that or mere word order as expressions of the relative (e.g., The man that I referred to or The man I referred to).  This drift,